This session explores how open-source analytics technologies are transforming the public sector through the lens of Electronic Income Verification (EIV) systems—platforms that process over 850,000 real-time verifications daily, integrate 40+ data sources, and maintain 99.95% uptime to support equitable, efficient public benefit delivery.
We’ll dive into the open-source stack behind these systems: event streaming with Apache Kafka, data orchestration with Airflow, analytics with Apache Superset and DuckDB, and ML-powered fraud detection using tools like scikit-learn and Hugging Face NLP. You’ll learn how public agencies are building scalable, secure, and cost-effective solutions by leveraging community-driven technologies and standards.
Topics include building modular data pipelines, real-time dashboards, anomaly detection, and managing governance and compliance (GDPR, HIPAA) in open environments. The talk will also highlight DevOps practices such as IaC, GitOps, and monitoring with Prometheus and Grafana to maintain visibility, security, and auditability in high-trust systems.
Ideal for data engineers, open-source practitioners, and civic tech innovators, this session offers a real-world case study on how open analytics infrastructure can power large-scale, high-impact digital public services.