Introduction to Apache Airflow at PyData Delhi meetup #25

Related blog post: https://blog.socialcops.com/engineering/apache-airflow-disease-outbreaks-india/

The presentation begins with a general introduction to Apache Airflow and then goes into how the audience can develop their own ETL workflows using the framework, with the help of an example use case of "tracking disease outbreaks in India". It also talks about briefly how SocialCops has used Apache Airflow to power the DISHA dashboard (https://blog.socialcops.com/inside-sc/announcements/disha-dashboard-good-governance/) and to move data across their internal systems.

The full code is present here: https://github.com/socialcopsdev/airflow_blog

Transcript

1. Introduction to Apache Airflow A workflow to track disease outbreaks in India Vinayak Mehta, Data Engineer at SocialCops
2. What is Data Engineering?
3. Extract, Transform, Load (ETL) workflows
4. Scheduling Cron, anyone? An example workflow
5. Problems • Workflow management is a nightmare • Can’t scale as team sizes grow • Can’t handle dependencies between tasks • “I want to debug my workflow, where are the logs?!”
6. A wild Apache Airflow appears! "Airflow is a platform to programmatically author, schedule and monitor workflows." - Airflow docs • created by Maxime Beauchemin at Airbnb in 2014, open source from the very first commit • used by Airbnb, Lyft, Spotify, Quora, Reddit, Stripe, Yahoo! ...
7. Why Airflow?
8. Workflows are defined as Directed Acyclic Graphs, which means dependency management is sorted Workflows as DAGs
9. Extensible • Each node/task in the DAG is an “operator” • A lot of operators right out of the box which let you connect directly to MySQL, Postgres, Hive ... • PythonOperator, which means you can write what your task will do in the language that we all love!
10. Scalable • Tasks are executed by “executors” • Comes with CeleryExecutor, which lets you horizontally scale your workflow to multiple worker nodes
11. Open source • Under incubation at the Apache Software Foundation • New operators being added regularly! • Actively maintained
12. Architecture
13. The Holy Trinity • Webserver • Scheduler • Worker
14. A DAG!
15. What is this?
16. An orchestra!
17. Enough about Airflow
18. The data source • The Ministry of Health and Family Welfare has the IDSP scheme in place to track disease outbreaks • Tracking done at sub-district and village level across India • Outbreak data released as a PDF document on a weekly basis
19. The workflow • (E) Extract the PDFs from IDSP website • (T) Transform them into CSVs • (L) Load them into a data store
20. Extracting PDFs from IDSP website
23. Transforming PDFs into CSVs
24. Some minor cleaning
25. Tying it all together into a DAG
26. Where to go from here? • To the webserver! ◦ View task states ◦ View task logs • But what do we do with the code and data we have? ◦ Extend the code to extract from other sources. ◦ Use the data along with other sources to generate your own insights. ◦ *cough* predictive analytics *cough*
27. Where else do we use Airflow
28. Collect to Visualize workflows • Extract data from Collect, our data collection app • Transform using R/Python • Load data into Visualize, our visualization platform to derive better insights out of the data
29. DISHA • Data from 20 different ministries breaking silos to come together in one place • Bringing in accountability to a budget of over Rs. 2 lakh crore spent on 41 welfare schemes annually “DISHA is a crucial step towards good governance through which we will be able to monitor everything centrally. It will enable us to effectively monitor every village of the country.” – Narendra Modi, Prime Minister of India
30. Questions?
31. Thanks! SocialCops https://twitter.com/social_cops Email: de@socialcops.com Blog: blog.socialcops.com Website: www.socialcops.com