End-to-end automated data science process using Airflow.

End-to-end automated data science process using Airflow.

Transcript

1. End-to-end automated data science process using Airflow. Evive
2. About Evive • Data Driven benefit navigator • Founded in 2006 • 400 + employees
3. Evive Data 15 2.5M 400 Data team Evive Employee Total Active members
4. Data Usage 500+GB 50+ 30+ Total data per day Number of data channels Number of models running daily
5. Why Airflow THE WORKFLOW Ingestion Merge data from multiple sources Standardise Verify Publish
6. Airflow workers Data Sources Scheduler Database
7. Airflow Architecture
8. Functionalities • Scheduling • Dependency management • Error recovery • Monitoring • Versioning • Mailing and alerting
9. Creating a dag and an operator
10. Scheduling tasks
11. File sensor • Operator that listens to a particular directory and triggers the downstream task once the file lands on the corresponding directory. • Pynotify as operator.
12. Monitoring using airflow dashboard
13. Versioning • Versioning can be easily incorporated in airflow as the entire dag execution happens as one instance. • You can version your data as well as model outputs.
14. Mailing and alerting system
15. Future work • Integrating with the existing database architecture and ETL pipeline • Airflow Kubernetes executors