Orchestrating the Future: Navigating Today's Data Workflow Challenges with Airflow and Beyond | Budapest Data + ML Forum 2024

Navigating today's data landscape isn't just about managing workflows; it's about strategically propelling your business forward. Apache Airflow has stood out as the benchmark in this arena, driving data orchestration forward since its early days. As we dive into the complexities of our current data-rich environment, where the sheer volume of information and its timely, accurate processing are crucial for AI and ML applications, the role of Airflow has never been more critical.

In my journey as the Senior Engineering Director and a pivotal member of Apache Airflow's Project Management Committee (PMC), I've witnessed Airflow transform data handling, making agility and insight the norm in an ever-evolving digital space. At Astronomer, our collaboration with leading AI & ML teams worldwide has not only tested but also proven Airflow's mettle in delivering data reliably and efficiently—data that now powers not just insights but core business functions.

This session is a deep dive into the essence of Airflow's success. We'll trace its evolution from a budding project to the backbone of data orchestration it is today, constantly adapting to meet the next wave of data challenges, including those brought on by Generative AI. It's this forward-thinking adaptability that keeps Airflow at the forefront of innovation, ready for whatever comes next.

The ever-growing demands of AI and ML applications have ushered in an era where sophisticated data management isn't a luxury—it's a necessity. Airflow's innate flexibility and scalability are what makes it indispensable in managing the intricate workflows of today, especially those involving Large Language Models (LLMs).

This talk isn't just a rundown of Airflow's features; it's about harnessing these capabilities to turn your data workflows into a strategic asset. Together, we'll explore how Airflow remains at the cutting edge of data orchestration, ensuring your organization is not just keeping pace but setting the pace in a data-driven future.

Session in https://budapestdata.hu/2024/04/kaxil-naik-astronomer-io/ | https://dataml24.sessionize.com/session/667627

Transcript

1. Orchestrating the Future Navigating Today's Data Workflow Challenges with Airflow and Beyond Budapest Data + ML Forum June 2024
2. Kaxil Naik Apache Airflow Committer & PMC Member Senior Director of Engineering @ Astronomer @kaxil @kaxil @kaxil
3. • Orchestrator – The What & Why? • What is Apache Airflow? ◦ Why is Airflow the Industry Standard for Data Professionals? ◦ Evolution of Airflow • Today’s Data Workflow Challenges ◦ How Airflow addresses them – Real world case studies • The Future of Airflow Agenda
4. Orchestrator The What & Why?
5. What is Orchestration? Who is an Orchestrator?
6. Why Orchestration?
7. Orchestration in Engineering! Workflow Orchestrator Automates and manages interconnected tasks across various systems to streamline complex business processes. E.g Running bash script everyday to update packages on a laptop. Data Orchestrator Automates and manages interconnected tasks that deal with data across various systems to streamline complex business processes. E.g ETL for a BI dashboard.
8. What is Apache Airflow?
9. A Workflow Orchestrator, most commonly used for Data Orchestration Official Definition: A platform to programmatically author, schedule and monitor workflows What is Apache Airflow?
10. Python Native The language of data professionals (Data Engineers & Scientists). DAGs are defined in code: allowing more flexibility & observability of code changes when used with git. Pluggable Compute GPUs, Kubernetes, EC2, VMs etc. Integrates with Toolkit All data sources, all Python libraries, TensorFlow, SageMaker, MLFlow, Spark, Ray, etc. Common Interface Between Data Engineering, Data Science, ML Engineering and Operations. Data Agnostic But data aware. Cloud Native But cloud neutral. Monitoring & Alerting Built in features for logging, monitoring and alerting to external systems. Extensible Standardize custom operators and templates for common DS tasks across the organization. Key Features of Airflow
12. Example DAG
14. Why is Airflow the Industry Standard for Data Professionals?
15. 25M Monthly Downloads The Community 2.9K Contributors 35K GitHub Stars 47K Slack Community
16. Under …
17. Governed by Committers 33 PMC Members Project Management Committee 62
18. Integrations And ……
19. 90+ Providers
20. Docker Image docker pull apache/airflow
21. Helm Chart helm repo add apache-airflow https://airflow.apache.org/ helm install my-airflow apache-airflow/airflow
22. Conference & Meetups Attendees: Online Edition (2020-2022): 10k In-person (2023+): 500+ 15 Local Groups across the globe with 11k members
23. Managed Airflow Vendors
24. Airflow Survey and State of Apache Airflow report Infographic: https://airflow.apache.org/survey/ Report: https://www.astronomer.io/state-of-airflow/
25. Use cases for Airflow Ingestion and ETL/ELT related to business operations 0% 25% Source: 2023 Apache Airflow Survey, n=797 13% 90% of Apache Airflow usage is dedicated to ingestion and ETL/ELT tasks associated with analytics, followed by 68% for business operations. Additionally, there’s a growing adoption for MLOps (28%) and infrastructure management (13%), highlighting its versatility across various data workflow tasks. 50% 100% 90% 68% 28% Ingestion and ETL/ELT related to analytics Training, serving, or generally manage MLOps Spinning up and spinning down infrastructure Other 3% 75%
26. The Evolution of Airflow
27. Timeline: Major Milestones 2014 Oct Created at AirBnb 2016 March Donated to the Apache Software Foundation (ASF) as an Incubating project 2020 Dec Airflow 2.0 released 2015 June Open Sourced 2018 Dec Graduated as a top-level project 2025 Mar-Apr (Planned) Airflow 3.0 release 2020 July First Airflow Summit
28. Timeline: 2.x Minor Releases 2.1 2021-05 2.3 2022-05 2.2 2021-11 2.4 2022-09 2.5 2022-11 2.6 2023-04 2.7 2023-08 2.8 2023-12 2.9 2024-04
29. Code Contributions & downloads continue to grow! Downloads: 500K / month Downloads: 25M / month
30. Today’s Data Workflow Challenges
31. Today’s Data Workflow challenges Increasing Data Volumes Businesses generates more data than ever. Handling this data & its quality is critical. Need for near Real-time Processing Data Workflows are being used to drive critical business decisions in near real-time & hence requiring reliability & performance guarantees. Complexity in Data Workflows Modern workflows need handling data from multiple sources that require managing complex deps & dynamic schedules. Intelligent Infrastructure Infrastructure must be elastic & flexible to optimize for a modern workloads.
32. Today’s Data Workflow challenges Additional Interfaces Net-new teams- from ML to AI - want to get the best out of Airflow without learning a new framework. Licensing & Security in OSS OSS projects owned by a single company have changed licenses too often in recent past. Platform Governance Visibility, auditability, & lineage across a data platform is need-to-have. Cost Reduction Tight budgets have pushed teams to efficiently utilize the resources to drive operational costs down.
33. How does Airflow address these challenges?
34. Case Study: Texas Rangers Company: A professional baseball team in Major League Baseball (MLB), based in Arlington, Texas. The Rangers won their first World Series championship in 2023. Goal: Use data to gain unfair advantage, Moneyball style! Data to be collected: real-time game data streaming, comprehensive player health reporting, predictive analytics of everything from pitch spin to hit trajectory, and more Challenge: Scalability issues due to volume & unprecedented rate of data & infra bottleneck in their live game analytics pipeline. This impacted the timely delivery of analytics to their team and affected their competitive edge.
35. Case Study: Texas Rangers Solution: Use Airflow’s worker queues to create dedicated worker pools for CPU-intensive tasks while other tasks used cheaper workers. Using Data-aware Scheduling, they were able to start their DAGs when data was available instead of time-based scheduling. Result: Improved Scalability Using worker queues, DAG completion time reduced by 80% (from 20 mins to 3 mins) Increased Efficiency Optimizing compute resources allowed processing of 4 additional DAGs in parallel, enabling immediate post-game analytics delivery for a competitive edge.
36. Case Study: Bloomberg Company: Bloomberg is a leading source for financial & economic data: Equities, bonds, Index, Mortgages, currencies, etc. Founded in 1981 with subscribers in 170+ countries. Goal: Deliver a diverse array of information, news & analytics to facilitate decision-making Challenge: Maintaining custom pipelines for diverse datasets of different domains is expensive & time consuming. Their engineers lacked domain knowledge to aggregate data into client insights & their domain experts lack skills to maintain data pipelines in Production.
37. Case Study: Bloomberg Solution: Configuration-driven ETL platform leveraging Airflow & dynamic DAGs. User-defined configs are translated into Dynamic DAGs determining tasks & their dependencies with success/failure actions. Result: The Data Platform teams now supports 1600+ DAGs, 700+ datasets, 200+ users, 11 different product teams, 10k+ weekly file ingestions Source: https://airflowsummit.org/sessions/2023/airflow-at-bloomberg-leveraging-dynamic-dags-for-data-ingestion/
38. Case Study: Company: FanDuel Group is a sports betting company that lives on data with approx 17 million customers. Goal: Business growth led to higher daily data volumes, which fueled demand for new sources and richer analytics. Challenge: 2022 NFL season was fast approaching and FanDuel wanted a robust data architecture in anticipation of company’s busiest time in terms of daily volume of data.
39. Case Study: Solution: They worked with Astro professional services team to replace Operators with more efficient Deferrable Operators along with Astro’s auto-scaling features. Result: The number of worker nodes running on avg decreased by 35%, resulting in immediate infrastructure cost savings & average tasks per worker increased by 305%
40. Other Interesting Case Studies Grindr has saved $600,000 in Snowflake costs by monitoring their Snowflake usage across the organization with Airflow. Condé Nast has reduced costs by 54% by using deferrable operators. Airline: a tool powered by Airflow, built by Astronomer’s Customer Reliability Engineering (CRE) team, that monitors Airflow deployments and sends alerts proactively when issues arise.
41. Other Interesting Case Studies King uses ‘data reliability engineering as code’ tools such as SodaCore within Airflow pipelines to detect, diagnose and inform about data issues to create coverage, improve quality & accuracy and help eliminate data downtime. Laurel.ai: A pioneering AI company that automates time and billing for professional services. Uses multiple domain-specific LLMs to create billing timesheets from users’s footprints across their workflows & tools (Zoom, MS Teams etc). Airflow orchestrates their entire GenAI lifecycle: data extraction, model tuning & feedback loops. Ask Astro: An end-to-end example of a Q&A LLM application used to answer questions about Apache Airflow and Astronomer
42. The Future of Apache Airflow
43. Airflow 3 Make Airflow the foundation for Data, ML, and Gen AI orchestration for the next 5 years. 1. Enable secure remote task execution across network boundaries. 2. Integrate data awareness needed for governance and compliance 3. Enable non-python tasks, for integration with any language 4. Enable Versioning of Dags and Datasets 5. Single command local install for learning and experimentation.
44. Thank You A friendly reminder to RSVP to Airflow Summit 2024: • Celebrating 10 Years of Airflow • Sept. 10th-12th • The Westin St. Francis • San Francisco, CA @kaxil @kaxil @kaxil Airflow Summit Discount Code: 15DISC_MEETUP