Solving Multi-Tenant Challenges: Apache Airflow and Cloud Composer in Action

Join us and explore the world of cloud data platforms and see how Apache Airflow, an open-source workflow orchestrator, fits into the broader ecosystem. We'll look at the challenges we encountered while managing Airflow in a multi-tenant environment and share how Cloud Composer, Google's managed Apache Airflow solution, helped us overcome some of those hurdles. Finally, we will briefly touch upon some of Airflow's most exciting features, that allowed our tenants to efficiently create and schedule workflows in a standardized way.

Transcript

2. 2 WHO AM I? • Studied in Budapest & Munich • Settled in Vienna • Backend developer -> Platform engineer • Data Platform 2022+ • https://speakerdeck.com/posedio/ • Endurance sports ( ) • For more catch me @ Vienna Marathon • Owner of two Dogs ( ) Tamas
3. 3 1. Airflow 101 AGENDA
4. 4 1. Airflow 101 2. Do I need an orchestrator? AGENDA
5. 5 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) AGENDA
6. 6 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer AGENDA
7. 7 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer 5. Multi-tenancy in Airflow AGENDA
8. 8 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer 5. Multi-tenancy in Airflow AGENDA
9. AIRFLOW 1
10. 10 APACHE AIRFLOW • 2015: Airbnb needed a tool to author, iterate, and monitor batch data pipelines ( Article) • 2016: Airflow joined Apache Foundation • 2019: Airflow graduated • 2020+: Airflow 2.0 released, growing community
11. 11 AIRFLOW SUMMIT 2024 • 3 days ~200 Talks
12. 12 ASTRONOMER • 2018: SaaS offering for Airflow called Astro • Solid documentation on AirFlow • Integration of 3rd party tools (dbt ~ data build tool) • Enterprise-tier features for AirFlow (multi-tenancy with work-spaces etc.)
13. 13 AIRFLOW UI: DAG OVERVIEW
14. 14 AIRFLOW UI: GRAPH
15. 15 AIRFLOW UI: AUDIT LOG
16. DO I REALLY NEED IT? 2
17. 17 SESAME INC. • Inputs: • Recipe • Ingredients (~Stock level) • Processes • Configure mixer • Configure oven • Configure packaging • Start production • 23:00
18. 18 SESAME INC.
19. 19 AIRFLOW • DAG (directed acyclic graph) • Tasks: • Fetch recipe and stock • Configure mixer • Configure oven • Start production • Monitor packaging
20. 20 AIRFLOW • DAG (directed acyclic graph) • Tasks: • Fetch recipe and stock • Configure mixer • Configure oven • Start production • Monitor packaging
21. 21 CRON • 0 8 * * * /usr/bin/python start.py • Steps: • Fetch recipe and stock levels • Configure mixer • Configure oven • Start production • Monitor packaging
22. 22 CATCHUP • Machine didn’t work over weekend => Sat, Sun, (Mon) cookies are missing. • Cron: ??? • AirFlow:
23. 23 BACKFILL • While the cookies were stored in the storage room, they have gone bad. Can we re-run the batch? • Cron: ??? • AirFlow:
24. 24 LINKEDIN: AIRFLOW BACKFILL PLUGIN
25. 25 SENSORS • Checks if a condition is met at a specific interval. • timeout • poke_interaval • Types • GCS (files) • SQL • …
26. WHAT’S THE COST? 3
27. 27 AIRFLOW ARCHITECTURE
28. 28 AIRFLOW ARCHITECTURE: SCHEDULER
29. 29 AIRFLOW ARCHITECTURE: SCHEDULER
30. 30 AIRFLOW ARCHITECTURE
31. 31 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
32. 32 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
33. 33 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
34. 34 AIRFLOW EXECUTION ENVIRONMNET • Remote: • Celery • KubernetesPodOperator • GkeStartPodOperator
35. 35 AIRFLOW EXECUTION ENVIRONMNET • Remote: • Celery • KubernetesPodOperator • GkeStartPodOperator
36. 36 CRON VS AIRFLOW • Job monitoring (UI vs CRON-log) • Job overview (Graph vs 1000s of lines) • Business logic decouples from scheduling • Efficient remote execution (K8s Pods) • Comes at a price of having to manage Airflow…
37. CLOUD COMPOSER 4
38. 38 COMPOSER V2
39. 39 COMPOSER V3 ~ „SERVERLESS“
40. 40 COMPOSER V3 ~ „SERVERLESS“
41. MULTI TENANCY 5
42. 42 LET‘S ONBOARD! Guy with a shady use-case Business Intelligence Finance I don‘t even know who you are… Data Analysts Her friends…
43. 43 LIMITING ACCESS: VISIBILITY & EXECUTION
44. 44 LIMITING ACCESS: PER-FOLDER ROLE REGISTRATION • rbac_autoregister_per_folder_roles -> True • Default role: UserNoDags • Airflow now generates roles: finance, da
45. 45 LIMITING ACCESS TO RESOURCES
46. 46 DYNAMIC DAG GENERATION • Dynamic DAGs and Dynamic Tasks
47. 47 DYNAMIC DAG GENERATION CI/CD
48. 48 ASTRONOMER: DAG-FACTORY
49. 49 YAML? AGAIN?!
50. 50 YAML? AGAIN?! +
51. 51 SECURE? YES! BUT AT WHAT COST?
52. CLUSTER POLICIES 6
53. 53 AIRFLOW CLUSTER POLICIES • DAG/Task Validation
54. 54 AIRFLOW CLUSTER POLICIES • Mutation
55. 55 LIMITING ACCESS 1. Ensure DAG lands in correct folder 2. (Optionally) simulate DAG parsing & loading 3. Cluster Policies ~> Validate and/or mutate
56. 56 WRAPPING UP! • If you find yourself hand-crafting orchestration logic, consider frameworks (Airflow, Dagster, Prefect) • To reduce OpEx consider managed solutions (Cloud Composer ) • Look into the K8sPodOperator • Avoid YAML-hell • Use hooks and cherish the joy of Python
57. 21.11.2024