Solving Multi-Tenant Challenges: Apache Airflow and Cloud Composer in Action

Transcript

2. 2 WHO AM I? • Studied in Budapest & Munich • Settled in Vienna • Backend developer -> Platform engineer • Data Platform 2022+ • https://speakerdeck.com/posedio/ • Endurance sports ( ) • For more catch me @ Vienna Marathon • Owner of two Dogs ( ) Tamas
3. 3 1. Airflow 101 AGENDA
4. 4 1. Airflow 101 2. Do I need an orchestrator? AGENDA
5. 5 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) AGENDA
6. 6 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer AGENDA
7. 7 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer 5. Multi-tenancy in Airflow AGENDA
8. 8 1. Airflow 101 2. Do I need an orchestrator? 3. Airflow’s architecture (“the cost”) 4. Cloud Composer 5. Multi-tenancy in Airflow AGENDA
9. AIRFLOW 1
10. 10 APACHE AIRFLOW • 2015: Airbnb needed a tool to author, iterate, and monitor batch data pipelines ( Article) • 2016: Airflow joined Apache Foundation • 2019: Airflow graduated • 2020+: Airflow 2.0 released, growing community
11. 11 AIRFLOW SUMMIT 2024 • 3 days ~200 Talks
12. 12 ASTRONOMER • 2018: SaaS offering for Airflow called Astro • Solid documentation on AirFlow • Integration of 3rd party tools (dbt ~ data build tool) • Enterprise-tier features for AirFlow (multi-tenancy with work-spaces etc.)
13. 13 AIRFLOW UI: DAG OVERVIEW
14. 14 AIRFLOW UI: GRAPH
15. 15 AIRFLOW UI: AUDIT LOG
16. DO I REALLY NEED IT? 2
17. 17 SESAME INC. • Inputs: • Recipe • Ingredients (~Stock level) • Processes • Configure mixer • Configure oven • Configure packaging • Start production • 23:00
18. 18 SESAME INC.
19. 19 AIRFLOW • DAG (directed acyclic graph) • Tasks: • Fetch recipe and stock • Configure mixer • Configure oven • Start production • Monitor packaging
20. 20 AIRFLOW • DAG (directed acyclic graph) • Tasks: • Fetch recipe and stock • Configure mixer • Configure oven • Start production • Monitor packaging
21. 21 CRON • 0 8 * * * /usr/bin/python start.py • Steps: • Fetch recipe and stock levels • Configure mixer • Configure oven • Start production • Monitor packaging
22. 22 CATCHUP • Machine didn’t work over weekend => Sat, Sun, (Mon) cookies are missing. • Cron: ??? • AirFlow:
23. 23 BACKFILL • While the cookies were stored in the storage room, they have gone bad. Can we re-run the batch? • Cron: ??? • AirFlow:
24. 24 LINKEDIN: AIRFLOW BACKFILL PLUGIN
25. 25 SENSORS • Checks if a condition is met at a specific interval. • timeout • poke_interaval • Types • GCS (files) • SQL • …
26. WHAT’S THE COST? 3
27. 27 AIRFLOW ARCHITECTURE
28. 28 AIRFLOW ARCHITECTURE: SCHEDULER
29. 29 AIRFLOW ARCHITECTURE: SCHEDULER
30. 30 AIRFLOW ARCHITECTURE
31. 31 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
32. 32 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
33. 33 AIRFLOW EXECUTION ENVIRONMNET • Local machine: • BashOperator • PythonOperator • PythonVirtualenvOperator
34. 34 AIRFLOW EXECUTION ENVIRONMNET • Remote: • Celery • KubernetesPodOperator • GkeStartPodOperator
35. 35 AIRFLOW EXECUTION ENVIRONMNET • Remote: • Celery • KubernetesPodOperator • GkeStartPodOperator
36. 36 CRON VS AIRFLOW • Job monitoring (UI vs CRON-log) • Job overview (Graph vs 1000s of lines) • Business logic decouples from scheduling • Efficient remote execution (K8s Pods) • Comes at a price of having to manage Airflow…
37. CLOUD COMPOSER 4
38. 38 COMPOSER V2
39. 39 COMPOSER V3 ~ „SERVERLESS“
40. 40 COMPOSER V3 ~ „SERVERLESS“
41. MULTI TENANCY 5
42. 42 LET‘S ONBOARD! Guy with a shady use-case Business Intelligence Finance I don‘t even know who you are… Data Analysts Her friends…
43. 43 LIMITING ACCESS: VISIBILITY & EXECUTION
44. 44 LIMITING ACCESS: PER-FOLDER ROLE REGISTRATION • rbac_autoregister_per_folder_roles -> True • Default role: UserNoDags • Airflow now generates roles: finance, da
45. 45 LIMITING ACCESS TO RESOURCES
46. 46 DYNAMIC DAG GENERATION • Dynamic DAGs and Dynamic Tasks
47. 47 DYNAMIC DAG GENERATION CI/CD
48. 48 ASTRONOMER: DAG-FACTORY
49. 49 YAML? AGAIN?!
50. 50 YAML? AGAIN?! +
51. 51 SECURE? YES! BUT AT WHAT COST?
52. CLUSTER POLICIES 6
53. 53 AIRFLOW CLUSTER POLICIES • DAG/Task Validation
54. 54 AIRFLOW CLUSTER POLICIES • Mutation
55. 55 LIMITING ACCESS 1. Ensure DAG lands in correct folder 2. (Optionally) simulate DAG parsing & loading 3. Cluster Policies ~> Validate and/or mutate
56. 56 WRAPPING UP! • If you find yourself hand-crafting orchestration logic, consider frameworks (Airflow, Dagster, Prefect) • To reduce OpEx consider managed solutions (Cloud Composer ) • Look into the K8sPodOperator • Avoid YAML-hell • Use hooks and cherish the joy of Python
57. 21.11.2024