Apache Airflow: Synchronizing Datasets across Multiple instances

Transcript

1. Apache Airflow: Synchronizing Datasets across Multiple instances
2. Join at: slido.com #3371 950
3. Who Am I ? • Data Engineer @ Numberly • Airflow user since 2018 • French & Mexican • EuroPython 2023 Speaker ◦ Orchestrating Python Workflows in Apache Airflow • Lived in Taiwan for ~5 years
4. Intro to Apache Airflow
5. Orchestration • Run jobs automatically • At a preset frequency • Monitor their execution
6. Open Source & Community • Apache Foundation project → License & Management • Active developers
7. Open Source & Community
8. Open Source & Community • Apache Foundation project → License & Management • Active developers • Active Community : Slack, Stack Overflow, Reddit, etc…
9. Open Source & Community • Apache Foundation project → License & Management • Active developers • Active Community : Slack, Stack Overflow, Reddit, etc… • Global event → Airflow Summit
10. Basic Airflow Objects • DAGs • Tasks • Operators Check out the previous session:
11. Airflow Scheduling
12. Fixed Schedule Using crontab notation: Minute Hour DayOfMonth Month DayOfWeek Example: 0 8 1 * * → On the first day of each month at 8:00 AM
13. Fixed Schedule • Most commonly used • Easy to use • Requires extra features or workarounds for some use cases: ◦ Daylight Saving Time ✅ ◦ UTC ✅ ◦ Holidays ≈ • Not Dynamic
14. Event Driven Scheduling • Execution that follows external events • More complex to set up • 1 to 1 execution • Difficult issues: ◦ Order & Race conditions ◦ Ensuring deliverability ◦ Duplicate events
15. Data-Aware Scheduling • Execution that follows changes in Data Source • Execute if Data Asset has changed • Similar to Event Driven but with lower constraints ◦ 1 to 1 execution not guaranteed ◦ Order of events shouldn’t be important
16. Data-Aware Scheduling • Example DAG: depends on 3 datasets
17. Airflow Datasets (Data Assets) • It is just a string • Represent Data Asset • Maintaining link is YOUR responsibility
18. Upscale our Scheduler
19. When is it needed? • Machine performance ◦ High CPU and RAM usage ◦ Scheduler heartbeat failures ◦ dag_processing.total_parse_time is high • Slow scheduling ◦ Many tasks in “queued” or ”scheduled” state ◦ Long queue length ◦ Delayed DAG runs • Database limitations
20. Airflow Remote Executors Pros: • Robust: decoupling workers from scheduler process • Effective: Worker parallelization • Available: Low latency workers always running
21. Airflow Remote Executors Cons: • Requires Infrastructure setup • Can be Expensive: Cloud cost
22. Scheduler Replication
23. Separate Instances Pros: • Maximum scalability • Separation of Concerns • Better Access Control Just have 2 Airflows
24. Separate Instances Cons: • Even more work • Even more resources • Redundant operations Just have 2 Airflows ?
25. Synchronizing Airflow Datasets
26. Working Environment Older instance • Airflow 2.4 • 1000+ DAGs • Frozen Younger instance • Airflow 2.10 • • Active development
27. Implementation Candidates • Using custom "high-frequency" DAGs
28. The Hacky Solution • 1 DAG per instance ◦ Checks for new Data Asset Events ◦ Pushes to other instances via API • Requires complex error handling • N*(N-1) messages exchanged
29. Implementation Candidates • Using custom "high-frequency" DAGs • Using Airflow Listeners
30. Experimental features It Didn’t Work
31. Potential performance issues
32. Implementation Candidates • Using custom "high-frequency" DAGs • Using Airflow Listeners • Using an external service via the Airflow API
33. • Limited Airflow resource impact • Only 2*N messages • Easier error handling External Service
34. Implementation Candidates • Using custom "high-frequency" DAGs • Using Airflow Listeners • Using an external service via the Airflow API
35. Describing the current solution
36. External Service consumers producers
37. Implementation details • Single source of truth (Kafka)
38. Implementation details • Single source of truth (Kafka) • Separation of concerns (Consumer/Producer)
39. Implementation details • Single source of truth (Kafka) • Separation of concerns (Consumer/Producer) • Saved offsets ◦ Producer: Airflow Variable ◦ Consumer: Kafka Offset
40. Implementation details • Single source of truth (Kafka) • Separation of concerns (Consumer/Producer) • Saved offsets ◦ Producer: Airflow Variable ◦ Consumer: Kafka Offset • Polling done via API
41. Roadblocks & Solutions
42. Airflow API issues • Airflow 2.4 API does NOT have a POST endpoint for Dataset Events • Airflow 2.9 API Dataset Events POST endpoint does NOT create new Datasets
43. Airflow parsed object vs created objects • Parsed objects ◦ DAGs, Tasks, Datasets… • Created objects ◦ Dag runs, Dataset Events, XCOM… What happens when your files and your Database don’t agree?
44. DatasetAlias (Asset Alias) Dataset Alternative with following characteristics: • Dynamic • Can create Multiple Datasets at once Use if you don’t know which assets will be used at Design Time
45. External Service consumer 2.10 2.4 • Airflow API ◦ POST dataset/events • DatasetAlias DAG • Custom DAG
46. Changes in Airflow3.0
47. Naming changes • Dataset → Data Asset • DatasetAlias → Asset Alias
48. New functionalities • New API Endpoint: Materialize asset, solves Asset pushing issues • Event Driven Scheduling: New polling mechanism for scheduling ◦ AssetWatcher: monitors external event source • Cool New UI
49. At Numberly
51. Question time!