Defining data pipeline workflows using Apache Airflow

Transcript

1. Defining data pipeline workflows using Apache Airflow MAD · NOV

   23-24 · 2018 Juan Riaza ! jriaza@idealista.com " @juanriaza

2. • SoEware Developer • OSS enthusiast • Pythonista & Djangonaut

   • Now trying to tame Gophers • Reverse engineering apps • Hobbies: cooking and reading Who am I? MAD · NOV 23-24 · 2018

3. MAD · NOV 23-24 · 2018

4. In the beginning there was cron…

5. MAD · NOV 23-24 · 2018 Gather Data ETL Report

   22:00 00:00 02:00 Gather Data Report 22:00 23:30 01:00 ETL ☺

6. MAD · NOV 23-24 · 2018 Gather Data Report 22:00

   23:30 01:00 ETL Report 22:00 23:30 01:00 ETL Gather Data ⁉ ⏱⁉

7. MAD · NOV 23-24 · 2018 •Managing dependencies. Possible overlapping

   •Failure handling. Retries? •Error noXficaXons, metrics. Visibility? •Unified logs •Distributed cron… •Do you maintain a calendar of batch jobs? •What happens if…? Cron “hustle”

8. Every day is a new adventure…

9. MAD · NOV 23-24 · 2018 Meet Airflow

10. •“A pla`orm to programmaXcally author, schedule, and monitor workflows” •The

    glue that binds your data ecosystem together •Open source, wricen in Python •Was started in Oct 2014 by Max Beauchemin at •IncubaXng at since 2016 •550+ contributors, 5300+ commits, 9300+ stars Overview MAD · NOV 23-24 · 2018

11. MAD · NOV 23-24 · 2018

12. Use cases •ETL pipelines •Machine learning pipelines •PredicXve data pipelines:

    fraud detecXon, scoring/ ranking, classificaXon, recommender system, etc. •General job scheduling: DB back-ups •Anything… automate the garage door? MAD · NOV 23-24 · 2018

13. Airflow uses Operators as the fundamental unit of abstracXon to

    define tasks, and uses a DAG to define workflows using a set of operators MAD · NOV 23-24 · 2018

14. •Directed Acyclic Graph •Represents a workflow: set of tasks with

    a dependency structure •Each node represents some form of data processing What is a DAG MAD · NOV 23-24 · 2018

15. What does it look like? MAD · NOV 23-24 ·

    2018

16. err…

17. How it’s made MAD · NOV 23-24 · 2018

18. import datetime from airflow import DAG from airflow.operators.bash_operator import BashOperator

    from airflow.operators.python_operator import PythonOperator def print_commit(): print(' commit conf') with DAG('commit_dag', schedule_interval='@weekly') as dag: print_bash = BashOperator( task_id='print_bash', bash_command='echo " commit conf"') sleep = BashOperator( task_id='sleep', bash_command='sleep 5') print_python = PythonOperator( task_id='print_python', python_callable=print_commit) print_bash >> sleep >> print_python

19. Airflow UI MAD · NOV 23-24 · 2018

20. Airflow UI MAD · NOV 23-24 · 2018

21. Airflow UI MAD · NOV 23-24 · 2018

22. Airflow UI MAD · NOV 23-24 · 2018

23. •AcIon: perform an acXon locally or make a call to

    an external system to perform another acXon •Transfer: move data from one system to another •Sensor: wait for and detect some condiXon in a source system Operators MAD · NOV 23-24 · 2018

24. •Perform an acXon such as execuXng a Python funcXon or

    submilng a Spark Job •Built-in BashOperator, PythonOperator, DockerOperator, EmailOperator, … •Community contributed Databricks, AWS, GCP AcIon operators MAD · NOV 23-24 · 2018

25. •Move data between systems such as from Hive to Mysql

    or from S3 to Hive •Built-in HiveToMySqlTransfer, S3ToHiveTransfer •Community contributed Databricks, AWS, GCP Transfer operators MAD · NOV 23-24 · 2018

26. •Triggers downstream tasks in the dependency graph when a certain

    criteria is met. For example, checking for a certain file has become available on S3 before using it downstream •Built-in HiveParXXonSensor, HcpSensor, S3KeySensor, SqlSensor, FTPSensor, … •Community contributed Databricks, AWS, GCP Sensor operators MAD · NOV 23-24 · 2018

27. Advanced example MAD · NOV 23-24 · 2018 •Waits for

    a key (a file-like instance on S3) to be present in a S3 bucket •Add a new AWS Athena parXXon •Run an AWS Glue job and wait unXl its done •NoXfy the #dataservices channel at MicrosoE Teams S3KeySensor AwsAthenaQueryOperator AwsGlueScheduleJobOperator & AWSGlueJobSensor MSTeamsWebhookOperator

28. with DAG("commit_dag", default_args=default_args, schedule_interval="@weekly") as dag: s3_key_sensor = S3KeySensor( task_id="s3_key_sensor",

    bucket_name="dataservices-ingest", bucket_key="ingest/client_xyz/year={{ execution_date.year }}/" \ "week={{ execution_date.isocalendar()[1] }}/{{ ds_nodash }}.json.gz") aws_athena_update_raw_partition = AwsAthenaQueryOperator( query=""" ALTER TABLE raw_client_xyz ADD IF NOT EXISTS PARTITION (year='{{ execution_date.year }}', week='{{ execution_date.isocalendar()[1] }}');""", task_id="athena_update_raw_partition") aws_glue_job_schedule = AwsGlueScheduleJobOperator( job_name="commit_job", job_args={ "--source_database": "dataservices_staging", "--source_table_name": "raw_client_xyz", "--partition_year": "{{ execution_date.year }}", "--partition_week": "{{ execution_date.isocalendar()[1] }}"}, task_id="aws_glue_job_schedule") aws_glue_job_sensor = AWSGlueJobSensor( job_name="commit_job", job_run_id="{{ task_instance.xcom_pull(task_ids='aws_glue_job_schedule', key='aws_glue_job_run_id') }}", task_id="aws_glue_job_sensor") s3_processed_key_check = S3KeySensor( bucket_name="dataservices-processed", bucket_key="processed/client_xyz/year={{ execution_date.year }}/" \ "week={{ execution_date.isocalendar()[1] }}/{{ ds_nodash }}.json.gz", task_id="s3_processed_key_check") ms_teams_notify = MSTeamsWebhookOperator( task_id="ms_teams_notify", message="Data from client ✨ xyz ✨ has been processed") s3_key_sensor >> aws_athena_update_raw_partition >> aws_glue_job_schedule >> \ aws_glue_job_sensor >> s3_processed_key_check >> ms_teams_notify

29. What happens if… the ETL script has a bug …and

    has been running for weeks?

30. We can travel in Ime and rerun * only *

    the related tasks downstream without side effects. This is called backfilling.

31. How everything fits together

32. MAD · NOV 23-24 · 2018 Airflow UI (webserver) Metadata

    DB Scheduler Worker 1 Worker 2 Worker nth The big picture Airflow CLI Airflow REST API

33. •Webserver: Airflow's UI •Scheduler: responsible for scheduling jobs •Executor: Local,

    Celery, , •Metadata Database Core components MAD · NOV 23-24 · 2018

34. •Airflow webserver UI •Airflow CLI •Airflow Rest API server (experimental)

    Interfaces MAD · NOV 23-24 · 2018

35. Airflow CLI $ airflow [-h] <command> MAD · NOV 23-24

    · 2018

36. Airflow CLI scheduler -n $NUM Core services webserver -p 80

    Meta-DB operaXons initdb resetdb upgradedb Operate on DAGs pause unpause run trigger_dag backfill dag_state task_state clear Develop & test list_dags list_tasks variables render test MAD · NOV 23-24 · 2018

37. •A quick look into DAG and task progress •Error logging

    •Browse metadata: xcom, variables, SLAs •Historical stats Airflow webserver UI MAD · NOV 23-24 · 2018

38. Moving data MAD · NOV 23-24 · 2018 In theory,

    all data processing and storage should be done in external systems with Airflow only containing workflow metadata

39. •Variables: staXc values, config values, api keys •XComs: short for

    "cross communicaXon” CommunicaXon between tasks. Such as file name found by a sensor •ConnecIons: JDBCs, auth, etc. Airflow’s metadata storage MAD · NOV 23-24 · 2018

40. •Failure handling and monitoring: retry policies, SLAs •Smarter Cron allows

    to define more complex schedules. Even/odd days… •Complex dependencies (trigger rules) •Backfills •Template system: Jinja BaWeries included MAD · NOV 23-24 · 2018

41. •Idempotency (we’ll talk about this later…) •Logs can be piped

    to remote storage (S3, GCS …) •Backoff retries •Stage transformed data Best pracIces MAD · NOV 23-24 · 2018

42. Recipes

43. GeneraIng dynamic DAGs MAD · NOV 23-24 · 2018 Airflow

    checks every .py file in the DAGs folder… …and registers any available DAG defined as a global variable

44. for n in range(1, 10): dag_id = 'hello_world_{}'.format(str(n)) default_args =

    {'owner': 'airflow', 'start_date': datetime(2018, 1, 1)} schedule = '@daily' dag_number = n globals()[dag_id] = create_dag(dag_id, schedule, dag_number, default_args)

45. GeneraIng dynamic DAGs MAD · NOV 23-24 · 2018 •A

    Variable value •A staIc file (yaml, json, etc.) •Any external source (based on ConnecIons) It’s possible to create a DAG from:

46. Data Engineering

47. MAD · NOV 23-24 · 2018 Does anybody know about

    Maslow's hierarchy of needs?

48. MAD · NOV 23-24 · 2018 Data literacy, collecIon, and

    infrastructure

49. •Design, build, and maintain data warehouses •A data warehouse is

    a place where raw data is transformed and stored in query-able forms •Enable higher level analyXcs, be it business intelligence, online experimentaXon, or machine learning MAD · NOV 23-24 · 2018 The role

50. •SQL mastery: if english is the language of business, SQL

    is the language of data •Load data incrementally •Process historic data (backfilling) •ParXXon ingested data •Enforce the idempotency constrain MAD · NOV 23-24 · 2018 Key skills

51. Can't see the forest for the trees

52. MAD · NOV 23-24 · 2018 Airflow empowers data engineers

    FuncIonal data engineering

53. •Reproducible: determinisXc and idempotent •Re-running a task for the same

    date should always produce same output •Future proof: backfilling, versioning •Data can be repaired by rerunning the new code, either by clearing tasks or doing backfills MAD · NOV 23-24 · 2018 FuncIonal data engineering

54. MAD · NOV 23-24 · 2018

55. •Extract: where sensors wait for upstream data sources to land

    •Transform: where we apply business logic and perform acXons such as filtering, grouping, and aggregaXon to translate raw data into analysis- ready datasets •Load: we load the processed data and transport them to a final desXnaXon ETLs MAD · NOV 23-24 · 2018

56. Airflow deployment MAD · NOV 23-24 · 2018 https://github.com/idealista/airflow-role

57. Airflow deployment MAD · NOV 23-24 · 2018

58. Airflow deployment MAD · NOV 23-24 · 2018

59. Airflow at Idealista MAD · NOV 23-24 · 2018 •Deployment

    via Ansible role •AWS Glue & AWS Athena plugins •MicrosoE Teams plugin •AWS S3 to FTP sync operator

60. Q&A

61. Useful resources MAD · NOV 23-24 · 2018 •Useful SQL

    queries for Apache Airflow •Astronomer guides •Airflow maintenance dags •A Beginner’s Guide to Data Engineering by Robert Chang •The Rise of the Data Engineer by Maxime Beauchemin

62. CitaIons & AWribuIons MAD · NOV 23-24 · 2018 •

    Slide 3: image from XKCD (hcps://xkcd.com/2054/) • Slide 4: picture by Andrew Seaman (hcps://unsplash.com/photos/EuDUHo5yyAg) • Slide 8: image from Pinterest (hcps://www.pinterest.com/pin/377739487476469866/) • Slide 9: Apache Airflow logo (hcps://airflow.apache.org/) • Slide 11: logos belong to the respecXve companies • Slide 14: image from Wikipedia (hcps://en.wikipedia.org/wiki/Directed_acyclic_graph) • Slide 31: picture by Garec Mizunaka (hcps://unsplash.com/photos/xFjX9rYILo) • Slide 33: Flask logo (hcp://flask.pocoo.org/), Mesos logo (hcp://mesos.apache.org/), Kubernetes logo (hcps://kubernetes.io/) • Slide 42: picture by Dan Gold (hcps://unsplash.com/photos/5O1ddenSM4g) • Slide 43-45: hcps://www.astronomer.io/guides/dynamically-generaXng-dags/ • Slide 48: Monica RogaX’s “The AI Hierarchy of Needs” hcps://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007 • Slide 49: Robert Chang’s “A Beginner’s Guide to Data Engineering” hcps://medium.com/@rchang/a-beginners-guide-to-data-engineering-part- i-4227c5c457d7 • Slide 50: Maxime Beauchemin’s “The Rise of the Data Engineer” hcps://medium.freecodecamp.org/the-rise-of-the-data-engineer-91be18f1e603 • Slide 51: picture by Jesse Gardner (hcps://unsplash.com/photos/mERlBKFGJiQ) • Slide 54: Vineet Goel’s “Why Robinhood uses Airflow?” (hcps://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8) • Slide 56: Ansible logo (hcps://www.ansible.com/) • Slide 57: Astronomer logo (hcps://www.astronomer.io) • Slide 58: Google Cloud Pla`orm logo and Google Cloud Composer logo (hcps://cloud.google.com/composer/) • Slide 60: picture by Edwin Andrade (hcps://unsplash.com/photos/4V1dC_eoCwg) • Commit logo (hcps://2018.commit-conf.com)