Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Airflow: Data Workflow Management

054d9fa364a9f7db6985df0645b21b42?s=47 Gilson Filho
October 20, 2017
Programming
0
320

Airflow: Data Workflow Management

Apresentação do Airflow como Lightning Talk na Python Brasil[13] em Belo Horizonte

054d9fa364a9f7db6985df0645b21b42?s=128

Gilson Filho

October 20, 2017
Tweet

More Decks by Gilson Filho

See All by Gilson Filho
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
1.1k
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
1
550
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
300
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
290
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
300
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
30
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
2
350
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
2
77
054d9fa364a9f7db6985df0645b21b42?s=48 gilsondev
0
330

Other Decks in Programming

See All in Programming
7f408bc67dc9ae3b288ee92d16d3c4c2?s=48 deepu105
0
230
5155c9bc5b30d9e2a8fd677695887861?s=48 kaonash
2
1.9k
438799c739f93d00abc30fa2da33c543?s=48 rukiadia
3
870
8a43d544df4886ea5ef714e78f3420a7?s=48 chimerast
0
340
0e6b2b5b4861d0e4b136527c0d7d428e?s=48 imk2o
0
200
C90bac78c0fb61105cfd8239767f903d?s=48 o0h
PRO
3
1.6k
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
160
E7151ab8219e76672f7a7691dd2c88e6?s=48 seike460
7
2.2k
326530203cf3b827e41f68edb45e7509?s=48 bamboooo
1
130
B7bf4e80841a44f921665e88d874f2c3?s=48 treby
0
130
1ac5890739d082cd2f2f91e3c1799b4d?s=48 hamakou108
4
520
F77032adcbe77d2777bb0e0c30873159?s=48 etagwerker
1
130

Featured

See All Featured
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
440
28k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
119
16k
15f2b8221f247985a27a627d2ece72f0?s=48 pauljervisheath
195
15k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
146
20k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
193
8.7k
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
70
3.6k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
56
5.4k
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
176
14k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
25
1k
C51b39fa3c79ccb99dcb8a076bc98287?s=48 brianwarren
82
4.8k
282d599b414022eba5096510f675b6e2?s=48 chrislema
173
14k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
1346
200k

Transcript

  1. Airflow Data Workflow Management

  2. O que é? Execução de workflow de dados Escritos em

    Python Forma de DAGs Criado pelo Airbnb Open Source (Apache License) Desenvolvido em Python Flask, SQLAlchemy, Jinja, etc Alternativa ao Luigi do Spotify

  3. DAG Directed Acyclic Graphs Rede de jobs Possui agendamento próprio

    Código enxuto e fácil de entender quando criado

  4. DAG Airflow - Backfill

  5. DAG Airflow - Visualização de Grafos

  6. DAG Airflow - Detalhes

  7. Alguns processos feitos com Airflow Data warehousing: cleanse, organize, data

    quality check Growth analytics Experimentation: compute our A/B testing Email targeting Sessionization: compute clickstream Search: compute search ranking related metrics Data infrastructure maintenance: database scrapes, folder cleanup, applying data retention policies, … feitos pelo Airbnb

  8. meucandidato.me

  9. Blog: Telegram: Meetup: df.python.org.br @grupydf meetup.com/pt-BR/pythondf

  10. Email: me@gilsondev.in Blog: Telegram/Github: @gilsondev Twitter: gilsondev.in @gilsonfilho Em que

    posso ajudar?