Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Airflow: Data Workflow Management

Gilson Filho
October 20, 2017
Programming
0
460

Airflow: Data Workflow Management

Apresentação do Airflow como Lightning Talk na Python Brasil[13] em Belo Horizonte

Gilson Filho

October 20, 2017
Tweet

More Decks by Gilson Filho

See All by Gilson Filho
Como monitorar os erros da sua aplicação com o Sentry
gilsondev
0
2.1k
Airflow - Automatizando seu fluxo de trabalho
gilsondev
1
1.2k
Protocolo HTTP
gilsondev
0
520
Workshop Docker
gilsondev
0
410
Django: framework web para perfeccionistas com prazos
gilsondev
0
410
Django - Framework para perfeccionistas em prazos
gilsondev
0
46
Vim - Um editor onipresente e flexível
gilsondev
2
460
Vim - Um editor onipresente e flexível
gilsondev
2
120
Desenvolvendo aplicações web com python e web2py
gilsondev
0
590

Other Decks in Programming

See All in Programming
知られざるDMMデータエンジニアの生態 〜かつてツチノコと呼ばれし者〜
takaha4k
2
830
Scaling your build logic
antalmonori
1
120
『改訂新版 良いコード／悪いコードで学ぶ設計入門』活用方法−爆速でスキルアップする！効果的な学習アプローチ / effective-learning-of-good-code
minodriven
29
4.5k
Запуск 1С:УХ в крупном энтерпрайзе: мечта и реальность ПМа
lamodatech
0
970
functionalなアプローチで動的要素を排除する
ryopeko
1
570
見えないメモリを観測する: PHP 8.4 `pg_result_memory_size()` とSQL結果のメモリ管理
kentaroutakeda
0
960
AHC041解説
terryu16
0
470
ISUCON14感想戦で85万点まで頑張ってみた
ponyo877
1
750
React 19でお手軽にCSS-in-JSを自作する
yukukotani
5
580
令和7年版 あなたが使ってよいフロントエンド機能とは
mugi_uno
12
5.7k
PicoRubyと暮らす、シェアハウスハック
ryosk7
0
240
“あなた” の開発を支援する AI エージェント Bedrock Engineer / introducing-bedrock-engineer
gawa
7
870

Featured

See All Featured
GraphQLとの向き合い方2022年版
quramy
44
13k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
30
2.1k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
RailsConf 2023
tenderlove
29
980
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
Building a Modern Day 
E-commerce SEO Strategy
aleyda
38
7.1k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.3k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
20
2.4k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
970
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.2k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k

Transcript

  1. Airflow Data Workflow Management

  2. O que é? Execução de workflow de dados Escritos em

    Python Forma de DAGs Criado pelo Airbnb Open Source (Apache License) Desenvolvido em Python Flask, SQLAlchemy, Jinja, etc Alternativa ao Luigi do Spotify

  3. DAG Directed Acyclic Graphs Rede de jobs Possui agendamento próprio

    Código enxuto e fácil de entender quando criado

  4. DAG Airflow - Backfill

  5. DAG Airflow - Visualização de Grafos

  6. DAG Airflow - Detalhes

  7. Alguns processos feitos com Airflow Data warehousing: cleanse, organize, data

    quality check Growth analytics Experimentation: compute our A/B testing Email targeting Sessionization: compute clickstream Search: compute search ranking related metrics Data infrastructure maintenance: database scrapes, folder cleanup, applying data retention policies, … feitos pelo Airbnb

  8. meucandidato.me

  9. Blog: Telegram: Meetup: df.python.org.br @grupydf meetup.com/pt-BR/pythondf

  10. Email: [email protected] Blog: Telegram/Github: @gilsondev Twitter: gilsondev.in @gilsonfilho Em que

    posso ajudar?