Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Airflow: Data Workflow Management
Search
Gilson Filho
October 20, 2017
Programming
0
470
Airflow: Data Workflow Management
Apresentação do Airflow como Lightning Talk na Python Brasil[13] em Belo Horizonte
Gilson Filho
October 20, 2017
Tweet
Share
More Decks by Gilson Filho
See All by Gilson Filho
Como monitorar os erros da sua aplicação com o Sentry
gilsondev
0
2.1k
Airflow - Automatizando seu fluxo de trabalho
gilsondev
1
1.2k
Protocolo HTTP
gilsondev
0
540
Workshop Docker
gilsondev
0
420
Django: framework web para perfeccionistas com prazos
gilsondev
0
420
Django - Framework para perfeccionistas em prazos
gilsondev
0
47
Vim - Um editor onipresente e flexível
gilsondev
2
470
Vim - Um editor onipresente e flexível
gilsondev
2
130
Desenvolvendo aplicações web com python e web2py
gilsondev
0
600
Other Decks in Programming
See All in Programming
イベントストーミングから始めるドメイン駆動設計
jgeem
4
870
セキュリティマネジャー廃止とクラウドネイティブ型サンドボックス活用
kazumura
1
190
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
nrslib
1
120
Bytecode Manipulation 으로 생산성 높이기
bigstark
2
360
関数型まつりレポート for JuliaTokai #22
antimon2
0
130
Cline指示通りに動かない? AI小説エージェントで学ぶ指示書の書き方と自動アップデートの仕組み
kamomeashizawa
1
560
Effect の双対、Coeffect
yukikurage
5
1.4k
XP, Testing and ninja testing
m_seki
2
120
GraphRAGの仕組みまるわかり
tosuri13
7
450
Kotlin エンジニアへ送る:Swift 案件に参加させられる日に備えて~似てるけど色々違う Swift の仕様 / from Kotlin to Swift
lovee
1
250
Create a website using Spatial Web
akkeylab
0
290
ASP.NETアプリケーションのモダナイズ インフラ編
tomokusaba
1
390
Featured
See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Making Projects Easy
brettharned
116
6.3k
Raft: Consensus for Rubyists
vanstee
140
7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
GraphQLとの向き合い方2022年版
quramy
46
14k
We Have a Design System, Now What?
morganepeng
52
7.6k
A Tale of Four Properties
chriscoyier
160
23k
YesSQL, Process and Tooling at Scale
rocio
173
14k
KATA
mclloyd
29
14k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
It's Worth the Effort
3n
184
28k
Transcript
Airflow Data Workflow Management
O que é? Execução de workflow de dados Escritos em
Python Forma de DAGs Criado pelo Airbnb Open Source (Apache License) Desenvolvido em Python Flask, SQLAlchemy, Jinja, etc Alternativa ao Luigi do Spotify
DAG Directed Acyclic Graphs Rede de jobs Possui agendamento próprio
Código enxuto e fácil de entender quando criado
DAG Airflow - Backfill
DAG Airflow - Visualização de Grafos
DAG Airflow - Detalhes
Alguns processos feitos com Airflow Data warehousing: cleanse, organize, data
quality check Growth analytics Experimentation: compute our A/B testing Email targeting Sessionization: compute clickstream Search: compute search ranking related metrics Data infrastructure maintenance: database scrapes, folder cleanup, applying data retention policies, … feitos pelo Airbnb
meucandidato.me
Blog: Telegram: Meetup: df.python.org.br @grupydf meetup.com/pt-BR/pythondf
Email:
[email protected]
Blog: Telegram/Github: @gilsondev Twitter: gilsondev.in @gilsonfilho Em que
posso ajudar?