Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Airflow: Data Workflow Management

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Gilson Filho Gilson Filho
October 20, 2017
Programming
0
490

Airflow: Data Workflow Management

Apresentação do Airflow como Lightning Talk na Python Brasil[13] em Belo Horizonte

Avatar for Gilson Filho

Gilson Filho

October 20, 2017
Tweet

More Decks by Gilson Filho

See All by Gilson Filho
Como monitorar os erros da sua aplicação com o Sentry
Avatar for Gilson Filho gilsondev
0
2.1k
Airflow - Automatizando seu fluxo de trabalho
Avatar for Gilson Filho gilsondev
1
1.3k
Protocolo HTTP
Avatar for Gilson Filho gilsondev
0
570
Workshop Docker
Avatar for Gilson Filho gilsondev
0
460
Django: framework web para perfeccionistas com prazos
Avatar for Gilson Filho gilsondev
0
450
Django - Framework para perfeccionistas em prazos
Avatar for Gilson Filho gilsondev
0
52
Vim - Um editor onipresente e flexível
Avatar for Gilson Filho gilsondev
2
480
Vim - Um editor onipresente e flexível
Avatar for Gilson Filho gilsondev
2
140
Desenvolvendo aplicações web com python e web2py
Avatar for Gilson Filho gilsondev
0
600

Other Decks in Programming

See All in Programming
AIとペアプロして処理時間を97%削減した話 #pyconshizu
Avatar for Kashun Yoshida kashewnuts
1
210
米国のサイバーセキュリティタイムラインと見る Goの暗号パッケージの進化
Avatar for tom twinkle tomtwinkle
2
510
「やめとこ」がなくなった — 1月にZennを始めて22本書いた AI共創開発のリアル
Avatar for atani atani14
0
360
コードレビューをしない選択 #でぃーぷらすトウキョウ
Avatar for Takuma Kajikawa kajitack
0
220
20260228_JAWS_Beginner_Kansai
Avatar for Takuya Yonezawa takuyay0ne
5
460
ふつうのRubyist、ちいさなデバイス、大きな一年 / Ordinary Rubyists, Tiny Devices, Big Year
Avatar for chobishiba chobishiba
1
410
Ruby x Terminal
Avatar for Akira Matsuda a_matsuda
7
590
RubyとGoでゼロから作る証券システム： 高信頼性が求められるシステムのコードの外側にある設計と運用のリアル
Avatar for free_world21 free_world21
0
230
AWS×クラウドネイティブソフトウェア設計 / AWS x Cloud-Native Software Design
Avatar for nrs nrslib
13
2.7k
社内規程RAGの精度を73.3% → 100%に改善した話
Avatar for oharu121 oharu121
13
7.8k
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
330
朝日新聞のデジタル版を支えるGoバックエンド ー価値ある情報をいち早く確実にお届けするために
Avatar for JunkiIshida junkiishida
1
680

Featured

See All Featured
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.1k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
140
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8k
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
600
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.1k
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
122
21k
How to Grow Your eCommerce with AI & Automation
Avatar for Katarina Dahlin katarinadahlin
PRO
1
130
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
120

Transcript

  1. Airflow Data Workflow Management

  2. O que é? Execução de workflow de dados Escritos em

    Python Forma de DAGs Criado pelo Airbnb Open Source (Apache License) Desenvolvido em Python Flask, SQLAlchemy, Jinja, etc Alternativa ao Luigi do Spotify

  3. DAG Directed Acyclic Graphs Rede de jobs Possui agendamento próprio

    Código enxuto e fácil de entender quando criado

  4. DAG Airflow - Backfill

  5. DAG Airflow - Visualização de Grafos

  6. DAG Airflow - Detalhes

  7. Alguns processos feitos com Airflow Data warehousing: cleanse, organize, data

    quality check Growth analytics Experimentation: compute our A/B testing Email targeting Sessionization: compute clickstream Search: compute search ranking related metrics Data infrastructure maintenance: database scrapes, folder cleanup, applying data retention policies, … feitos pelo Airbnb

  8. meucandidato.me

  9. Blog: Telegram: Meetup: df.python.org.br @grupydf meetup.com/pt-BR/pythondf

  10. Email: [email protected] Blog: Telegram/Github: @gilsondev Twitter: gilsondev.in @gilsonfilho Em que

    posso ajudar?