Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Technology
0
180
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
100
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
46
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
120
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
810
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
150
Python and Life Hacking with Emacs
mfcabrera
2
220
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.3k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
160
Dictionary Learning for Music Genre Recognition
mfcabrera
0
180
Other Decks in Technology
See All in Technology
プルリク作ったらデプロイされる仕組み on ECS / SRE NEXT 2022
carta_engineering
1
470
IDOLY PRIDEにおけるAssetBundleビルドパイプラインについて
qualiarts
0
310
Dagu | オンプレ向けワークフローエンジン(WebUI 同梱)
yohamta
0
190
モダンデータスタックとかの話(データエンジニアのお仕事とは)
foursue
0
470
Whats new in Android Media?
myolwin00
2
120
個人的に使って良かったUiPathアクティビティ
saoritakita
0
120
Microsoft Build 2022 - Azure のデータ & 分析サービス 最新アップデート / Microsoft Build 2022 Updates on Azure Data and Analytics Services
nakazax
1
130
プロダクトの理想と現実はなぜ乖離しがち?プロダクト作りに潜む問題を考える
suzukentaro
0
200
一人から始めるプロダクトSRE / How to start SRE in a product team, all by yourself
vtryo
4
2.9k
SRENEXT2022 組織にSREを実装していくまでの道のり
marnie0301
1
650
Adopting Kafka for the #1 job site in the world
ymyzk
1
660
YAMLを書くだけで構築できる分散ストレージ
sat
PRO
0
190
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
337
17k
Building Better People: How to give real-time feedback that sticks.
wjessup
343
17k
Happy Clients
brianwarren
89
5.5k
The Power of CSS Pseudo Elements
geoffreycrofte
46
3.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
4
450
Mobile First: as difficult as doing things right
swwweet
212
7.5k
Put a Button on it: Removing Barriers to Going Fast.
kastner
56
2.3k
Visualization
eitanlees
124
11k
Fantastic passwords and where to find them - at NoRuKo
philnash
25
1.5k
Designing the Hi-DPI Web
ddemaree
272
32k
The Pragmatic Product Professional
lauravandoore
19
2.9k
The Language of Interfaces
destraynor
148
20k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera mfcabrera@gmail.com Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello” me@mail.org
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight