Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with P...
Search
Miguel Cabrera
October 31, 2016
Technology
0
320
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
280
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
120
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
160
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.2k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
190
Python and Life Hacking with Emacs
mfcabrera
2
330
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.9k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
280
Dictionary Learning for Music Genre Recognition
mfcabrera
0
250
Other Decks in Technology
See All in Technology
Talk to Someone At Delta Airlines™️ USA Contact Numbers
travelcarecenter
0
170
ObsidianをLLM時代のナレッジベースに! クリッピング→Markdown→CLI連携の実践
srvhat09
7
8.8k
DATA+AI SummitとSnowflake Summit: ユーザから見た共通点と相違点 / DATA+AI Summit and Snowflake Summit
nttcom
0
200
2025/07/22_家族アルバム みてねのCRE における生成AI活用事例
masartz
2
100
AIを使っていい感じにE2Eテストを書けるようになるまで / Trying to Write Good E2E Tests with AI
katawara
2
1.6k
会社もクラウドも違うけど 通じたコスト削減テクニック/Cost optimization strategies effective regardless of company or cloud provider
aeonpeople
2
140
組織内、組織間の資産保護に必要なアイデンティティ基盤と関連技術の最新動向
fujie
0
500
大規模組織にAIエージェントを迅速に導入するためのセキュリティの勘所 / AI agents for large-scale organizations
i35_267
6
210
Webの技術とガジェットで那須の子ども達にワクワクを! / IoTLT_20250720
you
PRO
0
120
MCP とマネージド PaaS で実現する大規模 AI アプリケーションの高速開発
nahokoxxx
1
1.4k
Microsoft Defender XDRで疲弊しないためのインシデント対応
sophiakunii
3
400
P2P ではじめる WebRTC のつまづきどころ
tnoho
1
190
Featured
See All Featured
How STYLIGHT went responsive
nonsquared
100
5.6k
Agile that works and the tools we love
rasmusluckow
329
21k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
What's in a price? How to price your products and services
michaelherold
246
12k
It's Worth the Effort
3n
185
28k
Side Projects
sachag
455
43k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Optimizing for Happiness
mojombo
379
70k
Writing Fast Ruby
sferik
628
62k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight