Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with P...
Search
Miguel Cabrera
October 31, 2016
Technology
0
290
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
250
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
110
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
150
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.2k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
180
Python and Life Hacking with Emacs
mfcabrera
2
320
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.8k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
250
Dictionary Learning for Music Genre Recognition
mfcabrera
0
250
Other Decks in Technology
See All in Technology
目標と時間軸 〜ベイビーステップでケイパビリティを高めよう〜
kakehashi
PRO
8
950
Amazon Aurora のバージョンアップ手法について
smt7174
2
180
技術スタックだけじゃない、業務ドメイン知識のオンボーディングも同じくらいの量が必要な話
niftycorp
PRO
0
130
サバイバルモード下でのエンジニアリングマネジメント
konifar
20
6k
RayでPHPのデバッグをちょっと快適にする
muno92
PRO
0
200
Ruby on Railsで持続可能な開発を行うために取り組んでいること
am1157154
3
160
[OpsJAWS Meetup33 AIOps] Amazon Bedrockガードレールで守る安全なAI運用
akiratameto
1
130
データベースの負荷を紐解く/untangle-the-database-load
emiki
2
550
Охота на косуль у древних
ashapiro
0
120
開発組織を進化させる!AWSで実践するチームトポロジー
iwamot
2
520
プルリクエストレビューを終わらせるためのチーム体制 / The Team for Completing Pull Request Reviews
nekonenene
1
920
Change Managerを活用して本番環境へのセキュアなGUIアクセスを統制する / Control Secure GUI Access to the Production Environment with Change Manager
yuj1osm
0
110
Featured
See All Featured
Fireside Chat
paigeccino
35
3.2k
Automating Front-end Workflow
addyosmani
1369
200k
The Language of Interfaces
destraynor
156
24k
Into the Great Unknown - MozCon
thekraken
35
1.6k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
129
19k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.5k
Scaling GitHub
holman
459
140k
Building an army of robots
kneath
303
45k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Code Reviewing Like a Champion
maltzj
521
39k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7.1k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera mfcabrera@gmail.com Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello” me@mail.org
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight