Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with Python
Search
Miguel Cabrera
October 31, 2016
Technology
0
230
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
210
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
110
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
140
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
170
Python and Life Hacking with Emacs
mfcabrera
2
280
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.7k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
200
Dictionary Learning for Music Genre Recognition
mfcabrera
0
230
Other Decks in Technology
See All in Technology
Python と Snowflake はズッ友だょ!~ Snowflake の Python 関連機能をふりかえる ~
__allllllllez__
1
120
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
1
300
LangSmith入門―トレース/評価/プロンプト管理などを担うLLMアプリ開発プラットフォーム
os1ma
3
170
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
5
530
アクセシビリティを考慮したUI/CSSフレームワーク・ライブラリ選定
yajihum
2
1k
Compose Compiler Metricsを使った実践的なコードレビュー
tomorrowkey
1
220
非同期推論システムによるコスト削減と信頼性向上
koki_nishihara
0
260
【NW X Security JAWS#3】L3-4:AWS環境のIPv6移行に向けて知っておきたいこと
shotashiratori
0
220
JAWS-UG Bedrock Claude Night
yamahiro
3
610
Vertex AI を中心に 生成AIのアップデートを共有します
kaz1437
0
310
長期間TiDBを使ってきた話 @ 私たちはなぜNewSQLを使うのかTiDB選定5社が語る選定理由と活用LT / Experiences with TiDB Over Time
chibiegg
2
900
20分で完全に理解するGrafanaダッシュボード
hamadakoji
3
640
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
4 Signs Your Business is Dying
shpigford
175
21k
How to train your dragon (web standard)
notwaldorf
73
5.2k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
19
1.7k
Art, The Web, and Tiny UX
lynnandtonic
289
19k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
121
39k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.6k
GraphQLとの向き合い方2022年版
quramy
32
12k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight