Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Data Pipelines in Python
Search
Marco Bonzanini
April 16, 2016
Programming
2
550
Building Data Pipelines in Python
Slides of my talk at PyCon7 in Florence (April 2016)
Marco Bonzanini
April 16, 2016
Tweet
Share
More Decks by Marco Bonzanini
See All by Marco Bonzanini
Pitfalls in Data Science Projects (and how to avoid them)
marcobonzanini
0
17
Is Your Open-source LLM Really Open?
marcobonzanini
0
31
Perambulations in Football Analytics
marcobonzanini
0
20
Natural Language Processing Expert Briefing @ PyData Global 2022
marcobonzanini
0
74
Natural Language Processing Expert Briefing @ PyData Global 2021
marcobonzanini
0
97
Getting into Data Science @ HisarCS 2021
marcobonzanini
0
220
Mining topics in documents with topic modelling and Python @ London Python meetup
marcobonzanini
1
200
Topic Modelling workshop @ PyCon UK 2019
marcobonzanini
2
94
Lies, Damned Lies, and Statistics @ PyCon UK 2019
marcobonzanini
0
100
Other Decks in Programming
See All in Programming
ファインディLT_ポケモン対戦の定量的分析
fufufukakaka
0
950
自力でTTSモデルを作った話
zgock999
0
120
CDKを使ったPagerDuty連携インフラのテンプレート化
shibuya_shogo
0
120
1年目の私に伝えたい!テストコードを怖がらなくなるためのヒント/Tips for not being afraid of test code
push_gawa
1
650
Go 1.24でジェネリックになった型エイリアスの紹介
syumai
2
300
メンテが命: PHPフレームワークのコンテナ化とアップグレード戦略
shunta27
0
320
Jakarta EE meets AI
ivargrimstad
0
650
Amazon Bedrockマルチエージェントコラボレーションを諦めてLangGraphに入門してみた
akihisaikeda
1
160
未経験でSRE、はじめました! 組織を支える役割と軌跡
curekoshimizu
1
200
SwiftUI Viewの責務分離
elmetal
PRO
2
280
⚪⚪の⚪⚪をSwiftUIで再現す る
u503
0
120
仕様変更に耐えるための"今の"DRY原則を考える
mkmk884
9
3.3k
Featured
See All Featured
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
440
A designer walks into a library…
pauljervisheath
205
24k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
227
22k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
Docker and Python
trallard
44
3.3k
Scaling GitHub
holman
459
140k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
How GitHub (no longer) Works
holman
314
140k
Transcript
Building Data Pipelines in Python Marco Bonzanini ! PyCon Italia
- Florence 2016
Nice to meet you • @MarcoBonzanini • “Type B” Data
Scientist • PhD in Information Retrieval • Book with PacktPub (July 2016) • Usually at PyData London
R&D ≠ Engineering R&D results in production = high value
None
Big Data Problems vs Big Data Problems
Data Pipelines Data ETL Analytics • Many components in a
data pipeline: • Extract, Clean, Augment, Join data
Good Data Pipelines Easy to reproduce Easy to productise
Towards Good Pipelines • Transform your data, don’t overwrite •
Break it down into components • Different packages (e.g. setup.py) • Unit tests vs end-to-end tests Good = Replicable and Productisable
Anti-Patterns • Bunch of scripts • Single run-everything script •
Hacky homemade dependency control • Don’t reinvent the wheel
Intermezzo Let me rant about testing Icon by Freepik from
flaticon.com
(Unit) Testing • Unit tests in three easy steps: •
import unittest • Write your tests • Quit complaining about lack of time to write tests
Benefits of (unit) testing • Safety net for refactoring •
Safety net for lib upgrades • Validate your assumptions • Document code / communicate your intentions • You’re forced to think
Testing: not convinced yet?
Testing: not convinced yet?
Testing: not convinced yet? f1 = fscore(p, r) min_bound,
max_bound = sorted([p, r]) assert min_bound <= f1 <= max_bound
Testing: I’m almost done • Unit tests vs Defensive Programming
• Say no to tautologies • Say no to vanity tests • Know the ecosystem: py.test, nosetests, hypothesis, coverage.py, …
</rant>
Intro to Luigi GNU Make + Unix pipes + Steroids
• Workflow manager in Python, by Spotify • Dependency management • Error control, checkpoints, failure recovery • Minimal boilerplate • Dependency graph visualisation $ pip install luigi
Luigi Task: unit of execution class MyTask(luigi.Task): ! def requires(self):
pass # list of dependencies def output(self): pass # task output def run(self): pass # task logic
Luigi Target: output of a task class MyTarget(luigi.Target): ! def
exists(self): pass # return bool Off the shelf support for local file system, S3, Elasticsearch, RDBMS (also via luigi.contrib)
Not only Luigi • More Python-based workflow managers: • Airflow
by Airbnb • Mrjob by Yelp • Pinball by Pinterest
When things go wrong • import logging • Say no
to print() for debugging • Custom log format / extensive info • Different levels of severity • Easy to switch off or change level
Who reads the logs? You’re not going to read the
logs, unless… • E-mail notifications • built-in in Luigi • Slack notifications $ pip install luigi_slack # WIP
Summary • R&D is not Engineering: can we meet halfway?
• Prototypes vs. Products • Automation and replicability matter • You need a workflow manager • Good engineering principles help: • Testing, logging, packaging, …
Vanity Slide • speakerdeck.com/marcobonzanini • github.com/bonzanini • marcobonzanini.com • @MarcoBonzanini