Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Scaling your data infrastructure
barrachri
April 20, 2018
Technology
1
100
Scaling your data infrastructure
Scaling your data infrastructure @ PyConNove
barrachri
April 20, 2018
Tweet
Share
More Decks by barrachri
See All by barrachri
Will Tech Save Us?
barrachri
0
85
How software can feed the World
barrachri
1
100
How to fight with yourself and win.
barrachri
0
180
Introduction to Statistics with Python
barrachri
0
220
EuroPython 2015 and the future
barrachri
2
98
Start with Flask
barrachri
3
160
Django & Docker
barrachri
6
840
Other Decks in Technology
See All in Technology
JAWS-UG 横浜 #54 資料
takakuni
0
220
Cloudflare Workersで動くOG画像生成器
aiji42
1
500
cdk deployに必要な権限ってなんだ?
kinyok
0
190
FlexScan HD2452Wの 後継を探して
tring
0
6.5k
CES_2023_FleetWise_demo.pdf
sparkgene
0
120
ChatGPT for Hacking
anugrahsr
0
4.5k
Kaggleシミュレーションコンペの動向
nagiss
0
280
ECテックカンファレンス2023 EC事業部のモバイル開発2023
tatsumi0000
0
320
Multi-Cloud Gatewayでデータを統治せよ!/ Data Federation with MCG
tutsunom
1
320
S3とCloudWatch Logsの見直しから始めるコスト削減 / Cost saving S3 and CloudWatch Logs
shonansurvivors
0
260
組織に対してSREを適用するとどうなるか
kuniim
9
2.9k
あつめたデータをどう扱うか
skrb
2
160
Featured
See All Featured
5 minutes of I Can Smell Your CMS
philhawksworth
198
18k
Intergalactic Javascript Robots from Outer Space
tanoku
261
26k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
349
27k
The Language of Interfaces
destraynor
149
21k
Pencils Down: Stop Designing & Start Developing
hursman
114
10k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
24
4.6k
WebSockets: Embracing the real-time Web
robhawkes
58
6k
The Web Native Designer (August 2011)
paulrobertlloyd
76
2.2k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
31
20k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
The World Runs on Bad Software
bkeepers
PRO
59
5.7k
Rebuilding a faster, lazier Slack
samanthasiow
69
7.5k
Transcript
Scaling your data infrastructure C H R I S T
I A N B A R R A @ P Y C O N N O V E
THE AGENDA 2 3 START THE DATA SCIENCE WORKFLOW SCALING
IS NOT JUST A MATTER OF MACHINE WHEN THE SIZE OF YOUR DATA MATTERS 1
THE AGENDA 4 5 CONTAINERIZED DATA SCIENCE CASSINY: PUT ALL
THE THINGS TOGETHER END
THE DATA SCIENCE WORKFLOW
HEXAGON PRESENTATION TEMPLATE
HOW YOU BUILD, ITERATE AND SHARE DEPENDS ON MANY THINGS
Your Users Your Product Your Team Your Company Your Tech Stack Your Domain
SCIKIT-LEARN DOCKER DATA SCIENCE TOOLBELT PANDAS JUPYTER RAY
SCALING IS NOT JUST A MATTER OF MACHINES
We all use it.
We really care about versioning. We have Untitled_1.ipynb, Untitled_2.ipynb and
Untitled_3.ipynb. HOMER SIMPSON C H I E F D A T A S C I E N T I S T D A T A B E E R I N C
Since JSON is a plain text format, they can be
version-controlled and shared with colleagues. E X I P Y T H O N N O T E B O O K D O C U M E N T A T I O N
THEY GOT IT RIGHT
BUT WE KEEP IMPROVING
90% OF JUPITER IS MADE BY HYDROGEN
THE HARD THING ABOUT STORAGE
PARQUET P A R Q U E T + O
B J E C T S T O R A G E = YO U C A N Q U E R Y I T U S I N G S Q L PA N DA S H A S N AT I V E S U P P O R T F O R G E T A B O U T C S V
WHEN THE SIZE OF YOUR DATA MATTERS
IT’S TOO SLOW DOESN’T FIT IN YOUR RAM
CODE OPTIMIZATION APPROACH SCALING FROM DIFFERENT SIDES A BIGGER MACHINE
USE MULTIPLE CORES MORE MACHINES FRAMEWORKS: DASK RAY SPARK PANDAS: READ BY CHUNKS SCIKIT-LEARN: PARTIAL FIT
chunks & partial_fit 1 M A C H I N
E
Multiple machines. n M A C H I N E
S
I don’t want to use Spark/JVM, what do you have
for me? H A P P Y P Y T H O N U S E R
WHAT IS RAY?
A high-performance distributed execution engine REDIS SCHEDULER WORKER ARROW &
PLASMA
Use pandas through ray to query parquet files in an
object storage. W O R K I N P R O G R E S S
CONTAINERIZED DATA SCIENCE
If you trained a model with scikit-learn 0.18.1, will the
same model work with 0.19.1? P R O B L E M # 1
How do you share your models? P R O B
L E M # 2
How do you put your models in production? P R
O B L E M # 3
Containerize everything. T H E A N S W E
R
1. It’s damn easy to move things around 2. You
get versioning for free 3. Stack agnostic 4. Move Docker images around T O R E C A P
CASSINY: PUT ALL THE THINGS TOGETHER
CLEAR REQUIREMENTS CONTAINERIZED EASY OBJECT STORAGE JUPYTER + IPYTHON PLATFORM
AGNOSTIC
OPEN SOURCE
DEMO
TAKEAWAYS UNIFIED DATA WAREHOUSE KEEP YOUR CODE RUNNING ON ONE
MACHINE USE DOCKER TRY RAY BRING CI/CD TO YOUR DATASCIENCE WORKFLOW OBJECT STORAGE IS COOL DISTRIBUTED COMPUTING IS HARD I DIDN’T HAVE ANOTHER POINT
None