Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling your data infrastructure
Search
barrachri
April 20, 2018
Technology
1
180
Scaling your data infrastructure
Scaling your data infrastructure @ PyConNove
barrachri
April 20, 2018
Tweet
Share
More Decks by barrachri
See All by barrachri
Will Tech Save Us?
barrachri
0
95
How software can feed the World
barrachri
1
160
How to fight with yourself and win.
barrachri
0
270
Introduction to Statistics with Python
barrachri
0
340
EuroPython 2015 and the future
barrachri
2
110
Start with Flask
barrachri
3
170
Django & Docker
barrachri
6
940
Other Decks in Technology
See All in Technology
メンバーがオーナーシップを発揮しやすいチームづくり
ham0215
2
150
あなたの人生も変わるかも?AWS認定2つで始まったウソみたいな話
iwamot
3
860
Git scrapingで始める継続的なデータ追跡 / Git Scraping
ohbarye
5
500
三菱電機で社内コミュニティを立ち上げた話
kurebayashi
1
360
AIアプリケーション開発でAzure AI Searchを使いこなすためには
isidaitc
1
120
AWSの生成AIサービス Amazon Bedrock入門!(2025年1月版)
minorun365
PRO
7
480
技術に触れたり、顔を出そう
maruto
1
160
Bring Your Own Container: When Containers Turn the Key to EDR Bypass/byoc-avtokyo2024
tkmru
0
860
EMConf JP の楽しみ方 / How to enjoy EMConf JP
pauli
2
150
KMP with Crashlytics
sansantech
PRO
0
240
自社 200 記事を元に整理した読みやすいテックブログを書くための Tips 集
masakihirose
2
330
Goで実践するBFP
hiroyaterui
1
120
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
33
2.5k
Typedesign – Prime Four
hannesfritz
40
2.5k
Making Projects Easy
brettharned
116
6k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
Testing 201, or: Great Expectations
jmmastey
41
7.2k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.2k
A better future with KSS
kneath
238
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
GraphQLとの向き合い方2022年版
quramy
44
13k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Docker and Python
trallard
43
3.2k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.6k
Transcript
Scaling your data infrastructure C H R I S T
I A N B A R R A @ P Y C O N N O V E
THE AGENDA 2 3 START THE DATA SCIENCE WORKFLOW SCALING
IS NOT JUST A MATTER OF MACHINE WHEN THE SIZE OF YOUR DATA MATTERS 1
THE AGENDA 4 5 CONTAINERIZED DATA SCIENCE CASSINY: PUT ALL
THE THINGS TOGETHER END
THE DATA SCIENCE WORKFLOW
HEXAGON PRESENTATION TEMPLATE
HOW YOU BUILD, ITERATE AND SHARE DEPENDS ON MANY THINGS
Your Users Your Product Your Team Your Company Your Tech Stack Your Domain
SCIKIT-LEARN DOCKER DATA SCIENCE TOOLBELT PANDAS JUPYTER RAY
SCALING IS NOT JUST A MATTER OF MACHINES
We all use it.
We really care about versioning. We have Untitled_1.ipynb, Untitled_2.ipynb and
Untitled_3.ipynb. HOMER SIMPSON C H I E F D A T A S C I E N T I S T D A T A B E E R I N C
Since JSON is a plain text format, they can be
version-controlled and shared with colleagues. E X I P Y T H O N N O T E B O O K D O C U M E N T A T I O N
THEY GOT IT RIGHT
BUT WE KEEP IMPROVING
90% OF JUPITER IS MADE BY HYDROGEN
THE HARD THING ABOUT STORAGE
PARQUET P A R Q U E T + O
B J E C T S T O R A G E = YO U C A N Q U E R Y I T U S I N G S Q L PA N DA S H A S N AT I V E S U P P O R T F O R G E T A B O U T C S V
WHEN THE SIZE OF YOUR DATA MATTERS
IT’S TOO SLOW DOESN’T FIT IN YOUR RAM
CODE OPTIMIZATION APPROACH SCALING FROM DIFFERENT SIDES A BIGGER MACHINE
USE MULTIPLE CORES MORE MACHINES FRAMEWORKS: DASK RAY SPARK PANDAS: READ BY CHUNKS SCIKIT-LEARN: PARTIAL FIT
chunks & partial_fit 1 M A C H I N
E
Multiple machines. n M A C H I N E
S
I don’t want to use Spark/JVM, what do you have
for me? H A P P Y P Y T H O N U S E R
WHAT IS RAY?
A high-performance distributed execution engine REDIS SCHEDULER WORKER ARROW &
PLASMA
Use pandas through ray to query parquet files in an
object storage. W O R K I N P R O G R E S S
CONTAINERIZED DATA SCIENCE
If you trained a model with scikit-learn 0.18.1, will the
same model work with 0.19.1? P R O B L E M # 1
How do you share your models? P R O B
L E M # 2
How do you put your models in production? P R
O B L E M # 3
Containerize everything. T H E A N S W E
R
1. It’s damn easy to move things around 2. You
get versioning for free 3. Stack agnostic 4. Move Docker images around T O R E C A P
CASSINY: PUT ALL THE THINGS TOGETHER
CLEAR REQUIREMENTS CONTAINERIZED EASY OBJECT STORAGE JUPYTER + IPYTHON PLATFORM
AGNOSTIC
OPEN SOURCE
DEMO
TAKEAWAYS UNIFIED DATA WAREHOUSE KEEP YOUR CODE RUNNING ON ONE
MACHINE USE DOCKER TRY RAY BRING CI/CD TO YOUR DATASCIENCE WORKFLOW OBJECT STORAGE IS COOL DISTRIBUTED COMPUTING IS HARD I DIDN’T HAVE ANOTHER POINT
None