Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling your data infrastructure
Search
barrachri
April 20, 2018
Technology
1
160
Scaling your data infrastructure
Scaling your data infrastructure @ PyConNove
barrachri
April 20, 2018
Tweet
Share
More Decks by barrachri
See All by barrachri
Will Tech Save Us?
barrachri
0
92
How software can feed the World
barrachri
1
160
How to fight with yourself and win.
barrachri
0
260
Introduction to Statistics with Python
barrachri
0
320
EuroPython 2015 and the future
barrachri
2
110
Start with Flask
barrachri
3
170
Django & Docker
barrachri
6
930
Other Decks in Technology
See All in Technology
Oracle Autonomous Database:サービス概要のご紹介
oracle4engineer
PRO
1
7k
アプリをリリースできる状態に保ったまま 段階的にリファクタリングするための 戦略と戦術 / Strategies and tactics for incremental refactoring
yanzm
6
610
技術ブログや登壇資料を秒で作るコツ伝授します
minorun365
PRO
23
5.4k
AWS SAW を広めたい @四国クラウドお遍路
kazzpapa3
0
220
SORACOMで実現するIoTのマルチクラウド対応 - IoTでのクリーンアーキテクチャの実現 -
kenichirokimura
0
340
リアルお遍路+SORACOM IoT
ozk009
1
110
音声AIエージェントの世界とRetell AI入門 / Introduction to the World of Voice AI Agents and Retell AI
rkaga
4
890
AWSを始めた頃に陥りがちなポイントをまとめてみた
oshanqq
1
3.4k
効果的なオンコール対応と障害対応
ryuichi1208
5
2.6k
Fediverse Discovery Providers overview
andypiper
0
110
サプライチェーン攻撃に備える
ryunen344
0
150
Swift Testingのconfirmationを コードリーディング/Dive into Swift Testing confirmation
laprasdrum
1
220
Featured
See All Featured
RailsConf 2023
tenderlove
27
800
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
190
16k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Testing 201, or: Great Expectations
jmmastey
36
7k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
24
600
The Brand Is Dead. Long Live the Brand.
mthomps
53
37k
What's in a price? How to price your products and services
michaelherold
242
11k
Unsuck your backbone
ammeep
667
57k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Large-scale JavaScript Application Architecture
addyosmani
508
110k
Done Done
chrislema
180
16k
Building Flexible Design Systems
yeseniaperezcruz
325
37k
Transcript
Scaling your data infrastructure C H R I S T
I A N B A R R A @ P Y C O N N O V E
THE AGENDA 2 3 START THE DATA SCIENCE WORKFLOW SCALING
IS NOT JUST A MATTER OF MACHINE WHEN THE SIZE OF YOUR DATA MATTERS 1
THE AGENDA 4 5 CONTAINERIZED DATA SCIENCE CASSINY: PUT ALL
THE THINGS TOGETHER END
THE DATA SCIENCE WORKFLOW
HEXAGON PRESENTATION TEMPLATE
HOW YOU BUILD, ITERATE AND SHARE DEPENDS ON MANY THINGS
Your Users Your Product Your Team Your Company Your Tech Stack Your Domain
SCIKIT-LEARN DOCKER DATA SCIENCE TOOLBELT PANDAS JUPYTER RAY
SCALING IS NOT JUST A MATTER OF MACHINES
We all use it.
We really care about versioning. We have Untitled_1.ipynb, Untitled_2.ipynb and
Untitled_3.ipynb. HOMER SIMPSON C H I E F D A T A S C I E N T I S T D A T A B E E R I N C
Since JSON is a plain text format, they can be
version-controlled and shared with colleagues. E X I P Y T H O N N O T E B O O K D O C U M E N T A T I O N
THEY GOT IT RIGHT
BUT WE KEEP IMPROVING
90% OF JUPITER IS MADE BY HYDROGEN
THE HARD THING ABOUT STORAGE
PARQUET P A R Q U E T + O
B J E C T S T O R A G E = YO U C A N Q U E R Y I T U S I N G S Q L PA N DA S H A S N AT I V E S U P P O R T F O R G E T A B O U T C S V
WHEN THE SIZE OF YOUR DATA MATTERS
IT’S TOO SLOW DOESN’T FIT IN YOUR RAM
CODE OPTIMIZATION APPROACH SCALING FROM DIFFERENT SIDES A BIGGER MACHINE
USE MULTIPLE CORES MORE MACHINES FRAMEWORKS: DASK RAY SPARK PANDAS: READ BY CHUNKS SCIKIT-LEARN: PARTIAL FIT
chunks & partial_fit 1 M A C H I N
E
Multiple machines. n M A C H I N E
S
I don’t want to use Spark/JVM, what do you have
for me? H A P P Y P Y T H O N U S E R
WHAT IS RAY?
A high-performance distributed execution engine REDIS SCHEDULER WORKER ARROW &
PLASMA
Use pandas through ray to query parquet files in an
object storage. W O R K I N P R O G R E S S
CONTAINERIZED DATA SCIENCE
If you trained a model with scikit-learn 0.18.1, will the
same model work with 0.19.1? P R O B L E M # 1
How do you share your models? P R O B
L E M # 2
How do you put your models in production? P R
O B L E M # 3
Containerize everything. T H E A N S W E
R
1. It’s damn easy to move things around 2. You
get versioning for free 3. Stack agnostic 4. Move Docker images around T O R E C A P
CASSINY: PUT ALL THE THINGS TOGETHER
CLEAR REQUIREMENTS CONTAINERIZED EASY OBJECT STORAGE JUPYTER + IPYTHON PLATFORM
AGNOSTIC
OPEN SOURCE
DEMO
TAKEAWAYS UNIFIED DATA WAREHOUSE KEEP YOUR CODE RUNNING ON ONE
MACHINE USE DOCKER TRY RAY BRING CI/CD TO YOUR DATASCIENCE WORKFLOW OBJECT STORAGE IS COOL DISTRIBUTED COMPUTING IS HARD I DIDN’T HAVE ANOTHER POINT
None