Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling your data infrastructure
Search
barrachri
April 20, 2018
Technology
1
200
Scaling your data infrastructure
Scaling your data infrastructure @ PyConNove
barrachri
April 20, 2018
Tweet
Share
More Decks by barrachri
See All by barrachri
Will Tech Save Us?
barrachri
0
110
How software can feed the World
barrachri
1
170
How to fight with yourself and win.
barrachri
0
300
Introduction to Statistics with Python
barrachri
0
390
EuroPython 2015 and the future
barrachri
2
110
Start with Flask
barrachri
3
180
Django & Docker
barrachri
6
990
Other Decks in Technology
See All in Technology
メルカリIBIS:AIが拓く次世代インシデント対応
0gm
1
170
Rubyの国のPerlMonger
anatofuz
3
740
Strands Agents & Bedrock AgentCoreを1分でおさらい
minorun365
PRO
8
370
Delegate authentication and a lot more to Keycloak with OpenID Connect
ahus1
0
220
オブザーバビリティ文化を組織に浸透させるには / install observability culture
mackerelio
0
110
事業特性から逆算したインフラ設計
upsider_tech
0
150
LLMで構造化出力の成功率をグンと上げる方法
keisuketakiguchi
0
970
いかにして命令の入れ替わりについて心配するのをやめ、メモリモデルを愛するようになったか(改)
nullpo_head
7
2.7k
Telemetry APIから学ぶGoogle Cloud ObservabilityとOpenTelemetryの現在 / getting-started-telemetry-api-with-google-cloud
k6s4i53rx
0
160
[OCI Technical Deep Dive] OracleのAI戦略(2025年8月5日開催)
oracle4engineer
PRO
1
210
Findy Freelance 利用シーン別AI活用例
ness
0
630
はじめての転職講座/The Guide of First Career Change
kwappa
5
4.3k
Featured
See All Featured
Writing Fast Ruby
sferik
628
62k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Automating Front-end Workflow
addyosmani
1370
200k
Docker and Python
trallard
45
3.5k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.6k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.3k
Why Our Code Smells
bkeepers
PRO
338
57k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Transcript
Scaling your data infrastructure C H R I S T
I A N B A R R A @ P Y C O N N O V E
THE AGENDA 2 3 START THE DATA SCIENCE WORKFLOW SCALING
IS NOT JUST A MATTER OF MACHINE WHEN THE SIZE OF YOUR DATA MATTERS 1
THE AGENDA 4 5 CONTAINERIZED DATA SCIENCE CASSINY: PUT ALL
THE THINGS TOGETHER END
THE DATA SCIENCE WORKFLOW
HEXAGON PRESENTATION TEMPLATE
HOW YOU BUILD, ITERATE AND SHARE DEPENDS ON MANY THINGS
Your Users Your Product Your Team Your Company Your Tech Stack Your Domain
SCIKIT-LEARN DOCKER DATA SCIENCE TOOLBELT PANDAS JUPYTER RAY
SCALING IS NOT JUST A MATTER OF MACHINES
We all use it.
We really care about versioning. We have Untitled_1.ipynb, Untitled_2.ipynb and
Untitled_3.ipynb. HOMER SIMPSON C H I E F D A T A S C I E N T I S T D A T A B E E R I N C
Since JSON is a plain text format, they can be
version-controlled and shared with colleagues. E X I P Y T H O N N O T E B O O K D O C U M E N T A T I O N
THEY GOT IT RIGHT
BUT WE KEEP IMPROVING
90% OF JUPITER IS MADE BY HYDROGEN
THE HARD THING ABOUT STORAGE
PARQUET P A R Q U E T + O
B J E C T S T O R A G E = YO U C A N Q U E R Y I T U S I N G S Q L PA N DA S H A S N AT I V E S U P P O R T F O R G E T A B O U T C S V
WHEN THE SIZE OF YOUR DATA MATTERS
IT’S TOO SLOW DOESN’T FIT IN YOUR RAM
CODE OPTIMIZATION APPROACH SCALING FROM DIFFERENT SIDES A BIGGER MACHINE
USE MULTIPLE CORES MORE MACHINES FRAMEWORKS: DASK RAY SPARK PANDAS: READ BY CHUNKS SCIKIT-LEARN: PARTIAL FIT
chunks & partial_fit 1 M A C H I N
E
Multiple machines. n M A C H I N E
S
I don’t want to use Spark/JVM, what do you have
for me? H A P P Y P Y T H O N U S E R
WHAT IS RAY?
A high-performance distributed execution engine REDIS SCHEDULER WORKER ARROW &
PLASMA
Use pandas through ray to query parquet files in an
object storage. W O R K I N P R O G R E S S
CONTAINERIZED DATA SCIENCE
If you trained a model with scikit-learn 0.18.1, will the
same model work with 0.19.1? P R O B L E M # 1
How do you share your models? P R O B
L E M # 2
How do you put your models in production? P R
O B L E M # 3
Containerize everything. T H E A N S W E
R
1. It’s damn easy to move things around 2. You
get versioning for free 3. Stack agnostic 4. Move Docker images around T O R E C A P
CASSINY: PUT ALL THE THINGS TOGETHER
CLEAR REQUIREMENTS CONTAINERIZED EASY OBJECT STORAGE JUPYTER + IPYTHON PLATFORM
AGNOSTIC
OPEN SOURCE
DEMO
TAKEAWAYS UNIFIED DATA WAREHOUSE KEEP YOUR CODE RUNNING ON ONE
MACHINE USE DOCKER TRY RAY BRING CI/CD TO YOUR DATASCIENCE WORKFLOW OBJECT STORAGE IS COOL DISTRIBUTED COMPUTING IS HARD I DIDN’T HAVE ANOTHER POINT
None