Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Scale Agnostic Python
Sean T Allen
August 17, 2017
Programming
0
130
Scale Agnostic Python
Presented to the NYC Python meetup August 17, 2017
Sean T Allen
August 17, 2017
Tweet
Share
More Decks by Sean T Allen
See All by Sean T Allen
seantallen
0
27
seantallen
1
67
seantallen
0
75
seantallen
2
140
seantallen
0
35
seantallen
0
340
seantallen
0
350
seantallen
0
120
seantallen
0
140
Other Decks in Programming
See All in Programming
line_developers_tw
0
1.3k
zsmb
1
130
rshindo
2
300
dulltz
0
520
dictoss
0
170
hr01
1
1.3k
yagitatsu
3
1.5k
akatsukinewgrad
0
220
nbkouhou
0
950
line_developers_tw
0
550
line_developers_tw
0
540
kazuki19992
0
450
Featured
See All Featured
rasmusluckow
318
18k
chrislema
173
14k
wjessup
338
16k
reverentgeek
27
1.9k
shpigford
368
42k
geeforr
332
29k
dotmariusz
94
5.1k
nonsquared
81
3.3k
revolveconf
200
9.6k
schacon
145
6.6k
lara
172
9.5k
bermonpainter
342
26k
Transcript
SCALE AGNOSTIC PYTHON HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT
ANY CODE CHANGES*
SEAN T. ALLEN VP OF ENGINEERING AT SENDENCE AUTHOR OF
“STORM APPLIED” @SEANTALLEN
SCALE AGNOSTIC COMPUTING
THE PROBLEM THE LITTLE APPLICATION THAT COULD… UNTIL IT COULDN’T.
Scaling an application from “little data” to “a little more
data” to “big data” is hard.
OPTIONS
Rewrite the application Rewrite to scale across multiple processes Example:
“multiprocessing” Python library
Run more versions Smart loader to load data in. Perhaps
via a job queue like Celery.
Go scale agnostic Write you application against a scale agnostic
API that runs on a scale aware platform
SENDENCE WALLAROO SCALE AGNOSTIC COMPUTING FOR PYTHON
Sendence Wallaroo Simple Scale Agnostic API Managed in-memory state Guaranteed
Message Processing
“We handle the hard infrastructure plumbing so you don’t have
to.” – The Wallaroo Team
LEARN MORE… sendence.com Sendence Engineering on Twitter: @SendenceEng Pat Helland’s
“Life Beyond Distributed Transactions”