Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
2016 - Dillon Niederhut - What to do when your data is large, but not big
Search
PyBay
September 25, 2016
0
68
2016 - Dillon Niederhut - What to do when your data is large, but not big
PyBay
September 25, 2016
Tweet
Share
More Decks by PyBay
See All by PyBay
2017 - The Packaging Gradient
pybay
2
740
2017 - Building Bridges: Stopping Python 2 without damages
pybay
0
530
2017 - Bringing Python 3 to LinkedIn
pybay
1
470
2017 - Python Debugging with PUDB
pybay
0
480
2017 - Opening up to Open Source
pybay
0
170
2017 - A Gentle Introduction to Text Classification with Deep Learning
pybay
2
140
2017 - Performant Asynchronous Programming at Quora
pybay
1
280
2017 - latus - a Personal Cloud Storage App written in Python
pybay
2
410
2017 - Everything You Ever Wanted to Know About Web Authentication in Python
pybay
3
410
Featured
See All Featured
Atom: Resistance is Futile
akmur
260
25k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
188
16k
Fireside Chat
paigeccino
22
2.6k
Unsuck your backbone
ammeep
663
57k
Bash Introduction
62gerente
605
210k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
13
8.3k
10 Git Anti Patterns You Should be Aware of
lemiorhan
649
58k
Facilitating Awesome Meetings
lara
43
5.6k
Scaling GitHub
holman
457
140k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
Code Reviewing Like a Champion
maltzj
515
39k
Transcript
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
What to do when your data are large but not big Dillon Niederhut PyBay – the San Francisco Bay Area Python Conference 20 August 2016
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
about this talk • data at github.com/deniederhut/pybay 2016 • python libraries : celery, h5py, numpy, pandas, pymongo • other libraries : mongodb, rabbitmq, sqlite
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
about me • dlab.berkeley.edu • @DLabAtBerkeley
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
size concerns 1 1from xkcd
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
time concerns 2 2always relevant
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
code concerns 3 3thanks Randall!
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
sequential processing
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
parallel processing
Large data in python Dillon Niederhut Introduction Motivation Strategies Closing
contact • dillon.niederhut.us • @dillonniederhut