2016 - Dillon Niederhut - What to do when your data is large, but not big

Transcript

1. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing What to do when your data are large but not big Dillon Niederhut PyBay – the San Francisco Bay Area Python Conference 20 August 2016
2. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing about this talk • data at github.com/deniederhut/pybay 2016 • python libraries : celery, h5py, numpy, pandas, pymongo • other libraries : mongodb, rabbitmq, sqlite
3. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing about me • dlab.berkeley.edu • @DLabAtBerkeley
4. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing size concerns 1 1from xkcd
5. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing time concerns 2 2always relevant
6. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing code concerns 3 3thanks Randall!
7. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing sequential processing
8. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing parallel processing
9. Large data in python Dillon Niederhut Introduction Motivation Strategies Closing contact • dillon.niederhut.us • @dillonniederhut