Data Formats for Data Science @PyConDe

Talk @ **PyConDE**

The CSV is the most widely adopted data format. It used to
store and share *not-so-big* scientific data. However, this format is not particularly
suited in case data require any sort of internal
hierarchical structure, or if data are too big. To this end, other data formats must be considered.
These formats may include *general purpose* solutions, e.g. [No]SQL databases, HDFS (Hadoop File System);
or may be specifically designed for scientific data, e.g. hdf5, ROOT, NetCDF.

In this talk, the different data formats will be presented and compared w.r.t. their
usage for scientific computations along with corresponding Python libraries.

Finally, few notes about the new trends for **columnar databases** will be discussed for very fast
in-memory analytics (e.g. *MonetDB*).

Transcript

1. PyConDE DATA FORMATS FOR 
 DATA SCIENCE Data Scientist and Researcher Fondazione Bruno Kessler (FBK)
 Trento, Italy Valerio Maggio @leriomaggio Remastered Director’s Confidential now with different jokes!
3. Data Science is about data…
4. Data Science is about data… • First step in Data Science is 
 Data Loading and Processing • Data Loading Q: What’s the format in which data are saved ? Q: What’s the most Pythonic way to do that ? • Data Storage Q: What’s the best format to store (and so share) my data?
5. Data + Documentation NOTEBOOKS FOR REPRODUCIBLE SCIENCE
6. Raw Data
7. Numpy to the rescue Matrix as a Python list of lists Numpy Array
9. Raw Data + Metadata CSV
10. csv Module (in standard library)
11. Introduction to Time Series Analysis with Pandas by Alexander Hendorf
12. Pandas DataFrame
16. SPREADSHITS EE ???
17. When I got data shared in XSLX format
18. xlsxwriter.readthedocs.io
19. Storage matters BINARY FORMAT • Space is not the only concern (for text). Speed matters! • Python conversion to int() and float() are slow • costly atoi()/atof() C functions * A. Scopatz, K.D. Huff - Effective Computations in Physics - Field Guide to Research in Python, O’Reilly 2015 * Integers and floats in native and string representations
20. Unstructured Binary: 
 collection of binary chunks import pickle
21. HIERARCHICAL DATA FORMAT 5 (a.k.a. HDF5) • Free and open source file format specification • (+) Works great with both big or tiny datasets • (+) Storage friendly • Allows for Compression • (+) Dev. Friendly • Query DSL + Multiple-language support • Python: PyTables, hdf5, h5py
22. HIERARCHICAL DATA FORMAT 5 (a.k.a. HDF5) • Free and open source file format specification • (+) Works great with both big or tiny datasets • (+) Storage friendly • Allows for Compression • (+) Dev. Friendly • Query DSL + Multiple-language support • Python: PyTables, hdf5, h5py from Keras Documentation
24. with PyTables NUMPY ARRAYS TIGHT INTEGRATION Accessing the table
25. HIERARCHY AND GROUPS
26. DATA CHUNKING A. Scopatz, K.D. Huff - Effective Computations in Physics - Field Guide to Research in Python, O’Reilly 2015
27. DATA CHUNKING A. Scopatz, K.D. Huff - Effective Computations in Physics - Field Guide to Research in Python, O’Reilly 2015 • Small chunks are good for accessing only some of the data at a time. 
 • Large chunks are good for accessing lots of data at a time. 
 • Reading and writing chunks may happen in parallel
28. PARALLEL HDF5 MPI (mpi4py) integration
29. Particle Physics • Data Analysis Framework (and tool) dev. @CERN • Written in C++; Native extension in Python (aka PyROOT) • ROOT6 also ships a Jupyter Kernel • Definition of a new Binary Data Format (.root) • based on the serialisation of C++ Objects • Speed and Storage Matters!!
30. ROOT tool
31. rootpy root_numpy rootpy.github.io/root_numpy/ rootpy.github.io/ C++ style (inside the ROOT interactive shell) PyROOT (Pythonish style)
32. rootpy (Pythonic style)
33. Tight integration with PyROOT objects root_numpy examples
34. http://www.rootpy.org/commands/root2hdf5.html root2hdf5 (included in rootpy)
35. GeoData (NetCDF) http://xarray.pydata.org/en/stable/index.html MULTIDIMENSI ONAL LABELED ARRAY
36. when Pandas is not enough!
37. Information Architecture RELATIONAL DATABASES • Normalisation (No Duplicates) & Fixed Structure • SQL: Structured Query Language • Many different dialects! • ORM is the way!
38. SQL
41. Flexibility NO-SQL DATABASES • Your data requires a flexible (not fixed) structure • JSON-based data format (mostly) • e.g. MongoDB pymongo
42. JSON
43. Jupyter Notebook Data Format
44. Flexibility + Validation • Your data requires a flexible(~ish) 
 structure • But you want to validate your data • XML-based data format
46. Speed IN-MEMORY DB ANALYTICS • Normalisation (No Duplicates) & Fixed Structure • (Super effective) in-DB Analytics • Column-oriented DB
47. BIG DATA AND COLUMNAR DBS • Big Data World is shifting towards columnar DBs • better oriented to OLAP (analytics) rather than OLTP
50. DATA IN MULTIPLE FORMATS (Big) Data Lake
51. HDFS HADOOP
52. When I got to be working with Java staff
53. HDFS+CSV Opening a Single File on the HDFS hdfs3 matthewrocklin.com/blog/work/2016/02/22/dask-distributed-part-2
54. Wildcard opening of CSVs on the HDFS
56. No Java along the way!
57. Multiple Formats, Unique Interface http://blaze.pydata.org
58. Complicated data require complicated formats Complicated formats require good tools
59. https://twitter.com/peterhoffmann/status/792298757421408256 Just an example of what’s left behind
60. Thanks a lot for your kind attention +ValerioMaggio vmaggio@fbk.com it.linkedin.com/in/valeriomaggio @leriomaggio