Strongly typed datasets in a weakly typed world

Strongly typed Parquet Datasets / Hive Tables are often used to exchange and preserve data in a Pandas-driven environment, where types are rather unstable. This results in multiple issues and these as well as potential solutions will be presented, together with an RFC directed to the community.

Transcript

1. 1 Strongly Typed Datasets in a Weakly Typed World Marco Neumann @crepererum PyCon.DE 2018
2. 2 For Production Data Engineering
4. 4 Dataflow
5. 4 Dataflow
6. 4 Dataflow
7. 4 Dataflow
8. 4 Dataflow
9. 4 Dataflow
10. 4 Dataflow dynamic
11. 4 Dataflow static
12. 5 Numpy https://docs.scipy.org/doc/numpy/user/basics.types.html https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
13. 6 Pandas https://pandas.pydata.org/pandas-docs/stable/basics.html#dtypes https://pandas.pydata.org/pandas-docs/stable/extending.html https://pandas.pydata.org/pandas-docs/stable/categorical.html
14. 7 PyTorch https://pytorch.org/docs/stable/tensors.html
15. 8 Turbodbc https://turbodbc.readthedocs.io/en/latest/pages/getting_started.html https://turbodbc.readthedocs.io/en/latest/pages/advanced_usage.html
16. 9 Pyarrow https://arrow.apache.org/docs/python/api.html
17. 10 Parquet https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
18. 11 Perfect World
19. 12 Preservation dask_dataframe_1 ├── _common_metadata ├── month=1 │ └── 552da9ba018c48c1b8722643af3e081d.parquet └── month=2 └── cc686be9ffb54f21a5aaca7e9502e3c6.parquet
20. 13 Dask Types dask_dataframe_1 ├── _common_metadata ├── month=1 │ └── 552da9ba018c48c1b8722643af3e081d.parquet └── month=2 └── cc686be9ffb54f21a5aaca7e9502e3c6.parquet
21. 14 Dask Types dask_dataframe_1 ├── _common_metadata ├── month=1 │ └── 552da9ba018c48c1b8722643af3e081d.parquet └── month=2 └── cc686be9ffb54f21a5aaca7e9502e3c6.parquet
22. 15 Arrow Types
23. 16 Type Variance
24. 17 Turbodbc >>> cursor.execute("SELECT * FROM (VALUES(1), (2), (3))") >>> table = cursor.fetchallarrow() >>> table pyarrow.Table __COL0__: int64 >>> cursor.execute("SELECT * FROM (VALUES(1), (2), (3))") >>> table = cursor.fetchallarrow(adaptive_integers=True) >>> table pyarrow.Table __COL0__: int8
25. 18 Pandas dtype cast to integer float boolean object
26. 19 Pandas
27. 20 Numpy
28. 21 Type Compatibility
29. 22 int16 => int64 [-32768, 32767] [-9223372036854775808, 9223372036854775807]
30. 23 int16 <= int64 [-32768, 32767] [-9223372036854775808, 9223372036854775807]
31. 24 int64 => int64 [-9223372036854775808, 9223372036854775807] [-9223372036854775808, 9223372036854775807]
32. 25 int64 => float64 [-9223372036854775808, 9223372036854775807] [-1.7976931348623157e+308, 1.7976931348623157e+308]
33. 26 int64 => float64
34. 26 int64 => float64
35. 26 int64 => float64 numeric information IDs
36. 27 unicode => binary “München” “M\xc3\xbcnchen”
37. 28 Type Preservation
38. 29 Arrow Types RECAP
39. 30 Arrow Types (modified)
40. 31 Dask Types dask_dataframe_1 ├── _common_metadata ├── month=1 │ └── 552da9ba018c48c1b8722643af3e081d.parquet └── month=2 └── cc686be9ffb54f21a5aaca7e9502e3c6.parquet RECAP
41. 32 Type Normalization data AType = ABinary | ABool | ADate Int | ADecimal128 Int Int | ADictionary AType AType Bool | AFloat Int | AInt Int | AList AType | ANull | AString | ATime Int String | ATimestamp String String | AUInt Int deriving (Show) convert :: AType -> AType convert (AInt w) = AInt 64 convert (AUInt w) = AUInt 64 convert (AFloat w) = AFloat 64 convert (ADate w) = ADate 64 convert (ATime w u) = ATime 64 u convert (AList t) = AList (convert t) convert (ADictionary t i s) = t convert x = x
42. 33 Takeaways 1. Dynamic Typed Runtime, but Statically Typed Datasets 2. Bit-Width: 1. Allow Variance during Read/Write 2. (if required) Store Maximum-Width Type 3. Categoricals: 1. Allow “Compressed” and “Uncompressed” Data 2. (if required) Store “Uncompressed”/Internal Type
43. 34 Thank You 1. Bottles, portions, liquid and perfume HD
 by Alex Sajan
 https://unsplash.com/photos/BNuxGwj-a24
 2. Sunset boulevard
 by Simon Matzinger
 https://unsplash.com/photos/twukN12EN7c
 3. Stages of a Monarch Butterfly
 by Suzanne D. Williams
 https://unsplash.com/photos/VMKBFR6r_jg
 4. Lovers bike ride
 by Sabina Ciesielska
 https://unsplash.com/photos/-2PrzYfgotw
 5. Jar, jam, market food HD
 by Viktor Forgacs
 https://unsplash.com/photos/5mGGOWD-Ths Marco Neumann @crepererum