Going beyond Apache Parquet's default settings

In the last decade, Apache Parquet has become the standard format to store tabular data on disk regardless of the technology stack used. This is due to its read/write performance, efficient compression technology, interoperability and especially outstanding performance with the default settings.

While these default settings and access patterns already provide decent performance, by understanding the format in more detail and using recent developments, one can get much better performance, smaller files, and utilise Parquet's newer partial reading features to read even smaller subsets of a file for a given query.

This talk aims to provide insight into the Parquet format and its recent development that are useful for end users' daily workflows. One only needs prior knowledge to know what a DataFrame/tabular data is.

Transcript

1. Going beyond Parquet’s default settings Uwe Korn – QuantCo – April 2024 🔎
2. About me • Uwe Korn
 https://mastodon.social/@xhochy / @xhochy • CTO at Data Science startup QuantCo • Previously worked as a Data Engineer • A lot of OSS, notably Apache {Arrow, Parquet} and conda-forge • PyData Südwest Co-Organizer
3. Apache Parquet 1. Data Frame storage? CSV? Why? 2. Use Parquet
4. Photo by Hansjörg Keller on Unsplash
5. Apache Parquet 1. Columnar, on-disk storage format 2. Started in 2012 by Cloudera and Twitter 3. Later, it became Apache Parquet 4. Fall 2016 brought full Python & C++ Support 5. State-of-the-art since the Hadoop era, still going strong
6. Clear bene fi ts 1. Columnar makes vectorized operations fast 2. E ffi cient encodings and compression make it small 3. Predicate-pushdown brings computation to the I/O layer 4. Language-independent and widespread; common exchange format
7. Constructing Parquet Files
8. Parquet with pandas
9. Parquet with polars
10. Anatomy of a fi le
11. Anatomy of a fi le
12. Anatomy of a fi le
13. Anatomy of a fi le
14. Anatomy of a fi le
15. Photo by Gabriel Dias Pimenta on Unsplash Tuning
16. Knobs to tune 1. Compression Algorithm 2. Compression Level 3. RowGroup size 4. Encodings
17. Data Types!? Photo by Patrick Fore on Unsplash
18. Data Types?
19. Data Types? • Well, actually…
20. Data Types? • Well, actually… • …it doesn’t save much on disk.
21. Data Types? • Well, actually… • …it doesn’t save much on disk. • By choosing the optimal types (lossless cast to e.g. fl oat32 or uint8) on a month of New York Taxi trips:
22. Data Types? • Well, actually… • …it doesn’t save much on disk. • By choosing the optimal types (lossless cast to e.g. fl oat32 or uint8) on a month of New York Taxi trips: Saves 963 bytes 😥 of 20.6 MiB
23. Compression Photo by cafeconcetto on Unsplash
24. Compression Algorithm
25. Compression Algorithm • Datasets:
26. Compression Algorithm • Datasets: • New York Yellow Taxi Trips 2021-01
27. Compression Algorithm • Datasets: • New York Yellow Taxi Trips 2021-01 • New York Yellow Taxi Trips 2021-01 with a custom prediction
28. Compression Algorithm • Datasets: • New York Yellow Taxi Trips 2021-01 • New York Yellow Taxi Trips 2021-01 with a custom prediction • gov.uk (House) Price Paid dataset
29. Compression Algorithm • Datasets: • New York Yellow Taxi Trips 2021-01 • New York Yellow Taxi Trips 2021-01 with a custom prediction • gov.uk (House) Price Paid dataset • COVID-19 Epidemiology
30. Compression Algorithm • Datasets: • New York Yellow Taxi Trips 2021-01 • New York Yellow Taxi Trips 2021-01 with a custom prediction • gov.uk (House) Price Paid dataset • COVID-19 Epidemiology • Time measurements: Pick the median of fi ve runs
31. Compression Algorithm
32. Compression Algorithm
33. Compression Algorithm
34. Compression Level 1. For Brotli, ZStandard and GZIP, we can tune the level 2. Snappy and „none“ have a fi xed compression level.
35. GZIP
36. Brotli
37. ZStandard
38. ZStandard 🔬
39. ZStandard & Brotli 🔬
40. Compression
41. Compression 1. Let’s stick for now with ZStandard, as it seems a good tradeo ff between speed and size.
42. Compression 1. Let’s stick for now with ZStandard, as it seems a good tradeo ff between speed and size. 2. In some cases (e.g. slow network drives), it might be worth to also considering Brotli
43. Compression 1. Let’s stick for now with ZStandard, as it seems a good tradeo ff between speed and size. 2. In some cases (e.g. slow network drives), it might be worth to also considering Brotli • …but Brotli is relatively slow to decompress.
44. RowGroup size 1. If you plan to partially access the data, RowGroups are the common place to fi lter. 2. If you want to read the whole data, less are better. 3. Compression & encoding also works better.
45. Single RowGroup
46. Encodings 1. https://parquet.apache.org/docs/ fi le-format/data-pages/encodings/ 2. We have been using RLE_DICTIONARY for all columns 3. DELTA_* encodings not implemented in pyarrow 4. Byte Stream Split a recent addition
47. Dictionary Encoding
48. RLE Encoding
49. Byte Stream Split Encoding
50. Encodings 1. Byte Stream Split sometimes is faster than dictionary encoding, but not signi fi cantly 2. For high entropy columns, BSS shines
51. Hand-Crafted Delta
52. Hand-Crafted Delta 1. Let’s take the timestamps in NYC Taxi Trip 2. Sort by pickup date 3. Compute a delta column for both dates 4. 17.5% saving on the whole fi le.
53. Order your data 1. With our hand-crafted delta, it was worth sorting the data 2. This can help, but only worked for the Price Paid dataset in tests, there it saved 25%, all others actually got larger
54. Summary 1. Adjusting your data types is helpful for in-memory, but have no signi fi cant e ff ect on-disk 2. Store high-entropy fl oats as Byte Stream Split encoded columns 3. Check whether sorting has an e ff ect 4. Delta Encoding in Parquet would be useful, use handcrafted for now 5. Zstd on level 3/4 seems like a good default compression setting
55. Cost Function for compression
56. What do we get? 1. Run once with the default settings 2. Test all compression settings, but also… 1. … use hand-crafted delta. 2. … use Byte Stream Split on predictions.
57. Cost Function for compression
58. Cost Function for compression
59. Code example available at https://github.com/xhochy/pyconde24-parquet
60. Questions?