Parquet - from a file format to a system architecture

Transcript

1. Parquet from a file format to a system architecture https://flow-php.com
2. Parquet Overview https://flow-php.com
3. Developed in 2013 by Twitter and Cloudera https://flow-php.com
4. Key Goals • Reduced I/O (Input/Output) for analytical queries • Optimized data compression to save storage space • Efficient data encoding to save storage space • Support for complex data structures (nested fields) • Strict schema with schema evolution support https://flow-php.com
5. Widely adopted in modern big data stacks https://flow-php.com
6. Comparison with other file formats https://flow-php.com
7. CSV Advantages • Simple, human-readable, easy to generate and parse • Supported almost universally across programming languages and tools • Quickly importable to any spreadsheet application • Simple to modify with just a text editor Disadvantages • Inefficient in storage and querying • No support for nested data structures or metadata • No support for metadata or schema • No standard for escaping or formatting • Data integrity issues https://flow-php.com
8. XML Advantages • Human-readable, text-based and can be easily read by humans and machines alike • Supports complex and nested data • Validation and schema enforcement, through XSD and DTD Disadvantages • Extremely large file sizes due to heavy tagging, leading to inefficient storage. • Complexity increases significantly for deeply nested structures, making it harder to manage and debug • Not designed for efficient querying and processing of large datasets in analytical systems https://flow-php.com
9. JSON Advantages • Supports nested structures, arrays, and complex types natively • Native support in most modern programming languages • Easily readable and writable for humans, making debugging and manual inspection straightforward • Universally accepted in web APIs, configuration files, and data interchange formats Disadvantages • Lack of compression and indexing • No strict schema enforcement • JSON's structure isn't well-suited for analytical queries and partial data access https://flow-php.com
10. Avro Advantages • Designed for efficient binary serialization • Support for schema and schema evolution • Support for nested structures • Good for streaming use cases Disadvantages • Not human-readable, due to it’s binary nature • Not as performant for analytical queries since it requires scanning entire rows, even if only a few columns are needed https://flow-php.com
11. Parquet Advantages • Excellent compression, reducing storage costs • Handles nested structures and data types, including maps, lists and structures • Strict schema and analytical metadata • Supports evolution • Optimized for Parallel Processing Disadvantages • Not human-readable, due to it’s binary nature • Not easily editable • Not as performant for real-time streaming or transactional systems where row based approach is needed https://flow-php.com
12. Key Differences Format Type Schema Nested Structures Compression Encoding CSV Text / Row No No No No JSON Text / Row Partial Yes No No XML Text / Row Yes Yes No No Avro Binary / Row Yes Yes Yes Yes Parquet Binary / Column Yes Yes Yes Yes https://flow-php.com
13. Row vs Colum formats https://flow-php.com
14. Users File Schema • id – int32 • name – string • email – string • active – bool https://flow-php.com
15. Rows id name email active 1 user-01 user- 01@email.com true 2 user-02 user- 02@email.com true 3 user-03 user- 03@email.com true 4 user-04 user- 04@email.com true 5 user-05 user- 05@email.com false https://flow-php.com
16. Rows id name email active 1 user-01 user- 01@email.com true 2 user-02 user- 02@email.com true 3 user-03 user- 03@email.com true 4 user-04 user- 04@email.com true 5 user-05 user- 05@email.com false Columns id 1 2 3 4 5 name user-01 user-02 user-03 user-04 user-05 email user- 01@email.com user- 02@email.com user- 03@email.com user- 04@email.com user- 05@email.com active true true true true false https://flow-php.com
17. Parquet Building Blocks https://flow-php.com
18. File Footer • schema • row group metadata • column metadata • key-value metadata • created by / version / total rows • encryption metadata • footer length • magic number https://flow-php.com
19. File Metadata https://flow-php.com
20. Page Header Metadata https://flow-php.com
21. File Content • row groups • columns • pages https://flow-php.com
22. Parquet File Structure https://flow-php.com
23. Metadata is like a book index https://flow-php.com
24. Metadata in real world Using metadata in systems architecture can significantly improve data accessibility and processing. Metadata Storage: Data Storage: https://flow-php.com
25. File Metadata https://flow-php.com
26. Binary Serialization with Thrift https://flow-php.com
27. Thrift definition to PHP https://flow-php.com
28. Reading Thrift https://flow-php.com
29. Benefits • cross platform data structures • 50-70% size reduction • allows for additional encoding (like RLE/BitPack) • support for multiple protocols like binary/compact/json https://flow-php.com
30. Apache Thrift Alternatives • gRPC • Protobuf • MessagePack • FlatBuffers https://flow-php.com
31. Parquet Column Types https://flow-php.com
32. Physical Types • BOOLEAN • INT32 • INT64 • INT96 (deprecated) • FLOAT • DOUBLE • BYTE_ARRAY • FIXED_LEN_BYTE_ARRAY Logical Types • STRING • MAP • LIST • ENUM • DECIMAL • DATE • TIME • TIMESTAMP • INTEGER • BSON • JSON • UUID • FLOAT16 https://flow-php.com
33. Logical types are physical types with metadata Logical • STRING • DATE • TIMESTAMP • UUID Physical • BYTE_ARRAY • INT32 • INT64 (used to be INT96) • FIXED_LEN_BYTE_ARRAY https://flow-php.com
34. Types are used by two types of Columns • Flat Column • All physical types • Logical types except Map and List • Nested Column • Map • List • Structure* • Nested columns are collection of flat / nested columns https://flow-php.com
35. Advanced Structures https://flow-php.com
36. Structure – collection of columns https://flow-php.com
37. Parquet column Repetition https://flow-php.com
38. Repetition Explained https://flow-php.com
39. Examples https://flow-php.com
40. Parquet Schema https://flow-php.com
41. Schema DDL (Data Definition Language) https://flow-php.com
42. But wait a moment can’t we do the same in JSON or XML? https://flow-php.com
43. Well, yes, but also no... https://flow-php.com
44. Parquet is more efficient thanks to how the data is stored https://flow-php.com
45. It’s year 2010 "Dremel: Interactive Analysis of Web-Scale Datasets” was presented by Google at 36th International Conference on Very Large Data Bases (VLDB) https://flow-php.com
46. Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data https://flow-php.com
47. Orders Schema DDL https://flow-php.com
48. JSON Order https://flow-php.com
49. XML Order https://flow-php.com
50. Parquet Order Preview generated with: flow-php/cli https://flow-php.com
51. Parquet uses the columnar storage approach that was first introduced by Google Dremel https://flow-php.com
52. Address – Structure{street:string, city:string, zip: string, country:string } Json Parquet https://flow-php.com
53. Notes – List<string> Json Parquet https://flow-php.com
54. Items – List<Structure{sku:string, quantity:int, price: float}> Json Parquet https://flow-php.com
55. Dremel Shred • Column – Int32 • Rows – [1, 2, 3, null, 4, 5] https://flow-php.com
56. Dremel Shred • Column – Int32 • Rows – [1, 2, 3, null, 4, 5] What are Repetition and Definition Levels? https://flow-php.com
57. Repetitions and Definition levels are taken from file schema https://flow-php.com
58. Parquet Order Preview generated with: flow-php/cli https://flow-php.com
59. Dremel Shred • Column – Int32 • Rows – [1, 2, 3, null, 4, 5] • Repetitions: [0, 0, 0, 0, 0, 0] • Definitions: [1, 1, 1, 0, 1, 1] • Values: [1, 2, 3, 4, 5] https://flow-php.com
60. Dremel Shred • Column – Int32 • Rows – [1, 2, 3, null, 4, 5] • Repetitions: [0, 0, 0, 0, 0, 0] • Definitions: [1, 1, 1, 0, 1, 1] • Values: [1, 2, 3, 4, 5] https://flow-php.com
61. Dremel Shred • Column – Int32 • Rows – [1, 2, 3, null, 4, 5] • Repetitions: [0, 0, 0, 0, 0, 0] • Definitions: [1, 1, 1, 0, 1, 1] • Values: [1, 2, 3, 4, 5] https://flow-php.com
62. Dremel Shred • Column – list<Int32> • Rows – [[1, 2, 3], [1, 4, 5]] • Repetitions: [0, 1, 1, 0, 1, 1] • Definitions: [2, 2, 2, 2, 2, 2] • Values: [1, 2, 3, 1, 4, 5] https://flow-php.com
63. Dremel Shred • Column – list<Int32> • Rows – [[1, 2, 3], [1, null, 5]] • Repetitions: [0, 1, 1, 0, 1, 1] • Definitions: [3, 3, 3, 3, 2, 3] • Values: [1, 2, 3, 1, 5] https://flow-php.com
64. Where parquet stores repetitions definitions and values • row group • column chunk • page (header & data) • repetitions • definitions • values https://flow-php.com
65. Repetitions and Definitions are encoded Run Length Encoding / Bit-Packing Hybrid https://flow-php.com
66. Dremel Shred • Column – list<Int32> • Rows – [[1, 2, 3], [1, null, 5]] • Repetitions: [0, 1, 1, 0, 1, 1] • Definitions: [3, 3, 3, 3, 2, 3] • Values: [1, 2, 3, 1, 5] https://flow-php.com
67. Run Length Encoding / Bit-Packing - Hybrid https://flow-php.com
68. For highly repetitive data, the Run-Length Encoding (RLE) / Bit- Packing - Hybrid is extremely effective. https://flow-php.com
69. Encoding Visualization Before After https://flow-php.com
70. RLE Bit Packing Before After https://flow-php.com
71. Values are also encoded encoding algorithm is used based on a column type https://flow-php.com
72. Parquet supported encodings https://flow-php.com
73. Plain • BOOLEAN - 1 bit per value. 0 is false; 1 is true. • INT32 - 4 bytes per value. Stored as little-endian. • INT64 - 8 bytes per value. Stored as little-endian. • FLOAT - 4 bytes per value. IEEE. Stored as little-endian. • DOUBLE - 8 bytes per value. IEEE. Stored as little-endian. • BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes. • FIXED_LEN_BYTE_ARRAY - Just the bytes. Note: Default encoding https://flow-php.com
74. Plain Dictionary Dictionary encoding. The values in the dictionary are encoded in the plain type. Note: Deprecated • in a data page use RLE_DICTIONARY instead • in a Dictionary page use PLAIN instead https://flow-php.com
75. RLE Group packed run length encoding. Usable for definition/repetition levels encoding and Booleans (on one bit: 0 is false; 1 is true.) https://flow-php.com
76. Bit Packed Bit packed encoding. This can only be used if the data has a known max width. Usable for definition/repetition levels encoding https://flow-php.com
77. Delta Binary Packed Encoding for byte arrays to separate the length values and the data. The lengths are encoded using DELTA_BINARY_PACKED https://flow-php.com
78. Delta Byte Array Incremental-encoded byte array. Prefix lengths are encoded using DELTA_BINARY_PACKED. Suffixes are stored as delta length byte arrays. https://flow-php.com
79. RLE Dictionary Dictionary encoding: the ids are encoded using the RLE encoding https://flow-php.com
80. RLE Dictionary Dictionary encoding: the ids are encoded using the RLE encoding row value 1 apple 2 orange 3 banana 4 apple 5 orange 6 banana 7 orange Raw Values Dictionary index value 0 apple 1 orange 2 banana Indices row value 1 0 2 1 3 2 4 0 5 1 6 2 7 1 https://flow-php.com
81. Byte Stream Split Encoding for fixed-width data (FLOAT, DOUBLE, INT32, INT64, FIXED_LEN_BYTE_ARRAY). K byte-streams are created where K is the size in bytes of the data type. The individual bytes of a value are scattered to the corresponding stream and the streams are concatenated. This itself does not reduce the size of the data but can lead to better compression afterwards. Note: Added in 2.8 for FLOAT and DOUBLE. Support for INT32, INT64 and FIXED_LEN_BYTE_ARRAY added in 2.11. https://flow-php.com
82. Pages are also compressed each page is compressed independently https://flow-php.com
83. Compression Algorithms https://flow-php.com
84. Oversimplified comparison https://flow-php.com
85. Google Snappy is the most popular due to speed https://flow-php.com
86. Why parquet is so efficient? • Flat storage for nested structures • Advanced encodings system chosen by the data type • Compression applied on each page https://flow-php.com
87. Repetitive data can be encoded and then compressed reducing storage space https://flow-php.com
88. Flat columnar storage is easier to query but can we optimize it even more? https://flow-php.com
89. Row Group - configuration • Most parquet libraries will let developer to define the size of the Row Group. • Size of the Row Group defines how much data writer will hold in memory before dumping to a file • Size of the Row Group also defines how much data reader will put into the memory at once row group size rows 1 256Mb 1 000 000 2 256Mb 1 000 000 3 256Mb 1 000 000 4 128Mb 500 000 Parquet File https://flow-php.com
90. Bigger Row Group higher memory consumption lower I/O operations number https://flow-php.com
91. Smaller Row Group lower memory consumption higher I/O operations number https://flow-php.com
92. Row group needs to be written at once but we can select columns to read https://flow-php.com
93. Row Group Row Group column: id id name email active 1 user-01 user- 01@email.com true 2 user-02 user- 02@email.com true 3 user-03 user- 03@email.com true 4 user-04 user- 04@email.com true 5 user-05 user- 05@email.com false column: name column: email column: active Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data For simplicity let’s look at it from a rows point of view https://flow-php.com
94. Parquet is ideal for parallel reading https://flow-php.com
95. Read only selected columns Row Group column: id column: name column: email column: active Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data Page 0 =========== header repetitions definitions data Page 1 =========== header repetitions definitions data Page ... n =========== header repetitions definitions data https://flow-php.com
96. That’s not even all! remember column metadata? https://flow-php.com
97. Parquet Metadata Structure • Row Group • Column Chunk (one chunk per column) • Metadata for column in a row group https://flow-php.com
98. Column Chunk Statistics • number of nulls • unique values count • max value • min value https://flow-php.com
99. How can we benefit from that? https://flow-php.com
100. How to query parquet file Parquet File Metadata Row Group 1 Metadata Column “date” Statistics ======= min: 2024-01-01 max: 2024-02-01 Row Group 2 Metadata Column “date” Statistics ======= min: 2024-03-01 max: 2024-04-01 Row Group 3 Metadata Column “date” Statistics ======= min: 2024-05-01 max: 2024-06-01 https://flow-php.com
101. How to query parquet file Parquet File Metadata Row Group 1 Metadata Column “date” Statistics ======= min: 2024-01-01 max: 2024-02-01 Row Group 2 Metadata Column “date” Statistics ======= min: 2024-03-01 max: 2024-04-01 Row Group 3 Metadata Column “date” Statistics ======= min: 2024-05-01 max: 2024-06-01 This is a technique known from RDBMs world Partition Pruning Filter out out unnecessary data partitions based on their statistics https://flow-php.com
102. Can it get better? https://flow-php.com
103. Remember flat columnar storage? https://flow-php.com
104. Parquet can filter out row groups based on nested structures properties https://flow-php.com
105. What about the downsides? https://flow-php.com
106. All parquet viewers sucks! https://flow-php.com
107. Row Group Row Group Immutability - Can’t be “modified” Row Group column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page https://flow-php.com
108. Row Group Row Group Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group Row Group column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page https://flow-php.com
109. Row Group Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group 2. load row group to memory and modify column: id column: name column: email column: active page page 0 page page https://flow-php.com
110. Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group 2. load row group to memory and modify 3. keep moving row groups to the new file Row Group column: id column: name column: email column: active page page 0 page page https://flow-php.com
111. Row Group Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group 2. load row group to memory and modify 3. keep moving row groups to the new file 4. dump modified row group from memory Row Group column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page https://flow-php.com
112. Row Group Row Group Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group 2. load row group to memory and modify 3. keep moving row groups to the new file 4. dump modified row group from memory 5. recalculate file metadata Row Group column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page https://flow-php.com
113. Row Group Row Group Immutability To modify parquet single cell, we need to rewrite the entire file: 1. localize row group 2. load row group to memory and modify 3. keep moving row groups to the new file 4. dump modified row group from memory 5. recalculate file metadata 6. dump metadata to file footer Row Group column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page column: id column: name column: email column: active page page 0 page page https://flow-php.com
114. Thank you! https://flow-php.com
115. https://flow-php.com Norbert Orzechowicz WHOAMI Software architect Consultant Open-source enthusiast