Hardwood: Building a Parquet Library From Scratch (With a Little Help From AI)

Apache Parquet has become the lingua franca of the modern data ecosystem, powering data lakes and table formats like Apache Iceberg—but for Java, the go-to library parquet-java pulls in Hadoop and a truckload of other dependencies, and its reader is single-threaded. This was bugging me enough to start Hardwood, a brand-new Parquet parser written from scratch in modern Java, applying some of the performance lessons learned from the One Billion Row Challenge. Come and join me for this talk, where we'll look at:

- The internals of the Parquet format and what makes parallelizing its decoding surprisingly tricky
- Techniques for achieving high throughput, such as page-level parallelism, adaptive prefetching, and avoiding auto-boxing in hot loops
- How to use JDK Flight Recorder for identifying performance bottlenecks
- Practical learnings from using AI (specifically, Claude Code) as a coding companion—what works well, where you need to stay sharp, and why "built with AI" doesn't mean "vibe-coded"

Whether you're interested in file formats, Java performance, or getting a realistic take on AI-assisted development, there should be something in here for you.

Video recording: https://vimeo.com/1223604291

Transcript

1. Library Hardwood: Building a Parquet Parser From Scratch (With a Little Help From AI) Gunnar Morling @gunnarmorling © Dale Cruse https://flic.kr/p/2s68Vpa (CC BY 4.0)
2. What if accessing Parquet in Java didn’t need Hadoop — and used all your CPU cores? #Hardwood #Parquet #Java · @gunnarmorling
3. Agenda #Hardwood #Parquet #Java · @gunnarmorling
4. Gunnar Morling Technologist at Confluent • Creator of Hardwood • Former project lead of Debezium • kcctl 🧸, JfrUnit, ModiTect, MapStruct • One Billion Row Challenge 1⃣🐝🏎 • Java Champion #Hardwood #Parquet #Java · @gunnarmorling
5. Apache Parquet © Barry Silver https://flic.kr/p/8A6XQV (CC BY 2.0)
6. Row-Oriented vs. Columnar-Oriented Row-oriented (CSV, JSON, Avro) R1: id=1, name=Alice, fare=12.5 R2: id=2, name=Bob, fare=8.3 R3: id=3, name=Carol, fare=21.0 • Best for: per-row writes, replay one record | | | | | Columnar (Parquet, ORC, Arrow) id: name: fare: 1 2 Alice Bob 12.5 8.3 3 ... Carol 21.0 • Best for: scan a few columns across millions of rows #Hardwood #Parquet #Java · @gunnarmorling
7. Why Columnar? Example ----------------------------------------------------------------SELECT AVG(fare) FROM trips WHERE pickup_date > '2025-01-01' Row format: read every byte of every row Columnar: read 2 columns only -> 10-100x less I/O similar values cluster -> 3-10x compression ----------------------------------------------------------------- • Where you find it ◦ Data lakes on S3 etc., Open Table formats ◦ OLAP engines: Spark, Trino, DuckDB, etc. ◦ Streaming systems: Apache Flink ◦ ML feature stores; analytical mirrors of OLTP databases #Hardwood #Parquet #Java · @gunnarmorling
8. Inside a Parquet File +------------------------------------------------------------+ | PAR1 (4-byte magic, file start) | +------------------------------------------------------------+ | | | +=============== ROW GROUP 1 ======================+ | | | Column Chunk A | Column Chunk B | ... | | | | +--------------+ | +--------------+ | | | | | | DictPage | | | DataPage v2 | | | | | | | DataPage | | | DataPage v2 | | | | | | | DataPage | | | DataPage v2 | | | | | | +--------------+ | +--------------+ | | | | +====================================================+ | | | | +=============== ROW GROUP 2 ======================+ | | | ... | | | +====================================================+ | | | +------------------------------------------------------------+ | FOOTER (Thrift) schema | rg meta | stats | page index | +------------------------------------------------------------+ | footer length (4B) | PAR1 | +------------------------------------------------------------+ Read back-to-front: last 8 bytes -> footer -> jump to pages #Hardwood #Parquet #Java · @gunnarmorling
9. Nested Data: The Dremel Trick Definition & Repetition Levels Schema message Doc { required int64 id; repeated group authors { required string name; } } Records Doc(id=1, authors=[Alice, Bob]) Doc(id=2, authors=[]) Doc(id=3, authors=[Carol]) max D = 1 max R = 1 (authors is the one repeated parent) Column 'authors.name' on disk: value: D-level: R-level: Alice 1 0 Bob 1 1 D=1 -> a name is present R=0 -> start of a new Doc NULL 0 0 Carol 1 0 D=0 -> the list was empty R=1 -> next author in same Doc #Hardwood #Parquet #Java · @gunnarmorling
10. Apache Parquet A Long Tail of Capabilities • Different compression algorithms • Many different encodings • Predicate push-down (statistics, bloom filters) • VARIANT column type • Encryption #Hardwood #Parquet #Java · @gunnarmorling
11. Hardwood © Michael Coghlan https://flic.kr/p/PDwXXS (CC BY-SA 2.0)
12. Why Build a New Parquet Library? parquet-java’s Default Class Path: > 100 JARs, 65 MB lib/accessors-smart-1.2.jar:lib/aircompressor-2.0.2.jar:lib/animal-sniffer-annotations-1.17.jar:lib/asm-5.0.4.jar:lib/avro-1.7.7.jar:lib/checker-qual-2.5.2.ja r:lib/commons-beanutils-1.9.4.jar:lib/commons-cli-1.2.jar:lib/commons-codec-1.11.jar:lib/commons-collections-3.2.2.jar:lib/commons-compress-1.19.jar:lib/commo ns-configuration2-2.1.1.jar:lib/commons-io-2.5.jar:lib/commons-lang3-3.7.jar:lib/commons-logging-1.1.3.jar:lib/commons-math3-3.1.1.jar:lib/commons-net-3.6.jar :lib/commons-pool-1.6.jar:lib/commons-text-1.4.jar:lib/curator-client-4.2.0.jar:lib/curator-framework-4.2.0.jar:lib/curator-recipes-4.2.0.jar:lib/dnsjava-2.1. 7.jar:lib/failureaccess-1.0.jar:lib/gson-2.2.4.jar:lib/guava-27.0-jre.jar:lib/hadoop-annotations-3.3.0.jar:lib/hadoop-auth-3.3.0.jar:lib/hadoop-client-3.3.0.j ar:lib/hadoop-common-3.3.0.jar:lib/hadoop-hdfs-client-3.3.0.jar:lib/hadoop-mapreduce-client-common-3.3.0.jar:lib/hadoop-mapreduce-client-core-3.3.0.jar:lib/ha doop-mapreduce-client-jobclient-3.3.0.jar:lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:lib/hadoop-yarn-api-3.3.0.jar:lib/hadoop-yarn-client-3.3.0.jar:lib/hadoop-y arn-common-3.3.0.jar:lib/htrace-core4-4.1.0-incubating.jar:lib/httpclient-4.5.6.jar:lib/httpcore-4.4.10.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotat ions-2.10.3.jar:lib/jackson-core-2.10.3.jar:lib/jackson-core-asl-1.9.13.jar:lib/jackson-databind-2.10.3.jar:lib/jackson-jaxrs-base-2.10.3.jar:lib/jackson-jaxr s-json-provider-2.10.3.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/jackson-module-jaxb-annotations-2.10.3.jar:lib/jakarta.activation-api-1.2.1.jar:lib/jakarta.x ml.bind-api-2.3.2.jar:lib/javax.activation-api-1.2.0.jar:lib/javax.annotation-api-1.3.2.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-api-2.2.11.jar:lib/jcip-a nnotations-1.0-1.jar:lib/jersey-client-1.19.jar:lib/jersey-core-1.19.jar:lib/jersey-servlet-1.19.jar:lib/jetty-client-9.4.20.v20190813.jar:lib/jetty-http-9.4. 20.v20190813.jar:lib/jetty-io-9.4.20.v20190813.jar:lib/jetty-security-9.4.20.v20190813.jar:lib/jetty-servlet-9.4.20.v20190813.jar:lib/jetty-util-9.4.20.v20190 813.jar:lib/jetty-webapp-9.4.20.v20190813.jar:lib/jetty-xml-9.4.20.v20190813.jar:lib/jline-3.9.0.jar:lib/json-smart-2.3.jar:lib/jsp-api-2.1.jar:lib/jsr305-3.0 .2.jar:lib/jsr311-api-1.1.1.jar:lib/jts-core-1.20.0.jar:lib/kerb-admin-1.0.1.jar:lib/kerb-client-1.0.1.jar:lib/kerb-common-1.0.1.jar:lib/kerb-core-1.0.1.jar:l ib/kerb-crypto-1.0.1.jar:lib/kerb-identity-1.0.1.jar:lib/kerb-server-1.0.1.jar:lib/kerb-simplekdc-1.0.1.jar:lib/kerb-util-1.0.1.jar:lib/kerby-asn1-1.0.1.jar:l ib/kerby-config-1.0.1.jar:lib/kerby-pkix-1.0.1.jar:lib/kerby-util-1.0.1.jar:lib/kerby-xdr-1.0.1.jar:lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-g uava.jar:lib/log4j-1.2.17.jar:lib/nimbus-jose-jwt-7.9.jar:lib/okhttp-2.7.5.jar:lib/okio-1.6.0.jar:lib/paranamer-2.3.jar:lib/parquet-column-1.17.1.jar:lib/parq uet-common-1.17.1.jar:lib/parquet-encoding-1.17.1.jar:lib/parquet-format-structures-1.17.1.jar:lib/parquet-hadoop-1.17.1.jar:lib/parquet-jackson-1.17.1.jar:li b/protobuf-java-2.5.0.jar:lib/re2j-1.1.jar:lib/slf4j-api-1.7.33.jar:lib/snappy-java-1.1.10.7.jar:lib/stax2-api-3.1.4.jar:lib/token-provider-1.0.1.jar:lib/webs ocket-api-9.4.20.v20190813.jar:lib/websocket-client-9.4.20.v20190813.jar:lib/websocket-common-9.4.20.v20190813.jar:lib/woodstox-core-5.0.3.jar:lib/zstd-jni-1. 5.7-3.jar #Hardwood #Parquet #Java · @gunnarmorling
13. Why Build a New Parquet Library? parquet-java Pain Points • Pulls in Hadoop: …and a truckload of transitive deps • Reader is single-threaded: leaves cores idle on modern hardware But also: Explore how far LLMs will take you? #Hardwood #Parquet #Java · @gunnarmorling
14. Hardwood Goals What We Set Out to Build • Light-weight: zero transitive dependencies beyond optional compression libraries • Fast: be the fastest Parquet reader and writer for the JVM • Concurrent: multi-threaded at the core • Compatible: read every file that parquet-java reads • Embeddable: usable in GraalVM native binaries #Hardwood #Parquet #Java · @gunnarmorling
15. Reading a Parquet File try (ParquetFileReader reader = ParquetFileReader.open( InputFile.of(path)); RowReader rows = reader.rowReader()) { while (rows.hasNext()) { rows.next(); long id = rows.getLong("id"); String name = rows.getString("name"); LocalDate birth = rows.getDate("birth_date"); // ... typed primitives, no auto-boxing } } #Hardwood #Parquet #Java · @gunnarmorling
16. Query Controls: Projection // Column projection try (RowReader r = reader.buildRowReader() .projection(ColumnProjection.columns("id", "name", "created_at")) .build()) { while (r.hasNext()) { r.next(); long id = r.getLong("id"); String name = r.getString("name"); Instant ts = r.getTimestamp("created_at"); } } #Hardwood #Parquet #Java · @gunnarmorling
17. Query Controls: Filters // Predicate pushdown (row group + page + record) FilterPredicate after = FilterPredicate.gt("birth_date", LocalDate.of(2000, 1, 1)); FilterPredicate amount = FilterPredicate.gtEq("amount", new BigDecimal("99.99")); FilterPredicate where = FilterPredicate.and( FilterPredicate.gtEq("salary", 50_000L), FilterPredicate.lt("age", 65), FilterPredicate.isNotNull("email")); #Hardwood #Parquet #Java · @gunnarmorling
18. Reading From S3 // S3: static credentials (hardwood-s3) S3Source source = S3Source.builder() .region("us-east-1") .credentials(S3Credentials.of("AKIA...", "secret")) .build(); try (ParquetFileReader fr = ParquetFileReader.open( source.inputFile("s3://my-bucket/data/trips.parquet")); RowReader r = fr.rowReader()) { /* ... */ } #Hardwood #Parquet #Java · @gunnarmorling
19. Columnar API for Hot Loops try (ColumnReader fare = reader.columnReader("fare_amount")) { double sum = 0; while (fare.nextBatch()) { int count = fare.getValueCount(); double[] values = fare.getDoubles(); Validity validity = fare.getLeafValidity(); boolean hasNulls = validity.hasNulls(); for (int i = 0; i < count; i++) { if (!hasNulls || validity.isNotNull(i)) { sum += values[i]; } } }} #Hardwood #Parquet #Java · @gunnarmorling
20. Writing a Parquet File try (ParquetFileWriter writer = ParquetFileWriter.create(OutputFile.of(path), schema)) { RowWriter rows = writer.rowWriter(); for (Person person : people) { rows.writeRow(row -> row .setLong("id", person.id()) .setString("name", person.name()) .setDate("birth_date", person.birthDate()) .setList("phones", phones -> { for (String phone : person.phones()) { phones.addString(phone); } })); }} #Hardwood #Parquet #Java · @gunnarmorling
21. Implementation © formulanone https://flic.kr/p/woErp4 (CC BY-SA 2.0)
22. Page-Level Parallelism Why Pages, Not Row Groups or Column Chunks Row-group parallel Column-chunk parallel Page parallel +--------+ <- W1 | RG 1 | +--------+ +--------+ <- W2 | RG 2 | +--------+ +--------+ <- W3 | RG 3 | +--------+ +--------+ <- W1 | Col A | fast +--------+ +--------+ <- W2 | Col B | SLOW +--------+ +--------+ <- W3 | Col C | fast +--------+ Many files have only 1-2 row groups -> tiny fan-out; huge RGs -> memory pressure. Capped at # projected columns; columns decode at very different speeds -> fast workers idle waiting on slow. <-- Hardwood P1 P2 P3 P4 ... (100s) ^ ^ ^ ^ W1 W2 W3 W4 ... (within one row group) + bounded memory + virtual threads coordinate + adaptive prefetch: slow columns get more workers #Hardwood #Parquet #Java · @gunnarmorling
23. Modern Java FTW Some Features We Lean On • Virtual threads — per-column retriever/ drain coordinators • Vector API — SIMD bit-unpacking & dict lookups • FFM + libdeflate — native-speed gzip on Java 22+ • GraalVM — AOT-compiled native CLI binary • JFR — ship-with-the-binary profiling #Hardwood #Parquet #Java · @gunnarmorling
24. Modern Java FTW Some Features We Lean On • Virtual threads — per-column retriever/drain coordinators; decode itself runs on a fixed OS-thread pool • Vector API — SIMD bit-unpacking & dict lookups • FFM + libdeflate — native-speed gzip on Java 22+ • GraalVM — AOT-compiled native CLI binary • JFR — ship-with-the-binary profiling #Hardwood #Parquet #Java · @gunnarmorling
25. S3 I/O Path #Hardwood #Parquet #Java · @gunnarmorling
26. S3 I/O Path #Hardwood #Parquet #Java · @gunnarmorling
27. S3 I/O Path #Hardwood #Parquet #Java · @gunnarmorling
28. Zero-Copy Backings: Local mmap, Remote Range Cache Local files MappedInputFile file on disk | v +------------------------------------------------+ | mmap into the process (MappedByteBuffer) | +------------------------------------------------+ ^ ^ ^ slice slice slice <- ChunkHandle reads from the mapping directly Zero-copy ByteBuffer slices, no read() syscalls, OS page cache does the rest. first hit on range A..B subsequent hits on A..B +-----------------+ +-------------------+ | served from mmap | Remote files (S3, ...) | GET Range A..B | +-----------------+ | (no GET, no copy) | RangeBackedInputFile | +-------------------+ v ^ +-----------------------------------------------+ | sparse temp file, mmapped (cap: 2 GB) | | holes for unread regions | | populated on demand | +-----------------------------------------------+ #Hardwood #Parquet #Java · @gunnarmorling
29. Performance Numbers—Columnar Reader #Hardwood #Parquet #Java · @gunnarmorling
30. Performance Numbers—Row Reader #Hardwood #Parquet #Java · @gunnarmorling
31. Performance Numbers—Filtered Scan #Hardwood #Parquet #Java · @gunnarmorling
32. Fixed-Length Lists Same Length in Every Row—Encoded as if It Varied • Embeddings, coordinates, RGBA: lists whose length never changes • Dremel spells the structure out anyway: def levels a constant run, rep levels 0,1,1,…,1 for every single record • Costs ~3x a flat column holding the same values #Hardwood #Parquet #Java · @gunnarmorling
33. Detection Without Decoding Pattern-Matching the Encoded Level Streams • Example: 3D coordinates, i.e. fixed-length lists with n = 3 • 4-byte floats → ~1 MB page holds roughly 87,000 records of 3 coordinates each—261,000 leaf entries in total • Def levels: one single RLE run; Rep levels: stamped matching #Hardwood #Parquet #Java · @gunnarmorling
34. Avoiding Array Materialization The Benefits #Hardwood #Parquet #Java · @gunnarmorling
35. All the Details #Hardwood #Parquet #Java · @gunnarmorling
36. CLI © FlipFlopFlorida https://flic.kr/p/5ZWGsP (CC BY 2.0)
37. Hardwood CLI: Swiss Army Knife for Parquet $ hardwood print -n 20 -f data.parquet $ hardwood convert --format json -f s3://bucket/data.parquet $ hardwood inspect columns -f data.parquet #Hardwood #Parquet #Java · @gunnarmorling
38. dive: Interactive Parquet Exploration #Hardwood #Parquet #Java · @gunnarmorling
39. dive in Action #Hardwood #Parquet #Java · @gunnarmorling
40. dive in Action #Hardwood #Parquet #Java · @gunnarmorling
41. Hardwood CLI: Swiss Army Knife for Parquet Binaries for macOS, Linux & Windows https://github.com/hardwood-hq/hardwood/releases/ #Hardwood #Parquet #Java · @gunnarmorling
42. AI-Assisted Coding © Quinn Dombrowski https://flic.kr/p/4NnoZ5 (CC BY-SA 2.0)
43. What AI Does Well • Implementing a spec — encodings, page headers, Thrift Compact Protocol • Driving test suites — parquet-testing has hundreds of files; great oracle • Triaging failures — “what does this hex dump tell us?” • Boilerplate — JMH harnesses, Testcontainers setup, GraalVM hints • Pair-debugging — the rubber duck talks back #Hardwood #Parquet #Java · @gunnarmorling
44. Identifying Performance Bottlenecks “Make it faster, Claude!” Works Pretty Well! #Hardwood #Parquet #Java · @gunnarmorling
45. Where You Need to Stay Sharp • Duplicated logic — two near-identical decoders instead of one, because it’s easier than refactoring • Paper-over corner cases — another if/else instead of fixing the underlying bug • Quietly excluding test cases — instead of figuring out why the unexpected result happened • “Plausible” performance “wins” — that benchmarks reveal as no-ops or regressions #Hardwood #Parquet #Java · @gunnarmorling
46. Make or Buy—Reframed by AI Example: S3 Request Signer • The classical answer: pull in the AWS Java SDK • The Hardwood answer: write SigV4 from scratch ◦ 289 lines, JDK crypto only ◦ Validated against official AWS SigV4 test vectors What it bought • Zero mandatory deps for the S3 reader. • No transitive surface, no SDK version pin, no shading #Hardwood #Parquet #Java · @gunnarmorling
47. Code Reviews #Hardwood #Parquet #Java · @gunnarmorling
48. Code Reviews Custom Skills • /hardwood-review PR xyz • /hardwood-address-review xyz #Hardwood #Parquet #Java · @gunnarmorling
49. Built with AI ≠ Vibe-Coded LLM-assisted contributions are welcome. Vibe coding is not. • You read every diff. Every. Diff. • You own the architecture: invariants, threading, allocation budget. • Benchmarks are the ground truth, not Claude’s narration. AI-generated code is a starting point, not an end state. #Hardwood #Parquet #Java · @gunnarmorling
50. Wrap Up © Jennifer Kramer https://flic.kr/p/24Uqs7 (CC BY 2.0)
51. Status & Roadmap • • 1.0.0.Final ◦ Row + columnar reader; flat and nested files ◦ All types including VARIANT, encodings, compression schemes ◦ Local and remote files, predicate push-down ◦ CLI 1.1.0.Beta1 ◦ Writer preview, Bloom filters, fixed-list fast path (opt in) • 1.1 ◦ Full write support, late materialization, Flink integration #Hardwood #Parquet #Java · @gunnarmorling
52. hardwood.dev #Hardwood #Parquet #Java · @gunnarmorling
53. Summary • Apache Parquet: a richer file format than “just columnar” • Hardwood: ◦ Fast, zero-dependency Parquet read… library for the JVM ◦ Modern Java pays off in real numbers ◦ AI is a force multiplier — but you stay the engineer ◦ ❤❤❤ #Hardwood #Parquet #Java · @gunnarmorling
54. Get In Touch 📧 gmorling@confluent.io @gunnarmorling @gunnarmorling.dev �� morling.dev �� #Hardwood #Parquet #Java · @gunnarmorling
55. #Hardwood #Parquet #Java · @gunnarmorling