Hardwood: Building a Parquet Library From Scratch (With a Little Help From AI) -- Devoxx Belgium 2026

Apache Parquet has become the lingua franca of the modern data ecosystem, powering data lakes and table formats like Apache Iceberg—but for Java, the go-to library parquet-java pulls in Hadoop and a truckload of other dependencies, and its reader is single-threaded. This was bugging me enough to start Hardwood, a brand-new Parquet parser written from scratch in modern Java, applying some of the performance lessons learned from the One Billion Row Challenge. Come and join me for this talk, where we'll look at:

- The internals of the Parquet format and what makes parallelizing its decoding surprisingly tricky
- Techniques for achieving high throughput, such as page-level parallelism, adaptive prefetching, and avoiding auto-boxing in hot loops
- How to use JDK Flight Recorder for identifying performance bottlenecks
- Practical learnings from using AI (specifically, Claude Code) as a coding companion—what works well, where you need to stay sharp, and why "built with AI" doesn't mean "vibe-coded"

Whether you're interested in file formats, Java performance, or getting a realistic take on AI-assisted development, there should be something in here for you.

Transcript

1. Library Hardwood: Building a Parquet Parser From Scratch (With a Little Help From AI) Gunnar Morling @gunnarmorling © Dale Cruse https://flic.kr/p/2s68Vpa (CC BY 4.0)
2. What if accessing Parquet in Java didn’t need Hadoop — and used all your CPU cores? #Hardwood #Parquet #Java · @gunnarmorling
3. Gunnar Morling Technologist at Confluent • Creator of Hardwood • Former project lead of Debezium • kcctl 🧸, JfrUnit, ModiTect, MapStruct • One Billion Row Challenge 󰍹🐝🏎 • Java Champion #Hardwood #Parquet #Java · @gunnarmorling
4. Apache Parquet © Barry Silver https://flic.kr/p/8A6XQV (CC BY 2.0)
5. Row-Oriented vs. Columnar-Oriented Row-oriented (CSV, JSON, Avro) R1: id=1, name=Alice, fare=12.5 R2: id=2, name=Bob, fare=8.3 R3: id=3, name=Carol, fare=21.0 • Best for: per-row writes, replay one record | | | | | Columnar (Parquet, ORC, Arrow) id: name: fare: 1 2 Alice Bob 12.5 8.3 3 ... Carol 21.0 • Best for: scan a few columns across millions of rows #Hardwood #Parquet #Java · @gunnarmorling
6. Why Columnar? Example ----------------------------------------------------------------SELECT AVG(fare) FROM trips WHERE pickup_date > '2025-01-01' Row format: read every byte of every row Columnar: read 2 columns only -> 10-100x less I/O similar values cluster -> 3-10x compression ----------------------------------------------------------------- • Where you find it ◦ Data lakes on S3 etc., Open Table formats ◦ OLAP engines: Spark, Trino, DuckDB, etc. ◦ Streaming systems: Apache Flink ◦ ML feature stores; analytical mirrors of OLTP databases #Hardwood #Parquet #Java · @gunnarmorling
7. Inside a Parquet File Read Back-to-front: last 8 bytes -> footer -> jump to pages #Hardwood #Parquet #Java · @gunnarmorling
8. Nested Data: The Dremel Trick Definition & Repetition Levels Schema Records Column 'authors.name' on disk: #Hardwood #Parquet #Java · @gunnarmorling
9. Apache Parquet A Long Tail of Capabilities • Different compression algorithms • Many different encodings • Predicate push-down (statistics, bloom filters) • VARIANT column type • Encryption #Hardwood #Parquet #Java · @gunnarmorling
10. Hardwood © Michael Coghlan https://flic.kr/p/PDwXXS (CC BY-SA 2.0)
11. Why Build a New Parquet Library? parquet-java’s Default Class Path: > 100 JARs, 65 MB lib/accessors-smart-1.2.jar:lib/aircompressor-2.0.2.jar:lib/animal-sniffer-annotations-1.17.jar:lib/asm-5.0.4.jar:lib/avro-1.7.7.jar:lib/checker-qual-2.5.2.ja r:lib/commons-beanutils-1.9.4.jar:lib/commons-cli-1.2.jar:lib/commons-codec-1.11.jar:lib/commons-collections-3.2.2.jar:lib/commons-compress-1.19.jar:lib/commo ns-configuration2-2.1.1.jar:lib/commons-io-2.5.jar:lib/commons-lang3-3.7.jar:lib/commons-logging-1.1.3.jar:lib/commons-math3-3.1.1.jar:lib/commons-net-3.6.jar :lib/commons-pool-1.6.jar:lib/commons-text-1.4.jar:lib/curator-client-4.2.0.jar:lib/curator-framework-4.2.0.jar:lib/curator-recipes-4.2.0.jar:lib/dnsjava-2.1. 7.jar:lib/failureaccess-1.0.jar:lib/gson-2.2.4.jar:lib/guava-27.0-jre.jar:lib/hadoop-annotations-3.3.0.jar:lib/hadoop-auth-3.3.0.jar:lib/hadoop-client-3.3.0.j ar:lib/hadoop-common-3.3.0.jar:lib/hadoop-hdfs-client-3.3.0.jar:lib/hadoop-mapreduce-client-common-3.3.0.jar:lib/hadoop-mapreduce-client-core-3.3.0.jar:lib/ha doop-mapreduce-client-jobclient-3.3.0.jar:lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:lib/hadoop-yarn-api-3.3.0.jar:lib/hadoop-yarn-client-3.3.0.jar:lib/hadoop-y arn-common-3.3.0.jar:lib/htrace-core4-4.1.0-incubating.jar:lib/httpclient-4.5.6.jar:lib/httpcore-4.4.10.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotat ions-2.10.3.jar:lib/jackson-core-2.10.3.jar:lib/jackson-core-asl-1.9.13.jar:lib/jackson-databind-2.10.3.jar:lib/jackson-jaxrs-base-2.10.3.jar:lib/jackson-jaxr s-json-provider-2.10.3.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/jackson-module-jaxb-annotations-2.10.3.jar:lib/jakarta.activation-api-1.2.1.jar:lib/jakarta.x ml.bind-api-2.3.2.jar:lib/javax.activation-api-1.2.0.jar:lib/javax.annotation-api-1.3.2.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-api-2.2.11.jar:lib/jcip-a nnotations-1.0-1.jar:lib/jersey-client-1.19.jar:lib/jersey-core-1.19.jar:lib/jersey-servlet-1.19.jar:lib/jetty-client-9.4.20.v20190813.jar:lib/jetty-http-9.4. 20.v20190813.jar:lib/jetty-io-9.4.20.v20190813.jar:lib/jetty-security-9.4.20.v20190813.jar:lib/jetty-servlet-9.4.20.v20190813.jar:lib/jetty-util-9.4.20.v20190 813.jar:lib/jetty-webapp-9.4.20.v20190813.jar:lib/jetty-xml-9.4.20.v20190813.jar:lib/jline-3.9.0.jar:lib/json-smart-2.3.jar:lib/jsp-api-2.1.jar:lib/jsr305-3.0 .2.jar:lib/jsr311-api-1.1.1.jar:lib/jts-core-1.20.0.jar:lib/kerb-admin-1.0.1.jar:lib/kerb-client-1.0.1.jar:lib/kerb-common-1.0.1.jar:lib/kerb-core-1.0.1.jar:l ib/kerb-crypto-1.0.1.jar:lib/kerb-identity-1.0.1.jar:lib/kerb-server-1.0.1.jar:lib/kerb-simplekdc-1.0.1.jar:lib/kerb-util-1.0.1.jar:lib/kerby-asn1-1.0.1.jar:l ib/kerby-config-1.0.1.jar:lib/kerby-pkix-1.0.1.jar:lib/kerby-util-1.0.1.jar:lib/kerby-xdr-1.0.1.jar:lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-g uava.jar:lib/log4j-1.2.17.jar:lib/nimbus-jose-jwt-7.9.jar:lib/okhttp-2.7.5.jar:lib/okio-1.6.0.jar:lib/paranamer-2.3.jar:lib/parquet-column-1.17.1.jar:lib/parq uet-common-1.17.1.jar:lib/parquet-encoding-1.17.1.jar:lib/parquet-format-structures-1.17.1.jar:lib/parquet-hadoop-1.17.1.jar:lib/parquet-jackson-1.17.1.jar:li b/protobuf-java-2.5.0.jar:lib/re2j-1.1.jar:lib/slf4j-api-1.7.33.jar:lib/snappy-java-1.1.10.7.jar:lib/stax2-api-3.1.4.jar:lib/token-provider-1.0.1.jar:lib/webs ocket-api-9.4.20.v20190813.jar:lib/websocket-client-9.4.20.v20190813.jar:lib/websocket-common-9.4.20.v20190813.jar:lib/woodstox-core-5.0.3.jar:lib/zstd-jni-1. 5.7-3.jar #Hardwood #Parquet #Java · @gunnarmorling
12. Why Build a New Parquet Library? parquet-java Pain Points • Pulls in Hadoop: …and a truckload of transitive deps • Reader is single-threaded: leaves cores idle on modern hardware But also: Explore how far LLMs will take you? #Hardwood #Parquet #Java · @gunnarmorling
13. Hardwood Goals What We Set Out to Build • Light-weight: zero transitive dependencies beyond optional compression libraries • Fast: be the fastest Parquet reader and writer for the JVM • Concurrent: multi-threaded at the core • Compatible: read every file that parquet-java reads • Embeddable: usable in GraalVM native binaries #Hardwood #Parquet #Java · @gunnarmorling
14. Reading a Parquet File try (ParquetFileReader reader = ParquetFileReader.open( InputFile.of(path)); RowReader rows = reader.rowReader()) { while (rows.hasNext()) { rows.next(); long id = rows.getLong("id"); String name = rows.getString("name"); LocalDate birth = rows.getDate("birth_date"); // ... typed primitives, no auto-boxing } } #Hardwood #Parquet #Java · @gunnarmorling
15. Query Controls: Projection // Column projection try (RowReader r = reader.buildRowReader() .projection(ColumnProjection.columns("id", "name", "created_at")) .build()) { while (r.hasNext()) { r.next(); long id = r.getLong("id"); String name = r.getString("name"); Instant ts = r.getTimestamp("created_at"); } } #Hardwood #Parquet #Java · @gunnarmorling
16. Query Controls: Filters // Predicate pushdown (row group + page + record) FilterPredicate after = FilterPredicate.gt("birth_date", LocalDate.of(2000, 1, 1)); FilterPredicate amount = FilterPredicate.gtEq("amount", new BigDecimal("99.99")); FilterPredicate where = FilterPredicate.and( FilterPredicate.gtEq("salary", 50_000L), FilterPredicate.lt("age", 65), FilterPredicate.isNotNull("email")); #Hardwood #Parquet #Java · @gunnarmorling
17. Reading From S3 // S3: static credentials (hardwood-s3) S3Source source = S3Source.builder() .region("us-east-1") .credentials(S3Credentials.of("AKIA...", "secret")) .build(); try (ParquetFileReader fr = ParquetFileReader.open( source.inputFile("s3://my-bucket/data/trips.parquet")); RowReader r = fr.rowReader()) { /* ... */ } #Hardwood #Parquet #Java · @gunnarmorling
18. Columnar API for Hot Loops try (ColumnReader fare = reader.columnReader("fare_amount")) { double sum = 0; while (fare.nextBatch()) { int count = fare.getValueCount(); double[] values = fare.getDoubles(); Validity validity = fare.getLeafValidity(); boolean hasNulls = validity.hasNulls(); for (int i = 0; i < count; i++) { if (!hasNulls || validity.isNotNull(i)) { sum += values[i]; } } }} #Hardwood #Parquet #Java · @gunnarmorling
19. Writing a Parquet File try (ParquetFileWriter writer = ParquetFileWriter.create(OutputFile.of(path), schema)) { RowWriter rows = writer.rowWriter(); for (Person person : people) { rows.writeRow(row -> row .setLong("id", person.id()) .setString("name", person.name()) .setDate("birth_date", person.birthDate()) .setList("phones", phones -> { for (String phone : person.phones()) { phones.addString(phone); } })); }} #Hardwood #Parquet #Java · @gunnarmorling
20. Implementation © formulanone https://flic.kr/p/woErp4 (CC BY-SA 2.0)
21. Page-Level Parallelism Why Pages, Not Row Groups or Column Chunks Row-group parallel Column-chunk parallel Page parallel +--------+ <- W1 | RG 1 | +--------+ +--------+ <- W2 | RG 2 | +--------+ +--------+ <- W3 | RG 3 | +--------+ +--------+ <- W1 | Col A | fast +--------+ +--------+ <- W2 | Col B | SLOW +--------+ +--------+ <- W3 | Col C | fast +--------+ Many files have only 1-2 row groups -> tiny fan-out; huge RGs -> memory pressure. Capped at # projected columns; columns decode at very different speeds -> fast workers idle waiting on slow. <-- Hardwood P1 P2 P3 P4 ... (100s) ^ ^ ^ ^ W1 W2 W3 W4 ... (within one row group) + bounded memory + virtual threads coordinate + adaptive prefetch: slow columns get more workers #Hardwood #Parquet #Java · @gunnarmorling
22. Modern Java FTW Some Features We Lean On • Virtual threads — per-column retriever/ drain coordinators • Vector API — SIMD bit-unpacking & dict lookups • FFM + libdeflate — native-speed gzip on Java 22+ • GraalVM — AOT-compiled native CLI binary • JFR — ship-with-the-binary profiling #Hardwood #Parquet #Java · @gunnarmorling
23. Modern Java FTW Some Features We Lean On • Virtual threads — per-column retriever/drain coordinators; decode itself runs on a fixed OS-thread pool • Vector API — SIMD bit-unpacking & dict lookups • FFM + libdeflate — native-speed gzip on Java 22+ • GraalVM — AOT-compiled native CLI binary • JFR — ship-with-the-binary profiling #Hardwood #Parquet #Java · @gunnarmorling
24. Performance Numbers—Columnar Reader #Hardwood #Parquet #Java · @gunnarmorling
25. Performance Numbers—Row Reader #Hardwood #Parquet #Java · @gunnarmorling
26. Skipping Work What the Metadata Can Prove Predicate: fare_amount > 50 • • • No value in min/max satisfies it (max ≤ 50, row group or page) → CANNOT_MATCH Whole min/max range satisfies it (min > 50), no nulls → ALWAYS_MATCHES Otherwise → MIGHT_MATCH Predicate: payment_type = 'CASH' • • • Predicate value out of min/max range → CANNOT_MATCH Bloom filter proves absence → CANNOT_MATCH Not in the dictionary of a fully dictionary-encoded chunk → CANNOT_MATCH CANNOT_MATCH ALWAYS_MATCHES MIGHT_MATCH Skip the bytes Read but skip per-row filtering Evaluate the rows #Hardwood #Parquet #Java · @gunnarmorling
27. Performance Numbers—Filtered Scan #Hardwood #Parquet #Java · @gunnarmorling
28. Fixed-Length Lists Same Length in Every Row—Encoded as if It Varied • Embeddings, coordinates, RGBA: lists whose length never changes • Dremel spells the structure out anyway: def levels a constant run, rep levels 0,1,1,…,1 for every single record • Costs ~3x a flat column holding the same values #Hardwood #Parquet #Java · @gunnarmorling
29. Detection Without Decoding Pattern-Matching the Encoded Level Streams • Example: 3D coordinates, i.e. fixed-length lists with n = 3 • 4-byte floats → ~1 MB page holds roughly 87,000 records of 3 coordinates each—261,000 leaf entries in total • Def levels: one single RLE run; Rep levels: stamped matching #Hardwood #Parquet #Java · @gunnarmorling
30. Rep Levels, One Word at a Time SWAR over the bit-packed level stream • n ≤ 8: stream is bit-packed and repeats every n / gcd(n, 8) bytes • Compare one long against the tiled pattern → first mismatch falls back to regular decoding • Large n: derive the byte stride from the first record, then one bulk Arrays.equals of the stream against itself shifted by one stride • Detection cost typically neglectable #Hardwood #Parquet #Java · @gunnarmorling
31. Avoiding Array Materialization The Benefits #Hardwood #Parquet #Java · @gunnarmorling
32. All the Details #Hardwood #Parquet #Java · @gunnarmorling
33. CLI © FlipFlopFlorida https://flic.kr/p/5ZWGsP (CC BY 2.0)
34. Hardwood CLI: Swiss Army Knife for Parquet $ hardwood print -n 20 -f data.parquet $ hardwood convert --format json -f s3://bucket/data.parquet $ hardwood inspect columns -f data.parquet #Hardwood #Parquet #Java · @gunnarmorling
35. dive: Interactive Parquet Exploration #Hardwood #Parquet #Java · @gunnarmorling
36. dive in Action #Hardwood #Parquet #Java · @gunnarmorling
37. Hardwood CLI: Swiss Army Knife for Parquet Binaries for macOS, Linux & Windows https://github.com/hardwood-hq/hardwood/releases/ #Hardwood #Parquet #Java · @gunnarmorling
38. AI-Assisted Coding © Quinn Dombrowski https://flic.kr/p/4NnoZ5 (CC BY-SA 2.0)
39. What AI Does Well • Implementing a spec — encodings, page headers, Thrift Compact Protocol • Driving test suites — parquet-testing corpus, plus parquet-java & DuckDB as test-scope oracles • Triaging failures — “what does this hex dump tell us?” • Boilerplate — JMH harnesses, Testcontainers setup, GraalVM hints #Hardwood #Parquet #Java · @gunnarmorling
40. Identifying Performance Bottlenecks “Make it faster, Claude!” Works Pretty Well! #Hardwood #Parquet #Java · @gunnarmorling
41. Where You Need to Stay Sharp • Duplicated logic — two near-identical decoders instead of one, because it’s easier than refactoring • Paper-over corner cases — another if/else instead of fixing the underlying bug • Quietly excluding test cases — instead of figuring out why the unexpected result happened • “Plausible” performance “wins” — that benchmarks reveal as no-ops or regressions #Hardwood #Parquet #Java · @gunnarmorling
42. Make or Buy—Reframed by AI Example: S3 Request Signer • The classical answer: pull in the AWS Java SDK • The Hardwood answer: write SigV4 from scratch ◦ 289 lines, JDK crypto only ◦ Validated against official AWS SigV4 test vectors What it bought • Zero mandatory deps for the S3 reader. • No transitive surface, no SDK version pin, no shading #Hardwood #Parquet #Java · @gunnarmorling
43. Make or Buy—Reframed by AI Example: S3 Request Signer Before · 30 JARs, 8.3 MB After · 0 JARs • HTTP: java.net.http.HttpClient, in the JDK • Signing: 289 lines of SigV4, JDK crypto • Checked against AWS's own signing test vectors #Hardwood #Parquet #Java · @gunnarmorling
44. Code Reviews #Hardwood #Parquet #Java · @gunnarmorling
45. Code Reviews Custom Skills • /hardwood-review PR xyz • /hardwood-address-review xyz #Hardwood #Parquet #Java · @gunnarmorling
46. Built with AI ≠ Vibe-Coded LLM-assisted contributions are welcome. Vibe coding is not. • You must be able to explain the change • You own the architecture: invariants, threading, allocation budget • Benchmarks are the ground truth, not Claude’s narration AI-generated code is a starting point, not an end state. #Hardwood #Parquet #Java · @gunnarmorling
47. Wrap Up © Jennifer Kramer https://flic.kr/p/24Uqs7 (CC BY 2.0)
48. Status & Roadmap • • 1.0.0.Final ◦ Row + columnar reader; flat and nested files ◦ All types including VARIANT, encodings, compression schemes ◦ Local and remote files, predicate push-down ◦ CLI 1.1.0.Beta1 Write support, Bloom filter & dictionary pruning, fixed-list fast path, Avro bindings Exploring ◦ Late materialization, Flink integration ◦ Iceberg REST catalog, Arrow interop, Valhalla ◦ • #Hardwood #Parquet #Java · @gunnarmorling
49. Summary Learn more on hardwood.dev • Hardwood: ◦ Fast, zero-dependency Parquet library for the JVM ◦ Modern Java pays off in real numbers ◦ AI is a force multiplier — but you stay the engineer #Hardwood #Parquet #Java · @gunnarmorling
50. Get In Touch 📧gmorling@confluent.io @gunnarmorling @gunnarmorling.dev �� morling.dev �� #Hardwood #Parquet #Java · @gunnarmorling
51. #Hardwood #Parquet #Java · @gunnarmorling