Serverless Watershed Extraction: Benchmarking Zarr vs COG for Large-Scale Flow Direction Data

Transcript

1. FOSS4G HIROSHIMA 2026 — Sep 3, Room 1 Serverless Watershed Extraction Benchmarking Zarr vs COG for Large-Scale Flow Direction Data Shinsuke Nakamori
2. About me Shinsuke Nakamori ・ Web GIS / QGIS plugin developer at MIERUNE Inc. (Sapporo, Japan) ・ Previously: iRIC Software — river flow & flood simulation
3. Japan floods. A lot. ・ Heavy rainfall nationwide, every year ・ Flood-risk simulation is essential to reduce damage Flood simulation of the Kuma River, July 2020 — computed with iRIC © 2020, iRIC organization
4. Simulation needs the watershed first THE TRADITIONAL WORKFLOW WITH A NATIONAL DEM 01 02 03 04 Download DEM tiles for the area Fill sinks (depression filling) Compute flow direction raster Compute watershed raster Repeating this for every site blocks rapid simulation
5. A watershed should be one click away
6. So I built this ・ Click a river → the upstream watershed appears in seconds ・ Works anywhere in Japan — the largest basin included ・ Small streams: under a second ・ No downloads, no preprocessing on the user side
7. Two ingredients THE DATA THE ALGORITHM J-FlwDir Reverse BFS Nationwide flow direction map Breadth-first search, walked upstream
8. The data: J-FlwDir Japan Flow Direction Map — Yamazaki Lab, The University of Tokyo Resolution 1 arcsec (~30 m) Coverage All of Japan Distribution 1°×1° GeoTIFF tiles Encoding D8 flow direction Nationwide mosaic 97,200 × 79,200 px = 7.7 billion cells Flow directions are hydrologically corrected, so river networks stay connected. Ready for watershed extraction anywhere in Japan.
9. The algorithm: Reverse BFS From the clicked cell, collect the neighbors that flow into the visited set. One step at a time. STEP 1 STEP 2 STEP 3
10. Reverse BFS makes two opposing demands THE RANGE IT MAY TOUCH THE DATA IT ACTUALLY READS Unpredictable Tiny The search can reach anywhere Even the Tone touches under 1% of it: upstream, so the whole nationwide grid 114 chunks of 512². A small stream has to be available reads one. Everything within reach, almost nothing read
11. That's exactly what COG and Zarr are for Read only the parts you need, the moment you need them, without downloading 7.7 billion cells ・ Both formats store the raster as a grid of chunks ・ Any chunk is retrievable via HTTP range requests ・ 7.7B cells stay on Amazon S3; each request reads only a fraction of them
12. Read only the chunks the search reaches COG stores tiles, Zarr stores chunks, so the same partial read works on both. chunks fetched extracted watershed never read
13. System architecture
14. Fast enough. But what makes it fast? WHAT WORKS WHAT COULD BE CHANGING IT ~20 s for the Tone ・ Chunk size 20 million cells, no preprocessing. A small stream comes back in under a ・ Watershed size second. So I measured it ・ File format
15. Benchmark design 8 variants, all derived from the same source GeoTIFF FORMAT CHUNK SIZE EXTRA COG 256 / 512 / 1024 / 2048 LZW Zarr v3 256 / 512 / 1024 / 2048 zstd 4 basin sizes — 20M (Tone, Ishikari) / 2.1M / 1,851 cells Every measurement is validated: all variants return the identical watershed (cell-exact). I compare speed of the same answer.
16. Chunk size beats format 256 → 2048 moves COG 1.5× and Zarr 2.6× · at large chunks the formats sit within 4% Tone River basin (20M cells) · AWS Lambda, same region as S3 · warm · median of 3 runs
17. Small chunks: the stack, not the format At the same chunk size, both formats read the same chunks, decompress the same bytes, and run the same BFS. Only the fetching differs. MEASURED AT CHUNK SIZE 512, TONE BASIN COG Zarr DATA GETS 114 — one per chunk 114 — one per chunk HTTP ROUND TRIPS 114 251 — a HEAD per GET READER STACK C++ (GDAL) Python (zarr-python + s3fs) Both costs scale with request count, so they only bite at small chunks. At the sizes you'd actually tune to, the two formats sit within 4%.
18. And big chunks are a safe default Tiny basins do pay for the extra bytes · a whole 4 MB chunk, 1.9× slower at 2048 Zarr · AWS Lambda, same region as S3 · warm But 1.9× of a tenth of a second is still a tenth of a second
19. Takeaways 01 J-FlwDir covers all of Japan, so a watershed can be extracted anywhere in the country 02 COG and Zarr are cloud-optimized, so the search can fetch just what it needs, as it needs it 03 In practice, choosing the right chunk size matters more than choosing the format
20. Beyond Japan: MERIT Hydro ・ Same lab (Yamazaki Lab) publishes MERIT Hydro — global hydrography, 3 arcsec (~90 m), same flow-direction lineage ・ The same approach should apply to any river in the world ・ Lower resolution cells and much larger basins, so the right chunk size may differ by region — worth measuring again Try it on your rivers.
21. Everything is open Code and benchmark harness github.com/nakamori1024/ jflwdir-extract-api Data: J-FlwDir (Yamazaki Lab) — registration required Thank you! MIERUNE Inc. / Shinsuke Nakamori
22. Appendix
23. Zarr sharding backfires here Slower at both chunk sizes · a shard-index fetch, then still a GET per chunk Tone River basin (20M cells) · AWS Lambda, same region as S3 · warm · median of 3 runs Sharding is a write-side optimization, not a read-side one
24. You can't buy single-core speed 5× the Lambda memory, and vCPU with it · runtime unchanged within a second Tone River basin (20M cells) · AWS Lambda, warm · single-core bound The extraction is single-threaded — extra cores just idle