Slide 14
Slide 14 text
Pangeo cloud data access stack
2024
- netCDF -> kerchunk -> kerchunk json ->
xr.open_dataset(‘refs.json’, engine=’kerchunk’)
- Only combine once, faster to read, not duplicated, pain to use
- netCDF -> VirtualiZarr -> Icechunk ->
xr.open_zarr(icechunkstore)
- Painless, even faster to read, and version-controlled!
2018
2021
- netCDF -> fsspec -> xr.open_mfdataset(‘*.nc’)
- Slow to open and combine, slow to read
- Or… you duplicate your entire dataset as Zarr