Short 4-minute talk on using the Cubed package as an alternative to dask.array for processing large datasets in Xarray.
Given as a lightning talk at the SciPy Conference 2023 in Austin, TX.
See this blog post for more details (https://xarray.dev/blog/cubed-xarray)
Serverless Array Processing
Big science means *Big* arrays
So use dask.array!
Dask is great, but it doesn’t always succeed…
Sometimes unexpectedly exceeds your
Q: Can we guarantee
distributed array execution
respects RAM constraints?
A: Yes! For certain operations…
Deploy one serverless container per chunk - read from / write to Zarr
Xarray wraps Cubed OR Dask OR [new things??]
Read the blog post! https://xarray.dev/blog/cubed-xarray
Also thanks Tom White
for writing Cubed!
Join the discussion!