Cubed-xarray lightning talk at SciPy2023

Short 4-minute talk on using the Cubed package as an alternative to dask.array for processing large datasets in Xarray.

Given as a lightning talk at the SciPy Conference 2023 in Austin, TX.

See this blog post for more details (https://xarray.dev/blog/cubed-xarray)

Transcript

1. Cubed: Bounded-Memory Serverless Array Processing (in Xarray) *Tom Nicholas Tom White *thomas.nicholas@columbia.edu *github.com/TomNicholas
2. Big science means *Big* arrays 😍 😬 PBs??
3. So use dask.array! Dask is great, but it doesn’t always succeed… Sometimes unexpectedly exceeds your RAM budget 😕 Q: Can we guarantee distributed array execution respects RAM constraints?
4. Rechunker ✨Cubed✨ (Bounded-memory) A: Yes! For certain operations… 🤔
5. Invented by Cubed’s Design
6. Coiled Functions … Serverless execution Deploy one serverless container per chunk - read from / write to Zarr
7. Xarray wraps Cubed OR Dask OR [new things??] Executes via Executes via Cubed ?? Tabular data: Array data:
8. Read the blog post! https://xarray.dev/blog/cubed-xarray Also thanks Tom White for writing Cubed! Join the discussion!