Slide 1

Slide 1 text

Cubed: Bounded-Memory Serverless Array Processing (in Xarray) *Tom Nicholas Tom White *[email protected] *github.com/TomNicholas

Slide 2

Slide 2 text

Big science means *Big* arrays šŸ˜ šŸ˜¬ PBs??

Slide 3

Slide 3 text

So use dask.array! Dask is great, but it doesnā€™t always succeedā€¦ Sometimes unexpectedly exceeds your RAM budget šŸ˜• Q: Can we guarantee distributed array execution respects RAM constraints?

Slide 4

Slide 4 text

Rechunker āœØCubedāœØ (Bounded-memory) A: Yes! For certain operationsā€¦ šŸ¤”

Slide 5

Slide 5 text

Invented by Cubedā€™s Design

Slide 6

Slide 6 text

Coiled Functions ā€¦ Serverless execution Deploy one serverless container per chunk - read from / write to Zarr

Slide 7

Slide 7 text

Xarray wraps Cubed OR Dask OR [new things??] Executes via Executes via Cubed ?? Tabular data: Array data:

Slide 8

Slide 8 text

Read the blog post! https://xarray.dev/blog/cubed-xarray Also thanks Tom White for writing Cubed! Join the discussion!