Cubed: Bounded-Memory
Serverless Array Processing
(in Xarray)
*Tom Nicholas
Tom White
*thomas.nicholas@columbia.edu
*github.com/TomNicholas
Slide 2
Slide 2 text
Big science means *Big* arrays
š
š¬
PBs??
Slide 3
Slide 3 text
So use dask.array!
Dask is great, but it doesnāt always succeedā¦
Sometimes unexpectedly exceeds your
RAM budget
š
Q: Can we guarantee
distributed array execution
respects RAM constraints?
Slide 4
Slide 4 text
Rechunker
āØCubedāØ
(Bounded-memory)
A: Yes! For certain operationsā¦
š¤
Slide 5
Slide 5 text
Invented
by
Cubedās Design
Slide 6
Slide 6 text
Coiled Functions
ā¦
Serverless execution
Deploy one serverless container per chunk - read from / write to Zarr
Slide 7
Slide 7 text
Xarray wraps Cubed OR Dask OR [new things??]
Executes via
Executes via
Cubed
??
Tabular
data:
Array
data:
Slide 8
Slide 8 text
Read the blog post! https://xarray.dev/blog/cubed-xarray
Also thanks Tom White
for writing Cubed!
Join the discussion!