Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cubed-xarray lightning talk at SciPy2023

Avatar for Tom Nicholas Tom Nicholas
July 17, 2023
Programming
0
140

Cubed-xarray lightning talk at SciPy2023

Short 4-minute talk on using the Cubed package as an alternative to dask.array for processing large datasets in Xarray.

Given as a lightning talk at the SciPy Conference 2023 in Austin, TX.

See this blog post for more details (https://xarray.dev/blog/cubed-xarray)
Avatar for Tom Nicholas

Tom Nicholas

July 17, 2023
Tweet

More Decks by Tom Nicholas

See All by Tom Nicholas
VirtualiZarr & Icechunk: Build a cloud-optimized datacube in 3 lines
Avatar for Tom Nicholas tomnicholas
0
32
FROST: Federated Registry of Scientific Things (@ Pangeo Showcase)
Avatar for Tom Nicholas tomnicholas
0
40
VirtualiZarr talk at MET Office
Avatar for Tom Nicholas tomnicholas
0
180
VirtualiZarr: Create virtual Zarr stores using xarray syntax
Avatar for Tom Nicholas tomnicholas
0
190
What's next for Pangeo?
Avatar for Tom Nicholas tomnicholas
0
110
Cubed: Bounded-Memory Serverless Array Processing (Pangeo showcase)
Avatar for Tom Nicholas tomnicholas
0
92
Pangeo for Plasma
Avatar for Tom Nicholas tomnicholas
0
160
xGCM: Staggered grids, topologies, and ufuncs in python
Avatar for Tom Nicholas tomnicholas
0
310
xarray-Datatree: Hierarchical Data Structures for Multi-Model Science
Avatar for Tom Nicholas tomnicholas
0
1.4k

Other Decks in Programming

See All in Programming
LRパーサーはいいぞ
Avatar for ydah ydah
5
1.1k
Storybookの情報をMCPサーバー化する
Avatar for Shota Iwamatsu shota_tech
1
200
Flutterでllama.cppをつかってローカルLLMを試してみた
Avatar for sakuraidayo sakuraidayo
0
130
AIコーディングの理想と現実
Avatar for Tomohisa Takaoka tomohisa
35
37k
KANNA Android の技術的課題と取り組み
Avatar for Yosuke Watanabe watabee
0
200
監視 やばい
Avatar for syossan27 syossan27
12
10k
インプロセスQAにおいて大事にしていること / In-process QA Meetup
Avatar for MEDLEY, INC. medley
0
140
Optimizing JRuby 10
Avatar for headius headius
0
580
Browser and UI #2 HTML/ARIA
Avatar for ken7253 ken7253
2
170
エンジニアが挑む、限界までの越境
Avatar for Nealle nealle
1
320
オープンソースコントリビュート入門
Avatar for katsuma _katsuma
0
120
The Missing Link in Angular’s Signal Story: Resource API and httpResource
Avatar for Manfred Steyer manfredsteyer
PRO
0
140

Featured

See All Featured
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
120
52k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
32
5.6k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
331
21k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
14
1.4k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
299
50k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
798
220k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
13
840
Fontdeck: Realign not Redesign
Avatar for Paul Robert Lloyd paulrobertlloyd
84
5.5k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
411
22k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
230
18k

Transcript

  1. Cubed: Bounded-Memory Serverless Array Processing (in Xarray) *Tom Nicholas Tom

    White *[email protected] *github.com/TomNicholas

  2. Big science means *Big* arrays 😍 😬 PBs??

  3. So use dask.array! Dask is great, but it doesn’t always

    succeed… Sometimes unexpectedly exceeds your RAM budget 😕 Q: Can we guarantee distributed array execution respects RAM constraints?

  4. Rechunker ✨Cubed✨ (Bounded-memory) A: Yes! For certain operations… 🤔

  5. Invented by Cubed’s Design

  6. Coiled Functions … Serverless execution Deploy one serverless container per

    chunk - read from / write to Zarr

  7. Xarray wraps Cubed OR Dask OR [new things??] Executes via

    Executes via Cubed ?? Tabular data: Array data:

  8. Read the blog post! https://xarray.dev/blog/cubed-xarray Also thanks Tom White for

    writing Cubed! Join the discussion!