Xarray flexible indexes

Xarray User Forum / Dask Summit 2021

Transcript

1. Xarray explicit/flexible indexes Benoît Bovy Dask Distributed Summit (May 2021)
2. Xarray data model
3. Some (many) of Xarray’s core features…
4. Label-based data selection
5. Automatic alignment
6. Group-by, interpolate, resample, di ff erentiate, integrate…
7. … (internally) rely on pandas.Index
8. Works well but has some limitations…
9. Irregular data X
10. Irregular data X Only one index per dimension
11. Irregular data X pd.MultiIndex does not support method=‘nearest’
12. Large size dimension(s)
13. Large size dimension(s) Unlike Dask DataFrame, no index partitioning in Xarray
14. Other (domain-speci fi c) cases • Geospatial data: Coordinate Reference System (CRS) pd.Index does not make any assumption about CRS • Staggered grids (cell centers vs. cell edges) • Any other case? Let’s hear from you! (src: http://thevisualroom.com) (src: QGIS documentation)
15. Workarounds
16. “Hacking” Xarray Use extra coordinates, attributes and/or Dataset (DataArray) accessors to store some objects…
17. Explicit, flexible indexes! (roadmap, ongoing, CZI grant)
18. “Explicit” indexes Make indexes 1st-class citizens of the Xarray data model Indexes: lat lon time Float64Index Float64Index DatetimeIndex
19. “Flexible” indexes Provide xarray.Index API (data selection, alignment… more?) + extension mechanism (e.g., entrypoints) Indexes: x, y KDTreeIndex An index may be built from several coordinates (possibly also from multi-dimension coordinates and/or coordinates with different dimensions)
20. More info pydata/xarray: design_notes/flexible_indexes_notes.md Projects - > Explicit Indexes
21. Use case: Point-wise selection of irregular data xarray - contrib/xoak
22. An example: 2D irregular mesh with lat/lon coordinates
23. Select nearest-neighbors (1D query point dataset)
24. this won’t be needed anymore with Xarray fl exible indexes Select nearest-neighbors (1D query point dataset)
25. Select nearest-neighbors (2D query point dataset) Xarray advanced indexing is really powerful!
26. Experimental Dask support (chunked point coordinates) 1st stage: “map” index lookup query points chunks Index points chunks index<1> .query()
27. Experimental Dask support (chunked point coordinates) 2nd stage: “reduce” brute-force lookup query points chunks Index points chunks dask.array.argmin
28. Experimental Dask support (chunked point coordinates) It works well, sometimes… …but often fails (miserably) It is challenging! - Chunk size matters - Dask inter-worker communication - Indexes are often complex objects - C/C++ native & dynamic data structures - Memory footprint? - Serialization?