at e d ata? time longitude latitude elevation Data variables used for computation Coordinates describe data Attributes + land_cover Metadata Standard: http://cfconventions.org/ Image Credit: Stephan Hoyer (xarray)
k s t o r a g e Image credit: https://blog.ubuntu.com/2015/05/18/what-are-the-different-types-of-storage-block-object-and-file • Operating system provides mechanism to read / write files and directories (e.g. POSIX). • Seeking and random access to bytes within files is fast. • “Most file systems are based on a block device, which is a level of abstraction for the hardware responsible for storing and retrieving specified blocks of data”
r a g e Image credit: https://blog.ubuntu.com/2015/05/18/what-are-the-different-types-of-storage-block-object-and-file • An object is a collection of bytes associated with a unique identifier • Bytes are read and written with http calls • Significant latency for each individual operation • Application level (not OS dependent) • Implemented by S3, GCS, Azure, Ceph, etc. • Underlying hardware…who knows?
b l e m : !11 We don’t know how issue an HTTP request to peek into an HDF file. HDF client library does not support this*. If we want to know what’s in it, we have to download the whole thing.
array_name 0.0 0.1 2.0 1.0 1.1 2.1 https://zarr.readthedocs.io/ • Open source library for storage of chunked, compressed ND-arrays • Created by Alistair Miles (Imperial) for genomics research (@alimanfoo); now community supported standard • Arrays are split into user-defined chunks; each chunk is optional compressed (blosc, zlib, zstd, lz4, etc., + easy to extend) • Can store arrays in memory, directories, zip files, or any python mutable mapping interface (dictionary) • External libraries provide a way to store directly into cloud object storage; easy to implement an interface for custom storage • Implementations in Python, C++, Java (N5), Julia, (C coming)
fields suddenly have many PB of data to analyze • Workflows vary widely, but most exciting applications involve processing all the data many times over in an interactive context • This work is strongly limited by storage / network throughput • We can get HPC-like performance on commodity cloud object storage with proper software abstractions
.zarray .zattrs Zarr Array: array_name 0.0 0.1 2.0 1.0 1.1 2.1 { "_ARRAY_DIMENSIONS": [ "time", "latitude", "longitude" ], "comment": "The sea level anomaly is the sea surface height above mean sea surface; it is referenced to the [1993, 2012] period; see the product user manual for details", "coordinates": "crs", "grid_mapping": "crs", "long_name": "Sea level anomaly", "standard_name": "sea_surface_height_above_sea_level", "units": "m" } Example .attrs file (json)
and write directly to a Zarr store (with @jhamman) • It was pretty easy! Data models are quite similar • Automatic mapping between zarr chunks <—> dask chunks • We needed to add a custom, “hidden” attribute (_ARRAY_DIMENSIONS) to give the zarr arrays dimensions !19 X a r r ay + z a r r
dataset with reasonable chunks ds = xr.open_mfdatset(‘bunch_o_files_*.nc’, chunks={‘time’: 1}) 2. Export to zarr ds.to_zarr(‘/path/to/zarr/directory’) ——or—— ds.to_zarr(gcsamp_object) 3. [maybe] upload to cloud storage $ gcsutil -m cp -r /path/to/zarr/directory gs://pangeo-data/path !20 P r e pa r i n g d ata s e t s f o r z a r r c l o u d s t o r a g e
w i t h T H R E D D S • Worked with Luca Cinquini, Hans Vahlenkamp, Aparna Radhakrishnan to connect pangeo to ESGF server running in Google Cloud • Used Dask to issue parallel OpenDAP reads from a cluster
w i t h T H R E D D S • Worked with Luca Cinquini, Hans Vahlenkamp, Aparna Radhakrishnan to connect pangeo to ESGF server running in Google Cloud • Used Dask to issue parallel OpenDAP reads from a cluster
rabernat
taishin
shotashiratori
nakasho
lycorptech_jp
mh4gf
dayjournal
uhyo
poteboy
infiniteloop_inc
nagix
manarobot
kneath
trishagee
scottboms
morganepeng
robhawkes
skipperchong
bluesmoon
yeseniaperezcruz
swwweet
sferik
myddelton
chriscoyier
