Pangeo @ 2018 Cornell Cloud Forum

Pangeo: A Community Driven Platform for Big Scientific Data Analytics on HPC and Cloud
Many fields of science today involve large groups of scientists all studying a similar set of large, shared datasets. Astronomy is a prime example, as are atmospheric science, oceanography, neuroscience, and genomics. As these shared datasets grow into the petabyte range, traditional analysis tools are struggling to keep up, holding back the progress of research. A central goal of the Pangeo project is to meet this challenge by developing data and software infrastructure to enable interactive-speed analysis of the largest geoscience datasets. This is accomplished through the integration of existing open source scientific Python technologies within an HPC or cloud environment. The software packages include xarray, a Python package for working with labeled, multidimensional array data; Dask, a parallel computing library for Python that helps xarray represent huge datasets and distribute computations across clusters; JupyterHub and JupyterLab, computing environments that enable users to interact with cloud-based resources; and Kubernetes, a versatile, cloud-agnostic scheduler for running interactive and batch workloads. My talk will explain how these tools are being deployed used today in climate and oceanography and speculate about how they might be useful for other fields

http://blogs.cornell.edu/cloudforum/home/agenda/

Transcript

1. Pa n g e o A c o m m u n i t y- d r i v e n e f f o r t f o r 
 B i g D ata g e o s c i e n c e
2. G l o b a l w a r m i n g i s h a p p e n i n g ! !2
3. !3 W h at D r i v e s P r o g r e s s i n G E O S c i e n c e ? New Ideas New Observations New Simulations E 5 r 0 jUj p ðN/jUj jfj/jUj P 1D (k) ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi N2 2 jUj2k2 q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jUj2k2 2 f2 q dk, (3) where k 5 (k, l) is now the wavenumber in the reference frame along and across the mean flow U and P 1D (k) 5 1 2p ð1‘ 2‘ jkj jkj P 2D (k, l) dl (4) is the effective one-dimensional (1D) topographic spectrum. Hence, the wave radiation from 2D topogra- phy reduces to an equivalent problem of wave radiation from 1D topography with the effective spectrum given by P1D (k). The effective 1D spectrum captures the effects of 2D c. Bottom topography Simulations are configured with multiscale topogra- phy characterized by small-scale abyssal hills a few ki- lometers wide based on multibeam observations from Drake Passage. The topographic spectrum associated with abyssal hills is well described by an anisotropic parametric representation proposed by Goff and Jordan (1988): P 2D (k, l) 5 2pH2(m 2 2) k 0 l 0 1 1 k2 k2 0 1 l2 l2 0 !2m/2 , (5) where k0 and l0 set the wavenumbers of the large hills, m is the high-wavenumber spectral slope, related to the pa- FIG. 3. Averaged profiles of (left) stratification (s21) and (right) flow speed (m s21) in the bottom 2 km from observations (gray), initial condition in the simulations (black), and final state in 2D (blue) and 3D (red) simulations.
4. !4 T h e T r a d i t i o n a l A p p r o a c h data provider FTP Service $ wget ftp://all/the/files/* $ python download_script.py Weird API Weird GUI data browser Let’s work on something else… ? My Workstation result = [] for file in all_files: result.append(process(file))
5. !5 D a r k R e p o s i t o r y * $ wget ftp://all/the/files/* * Balaji et al., 2018. Requirements for a global data infrastructure in support of CMIP6. Geoscientific Model Development Discussions. “Local copy of a dataset created to enable users to actually compute on the data.”
6. M y B i g D ata J o u r n e y !6 2013 2014 2015 2016 2017 2018 started at Columbia wandered the desert discovered
 Big Data
7. M y B i g D ata J o u r n e y !7 2013 2014 2015 2016 2017 2018 discovered
 Big Data started at Columbia wandered the desert discovered xarray!
8. S c i e n t i f i c P y t h o n f o r D ata S c i e n c e !8 source: stackoverflow.com
9. aospy S c i e n t i f i c P y t h o n f o r D ata S c i e n c e !9 SciPy Credit: Stephan Hoyer, Jake Vanderplas (SciPy 2015)
10. X a r r ay D ata s e t: M u lt i d i m e n s i o n a l Va r i a b l e s 
 w i t h c o o r d i n at e s a n d m e ta d ata !10 time longitude latitude elevation Data variables used for computation Coordinates describe data Indexes align data Attributes metadata ignored by operations + land_cover “netCDF meets pandas.DataFrame” Credit: Stephan Hoyer
11. d a s k !11 Complex computations represented as a graph of individual tasks. 
 Scheduler optimizes execution of graph. https://github.com/dask/dask/ ND-Arrays are split into chunks that comfortably fit in memory
12. M y B i g D ata J o u r n e y !12 2013 2014 2015 2016 2017 2018 discovered
 Big Data started at Columbia wandered the desert discovered xarray! used xarray on “medium data” connected with xarray community first Pangeo workshop
13. • Foster collaboration around the open source scientific python ecosystem for ocean / atmosphere / land / climate science. • Support the development with domain-specific geoscience packages. • Improve scalability of these tools to to handle petabyte-scale datasets on HPC and cloud platforms. !13 Pa n g e o P r o j e c t g o a l s
14. M y B i g D ata J o u r n e y !14 2013 2014 2015 2016 2017 2018 discovered
 Big Data started at Columbia wandered the desert discovered xarray! used xarray on datasets up to ~200 GB connected with fantastic xarray community first Pangeo workshop Earthcube proposal awarded pangeo.pydata.org
15. E a r t h c u b e A w a r d T e a m !15 Ryan Abernathey, Chiara Lepore, Michael Tippet, Naomi Henderson, Richard Seager 
 
 
 Kevin Paul, Joe Hamman, Ryan May, Davide Del Vento 
 
 
 Matthew Rocklin
16. O t h e r C o n t r i b u t o r s !16 Jacob Tomlinson, Niall Roberts, Alberto Arribas Developing and operating Pangeo environment to support analysis of UK Met office products Rich Signell Deploying Pangeo on AWS to support analysis of coastal ocean modeling Justin Simcock Operating Pangeo in the cloud to support Climate Impact Lab research and analysis Supporting Pangeo via SWOT mission and recently funded ACCESS award to UW / NCAR Yuvi Panda, Chris Holdgraf Spending lots of time helping us make things work on the cloud
17. • Lazy data access / just-in-time computation • Scalable, volatile compute • Parallel data access • Thin client !17 PA N G E O P R I N C I P L E S • Open Source / Community Development • Specialize late • Avoid writing / maintaining code as much as possible • Franchisable
18. !18 Pa n g e o E n v i r o n m e n t commercial cloud / large HPC data_chunk.0000 My Laptop import dask.dataframe as ddf df = ddf.read_parquet(‘s3://bckt‘) df.foo.value_counts() import xarray as xr ds = xr.open_zar(‘gs://my/bucket‘) ds.mean(dim=[‘time', ‘lon']) High performance
 file / object storage Raw data data_chunk.0001 data_chunk.0002 data_chunk.0003 compute scientist’s compute nodes Dask pod Dask pod Dask pod Jupyter pod
19. !19 B u i l d y o u r o w n pa n g e o Storage Formats Cloud Optimized COG/Zarr/Parquet/etc. ND-Arrays More coming… Data Models Processing Mode Interactive Batch Serverless Compute Platform HPC Cloud Local
20. !20 Pa n g e o D e p l o y m e n t s NASA Pleiades pa n g e o . p y d ata . o r g NCAR Cheyenne Over 1000 unique users since March http://pangeo.io/deployments.html
21. • Open to anyone with a GitHub account! … but highly experimental / unstable • Deployed on Google Cloud Platform • Based on zero-to-jupyerhub-k8s • Customizations to allow users to launch dask clusters interactively • Pre-loaded example notebooks • Lots of data available in GCS (mostly zarr format) • Huge learning experience for everyone involved! !21 pa n g e o . p y d ata . o r g
22. pa n g e o . p y d ata . o r g u s a g e s tat s !22 since March 2017
23. • Binder allows you to turn any code (esp. notebooks) stored in a GitHub repo into an interactive Jupyter Lab. • Can now launch directly into pangeo cloud cluster !23 Pa n g e o + B i n d e r
24. • This is a general problem: let’s build general solutions • Close partnership with Jupyter team • Production environments are being deployed: 
 ocean.pangeo.io, atmos.pangeo.io, astro.pangeo.io, neuro.pangeo.io • Lack of cloud optimized data is a major roadblock !24 W h e r e i s Pa n g e o G o i n g ?
25. D ata i n t h e C l o u d E R A !25 Traditional Approach: A Data Access Portal Data Access Server file.0001.nc file.0002.nc file.0003.nc file.0004.nc Data Granules (netCDF files) Client Client Client Data Center Internet
26. !26 F i l e / B l o c k s t o r a g e Image credit: https://blog.ubuntu.com/2015/05/18/what-are-the-different-types-of-storage-block-object-and-file • Operating system provides mechanism to read / write files and directories (e.g. POSIX). • Seeking and random access to bytes within files is fast. • “Most file systems are based on a block device, which is a level of abstraction for the hardware responsible for storing and retrieving specified blocks of data”
27. !27 O b j e c t s t o r a g e Image credit: https://blog.ubuntu.com/2015/05/18/what-are-the-different-types-of-storage-block-object-and-file • An object is a collection of bytes associated with a unique identifier • Bytes are read and written with http calls • Significant overhead each individual operation • Application level (not OS dependent) • Implemented by S3, GCS, Azure, Ceph, etc.
28. D ata i n t h e C l o u d E R A !28 Direct Access to Cloud Object Storage Catalog chunk.0.0.0 chunk.0.0.1 chunk.0.0.2 chunk.0.0.3 Data Granules
 (netCDF files or something new) Cloud Object Storage Client Client Client Cloud Data Center Cloud Compute Instances
29. !29 C l o u d n at i v e D ata F o r m at s Cloud Optimized Geotiff Geospatial imagery data https://www.cogeo.org/ Apache Parquet Columnar Data
 Serialization for Apache Arrow
 Developed for Hadoop ecosystem https://parquet.apache.org/ Zarr New format for chunked, compressed, multidimensional numerical array data (plus metadata) http://zarr.readthedocs.io
30. • Python library for storage of chunked, compressed ND- arrays • Developed by Alistair Miles (Imperial) for genomics research (@alimanfoo) • Arrays are split into user-defined chunks; each chunk is optional compressed (zlib, zstd, etc.) • Can store arrays in memory, directories, zip files, or any python mutable mapping interface (dictionary) • External libraries (s3fs, gcsf) provide a way to store directly into cloud object storage !30 z a r r Zarr Group: group_name .zgroup .zattrs .zarray .zattrs Zarr Array: array_name 0.0 0.1 2.0 1.0 1.1 2.1
31. D e m o w i t h Pa n g e o B i n d e r !31 https://github.com/pangeo-data/pangeo-example-notebooks
32. • Follow our blog: https://medium.com/pangeo • Access an existing Pangeo deployment on an HPC cluster, or cloud resources (eg. pangeo.pydata.org) • Adapt Pangeo elements to meet your projects needs (data portals, etc.) and give feedback via github: github.com/pangeo-data/pangeo • Contribute to open source! xarray, dask, zarr, jupyterhub, etc. !32 H o w t o g e t i n v o lv e d http://pangeo.io