Share
Embed
State of Pangeo - August 2019
My presentation from the August 2019 Pangeo Community Meeting
http://pangeo.io/meetings/2019_summer-meeting.html
Ryan Abernathey
August 21, 2019
Transcript
24 slides
-
S tat e o f Pa n g e o
A u g u s t 2 0 1 9 C o m m u n i t y M e e t i n g -
!2 “Pangeo is first and foremost a community promoting open,
reproducible, and scalable science. “This community provides documentation, develops and maintains software, and deploys computing infrastructure to make scientific research and programming easier.” http://pangeo.io/ -
Community Infrastructure Software Documentation Pa n g e o R
e p o r t C a r D * !3 A A B C *Just one opinion Education? 2018-2019 School Year -
Pa n g e o C o m m u
n i t y !4 We have a growing group of engaged participants in discussion. ✓GitHub Discussions ✓Weekly Checkin Meetings ✓Medium Blog ✓Working Groups ✓US / UK / Europe ✓Students / Postdocs / Faculty / Software Devs / Data Scientists ✓Academia / National Labs / Industry / NGO ✓Weather / Climate / Oceans / Geoscience / Neuroscience / Bioinformatics? / Astronomy? -
Pa n g e o C o m m u
n i t y !5 We have a growing group of engaged participants in discussion. ✓GitHub Discussions ✓Weekly Checkin Meetings ✓Medium Blog ✓Working Groups ✓US / UK / Europe ✓Students / Postdocs / Faculty / Software Devs / Data Scientists ✓Academia / National Labs / Industry / NGO ✓Weather / Climate / Oceans / Geoscience / Neuroscience / Bioinformatics? / Astronomy? -
✓Unidata NetCDF roadmap calls for Zarr backend ✓NCAR NCL end-of-life
plan and the “pivot to python” cites Pangeo as a key technology for the future of data analysis ✓NASA DAACs publicly exploring Pangeo-style approaches to data distribution ✓CSIRO adoption ✓ECMWF adoption C o m m u n i t y M i l e s t o n e S !6 -
✓Unidata NetCDF roadmap calls for Zarr backend ✓NCAR NCL end-of-life
plan and the “pivot to python” cites Pangeo as a key technology for the future of data analysis ✓NASA DAACs publicly exploring Pangeo-style approaches to data distribution ✓CSIRO adoption ✓ECMWF adoption C o m m u n i t y M i l e s t o n e S !7 -
H y p e C y c l e !8
Where are we in the Pangeo hype cycle? How can we manage such rapid growth? How can we harness community enthusiasm most effectively? -
Pa n g e o I n f r a
s t r u c t u r e !9 Interactive, scalable, data-proximate computing environments for real scientific analysis NASA Pleiades HPC Cloud -
✓Major progress on dask HPC compatibility (dask-jobqueue) ✓Jupyter portal on
NCAR Cheyenne ✓Growing experimentation with Zarr format at HPC centers H P C M i l e s t o n e S !10 -
✓Pangeo binder has served 9,775 repos ✓Pangeo Cloud Federation: automated
management of many complex JupyterHub deployments across multiple clouds https://github.com/pangeo-data/pangeo-cloud-federation/ ✓Pangeo Cloud Datastore: nested catalog of Zarr datasets in cloud storage, with intake and html interfaces https://pangeo-data.github.io/pangeo-datastore/ ✓NCAR Large Ensemble in AWS ✓CMIP6 Data Catalog is growing C l o u d M i l e s t o n e S !11 -
o c e a n . pa n g e
o . i o !12 Deployed since mid-May -
o c e a n . pa n g e
o . i o !13 Deployed since mid-May -
!14 How do we sustain our cloud infrastructure? Can’t use
free credits forever… -
S o f t w a r e !15 Intake
Supporting community-driven open-source -
!16 NASA Panoply INGRID L e g a c y
s o f t w a r e -
S o f t w a r e !17 What
about domain-specific software tools? Geographically-aware indexing Regridding of regular and unstructured grids Domain-specific functions Vector calculus operations xgcm xesmf regionmask salem geoxarray? xrft climpred Machine Learning interfaces -
S o f t w a r e !18 My
recommentation: Write a white paper on best practices for domain- specific package development within Pangeo ecosystem. -
D o c u m e n tat i o
n !19 We have made huge strides in the past two years. How can we help the rest of our community catch up? -
D o c u m e n tat i o
n !20 •About Pangeo ◦Motivation ◦Mission Statement ◦Goals ◦Get Involved •Frequently Asked Questions •Guide for Scientists ◦Learn About Pangeo Software ◦Explore the Use Cases ◦Try Out a Pangeo Deployment ◦Give Feedback ◦Contribute a Use Case ◦Contribute Data ◦Become an Open Source Contributor •Packages ◦Pangeo Core Packages ◦Pangeo Affiliated Packages ◦Guidelines for New Packages •Geoscience Use Cases ◦Physical Oceanography ◦Climate modeling ◦Meteorology •Technical Architecture ◦Where we began ◦Interoperability in Pangeo ◦Software ◦Compute Platforms ◦Storage Formats •Deployment Setup Guides ◦Setting up Pangeo on HPC Systems ◦Setting up Pangeo on Cloud Systems •Deployments ◦Pangeo provided Cloud Deployments ◦Other Cloud deployments ◦High Performance Computing Deployments •Pangeo and Data ◦Data on HPC ◦Data in the Cloud •Pangeo Data Catalog •Collaborators ◦Funding Agencies ◦Institutions ◦People •Pangeo Weekly Meeting Notes ◦Meeting Calendar ◦Meeting Notes •Pangeo Conferences ◦Future Meetings ◦Past Meetings •Contact Pangeo -
D o c u m e n tat i o
n !21 •About Pangeo ◦Motivation ◦Mission Statement ◦Goals ◦Get Involved •Frequently Asked Questions •Guide for Scientists ◦Learn About Pangeo Software ◦Explore the Use Cases ◦Try Out a Pangeo Deployment ◦Give Feedback ◦Contribute a Use Case ◦Contribute Data ◦Become an Open Source Contributor •Packages ◦Pangeo Core Packages ◦Pangeo Affiliated Packages ◦Guidelines for New Packages •Geoscience Use Cases ◦Physical Oceanography ◦Climate modeling ◦Meteorology •Technical Architecture ◦Where we began ◦Interoperability in Pangeo ◦Software ◦Compute Platforms ◦Storage Formats •Deployment Setup Guides ◦Setting up Pangeo on HPC Systems ◦Setting up Pangeo on Cloud Systems •Deployments ◦Pangeo provided Cloud Deployments ◦Other Cloud deployments ◦High Performance Computing Deployments •Pangeo and Data ◦Data on HPC ◦Data in the Cloud •Pangeo Data Catalog •Collaborators ◦Funding Agencies ◦Institutions ◦People •Pangeo Weekly Meeting Notes ◦Meeting Calendar ◦Meeting Notes •Pangeo Conferences ◦Future Meetings ◦Past Meetings •Contact Pangeo -
D o c u m e n tat i o
n R e fa c t o r !22 pangeo.io website Pangeo book Share general information about the project. Communicate with stakeholders and potential partners. Share technical information. Working title: Cloud Native Science I. User guide packages, data formats, dask tricks, etc. II. Administrator guide Hub deployment, dataset production, etc. -
• Education needs to be a central focus of our
project going forward • Model after the NCL group: a traveling Pangeo roadshow of end-user workshops • Partner closely with Software Carpentry, university educators, etc. D o c u m e n tat i o n E d u c at i o n !23 -
• Continue successful outreach and engagement with data providers •
Creatively brainstorm how to sustain our cloud-based environments • Work together on pangeo-interoperable domain-specific software • Improve our documentation S u m m a r y !24 My goals for the next year: