$30 off During Our Annual Pro Sale. View Details »

CMIP6 in the Cloud: Open, Fast, and Accessible Climate Science with Pangeo

Julius Busecke
January 26, 2022

CMIP6 in the Cloud: Open, Fast, and Accessible Climate Science with Pangeo

Slides to my AMS presentation in the "Joint J10B - Cloud Computing for Big Data in Atmosphere, Ocean, and Climate" session (https://ams.confex.com/ams/102ANNUAL/meetingapp.cgi/Session/59409)

Demo Video: https://youtu.be/bJNpfuW_ypU

Julius Busecke

January 26, 2022
Tweet

More Decks by Julius Busecke

Other Decks in Science

Transcript

  1. CMIP6 IN THE CLOUD OPEN, FAST, AND ACCESSIBLE CLIMATE SCIENCE

    WITH PANGEO | JULIUS BUSECKE | RYAN ABERNATHEY
  2. BIG DATA CLIMATE SCIENCE CMIP 6 Custom Analysis applied to

    each model and member
  3. DOWNLOAD FTP / OPeNDAP / etc.

  4. MB 😀 FTP / OPeNDAP / etc.

  5. GB 😐 FTP / OPeNDAP / etc.

  6. TB 😖 FTP / OPeNDAP / etc.

  7. PB 😱 FTP / OPeNDAP / etc.

  8. BRING THE COMPUTE TO THE DATA! Jupyter for interactive access

    remote systems Cloud / HPC Xarray provides data structures and intuitive interface for interacting with datasets Parallel computing system allows users deploy clusters of compute nodes for data processing. Dask tells the nodes what to do. Distributed storage “Analysis Ready Data” 
 stored on globally-available distributed storage. PANGEO ARCHITECTURE
  9. CLIMATE SCIENCE IN THE CLOUD

  10. CLIMATE SCIENCE IN THE CLOUD EASY COLLABORATION + PUBLIC ACCESS

    = REPRODUCIBLE SCIENCE
  11. CLIMATE SCIENCE IN THE CLOUD ACCESSIBLE BY ANYONE WITH AN

    INTERNET CONNECTION = MORE DIVERSE SCIENCE COMMUNITY
  12. CLIMATE SCIENCE IN THE CLOUD PERFORMANCE + DIVERSE COMMUNITY +

    OPEN SCIENCE = FAST SCIENTIFIC PROGRESS
  13. IT ALL STARTS WITH THE DATA Pangeo/ESGF is hosting a

    large part of the CMIP6 archive in public cloud buckets. Check out pangeo-data.github.io/pangeo-cmip6-cloud for more info. Wondering how to bring your data to the cloud? Check out Pangeo Forge Later today (2:30 EST) - Pangeo Forge: Crowdsourcing Analysis-Ready, Cloud Optimized Data for Ocean, Weather, and Climate Science Ocean Science Meeting [IN08C Open Ocean Science] - Pangeo Forge Mini-Hackathon
  14. PANGEO CMIP6 STACK Custom Code + Other packages Pangeo Foundation

    CMIP6 speci f ic Packages Project speci f ic Packages
  15. None
  16. But CMIP6 is not quite clean enough

  17. 😖

  18. 😖 😎

  19. DEMO

  20. SAY GOODBYE TO DOWNLOADS AND HELLO TO THE OPEN, FAST,

    AND FUN FUTURE OF CLIMATE SCIENCE FOR EVERYBODY.
  21. I ❤ FEEDBACK, QUESTIONS, CONTRIBUTIONS. CMIP data in the cloud

    Pangeo Cloud Deployments - https://pangeo.io/cloud.html Docs Github @JuliusBusecke jbusecke julius@ldeo.columbia.edu