CMIP6 in the Cloud: Open, Fast, and Accessible Climate Science with Pangeo

Slides to my AMS presentation in the "Joint J10B - Cloud Computing for Big Data in Atmosphere, Ocean, and Climate" session (https://ams.confex.com/ams/102ANNUAL/meetingapp.cgi/Session/59409)

Demo Video: https://youtu.be/bJNpfuW_ypU

Transcript

1. CMIP6 IN THE CLOUD OPEN, FAST, AND ACCESSIBLE CLIMATE SCIENCE WITH PANGEO | JULIUS BUSECKE | RYAN ABERNATHEY
2. BIG DATA CLIMATE SCIENCE CMIP 6 Custom Analysis applied to each model and member
3. DOWNLOAD FTP / OPeNDAP / etc.
4. MB 😀 FTP / OPeNDAP / etc.
5. GB 😐 FTP / OPeNDAP / etc.
6. TB 😖 FTP / OPeNDAP / etc.
7. PB 😱 FTP / OPeNDAP / etc.
8. BRING THE COMPUTE TO THE DATA! Jupyter for interactive access remote systems Cloud / HPC Xarray provides data structures and intuitive interface for interacting with datasets Parallel computing system allows users deploy clusters of compute nodes for data processing. Dask tells the nodes what to do. Distributed storage “Analysis Ready Data” 
 stored on globally-available distributed storage. PANGEO ARCHITECTURE
9. CLIMATE SCIENCE IN THE CLOUD
10. CLIMATE SCIENCE IN THE CLOUD EASY COLLABORATION + PUBLIC ACCESS = REPRODUCIBLE SCIENCE
11. CLIMATE SCIENCE IN THE CLOUD ACCESSIBLE BY ANYONE WITH AN INTERNET CONNECTION = MORE DIVERSE SCIENCE COMMUNITY
12. CLIMATE SCIENCE IN THE CLOUD PERFORMANCE + DIVERSE COMMUNITY + OPEN SCIENCE = FAST SCIENTIFIC PROGRESS
13. IT ALL STARTS WITH THE DATA Pangeo/ESGF is hosting a large part of the CMIP6 archive in public cloud buckets. Check out pangeo-data.github.io/pangeo-cmip6-cloud for more info. Wondering how to bring your data to the cloud? Check out Pangeo Forge Later today (2:30 EST) - Pangeo Forge: Crowdsourcing Analysis-Ready, Cloud Optimized Data for Ocean, Weather, and Climate Science Ocean Science Meeting [IN08C Open Ocean Science] - Pangeo Forge Mini-Hackathon
14. PANGEO CMIP6 STACK Custom Code + Other packages Pangeo Foundation CMIP6 speci f ic Packages Project speci f ic Packages
16. But CMIP6 is not quite clean enough
17. 😖
18. 😖 😎
19. DEMO
20. SAY GOODBYE TO DOWNLOADS AND HELLO TO THE OPEN, FAST, AND FUN FUTURE OF CLIMATE SCIENCE FOR EVERYBODY.
21. I ❤ FEEDBACK, QUESTIONS, CONTRIBUTIONS. CMIP data in the cloud Pangeo Cloud Deployments - https://pangeo.io/cloud.html Docs Github @JuliusBusecke jbusecke julius@ldeo.columbia.edu