Ocean Cloud (OceanShot NAS Plenary Talk)

Plenary talk presented at the 2021 February 3-4 launch meeting in response to the U.S. National Committee for the Ocean Decade call for disruptive advances in ocean science.

https://www.nationalacademies.org/our-work/us-national-committee-on-ocean-science-for-sustainable-development-2021-2030/ocean-shot-directory

1. O c e a n C l o u d Transforming oceanography with a new approach to data and computing Ryan Abernathey
2. Physical Oceanographer Ph.D. From MIT, 2012 Associate Prof. at Columbia / LDEO https://ocean-transport.github.io/ Co-founder of Pangeo Open Source Developer Open Science Advocate O c e a n C l o u d Transforming oceanography with a new approach to data and computing Ryan Abernathey
3. Problem: Ocean data are huge and complex! 🤯 This limits scientific inquiry and restricts participation. 😔 Solution: OceanCloud: a new approach to infrastructure based on cloud computing, open data, and open-source software. 😎 T h i s Ta l k 3
4. 4 Credit: NASA's Goddard Space Flight Center
5. 4 Credit: NASA's Goddard Space Flight Center
6. 4 Credit: NASA's Goddard Space Flight Center https://earthdata.nasa.gov/eosdis/cloud-evolution SWOT NISAR
7. 5 Credit: NASA's Goddard Space Flight Center
8. 5 Credit: NASA's Goddard Space Flight Center
9. 6 “Brb, let me just go download the data to my laptop…”
10. 6 PB “Brb, let me just go download the data to my laptop…”
11. P r i v i l e g e d I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 7 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons *Coined by Chelle Gentemann
12. P r i v i l e g e d I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 7 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons *Coined by Chelle Gentemann
13. P r i v i l e g e d I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 8 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons Data *Coined by Chelle Gentemann
14. P r i v i l e g e d I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 8 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons Data *Coined by Chelle Gentemann
15. P r i v i l e g e d I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 9 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons Data ❌ Results not reproducible outside fortress ❌ Barrier to collaboration ❌ Inefficient / duplicative ❌ Can’t scale to future data needs ❌ Limits inclusion and knowledge transfer *Coined by Chelle Gentemann
16. • Grass-roots collaboration between scientists, software developers around open-source tools for solving real problems • Foundational support from NSF EarthCube • International partners, industry connections 10 A C o m m u n i t y P l at f o r m f o r B i g - D ata G e o s c i e n c e http://pangeo.io
17. • Grass-roots collaboration between scientists, software developers around open-source tools for solving real problems • Foundational support from NSF EarthCube • International partners, industry connections 10 A C o m m u n i t y P l at f o r m f o r B i g - D ata G e o s c i e n c e http://pangeo.io
18. Scientific users / use cases Open-source software libraries HPC and cloud infrastructure • Define science questions • Use software / infrastructure • Identify bugs / bottlenecks • Provide feedback to developers • Contribute widely the the open source scientific python ecosystem • Maintain / extend existing libraries, start new ones reluctantly • Solve integration challenges • Deploy interactive analysis environments • Curate analysis-ready datasets • Platform agnostic Agile development 👩💻 Pa n g e o B u i l d s w i t h O p e n D e v e l o p m e n t 11
19. T h r e e P i l l a r s o f C l o u d D ata E n v i r o n m e n t s 12 “Analysis Ready Data” Cleaned, curated open- access datasets available via high-performance globally available strorage system “Elastic Scaling” Automatically provision many computers on demand to accelerate big data processing. cloud “Data Proximate Computing” Bring analysis to the data. Web- based access provides “on-click to compute” access.
20. T h r e e P i l l a r s o f C l o u d D ata E n v i r o n m e n t s 12 “Analysis Ready Data” Cleaned, curated open- access datasets available via high-performance globally available strorage system “Elastic Scaling” Automatically provision many computers on demand to accelerate big data processing. cloud “Data Proximate Computing” Bring analysis to the data. Web- based access provides “on-click to compute” access.
21. O c e a n - C l o u d W i l l b e a “ D ata W at e r i n g H o l e * ” 13 *Coined by Fernando Perez
22. 14 *Coined by Fernando Perez O c e a n - C l o u d W i l l b e a “ D ata W at e r i n g H o l e * ”
23. 15 👩💻👨💻👩💻 Group A: Air-Sea Interaction 👩💻👨💻👩💻 Group B: Seasonal Forecasting Research Education Industry *Coined by Fernando Perez O c e a n - C l o u d W i l l b e a “ D ata W at e r i n g H o l e * ”
24. 15 👩💻👨💻👩💻 Group A: Air-Sea Interaction 👩💻👨💻👩💻 Group B: Seasonal Forecasting Research Education Industry ✅ Faster science, more discoveries ✅ Inherently reproducible ✅ Allows seamless global collaboration ✅ Unleashes creativity ✅ Cost effective ✅ Accessible to all ✅ Connects with industry *Coined by Fernando Perez O c e a n - C l o u d W i l l b e a “ D ata W at e r i n g H o l e * ”
25. • Many agencies (e.g. NASA, NOAA) are already moving data distribution to cloud… • …but the missing link is accessible cloud computing environments for all. Pangeo and its partners can help with this. • We must avoid building new fortresses in the cloud and ensure interoperability from the start! • A National Oceanographic Partnership Program (NOPP) could provide funding, help facilitate inter-agency collaboration, and support user adoption. H o w c a n w e A c h i e v e t h i s ? 16
26. L e a r n M o r e 17 http://pangeo.io https://github.com/pangeo-data/ https://medium.com/pangeo @pangeo_data