Pangeo at Ocean Science Meeting 2022

Slide 1

Slide 1 text

Deepak Cherian + Julius Busecke Open Sesame! Open your science with Pangeo

Slide 2

Slide 2 text

Cloud-native   analysis Machine learning Crunching \Bigg datasets Analysis-Ready   Cloud Optimized Data pipelines Teaching earth   system data   science Cloud-analysis   platform Open source   scienti fi c   software Discourse   forum Data catalogs Dask Xarray Zarr

Slide 3

Slide 3 text

2010-2016 medium data Laptop/HPC   ~ 100 sims = 12 TB Reinventing many wheels 2017-2019 smol data laptop   ~GB 5 moorings!   2019- smol & big data MBs & ~TBs   laptop/server/HPC/cloud obs / models  

Slide 4

Slide 4 text

Before: for …: … After:   # groupby? (binning)   # rolling? (moving window)   # … 🤔 2017-2019 smol data laptop   ~GB 5 moorings!  

Slide 5

Slide 5 text

2019- smol & big data   laptop/server/HPC/cloud obs / models  

Slide 6

Slide 6 text

= community

Slide 7

Slide 7 text

= peer - to - peer   learning

Slide 8

Slide 8 text

Cloud-native   analysis Machine learning Crunching \Bigg datasets Analysis-Ready   Cloud Optimized Data pipelines Teaching earth   system data   science Cloud analysis   platform Open source   scienti fi c   software Discourse   forum = peer - to - peer   learning Data catalogs Dask Xarray Zarr

Slide 9

Slide 9 text

Citations in blue Community in blue

Slide 10

Slide 10 text

Custom Code A common language Project-speci fi c xskillscore … … Domain - speci fi c

Slide 11

Slide 11 text

Engage

Slide 12

Slide 12 text

Custom Code A common language Project-speci fi c xskillscore … … Domain - speci fi c

Slide 13

Slide 13 text

Custom Code A common language Project-speci fi c xskillscore … … Domain - speci fi c Project-speci fi c A new kind of science discourse.pangeo.io Github Twitter A common language shared Jupyter notebooks Domain - speci fi c

Slide 14

Slide 14 text

😍

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

An a lyzing Pet a byte sc a le clim a te d a t a in your browser with P a ngeo No Supercomputer, no problem! Custom Analysis applied to each model and member

Slide 17

Slide 17 text

What we want to do

Slide 18

Slide 18 text

What we want to do 💡 Have an idea

Slide 19

Slide 19 text

What we want to do Write some code

Slide 20

Slide 20 text

What we want to do Rock some science

Slide 21

Slide 21 text

What we want to do Rock some science

Slide 22

Slide 22 text

What we have to do instead Download Files

Slide 23

Slide 23 text

What we have to do instead FTP / OPeNDAP / etc. Download Files

Slide 24

Slide 24 text

MB 😀 FTP / OPeNDAP / etc.

Slide 25

Slide 25 text

GB 😐 FTP / OPeNDAP / etc.

Slide 26

Slide 26 text

TB 😖 FTP / OPeNDAP / etc.

Slide 27

Slide 27 text

PB 😱 FTP / OPeNDAP / etc.

Slide 28

Slide 28 text

Time to science?

Slide 29

Slide 29 text

Di ff erent dimension names in the CMIP data.     Not quite analysis -ready No! Time to clean data!

Slide 30

Slide 30 text

🤔 💡 No! Time to clean data!

Slide 31

Slide 31 text

🤔 💡 Competition for brain power

Slide 32

Slide 32 text

🤔 Isn’t there a better way?

Slide 33

Slide 33 text

There is! + Analysis Ready Data in the cloud Crowd-Sourced Data Cleaning   (peer-to-peer learning)

Slide 34

Slide 34 text

There is! + Analysis Ready Data in the cloud Crowd-Sourced Data Cleaning   (peer-to-peer learning) Less data wrangling, more 💡 =

Slide 35

Slide 35 text

Demo

Slide 36

Slide 36 text

Why the cloud?

Slide 37

Slide 37 text

Why the cloud? Fast

Slide 38

Slide 38 text

Why the cloud? Fast

Slide 39

Slide 39 text

Why the cloud? Open

Slide 40

Slide 40 text

Why the cloud? Open

Slide 41

Slide 41 text

Why the cloud? Collaborative + Reproducible science

Slide 42

Slide 42 text

Why the cloud? Collaborative + Reproducible science

Slide 43

Slide 43 text

Why the cloud? Low entry barrier to try new things!

Slide 44

Slide 44 text

Why the cloud? Low entry barrier to try new things!

Slide 45

Slide 45 text

= peer - to - peer   learning Why the cloud?

Slide 46

Slide 46 text

What’s next? Join the community! discourse.pangeo.io (forum) gallery.pangeo.io (science examples) projectpythia.org (open science resources) Github (go where the code lives) Twitter (great for quick questions) Weekly meetings

Slide 47

Slide 47 text

Dont be afraid to ask and share Nobody is going to shame your code. We all started somewhere! 🤗

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Start small

Slide 50

Slide 50 text

Start small Start your next project with pangeo tools

Slide 51

Slide 51 text

Start small Start your next project with pangeo tools Stay here and and dig into some CMIP6 data if you like

Slide 52

Slide 52 text

Start small Start your next project with pangeo tools Stay here and and dig into some CMIP6 data if you like