Earthcube 2020 - CMIP6 without the interpolation: Grid-native analysis with Pangeo in the cloud - Speaker Deck

Tweet

Tweet

Slide 1

Slide 1 text

CMIP6 WITHOUT THE INTERPOLATION GRID-NATIVE ANALYSIS WITH PANGEO IN THE CLOUD JULIUS BUSECKE | RYAN ABERNATHEY

Slide 2

Slide 2 text

EYRING ET AL., BOM 2003. … Each “MIP” consists of many numerical models

Slide 3

Slide 3 text

SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰

Slide 4

Slide 4 text

SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset (~600TB)

Slide 5

Slide 5 text

SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset (~600TB)

Slide 6

Slide 6 text

CURVILINEAR GRIDS DELANDMETER AND VAN SEBILLE, 2019 X0 , Y0 X1 , Y1 X2 , Y2 X3 , Y3 u0 u1 v0 v1 T0 F0 F1 F2 F3 x,y x y ⇠ ⌘ 0, 0 1, 0 1, 1 0, 1 F0 F1 F2 F3 U0 U1 V0 V1 T0 ⇠,⌘ (a) (b) ENES.ORG

Slide 7

Slide 7 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 8

Slide 8 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 9

Slide 9 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 10

Slide 10 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 11

Slide 11 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 12

Slide 12 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X Not too complicated but tedious and error prone

Slide 13

Slide 13 text

COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 14

Slide 14 text

COMPUTING A DERIVATIVE XGCM Starting from xarray dataset DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 15

Slide 15 text

COMPUTING A DERIVATIVE XGCM Starting from xarray dataset Create a `grid` object DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X

Slide 16

Slide 16 text

DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X COMPUTING A DERIVATIVE XGCM

Slide 17

Slide 17 text

DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X COMPUTING A DERIVATIVE XGCM Currently supported operations: diﬀerence interpolation cumulative sum min/max average integral cumulative integral derivative

Slide 18

Slide 18 text

CMIP6_PREPROCESSING

Slide 19

Slide 19 text

CMIP6_PREPROCESSING consistent renaming

Slide 20

Slide 20 text

CMIP6_PREPROCESSING consistent renaming

Slide 21

Slide 21 text

CMIP6_PREPROCESSING consistent renaming

Slide 22

Slide 22 text

CMIP6_PREPROCESSING consistent renaming

Slide 23

Slide 23 text

CMIP6_PREPROCESSING consistent renaming homogenize units

Slide 24

Slide 24 text

CMIP6_PREPROCESSING consistent renaming homogenize units consistent longitude convention

Slide 25

Slide 25 text

CMIP6_PREPROCESSING consistent renaming homogenize units xgcm grids consistent longitude convention

Slide 26

Slide 26 text

Sea Surface Temperature Gradient Magnitude

Slide 27

Slide 27 text

Sea Surface Temperature Gradient Magnitude

Slide 28

Slide 28 text

SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset

Slide 29

Slide 29 text

SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset

Slide 30

Slide 30 text

SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset cmip6_preprocessing

Slide 31

Slide 31 text

SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset cmip6_preprocessing ❤

Slide 32

Slide 32 text

@JULIUSBUSECKE JBUSECKE@PRINCETON.EDU INTERESTED? JOIN US ON GITHUB! HTTPS://XGCM.READTHEDOCS.IO/ HTTPS://GITHUB.COM/JBUSECKE/CMIP6_PREPROCESSING HTTPS://GITHUB.COM/XGCM/XGCM Check out the full tutorial in the Earthcube notebook: HTTPS://GITHUB.COM/EARTHCUBE2020/EC20_BUSECKE_ETAL