Navigating Weather and Climate Data

Slide 1

Slide 1 text

CONFIDENTIAL — PLEASE DO NOT DISTRIBUTE EARTHMOVER.IO NYC Clim a te Week 2024 N a vig a ting We a ther a nd Clim a te D a t a

Slide 2

Slide 2 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO outline  Wh a t a re we a ther a nd clim a te d a t a used for?  Where do we a ther a nd clim a te d a t a come from? • Types of d a t a : observ a tions, models, d a t a a ssimil a tion • Popul a r d a t a providers a nd d a t a products • CMIP6 spotlight  Technology for we a ther a nd clim a te d a t a systems • P a st / present / future

Slide 3

Slide 3 text

EARTHMOVER.IO CEO, Co-Founder Community cheerle a der Open source developer X a rr a y, Z a rr, Xgcm, Rechunker, P a ngeo Forge, etc. Ex Professor Physic a l Oce a nogr a pher & Clim a te Scientist NASA SWOT Science Te a m LEAP & M2LiNES who a m I?

Slide 4

Slide 4 text

Confidential — Please do not distribute 4 E a rthmover: a public benefit corpor a tion MISSION STATEMENT To empower people to use scientific d a t a to solve hum a nity’s gre a test ch a llenges Ry a n Abern a they CO - FOUNDER & CEO Joe H a mm a n CO - FOUNDER & CTO OUR TEAM  Experienced scientists.  Open source community leaders.  Seasoned engineers who have built production-grade scientific data infrastructure at top companies. OUR PRIOR EXPERIENCE OUR OPEN - SOURCE LEADERSHIP

Slide 5

Slide 5 text

5 Credit: NASA's Goddard Space Flight Center

Slide 6

Slide 6 text

Slide 7

Slide 7 text

EARTHMOVER.IO Wh a t a re we a ther a nd clim a te d a t a used for?

Slide 8

Slide 8 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO sp a ce a nd time sc a les of E a rth System phenomen a torn a does severe storms hurric a nes extr a tropic a l cyclones se a sons ENSO clim a te ch a nge Ad a pted from T a v a kolif a r et a l. 2017 https://doi.org/10.2166/wcc.2017.107 M IN U TES H O U R S D AYS W EEKS M O N TH S YEA R S D EC A D ES C EN TU R IES 1 KM 10 KM 100 KM 1000 KM 10,000 KM S EC O N D S GLOBAL SMALL BIG FAST SLOW m a rine he a tw a ves wild fi re drought flood

Slide 9

Slide 9 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO sp a ce a nd time sc a les torn a does severe storms hurric a nes extr a tropic a l cyclones se a sons ENSO clim a te ch a nge Ad a pted from T a v a kolif a r et a l. 2017 https://doi.org/10.2166/wcc.2017.107 M IN U TES H O U R S D AYS W EEKS M O N TH S YEA R S D EC A D ES C EN TU R IES 1 KM 10 KM 100 KM 1000 KM 10,000 KM S EC O N D S GLOBAL SMALL BIG FAST SLOW m a rine he a tw a ves wild fi re drought flood high-frequency tr a ders long-term a sset m a n a gers d a y tr a ders stocks, bonds, commodities, investment b a nks, priv a te equity, VC fin a nci a l sector

Slide 10

Slide 10 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO whose bottom line is imp a cted by we a ther a nd clim a te phenomen a ?  Agriculture  Forestry  Energy • Fossil Fuels (dem a nd) • Renew a bles (supply + dem a nd)  Tr a nsport a tion  He a lth  Ret a il  Hospit a lity  Construction  Re a l Est a te  Fin a nci a ls  Insur a nce  C a rbon m a rkets

Slide 11

Slide 11 text

EARTHMOVER.IO © 2025 EARTHMOVER PBC the d a t a v a lue ch a in  Body level one • Body level two 🛰 d a t a providers Gener a te r a w d a t a sets using direct observ a tions or simul a tion c a p a bilities. ex a mples: 🏭 d a t a refiners Process r a w d a t a to produce v a lue- a dded d a t a sets using modeling, st a tistics, AI / ML. ex a mples: ⛏ d a t a end users Comp a nies / institutions with a ctu a l a ssets a t risk to environment a l f a ctors. ex a mples:

Slide 12

Slide 12 text

EARTHMOVER.IO Where do we a ther a nd clim a te come from?

Slide 13

Slide 13 text

p a st present future wh a t h a ppened? wh a t’s h a ppening now? wh a t will h a ppen?

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO different types of s a tellite d a t a → “simple” s a tellite im a gery → a dv a nced remote sensing of physic a l v a ri a bles SWOT S a tellite Im a ge Credit: NASA, CNES Arctic Wildfires LANDSAT S a tellite Im a ge Credit: NASA E a rth Observ a tory

Slide 18

Slide 18 text

EARTHMOVER.IO 😱 But I just w a nt to know the wind speed in Chic a go on M a y 12, 1985?

Slide 19

Slide 19 text

EARTHMOVER.IO 😱 But I just w a nt to know the wind speed in Chic a go on M a y 12, 1985? Models to the rescue!

Slide 20

Slide 20 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO E a rth-system models Kot a m a rthi, R a o, et a l. “Glob a l Clim a te Models.” Downsc a ling Techniques for High-Resolution Clim a te Projections: From Glob a l Ch a nge to Loc a l Imp a cts. C a mbridge: C a mbridge University Press, 2021. 19 – 39. Print.  Equ a tions of physics  Numeric a l discretiz a tion of the E a rth (i.e. a “grid”)  Millions of lines of FORTRAN code

Slide 21

Slide 21 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO E a rth-system models → resolution a nd comput a tion a l cost IPCC AR4 WG1 Figure 1.4 1990 1996 2001 2007 Frontier Ex a sc a le Supercomputer. Credit OLCF Incre a sing Comput a tion a l Cost Incre a sing D a t a Volume

Slide 22

Slide 22 text

CONFIDENTIAL — PLEASE DO NOT DISTRIBUTE EARTHMOVER.IO CM2.6 Clim a te Model Credit: NOAA GFDL, Julius Busecke

Slide 23

Slide 23 text

Slide 24

Slide 24 text

EARTHMOVER.IO E a rth system d a t a a ssimil a tion All a v a il a ble observ a tions Physics Simul a tion forec a st a n a lysis re a n a lysis

Slide 25

Slide 25 text

CONFIDENTIAL — PLEASE DO NOT DISTRIBUTE EARTHMOVER.IO Credit NASA’s Glob a l Modeling a nd Assimil a tion O ff ice

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO popul a r forec a st d a t a products  Europe a n Center for Medium R a nge We a ther Forec a sting (ECMWF) • HRES - High-resolution 10-d a y forec a st • ENS - Ensemble 15-d a y forec a st • SEAS - Se a son a l Forec a st  US N a tion a l Oce a nogr a phic a nd Atmospheric Administr a tion (NOAA) • GFS - Glob a l Forec a st System • GEFS - Glob a l Ensemble Forec a st System • HRRR - High Resolution R a pid Refresh

Slide 29

Slide 29 text

Slide 30

Slide 30 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO re a n a lysis → consistent view of long-term historic a l record  C a reful qu a lity control of observ a tions  Physic a lly consistent, g a p-free, jump-free record  H a rmonized sp a tiotempor a l grid  Suit a ble for long-term clim a te studies Credit: ECMWF

Slide 31

Slide 31 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO popul a r re a n a lysis d a t a products  ECMWF ERA5  NASA MERRA2  NCEP North Americ a n Region a l Re a n a lysis  JRA - 55: J a p a nese 55-ye a r Re a n a lysis, Ne a r Re a l-Time D a t a  ECCO: Estim a ting the Circul a tion a nd Clim a te of the Oce a n  … a nd m a ny m a ny more! Active rese a rch topic

Slide 32

Slide 32 text

Slide 33

Slide 33 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO clim a te projections structur a l uncert a inty (model in a ccur a cies) forcing uncert a inty (emissions p a thw a y) Intern a l v a ri a bility Credit: IPCC AR4 Fig. 11.8

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP: Coupled Model Intercomp a rison Project  Intern a tion a l e ff ort by WCRP to coordin a te clim a te modeling e ff orts  Pre-de fi ned emissions scen a rios a nd experiment protocols  E a ch modeling center runs their own models  D a t a st a nd a rdiz a tion  Beg a n in 1995  CMIP6 out; CMIP7 beginning

Slide 37

Slide 37 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a  M a ny 100,000s of individu a l d a t a sets  E a ch d a t a set is identi fi ed by a unique id, consisting of 'f a cets'  F a cets a re p a rt of CMIP controlled voc a bul a ry (https://wcrp-cmip.github.io/CMIP6_CVs/) https://wcrp-cmip.org/cmip-data-access/#access-routes CMIP Cycle MIP activity Modelling Center Model Code Experiment/forcing scenarios Ensemble member Output Variable Model Grid CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-CM4.ssp585.r1i1p1f1.Omon.thetao.gn mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version di ff erent simulations components of a single simulation example MIP table used

Slide 38

Slide 38 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a → institution https://wcrp-cmip.org/cmip-data-access/#access-routes CMIP Cycle MIP activity Modelling Center Model Code Experiment/forcing scenarios Ensemble member Output Variable Model Grid CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-CM4.ssp585.r1i1p1f1.Omon.thetao.gn mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version di ff erent simulations components of a single simulation example MIP table used https://wcrp-cmip.org/map/#map_of_modelling_centres_and_esgf_nodes

Slide 39

Slide 39 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a → source https://wcrp-cmip.org/cmip-data-access/#access-routes CMIP Cycle MIP activity Modelling Center Model Code Experiment/forcing scenarios Ensemble member Output Variable Model Grid CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-CM4.ssp585.r1i1p1f1.Omon.thetao.gn mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version di ff erent simulations components of a single simulation example MIP table used https://wcrp-cmip.org/map/#map_of_modelling_centres_and_esgf_nodes

Slide 40

Slide 40 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a https://wcrp-cmip.org/cmip-data-access/#access-routes CMIP Cycle MIP activity Modelling Center Model Code Experiment/forcing scenarios Ensemble member Output Variable Model Grid CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-CM4.ssp585.r1i1p1f1.Omon.thetao.gn mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version di ff erent simulations components of a single simulation example MIP table used → experiment

Slide 41

Slide 41 text

Slide 42

Slide 42 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a  V a ri a ble n a mes a re st a nd a rdized using the CMOR (https://cmor.llnl.gov/) libr a ry. https://expearth.uib.no/?page_id=28 CMIP Cycle MIP activity Modelling Center Model Code Experiment/forcing scenarios Ensemble member Output Variable Model Grid CMIP6.ScenarioMIP.NOAA-GFDL.GFDL-CM4.ssp585.r1i1p1f1.Omon.thetao.gn mip_era.activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label.version di ff erent simulations components of a single simulation example MIP table used → d a t a set

Slide 43

Slide 43 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO E a rth System Grid Feder a tion → CMIP D a t a Distribution System  Feder a tion of n a tion a l l a bs a nd public-sector a gencies  Peer-to-peer d a t a sh a ring system b a sed on NetCDF fi les  D a t a c a t a log a nd se a rch c a p a bilities

Slide 44

Slide 44 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO CMIP d a t a : Ch a llenges  D a t a a rchive is 20 PB!  D a t a must be downlo a ded to loc a l stor a ge a nd org a nized in order to be used for a n a lysis.  This cre a tes a l a rge overhe a d for working with CMIP d a t a  Individu a l a ccess/d a t a cle a ning a ppro a ches might be incomp a tible, hindering reus a bility/reproducibility ESGF Custom Code Custom Code Custom Code University Lab Industry ❌ ✋🚫

Slide 45

Slide 45 text

Slide 46

Slide 46 text

EARTHMOVER.IO We a ther a nd Clim a te D a t a Systems

Slide 47

Slide 47 text

EARTHMOVER.IO 🧨 E a rth system d a t a volumes a re exploding

Slide 48

Slide 48 text

E a rth System d a t a workflows a re p a inful → the st a tus quo MY SERVER

Slide 49

Slide 49 text

E a rth System d a t a workflows a re p a inful 1 - DATA INGESTION Downlo a d a bunch of files from d a t a providers in form a ts like Grib, NetCDF, HDF. ☹ DATA PROVIDER → the st a tus quo

Slide 50

Slide 50 text

E a rth System d a t a workflows a re p a inful 1 - DATA INGESTION Downlo a d a bunch of files from d a t a providers in form a ts like Grib, NetCDF, HDF. ☹ DATA PROVIDER → the st a tus quo 2 - DATA CLEANING Wr a ngle downlo a ded files into some sort of org a nize structure. H a rmonize inconsistencies a cross files. 😫

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Wh a t if we could stop thinking a bout files? Wh a t if we could query E a rth System d a t a like a d a t a b a se?

Slide 54

Slide 54 text

Confidential — Please do not distribute 54 E a rth system d a t a a re multidimension a l, not t a bul a r ch a llenge D a t a systems designed for typic a l business d a t a (t a bles / d a t a fr a mes) don’t work for multidimension a l d a t a .

Slide 55

Slide 55 text

Slide 56

Slide 56 text

© 2025 EARTHMOVER PBC EARTHMOVER.IO E a rthmover’s Solution → The Cloud Pl a tform for We a ther a nd Clim a te D a t a connectors for common we a ther a nd clim a te d a t a sources a nd form a ts ingestion engine query engine stor a ge engine e ff icient, perform a nt cloud n a tive stor a ge a nd c a t a log integr a tions with popul a r d a t a science a nd GIS tools public cloud infr a structure

Slide 57

Slide 57 text

CONFIDENTIAL — PLEASE DO NOT DISTRIBUTE EARTHMOVER.IO

Slide 58

Slide 58 text

AMS 2024 PRESENTATION EARTHMOVER.IO who is Arr a yl a ke for?  Comp a nies l a rge a nd sm a ll System of record for your business-critic a l we a ther, clim a te a nd geosp a ti a l d a t a .  Public sector Use Arr a yl a ke to dissemin a te a n a lysis-re a dy, cloud-optimized d a t a to your t a rget users.  Ac a demi a Build a d a t a l a ke to support your dep a rtment’s rese a rch a nd educ a tion.

Slide 59

Slide 59 text

pl a net a ry-sc a le d a t a feder a tion in the cloud vision Se a mless exch a nge of d a t a a cross a c a demi a , government, a nd industry will empower reproducible open science a nd a cceler a te clim a te ch a nge solutions. University L a b AI St a rtup Big Tech Co. Gov’t Agency NGO