A Standardized Framework for Reproducible Model Skill Assessment

Slide 1

Slide 1 text

A Standardized Framework for Reproducible Model Skill Assessment Rich Signell (USGS, Woods Hole, MA) Filipe Fernandes (SECOORA) Kyle Wilcox (Axiom Data Science, Wickford, RI) 14th Conference on Estuarine and Coastal Modeling

Slide 2

Slide 2 text

US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations

Slide 3

Slide 3 text

IOOS Model Data Interoperability Design ROMS ADCIRC HYCOM SELFE SLOSH NcML NcML NcML NcML Common Data Model OPeNDAP+CF +UGRID|SGRID WCS NetCDF Subset THREDDS Data Server (TDS) Standardized (CF-1.6, UGRID-1.0, SGRID-0.3) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 pycsw-CKAN NcML Grid Sgrid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services CMG Portal Delft3D NcML sudo apt-get docker docker run –d axiom/docker-thredds

Slide 4

Slide 4 text

Catalog Search 4

Slide 5

Slide 5 text

NCTOOLBOX: test_cf_ugrid3.m

Slide 6

Slide 6 text

Skidaway “modena” glider: temperature

Slide 7

Slide 7 text

Interoperable Access in Python (Iris)

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Automated model comparison

Slide 11

Slide 11 text

2015 Boston Light Swim 2015 Aug 15, 7:00 am start 8 mile swim No wet suit How cold will the water be?

Slide 12

Slide 12 text

NECOFS Massbay Forecast

Slide 13

Slide 13 text

Reproducible Jupyter Notebook Go to: https://github.com/ioos/notebook_demos/boston_light_swim

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Final Result

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Forecasts Sent to Swimmers

Slide 21

Slide 21 text

Run it yourself

Slide 22

Slide 22 text

Summary • Standardized framework makes skill assessment easy and powerful • Services are now easy to install and maintain • Skill assessment notebooks are reproducible by others (for free) • More assessment leads to more appropriate use of modeling products • More assessment of models leads to better models