Slide 1

Slide 1 text

A Unified Framework for Managing and Accessing Forecast Model Data Rich Signell , USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA NOAA EDM Meeting, Washington, DC 2016-01-04

Slide 2

Slide 2 text

IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers 2 Customer Web access service Data Provider Observations Models

Slide 3

Slide 3 text

Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

Slide 4

Slide 4 text

NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)

Slide 5

Slide 5 text

Staggered Grid Models Ocean: ROMS HYCOM Delft3D POM ECOM Hydro: MODFLOW Atmos: WRF

Slide 6

Slide 6 text

Unstructured Grid models (tri, hex…) Ocean: ADCIRC FVCOM SELFE SCHISM Hydro: MODFLOW- USG Atmos: NCAR-MPAS OLAM UZIM

Slide 7

Slide 7 text

SGRID and UGRID conventions • Similar design patterns, building on CF, spearheaded by Bert Jagers at Deltares (Netherlands) with input from the community • SGRID: https://github.com/sgrid • UGRID: https://github.com/ugrid- conventions/ugrid-conventions

Slide 8

Slide 8 text

ROMS SGRID Example

Slide 9

Slide 9 text

IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP NetCDF Subset THREDDS Data Server Standardized (CF-1.6, SGRID-0.1, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python EDC NetCDF-Java SOS Geoportal Server GeoNetwork CKAN Observed data (buoy, gauge, ADCP, glider) Web Portals pycsw NcML Grid TimeSeries Profile Trajectory TimeSeriesProfile Sgrid Ugrid Nonstandard Data Files Catalog Services Rectilinear ERDDAP WCS

Slide 10

Slide 10 text

Catalog Search 10

Slide 11

Slide 11 text

Python packages for UGRID, SGRID 11 • pyugrid: allows model independent access to UGRID-compliant data in python (https://github.com/pyugrid) • pysgrid: allows model independent access to SGRID-compliant data in python (https://github.com/sgrid/pysgrid)

Slide 12

Slide 12 text

sci-wms 12 Scientific Web Map Service for Staggered and Unstructured Grids Capability: OGC Web map service for staggered and unstructured (e.g. triangular) grid models. Significance: Most models in IOOS (FVCOM, SELFE, ADCIRC, ROMS) are either unstructured or staggered. This is the first web map service that works with these models. Product: Freely-available , open-source python code at http://github.com/sci-wms. Applications: Deployed at IOOS COMT, IOOS Regional Associations (NERACOOS, SECOORA, MARACOOS, GLOS), USGS Coast and Marine Geology Portal, Hurricane Sandy Portal sci-wms map of GLOS unstructured surface currents for Lake St. Clair

Slide 13

Slide 13 text

Interoperable Access in Python (Iris)

Slide 14

Slide 14 text

IOOS System Test

Slide 15

Slide 15 text

Workflow for the USGS CMG Portal 15

Slide 16

Slide 16 text

Modeling Group System Administrator – Installs TDS, sci-wms and pycsw using Docker (https://hub.docker.com/u/axiom/) – Sets up a TDS datasetScan that includes NcML – Runs a script that periodically: • crawls thredds catalogs for NcML files, retrieving ISO metadata • injects UGRID and SGRID compliant data into sci- wms • writes modified ISO metadata with WMS endpoint • refreshes the pycsw database 16 Workflow (1/3)

Slide 17

Slide 17 text

Workflow (2/3) Modeler – Puts native model output files into a directory under the TDS datasetScan – Creates NcML to aggregate and CF, UGRID or SGRID-ize the data (using a template or YAML2NcML script) – Adds “CMG_Portal” to the “project” attribute if they want the dataset to show up in the portal – If necessary, edits the list of crawled catalogs on github and submits a PR 17

Slide 18

Slide 18 text

Workflow (3/3) Axiom Data Science – Runs a CSW search (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” – Datasets that have valid WMS services are added to the portal See for details of the workflow 18

Slide 19

Slide 19 text

19

Slide 20

Slide 20 text

WMS-driven Model Viewing Portal

Slide 21

Slide 21 text

Search and access at data.gov 21 Catalog services can be federated via OGC CSW (Catalog Service for the Web)

Slide 22

Slide 22 text

Displaying WMS in TerriaJS 22

Slide 23

Slide 23 text

Summary • We now have standardized web services for searching, accessing data, and displaying map images from all common ocean models (including all NOAA ocean forecast models) • Modeling groups can enable these services using free software: TDS, sci-wms, pycsw and automate management with some simple scripts

Slide 24

Slide 24 text

Priorities for Future Work • Establish formal linkage between CF, UGRID and SGRID • Make the framework more robust: improve testing, documentation, training • Improve performance of WMS services for SGRID and UGRID data • Investigate TerriaJS as a common infrastructure for model product browsing via standard services • Add UGRID and SGRID support to netcdf-java