A Unified Framework for Managing and Accessing Forecast Model Data

Talk to the NOAA Environmental Data Management meeting in DC

Transcript

1. A Unified Framework for Managing and Accessing Forecast Model Data Rich Signell , USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA NOAA EDM Meeting, Washington, DC 2016-01-04
2. IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers 2 Customer Web access service Data Provider Observations Models
3. Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
4. NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)
5. Staggered Grid Models Ocean: ROMS HYCOM Delft3D POM ECOM Hydro: MODFLOW Atmos: WRF
6. Unstructured Grid models (tri, hex…) Ocean: ADCIRC FVCOM SELFE SCHISM Hydro: MODFLOW- USG Atmos: NCAR-MPAS OLAM UZIM
7. SGRID and UGRID conventions • Similar design patterns, building on CF, spearheaded by Bert Jagers at Deltares (Netherlands) with input from the community • SGRID: https://github.com/sgrid • UGRID: https://github.com/ugrid- conventions/ugrid-conventions
8. ROMS SGRID Example
9. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP NetCDF Subset THREDDS Data Server Standardized (CF-1.6, SGRID-0.1, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python EDC NetCDF-Java SOS Geoportal Server GeoNetwork CKAN Observed data (buoy, gauge, ADCP, glider) Web Portals pycsw NcML Grid TimeSeries Profile Trajectory TimeSeriesProfile Sgrid Ugrid Nonstandard Data Files Catalog Services Rectilinear ERDDAP WCS
10. Catalog Search 10
11. Python packages for UGRID, SGRID 11 • pyugrid: allows model independent access to UGRID-compliant data in python (https://github.com/pyugrid) • pysgrid: allows model independent access to SGRID-compliant data in python (https://github.com/sgrid/pysgrid)
12. sci-wms 12 Scientific Web Map Service for Staggered and Unstructured Grids Capability: OGC Web map service for staggered and unstructured (e.g. triangular) grid models. Significance: Most models in IOOS (FVCOM, SELFE, ADCIRC, ROMS) are either unstructured or staggered. This is the first web map service that works with these models. Product: Freely-available , open-source python code at http://github.com/sci-wms. Applications: Deployed at IOOS COMT, IOOS Regional Associations (NERACOOS, SECOORA, MARACOOS, GLOS), USGS Coast and Marine Geology Portal, Hurricane Sandy Portal sci-wms map of GLOS unstructured surface currents for Lake St. Clair
13. Interoperable Access in Python (Iris)
14. IOOS System Test
15. Workflow for the USGS CMG Portal 15
16. Modeling Group System Administrator – Installs TDS, sci-wms and pycsw using Docker (https://hub.docker.com/u/axiom/) – Sets up a TDS datasetScan that includes NcML – Runs a script that periodically: • crawls thredds catalogs for NcML files, retrieving ISO metadata • injects UGRID and SGRID compliant data into sci- wms • writes modified ISO metadata with WMS endpoint • refreshes the pycsw database 16 Workflow (1/3)
17. Workflow (2/3) Modeler – Puts native model output files into a directory under the TDS datasetScan – Creates NcML to aggregate and CF, UGRID or SGRID-ize the data (using a template or YAML2NcML script) – Adds “CMG_Portal” to the “project” attribute if they want the dataset to show up in the portal – If necessary, edits the list of crawled catalogs on github and submits a PR 17
18. Workflow (3/3) Axiom Data Science – Runs a CSW search (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” – Datasets that have valid WMS services are added to the portal See <https://github.com/USGS-CMG/usgs-cmg- portal/wiki> for details of the workflow 18
20. WMS-driven Model Viewing Portal
21. Search and access at data.gov 21 Catalog services can be federated via OGC CSW (Catalog Service for the Web)
22. Displaying WMS in TerriaJS 22
23. Summary • We now have standardized web services for searching, accessing data, and displaying map images from all common ocean models (including all NOAA ocean forecast models) • Modeling groups can enable these services using free software: TDS, sci-wms, pycsw and automate management with some simple scripts
24. Priorities for Future Work • Establish formal linkage between CF, UGRID and SGRID • Make the framework more robust: improve testing, documentation, training • Improve performance of WMS services for SGRID and UGRID data • Investigate TerriaJS as a common infrastructure for model product browsing via standard services • Add UGRID and SGRID support to netcdf-java