Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Unified Framework for Managing and Accessing Forecast Model Data

Rich Signell
January 04, 2016

A Unified Framework for Managing and Accessing Forecast Model Data

Talk to the NOAA Environmental Data Management meeting in DC

Rich Signell

January 04, 2016
Tweet

More Decks by Rich Signell

Other Decks in Science

Transcript

  1. A Unified Framework for Managing and Accessing Forecast Model Data

    Rich Signell , USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA NOAA EDM Meeting, Washington, DC 2016-01-04
  2. IOOS Core Principles • Adopt open standards & practices •

    Avoid customer-specific stovepipes • Standardized access services implemented at data providers 2 Customer Web access service Data Provider Observations Models
  3. Ocean grids are often not regularly spaced! Stretched surface and

    terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
  4. NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP:

    Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)
  5. Unstructured Grid models (tri, hex…) Ocean: ADCIRC FVCOM SELFE SCHISM

    Hydro: MODFLOW- USG Atmos: NCAR-MPAS OLAM UZIM
  6. SGRID and UGRID conventions • Similar design patterns, building on

    CF, spearheaded by Bert Jagers at Deltares (Netherlands) with input from the community • SGRID: https://github.com/sgrid • UGRID: https://github.com/ugrid- conventions/ugrid-conventions
  7. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML

    NcML NcML NcML NcML Common Data Model OPeNDAP NetCDF Subset THREDDS Data Server Standardized (CF-1.6, SGRID-0.1, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python EDC NetCDF-Java SOS Geoportal Server GeoNetwork CKAN Observed data (buoy, gauge, ADCP, glider) Web Portals pycsw NcML Grid TimeSeries Profile Trajectory TimeSeriesProfile Sgrid Ugrid Nonstandard Data Files Catalog Services Rectilinear ERDDAP WCS
  8. Python packages for UGRID, SGRID 11 • pyugrid: allows model

    independent access to UGRID-compliant data in python (https://github.com/pyugrid) • pysgrid: allows model independent access to SGRID-compliant data in python (https://github.com/sgrid/pysgrid)
  9. sci-wms 12 Scientific Web Map Service for Staggered and Unstructured

    Grids Capability: OGC Web map service for staggered and unstructured (e.g. triangular) grid models. Significance: Most models in IOOS (FVCOM, SELFE, ADCIRC, ROMS) are either unstructured or staggered. This is the first web map service that works with these models. Product: Freely-available , open-source python code at http://github.com/sci-wms. Applications: Deployed at IOOS COMT, IOOS Regional Associations (NERACOOS, SECOORA, MARACOOS, GLOS), USGS Coast and Marine Geology Portal, Hurricane Sandy Portal sci-wms map of GLOS unstructured surface currents for Lake St. Clair
  10. Modeling Group System Administrator – Installs TDS, sci-wms and pycsw

    using Docker (https://hub.docker.com/u/axiom/) – Sets up a TDS datasetScan that includes NcML – Runs a script that periodically: • crawls thredds catalogs for NcML files, retrieving ISO metadata • injects UGRID and SGRID compliant data into sci- wms • writes modified ISO metadata with WMS endpoint • refreshes the pycsw database 16 Workflow (1/3)
  11. Workflow (2/3) Modeler – Puts native model output files into

    a directory under the TDS datasetScan – Creates NcML to aggregate and CF, UGRID or SGRID-ize the data (using a template or YAML2NcML script) – Adds “CMG_Portal” to the “project” attribute if they want the dataset to show up in the portal – If necessary, edits the list of crawled catalogs on github and submits a PR 17
  12. Workflow (3/3) Axiom Data Science – Runs a CSW search

    (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” – Datasets that have valid WMS services are added to the portal See <https://github.com/USGS-CMG/usgs-cmg- portal/wiki> for details of the workflow 18
  13. 19

  14. Search and access at data.gov 21 Catalog services can be

    federated via OGC CSW (Catalog Service for the Web)
  15. Summary • We now have standardized web services for searching,

    accessing data, and displaying map images from all common ocean models (including all NOAA ocean forecast models) • Modeling groups can enable these services using free software: TDS, sci-wms, pycsw and automate management with some simple scripts
  16. Priorities for Future Work • Establish formal linkage between CF,

    UGRID and SGRID • Make the framework more robust: improve testing, documentation, training • Improve performance of WMS services for SGRID and UGRID data • Investigate TerriaJS as a common infrastructure for model product browsing via standard services • Add UGRID and SGRID support to netcdf-java