Catalog-driven workflows using CSW

Presentation at ESIP Winter Meeting

Transcript

1. Catalog-driven workflows using CSW Rich Signell , USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA ESIP Winter Meeting, Washington, DC 2016-01-08
2. The 4th Network Layer: Data • “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” • “We need open standards… and above all, we need to teach scientists to work in this new layer of data” 2 From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” Data Web TCP/IP Ethernet
3. US Integrated Ocean Observing System (IOOS® ) • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations
4. IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers 4 Customer Web access service Data Provider Observations Models
5. Numerical model Output
6. Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
7. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP NetCDF Subset THREDDS Data Server Standardized (CF-1.6, SGRID-0.1, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python EDC NetCDF-Java SOS Geoportal Server GeoNetwork CKAN Observed data (buoy, gauge, ADCP, glider) Web Portals pycsw NcML Grid TimeSeries Profile Trajectory TimeSeriesProfile Sgrid Ugrid Nonstandard Data Files Catalog Services Rectilinear ERDDAP WCS
8. Catalog Search 8
9. Interoperable Access in Python (Iris)
10. IOOS System Test
11. 2015 Boston Light Swim 2015 Aug 15, 7:00 am start 8 mile swim No wet suit How cold will the water be?
12. NECOFS Massbay Forecast
13. Reproducible Jupyter Notebook Go to https://github.com/ocefpaf/boston_light_swim, click on “launch binder” to run on cloud
17. Final Result
20. pycsw 20
21. Workflow for the USGS CMG Portal 21
22. Workflow (3/3) Axiom Data Science – Runs a CSW search (in a cron job) on the modeling groups pycsw services, filtering on datasets that contain a project called “CMG_Portal” – Datasets that have valid WMS services are added to the portal See <https://github.com/USGS-CMG/usgs-cmg- portal/wiki> for details of the workflow 22
24. WMS-driven Model Viewing Portal
26. Interoperable access in Matlab (nctoolbox)
29. Catalog-driven dynamic portals 29
31. Benefits of catalog-driven applications • Dynamically adapt to new or changing data • Find the machine-to-machine issues – Easy problems that can be fixed in minutes to day – Harder problems to guide future work • Fixes for your workflow benefit everyone • Build success stories • Create reproducible workflows that others can learn from, expand on, or transform • Standardized workflows help develop the 4th network layer for data