Reducing Data Friction in Ocean Science

Talk at WHOI's Coastal Ocean and Fluid Dynamics Lab (COFDL), December 2, 2016

Transcript

1. On Reducing Data Friction in Ocean Science Rich Signell (USGS, Woods Hole, MA) Filipe Fernandes (SECOORA) Kyle Wilcox (Axiom Data Science, Wickford, RI) COFDL, Dec 2, 2016
2. The Fourth Paradigm (2009)
4. Jim Gray: “The suggestion that I have been making is that we now have terrible data management tools for most of the science disciplines. Commercial organizations like Walmart can afford to build their own data management software, but in science we do not have that luxury. … The funding agencies in the U.S. and elsewhere need to do a lot more to foster the building of tools to make scientists more productive.”
6. Mark Abbott: • “The architecture for data-intensive computing should be based on storage, computing and presentation services at every node of an interconnected network. Providing standard, extensible frameworks that accommodate innovation at the network edges should enable these knowledge “ecosystems” to form and evolve as the needs of science and policy changes.”
8. Wilbanks: The Fourth Network Layer • “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” • “We need open standards… and above all, we need to teach scientists to work in this new layer of data” From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” 4. Data 3. Web 2. TCP/IP 1. Ethernet
9. US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations
10. IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers Customer Web access service Data Provider Observations Models
11. IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
12. OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena
13. NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal OGC: Open Geospatial Consortium IOOS: Integrated Ocean Observing System NCEI: National Centers for Environmental Information
14. Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
15. Time Series, Trajectories, Profiles Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
19. IOOS Model Data Interoperability Design ROMS ADCIRC HYCOM SELFE SLOSH NcML NcML NcML NcML Common Data Model OPeNDAP+CF +UGRID|SGRID WCS NetCDF Subset THREDDS Data Server (TDS) Standardized (CF-1.6, UGRID-1.0, SGRID-0.3) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 pycsw-CKAN NcML Grid Sgrid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services CMG Portal Delft3D NcML sudo apt-get docker docker run –d axiom/docker-thredds
20. Catalog Search 23
21. NCTOOLBOX: test_cf_ugrid3.m
22. Skidaway “modena” glider: temperature
24. Interoperable Access in Python (Iris)
25. 2015 Boston Light Swim 2015 Aug 15, 7:00 am start 8 mile swim No wet suit How cold will the water be?
26. NECOFS Massbay Forecast
27. Reproducible Jupyter Notebook Go to: https://github.com/ioos/notebook_demos/boston_light_swim
31. Final Result
34. Forecasts Sent to Swimmers
35. Run it yourself
38. Demo time?
39. Sensor Map on IOOS.US
40. ERDDAP access from Sensor Map
41. ERDDAP access from Python
42. NERACOOS ERDDAP
46. Summary • A standardized framework helps build the Fourth Network Layer, reducing data friction • Standards exist for: models, remotely-sensed data, points, time series, ADCP, trajectories (glider, AUV) • Science notebooks can be dynamic and reusable • Easy to install standard services or just pass to IOOS or NCEI • Your data should live here