Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reducing Data Friction in Ocean Science

Cdbcc920e73869b6436479419b3a1841?s=47 Rich Signell
December 02, 2016

Reducing Data Friction in Ocean Science

Talk at WHOI's Coastal Ocean and Fluid Dynamics Lab (COFDL), December 2, 2016


Rich Signell

December 02, 2016

More Decks by Rich Signell

Other Decks in Science


  1. On Reducing Data Friction in Ocean Science Rich Signell (USGS,

    Woods Hole, MA) Filipe Fernandes (SECOORA) Kyle Wilcox (Axiom Data Science, Wickford, RI) COFDL, Dec 2, 2016
  2. The Fourth Paradigm (2009)

  3. None
  4. Jim Gray: “The suggestion that I have been making is

    that we now have terrible data management tools for most of the science disciplines. Commercial organizations like Walmart can afford to build their own data management software, but in science we do not have that luxury. … The funding agencies in the U.S. and elsewhere need to do a lot more to foster the building of tools to make scientists more productive.”
  5. None
  6. Mark Abbott: • “The architecture for data-intensive computing should be

    based on storage, computing and presentation services at every node of an interconnected network. Providing standard, extensible frameworks that accommodate innovation at the network edges should enable these knowledge “ecosystems” to form and evolve as the needs of science and policy changes.”
  7. None
  8. Wilbanks: The Fourth Network Layer • “We need an end-to-end,

    layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” • “We need open standards… and above all, we need to teach scientists to work in this new layer of data” From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” 4. Data 3. Web 2. TCP/IP 1. Ethernet
  9. US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines:

    • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations
  10. IOOS Core Principles • Adopt open standards & practices •

    Avoid customer-specific stovepipes • Standardized access services implemented at data providers Customer Web access service Data Provider Observations Models
  11. IOOS Recommended Web Services and Data Encodings In-situ data (buoys,

    piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
  12. OGC Sensor Observation Service (SOS) • Provides standard access to

    sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena
  13. NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP:

    Global Organization for Earth System Science Portal OGC: Open Geospatial Consortium IOOS: Integrated Ocean Observing System NCEI: National Centers for Environmental Information
  14. Ocean grids are often not regularly spaced! Stretched surface and

    terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
  15. Time Series, Trajectories, Profiles Meteorology and Wave Buoy in the

    Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
  16. None
  17. None
  18. None
  19. IOOS Model Data Interoperability Design ROMS ADCIRC HYCOM SELFE SLOSH

    NcML NcML NcML NcML Common Data Model OPeNDAP+CF +UGRID|SGRID WCS NetCDF Subset THREDDS Data Server (TDS) Standardized (CF-1.6, UGRID-1.0, SGRID-0.3) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 pycsw-CKAN NcML Grid Sgrid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services CMG Portal Delft3D NcML sudo apt-get docker docker run –d axiom/docker-thredds
  20. Catalog Search 23

  21. NCTOOLBOX: test_cf_ugrid3.m

  22. Skidaway “modena” glider: temperature

  23. None
  24. Interoperable Access in Python (Iris)

  25. 2015 Boston Light Swim 2015 Aug 15, 7:00 am start

    8 mile swim No wet suit How cold will the water be?
  26. NECOFS Massbay Forecast

  27. Reproducible Jupyter Notebook Go to: https://github.com/ioos/notebook_demos/boston_light_swim

  28. None
  29. None
  30. None
  31. Final Result

  32. None
  33. None
  34. Forecasts Sent to Swimmers

  35. Run it yourself

  36. None
  37. None
  38. Demo time?

  39. Sensor Map on IOOS.US

  40. ERDDAP access from Sensor Map

  41. ERDDAP access from Python


  43. None
  44. None
  45. None
  46. Summary • A standardized framework helps build the Fourth Network

    Layer, reducing data friction • Standards exist for: models, remotely-sensed data, points, time series, ADCP, trajectories (glider, AUV) • Science notebooks can be dynamic and reusable • Easy to install standard services or just pass to IOOS or NCEI • Your data should live here