Reducing Data Friction in Ocean Science

Slide 1

Slide 1 text

On Reducing Data Friction in Ocean Science Rich Signell (USGS, Woods Hole, MA) Filipe Fernandes (SECOORA) Kyle Wilcox (Axiom Data Science, Wickford, RI) COFDL, Dec 2, 2016

Slide 2

Slide 2 text

The Fourth Paradigm (2009)

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Jim Gray: “The suggestion that I have been making is that we now have terrible data management tools for most of the science disciplines. Commercial organizations like Walmart can afford to build their own data management software, but in science we do not have that luxury. … The funding agencies in the U.S. and elsewhere need to do a lot more to foster the building of tools to make scientists more productive.”

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Mark Abbott: • “The architecture for data-intensive computing should be based on storage, computing and presentation services at every node of an interconnected network. Providing standard, extensible frameworks that accommodate innovation at the network edges should enable these knowledge “ecosystems” to form and evolve as the needs of science and policy changes.”

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Wilbanks: The Fourth Network Layer • “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” • “We need open standards… and above all, we need to teach scientists to work in this new layer of data” From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” 4. Data 3. Web 2. TCP/IP 1. Ethernet

Slide 9

Slide 9 text

US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations

Slide 10

Slide 10 text

IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers Customer Web access service Data Provider Observations Models

Slide 11

Slide 11 text

IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding

Slide 12

Slide 12 text

OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena

Slide 13

Slide 13 text

NetCDF Climate and Forecast (CF) Conventions Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal OGC: Open Geospatial Consortium IOOS: Integrated Ocean Observing System NCEI: National Centers for Environmental Information

Slide 14

Slide 14 text

Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

Slide 15

Slide 15 text

Time Series, Trajectories, Profiles Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

IOOS Model Data Interoperability Design ROMS ADCIRC HYCOM SELFE SLOSH NcML NcML NcML NcML Common Data Model OPeNDAP+CF +UGRID|SGRID WCS NetCDF Subset THREDDS Data Server (TDS) Standardized (CF-1.6, UGRID-1.0, SGRID-0.3) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 pycsw-CKAN NcML Grid Sgrid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services CMG Portal Delft3D NcML sudo apt-get docker docker run –d axiom/docker-thredds

Slide 20

Slide 20 text

Catalog Search 23

Slide 21

Slide 21 text

NCTOOLBOX: test_cf_ugrid3.m

Slide 22

Slide 22 text

Skidaway “modena” glider: temperature

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Interoperable Access in Python (Iris)

Slide 25

Slide 25 text

2015 Boston Light Swim 2015 Aug 15, 7:00 am start 8 mile swim No wet suit How cold will the water be?

Slide 26

Slide 26 text

NECOFS Massbay Forecast

Slide 27

Slide 27 text

Reproducible Jupyter Notebook Go to: https://github.com/ioos/notebook_demos/boston_light_swim

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Final Result

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Forecasts Sent to Swimmers

Slide 35

Slide 35 text

Run it yourself

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Demo time?

Slide 39

Slide 39 text

Sensor Map on IOOS.US

Slide 40

Slide 40 text

ERDDAP access from Sensor Map

Slide 41

Slide 41 text

ERDDAP access from Python

Slide 42

Slide 42 text

NERACOOS ERDDAP

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Summary • A standardized framework helps build the Fourth Network Layer, reducing data friction • Standards exist for: models, remotely-sensed data, points, time series, ADCP, trajectories (glider, AUV) • Science notebooks can be dynamic and reusable • Easy to install standard services or just pass to IOOS or NCEI • Your data should live here