A free system for visualization and analysis of distributed model output in your browser

Presentation to NOAA Modeling Groups as part of the "Tuesday" lunch seminar series

Transcript

1. A free system for visualization and analysis of distributed model output in your browser Rich Signell , USGS, Woods Hole, MA, USA Filipe Fernandes, Centro Universidade Monte Serrat, Santos, Brazil Kyle Wilcox, Axiom Data Science, Wickford, RI, USA NOAA “Tuesday” Seminar Series 2015-12-16
2. The Fourth Paradigm: eScience 1. Thousand years ago: science was empirical describing natural phenomena 2. Last few hundred years: + theoretical branch using models, generalizations 3. Last few decades: + computational branch simulating complex phenomena 4. Today: + data exploration branch (eScience) • Data captured by instruments or simulations • Processed by software • Information/knowledge stored in computer • Scientist analyzes database / files using data management and statistics Ref: Slide from Turing Award Winner Jim Gray’s presentation to the NRC, Jan 11, 2007 (last presentation - lost at sea Jan 28)
3. The 4th Network Layer: Data • “We need an end-to-end, layer-by-layer, designed information technology … that are composed of no more than a stack of protocols” • “We need open standards… and above all, we need to teach scientists to work in this new layer of data” 3 From the essay: “I have seen the Paradigm Shift, and It Is Us”, byJohn Wilbanks, in the book “The Fourth Paradigm” Data Web TCP/IP Ethernet
4. US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations
5. IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers 5 Customer Web access service Data Provider Observations Models
6. Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
7. Unstructured (e.g. triangular) grid
8. NetCDF Climate and Forecast (CF) Conventions + UGRID + SGRID Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)
9. Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
10. OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena
11. IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
12. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Web Portals CKAN-pyCSW NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services
13. Catalog Search 13
14. WMS-driven Model Viewing Portal
15. Catalog Search 15 Catalog services can be federated via OGC CSW (Catalog Service for the Web)
16. Interoperable access in Matlab (nctoolbox)
17. Interoperable Access in Python (Iris)
18. Extra Slides 18
19. IOOS System Test
20. 2015 Boston Light Swim, Aug 15, 7:00am since 1907, 8 miles, no wet suit How cold will the water be?
21. NECOFS Massbay Forecast
22. Reproducible IPython/Jupyter Notebook
26. Final Result
29. Reproducible in Minutes for Free
30. 163 Python packages on IOOS channel!
31. Benefits of Standards-Based, Catalog-Driven, Reproducible Workflows • Find the real problems – Easy problems that can be fixed in minutes to day – Harder problems to guide future work • Fixes for specific workflows benefit everyone • Build success stories • Create reproducible workflows that others can learn from, expand on, or transform • Standardized workflows help develop the 4th network layer for data
32. [ rsignell-usgs | ocefpaf ] & github