SciPY 2014 Presentation, Austin, TX

An open source, free infrastructure is being developed by the US Integrated Ocean Observing System (IOOS) that allows anyone with a web browser to assess how well ocean models are performing by comparing with sensor data. Distributed datasets from multiple agencies and organizations are discovered via a catalog search, then data extracted via standard web services, and analyzed and visualized using IPython Notebooks.
Video: http://pyvideo.org/video/2795/ocean-model-assessment-for-everyone

Transcript

1. Ocean Model Assessment for Everyone Rich Signell , USGS, Woods Hole, MA, USA with Filipe Fernandes, Centro Universidade Monte Serrat, Santos, Brazil. Derrick Snowden, John Kupiec, NOAA Silver Springs, USA Dave Foster, Andy Bird, Kelly Knee, Bob Fratantonio, ASA/RPS Group, USA Kyle Wilcox and Will Koeppen, Axiom Consulting Hannah Dean, Consortium for Ocean Leadership, D.C., USA
2. How well do models simulate events? 2 Hurricane Sandy, Ocean Grove Pier - New Jersey, October 29, 2012 - Photograph by Bob Bowné Deepwater Horizon Oil Spill, Northern Gulf of Mexico, May 24, 2010. Image from MODIS on NASA’s Terra Satellite.
3. US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations SECOORA Model Skill- Assessment Project: Deborah Hernandez and Vembu Subramanian
4. IOOS Core Principles • Adopt open standards & practices • Avoid customer-specific stovepipes • Standardized access services implemented at data providers 4 Customer Web access service Data Provider Observations Models
5. Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
6. Unstructured (e.g. triangular) grid
7. Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
8. NetCDF Climate and Forecast (CF) Conventions provide a solution Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile) CF Convention Draft Spec for Unstructured Grid: http://bit.ly/ugrid_cf
9. IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
10. OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena
11. Example SOS GetObservation request: “http://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS? service=SOS& request=GetObservation&version=1.0.0& observedProperty=water_surface_height_above_reference_d atum& offering=urn:ioos:station:NOAA.NOS.CO-OPS:8454000& responseFormat=text/csv& eventTime=2014-07-04T00:00:00Z/2014-07-05T00:00:00Z& result=VerticalDatum==urn:ogc:def:datum:epsg::5103& dataType=PreliminarySixMinute”
12. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 CKAN-pyCSW NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services
13. Metadata harvest/query for search 13
14. Iris: Data Access using OPeNDAP+CF:
15. OWSLib for SOS and CSW
16. PyOOS to simplify met/ocean data access
17. Water Level Model Skill Assessment Link to NBVIEWER
19. Automated model comparison
22. Wakari – Scientific Python on the Cloud
23. “Share” in Wakari produced this URL
24. Free Wakari account (basic use)
25. Environment gets shared too!
26. Reproducible science with just a browser!
27. Where we are headed…
28. Where we are headed …
29. Where we are headed …
30. Surface Current Skill Assessment
31. Interactive plotting with mplleaflet Nbviewer Link
32. Interactive plotting with mplleaflet
33. Interactive plotting with Folium and Vincent Nbviewer link
34. Standardized access to Unstructured Grid data I would love to see integration of UGRID into Iris!!
35. Summary • We have a framework that allows ocean model assessment for everyone • Python gives us the scientific access, analysis and visualization environment • Ipython Notebook gives us documented workflows in a browser interface • Wakari (and tools like it) let anyone reproduce our workflows • Result: Better use of models, better models, better world
36. GitHub & Wakari + rsignell
37. Extra Slides
38. Client Software Stack • Environment – IPython Notebooks, Anaconda, Binstar, Wakari, Github • Search – CSW using OWSLib • Access – OPenDAP+CF using Iris and Pyugrid – Sensor Observation Service (SOS) using OWSLib and PyOOS • Analysis and Plotting – Scipy, Pandas, Matplotlib, Cartopy, Vincent, Folium
39. Using Github issues for everything
40. Using Github to Capture Successes & Lessons Learned
41. Example progress issue
42. Example progress issue
43. OWSlib CSW
44. OGC Catalog Services for the Web (CSW) • Provides standardized services for search – GetCapabilities : returns the list of queryables – GetRecords : allows geospatial, temporal, keyword and free text search (and other queryables) • http://ex.com/csw?service=CSW&version=2.0.2 &request=GetRecords&typeName=csw:Record &constraintlanguage=CQLTEXT&constraint="c sw:AnyText Like '%pollution%'"
45. OWSlib CSW
46. OWSlib CSW
47. OGC Web Map Service • http://wms_endpoint? REQUEST=GetCapabilities • http://wms_endpoint? REQUEST=GetMap&LAYERS=wave_height& BBOX=-72.3,40.3,67.1,44.7& ELEVATION=0&TIME=2014-07- 07T06:00:00.000Z& CRS=EPSG:4326&FORMAT=image/png& WIDTH=256&HEIGHT=256
48. SECOORA Model Skill Assessment Project Funding from IOOS SECOORA Director: Deborah Hernandez Project Lead: Vembu Subramanian
49. IOOS System Test Project • Test the IOOS data instrastructure to solve actual physical and biological scenarios: eat our own dog food • Demonstrates successes as well as issues with standards, metadata, services, tools and data providers • Using Ipython Notebooks as documentation, end-to-end examples and products • Shared Notebooks lead to more efficient, effective and widespread use of model and insitu data System Test Funding from US IOOS Program Office Director: Zdenka Willis Project Lead: Derrick Snowden