Reproducible Science

Presentation at 2014 Unidata Python Training Workshop

Transcript

1. Reproducible Science: The IOOS System Test Rich Signell , USGS, Woods Hole, MA, USA with Filipe Fernandes, Centro Universidade Monte Serrat, Santos, Brazil. Derrick Snowden, John Kupiec, NOAA Silver Springs, USA Dave Foster, Andy Bird, Kelly Knee, Bob Fratantonio, ASA/RPS Group, USA Kyle Wilcox and Will Koeppen, Axiom Consulting Hannah Dean, Consortium for Ocean Leadership, D.C., USA
2. Ideal Characteristics for Reproducible Science • Scientific workflow (data gathering, analysis, visualization) is available and documented • Data used in workflow is available via standardized web services • Workflow uses a software environment that is free and works on any platform • Workflow can be run on the cloud, using just your web browser, for free 2
3. US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations SECOORA Model Skill- Assessment Project: Deborah Hernandez and Vembu Subramanian
4. Curvilinear Grid Ocean Models Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
5. Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution
6. NetCDF Climate and Forecast (CF) Conventions provide a solution Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)
7. IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
8. OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena
9. Example SOS GetObservation request: “http://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS? service=SOS& request=GetObservation&version=1.0.0& observedProperty=water_surface_height_above_reference_d atum& offering=urn:ioos:station:NOAA.NOS.CO-OPS:8454000& responseFormat=text/csv& eventTime=2014-07-04T00:00:00Z/2014-07-05T00:00:00Z& result=VerticalDatum==urn:ogc:def:datum:epsg::5103& dataType=PreliminarySixMinute”
10. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 CKAN-pyCSW NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services
11. Metadata harvest/query for search 11
12. Iris: Data Access using OPeNDAP+CF:
13. OWSLib for SOS and CSW
14. PyOOS to simplify met/ocean data access
15. Water Level Model Skill Assessment Link to NBVIEWER
17. Automated model comparison
20. Wakari – Scientific Python on the Cloud
21. “Share” in Wakari produced this URL
22. Free Wakari account (basic use)
23. Environment gets shared too!
24. Reproducible science with just a browser!
25. More System Test Notebooks on GitHub
26. Where we are headed …
27. Where we are headed …
28. Surface Current Skill Assessment
29. Interactive plotting with mplleaflet Nbviewer Link
30. Interactive plotting with mplleaflet
31. Interactive plotting with Folium and Vincent Nbviewer link
32. Standardized access to Unstructured Grid data I would love to see integration of UGRID into Iris!!
33. Summary • We have a framework that allows ocean model assessment for everyone • Python gives us the scientific access, analysis and visualization environment • Ipython Notebook gives us documented workflows in a browser interface • Wakari (and tools like it) let anyone reproduce our workflows • Result: Better use of models, better models, better world
34. GitHub & Wakari + rsignell
35. Extra Slides
36. Client Software Stack • Environment – IPython Notebooks, Anaconda, Binstar, Wakari, Github • Search – CSW using OWSLib • Access – OPenDAP+CF using Iris and Pyugrid – Sensor Observation Service (SOS) using OWSLib and PyOOS • Analysis and Plotting – Scipy, Pandas, Matplotlib, Cartopy, Vincent, Folium
37. Using Github issues for everything
38. Using Github to Capture Successes & Lessons Learned
39. Example progress issue
40. Example progress issue
41. OWSlib CSW
42. OGC Catalog Services for the Web (CSW) • Provides standardized services for search – GetCapabilities : returns the list of queryables – GetRecords : allows geospatial, temporal, keyword and free text search (and other queryables) • http://ex.com/csw?service=CSW&version=2.0.2 &request=GetRecords&typeName=csw:Record &constraintlanguage=CQLTEXT&constraint="c sw:AnyText Like '%pollution%'"
43. OWSlib CSW
44. OWSlib CSW
45. OGC Web Map Service • http://wms_endpoint? REQUEST=GetCapabilities • http://wms_endpoint? REQUEST=GetMap&LAYERS=wave_height& BBOX=-72.3,40.3,67.1,44.7& ELEVATION=0&TIME=2014-07- 07T06:00:00.000Z& CRS=EPSG:4326&FORMAT=image/png& WIDTH=256&HEIGHT=256
46. SECOORA Model Skill Assessment Project Funding from IOOS SECOORA Director: Deborah Hernandez Project Lead: Vembu Subramanian
47. IOOS System Test Project • Test the IOOS data instrastructure to solve actual physical and biological scenarios: eat our own dog food • Demonstrates successes as well as issues with standards, metadata, services, tools and data providers • Using Ipython Notebooks as documentation, end-to-end examples and products • Shared Notebooks lead to more efficient, effective and widespread use of model and insitu data System Test Funding from US IOOS Program Office Director: Zdenka Willis Project Lead: Derrick Snowden