Reproducible Science - Speaker Deck

Slide 1

Slide 1 text

Reproducible Science: The IOOS System Test Rich Signell , USGS, Woods Hole, MA, USA with Filipe Fernandes, Centro Universidade Monte Serrat, Santos, Brazil. Derrick Snowden, John Kupiec, NOAA Silver Springs, USA Dave Foster, Andy Bird, Kelly Knee, Bob Fratantonio, ASA/RPS Group, USA Kyle Wilcox and Will Koeppen, Axiom Consulting Hannah Dean, Consortium for Ocean Leadership, D.C., USA

Slide 2

Slide 2 text

Ideal Characteristics for Reproducible Science • Scientific workflow (data gathering, analysis, visualization) is available and documented • Data used in workflow is available via standardized web services • Workflow uses a software environment that is free and works on any platform • Workflow can be run on the cloud, using just your web browser, for free 2

Slide 3

Slide 3 text

US Integrated Ocean Observing System (IOOS® ) IOOS® Plan defines: • Global Component • Coastal Component  17 Federal Agencies  11 Regional Associations SECOORA Model Skill- Assessment Project: Deborah Hernandez and Vembu Subramanian

Slide 4

Slide 4 text

Curvilinear Grid Ocean Models Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates

Slide 5

Slide 5 text

Time Series, Trajectories Meteorology and Wave Buoy in the Gulf of Maine. Image courtesy of NOAA. Ocean Glider. Photo by Dave Fratantoni, Woods Hole Oceanographic Institution

Slide 6

Slide 6 text

NetCDF Climate and Forecast (CF) Conventions provide a solution Groups using CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium (GALEON: WCS profile)

Slide 7

Slide 7 text

IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding

Slide 8

Slide 8 text

OGC Sensor Observation Service (SOS) • Provides standard access to sensor data – GetCapabilities: provides the means to access SOS service metadata – DescribeSensor - retrieves detailed information about the sensors and processes generating those measurements. – GetObservation - provides access to sensor observations and measurement data via a spatio- temporal query that can be filtered by phenomena

Slide 9

Slide 9 text

Example SOS GetObservation request: “http://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS? service=SOS& request=GetObservation&version=1.0.0& observedProperty=water_surface_height_above_reference_d atum& offering=urn:ioos:station:NOAA.NOS.CO-OPS:8454000& responseFormat=text/csv& eventTime=2014-07-04T00:00:00Z/2014-07-05T00:00:00Z& result=VerticalDatum==urn:ogc:def:datum:epsg::5103& dataType=PreliminarySixMinute”

Slide 10

Slide 10 text

IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 CKAN-pyCSW NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services

Slide 11

Slide 11 text

Metadata harvest/query for search 11

Slide 12

Slide 12 text

Iris: Data Access using OPeNDAP+CF:

Slide 13

Slide 13 text

OWSLib for SOS and CSW

Slide 14

Slide 14 text

PyOOS to simplify met/ocean data access

Slide 15

Slide 15 text

Water Level Model Skill Assessment Link to NBVIEWER

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Automated model comparison

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Wakari – Scientific Python on the Cloud

Slide 21

Slide 21 text

“Share” in Wakari produced this URL

Slide 22

Slide 22 text

Free Wakari account (basic use)

Slide 23

Slide 23 text

Environment gets shared too!

Slide 24

Slide 24 text

Reproducible science with just a browser!

Slide 25

Slide 25 text

More System Test Notebooks on GitHub

Slide 26

Slide 26 text

Where we are headed …

Slide 27

Slide 27 text

Where we are headed …

Slide 28

Slide 28 text

Surface Current Skill Assessment

Slide 29

Slide 29 text

Interactive plotting with mplleaflet Nbviewer Link

Slide 30

Slide 30 text

Interactive plotting with mplleaflet

Slide 31

Slide 31 text

Interactive plotting with Folium and Vincent Nbviewer link

Slide 32

Slide 32 text

Standardized access to Unstructured Grid data I would love to see integration of UGRID into Iris!!

Slide 33

Slide 33 text

Summary • We have a framework that allows ocean model assessment for everyone • Python gives us the scientific access, analysis and visualization environment • Ipython Notebook gives us documented workflows in a browser interface • Wakari (and tools like it) let anyone reproduce our workflows • Result: Better use of models, better models, better world

Slide 34

Slide 34 text

GitHub & Wakari + rsignell

Slide 35

Slide 35 text

Extra Slides

Slide 36

Slide 36 text

Client Software Stack • Environment – IPython Notebooks, Anaconda, Binstar, Wakari, Github • Search – CSW using OWSLib • Access – OPenDAP+CF using Iris and Pyugrid – Sensor Observation Service (SOS) using OWSLib and PyOOS • Analysis and Plotting – Scipy, Pandas, Matplotlib, Cartopy, Vincent, Folium

Slide 37

Slide 37 text

Using Github issues for everything

Slide 38

Slide 38 text

Using Github to Capture Successes & Lessons Learned

Slide 39

Slide 39 text

Example progress issue

Slide 40

Slide 40 text

Example progress issue

Slide 41

Slide 41 text

OWSlib CSW

Slide 42

Slide 42 text

OGC Catalog Services for the Web (CSW) • Provides standardized services for search – GetCapabilities : returns the list of queryables – GetRecords : allows geospatial, temporal, keyword and free text search (and other queryables) • http://ex.com/csw?service=CSW&version=2.0.2 &request=GetRecords&typeName=csw:Record &constraintlanguage=CQLTEXT&constraint="c sw:AnyText Like '%pollution%'"

Slide 43

Slide 43 text

OWSlib CSW

Slide 44

Slide 44 text

OWSlib CSW

Slide 45

Slide 45 text

OGC Web Map Service • http://wms_endpoint? REQUEST=GetCapabilities • http://wms_endpoint? REQUEST=GetMap&LAYERS=wave_height& BBOX=-72.3,40.3,67.1,44.7& ELEVATION=0&TIME=2014-07- 07T06:00:00.000Z& CRS=EPSG:4326&FORMAT=image/png& WIDTH=256&HEIGHT=256

Slide 46

Slide 46 text

SECOORA Model Skill Assessment Project Funding from IOOS SECOORA Director: Deborah Hernandez Project Lead: Vembu Subramanian

Slide 47

Slide 47 text

IOOS System Test Project • Test the IOOS data instrastructure to solve actual physical and biological scenarios: eat our own dog food • Demonstrates successes as well as issues with standards, metadata, services, tools and data providers • Using Ipython Notebooks as documentation, end-to-end examples and products • Shared Notebooks lead to more efficient, effective and widespread use of model and insitu data System Test Funding from US IOOS Program Office Director: Zdenka Willis Project Lead: Derrick Snowden