SciPy 2015 Presentation

A Distributed, Standards-Based Framework for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data: Application to Hurricane Sandy

Transcript

1. A Distributed, Standards-Based Framework for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data: Application to Hurricane Sandy Rich Signell and Andrew Yan, USGS, USA Filipe Fernandes, Centro Universidade Monte Serrat, Brazil. Kyle Wilcox, Axiom Data Science, USA
2. US Integrated Ocean Observing System (IOOS® )  17 Federal Agencies  11 Regional Associations
3. From pile of files to interoperability Search and Browse Datasets Access Datasets Dataset files
4. Ocean grids are often not regularly spaced! Stretched surface and terrain following vertical coordinates Curvilinear orthogonal horizontal coordinates
5. CF Conventions for met/ocean models Groups using CF: OGC: Open Geospatial Consortium GO-ESSP: Global Organization for Earth System Science Portal IOOS: Integrated Ocean Observing System float temp(ocean_time, s_rho, eta_rho, xi_rho) ; temp: standard_name = “sea_water_potential_temperature" ; temp: units = "Celsius" ; temp: coordinates = "lon_rho lat_rho s_rho ocean_time" ; double s_rho(s_rho) ; s_rho: long_name = "S-coordinate at RHO-points" ; s_rho: positive = "up" ; s_rho: standard_name = "ocean_s_coordinate_g1" ; s_rho: formula_terms = "s: s_rho C: Cs_r eta: zeta depth: h depth_c: hc "
6. IOOS Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OPeNDAP with Climate and Forecast Conventions XML or CSV Binary DAP using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles Data Type Web Service Encoding
7. IOOS Data Infrastructure Diagram ROMS ADCIRC HYCOM SELFE NCOM NcML NcML NcML NcML NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF-1.6, UGRID-0.9) Virtual Datasets Nonstandard Model Output Data Files Web Services Matlab Panoply IDV Clients NetCDF -Java Library or Broker WMS ncISO ArcGIS NetCDF4 -Python FVCOM Python ERDDAP NetCDF-Java SOS Geoportal Server GeoNetwork GI-CAT Observed data (buoy, gauge, ADCP, glider) Godiva2 CKAN-pyCSW NcML Grid Ugrid TimeSeries Profile Trajectory TimeSeriesProfile Nonstandard Data Files Catalog Services
8. WMS-driven Model Viewing Portal
9. Iris: Data Access using OPeNDAP+CF:
10. Interoperable Access “viridi”!!!
11. Two things that CF doesn’t do • Unstructured Grids • Staggered Grids
12. Standardized access to Unstructured Grid data
13. SGRID Conventions: github/sgrid
14. sci-wms (1.0 release)
15. UGRID browsing using sci-wms
16. SGRID browsing using sci-wms
17. UGRID data access using pyugrid
18. UGRID access using pyugrid
19. SGRID access using pysgrid
20. SGRID data access with pysgrid
21. pycsw for search
22. OWSlib CSW
23. Browse with Github.com/TerriaJS Consumes: WMS, ESRI Rest, OSM, KML, GeoJSON, GPX, even CSV files Can search CKAN to build dynamic catalog Supports both 3D and 2D in browser
24. IOOS binstar channel: 139 packages
25. yaml2ncml.py
26. Summary • We have now a complete open source framework for model data delivery, search, browse and access • Need to make it easy for modelers to connect their data • Standardized services and tools benefit both scientists and developers • Join us for sprint tomorrow on UGRID!
27. Search “GitHub” “Signell”
28. Client Software Stack • Environment: – IPython Notebooks, Anaconda, Binstar, Github • Delivery – THREDDS Data Server, – OPeNDAP +CF +UGRID +SGRID • Access – Iris, pyugrid, pysgrid • Search – CSW using pycsw (server) & OWSLib (client) • Browse – Sciwms (server) & terriaJS (client)
29. Interoperable Model Comparison in Matlab (using nctoolbox)