Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serving earth observation data with GeoServer: addressing real world requirements with Open Source

Serving earth observation data with GeoServer: addressing real world requirements with Open Source

Information on the latest developments in GeoServer for the support for Earth Observation data with support for the various OGC services and OpenSearch.

Bac74c17d65c22d0ae63915251f7750f?s=128

Simone Giannecchini
PRO

September 02, 2018
Tweet

Transcript

  1. Serving EO Data with GeoServer Addressing real-world EO requirements With

    GeoServer Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
  2. GeoSolutions ⚫ Founded in Italy in late 2006 ⚫ Expertise

    • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization ⚫ Supporting/Developing FOSS4G projects ⚫ GeoServer, MapStore ⚫ GeoNetwork, GeoNode, Ckan ⚫ Clients ⚫ Public Agencies ⚫ Private Companies ⚫ http://www.geo-solutions.it FOSS4G 2018, August 29th/31st, Dar Es Salaam
  3. Reference scenario FOSS4G 2018, August 29th/31st, Dar Es Salaam

  4. ImageMosaic – the basics

  5. Terminology ⚫ Granule/Tile ⚫ The individual raster element composing the

    mosaic ⚫ (Granule) Index ⚫ The collection of metadata records describing the location, spatial coverage and other attributes of each single granule FOSS4G 2018, August 29th/31st, Dar Es Salaam
  6. Complete freedom ⚫ Granules can: ⚫ Overlap as they please,

    can have different resolutions ⚫ Be in different file formats (faster if the format is uniform) ⚫ Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) ⚫ Be in different coordinate reference systems FOSS4G 2018, August 29th/31st, Dar Es Salaam
  7. The mosaic index ⚫ The file lookup system ⚫ A

    table of granules, their footprint, location, and extra attributes ⚫ Currently supported/tested sources: ⚫ PostGIS, Oracle Shapefile, h (no attr indexing, no timestamp) ⚫ H2 (local embedded db) ⚫ Can be customized to support custom granule indexes (e.g. legacy catalog) FOSS4G 2018, August 29th/31st, Dar Es Salaam
  8. The Dimensions ⚫ Maps to alphanumeric attributes in the index

    ⚫ TIME and ELEVATION receive special treatment for WMS and WCS ⚫ Custom/Additional dimensions ⚫ Everything besides TIME & ELEVATION ⚫ They can be dynamically discovered from GetCapabilities/DescribeCoverage FOSS4G 2018, August 29th/31st, Dar Es Salaam
  9. Filling the index ⚫ Available options ⚫ Have GeoServer do

    it for you from the UI (one shot setup) ⚫ Use the REST API to add granules and make it populate the index ⚫ Directly write the index (common for moving window of data) ⚫ Use the OpenSearch for EO extension (later in the presentation) FOSS4G 2018, August 29th/31st, Dar Es Salaam
  10. Documentation links ⚫ GeoServer reference docs: ⚫ http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html ⚫

    http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html ⚫ http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml ⚫ GeoSolutions training: ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/imagemosaic/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/rest/index.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
  11. ImageMosaic – the fun stuff

  12. Filtering and sorting ⚫ The mosaic index can be filtered

    upon with CQL ⚫ Can be sorted on using WFS like syntax ⚫ More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D FOSS4G 2018, August 29th/31st, Dar Es Salaam
  13. Masking Support ⚫ Mask allow finding valid data ⚫ Alternative

    to NODATA for lossy compressions ⚫ Vector and raster masks supported ⚫ Raster masks can be embedded (GDAL style) ⚫ Vector ones can be WKB or shapefiles ⚫ API to plug your own mask provider ⚫ Available for image mosaic and GDAL based data sources FOSS4G 2018, August 29th/31st, Dar Es Salaam
  14. Coverage Views ⚫ NetCDF/Grib/Mosaic: one reader, many coverages ⚫ Sometimes

    the coverages are strictly related ⚫ E.g., U and V components of wind/current vectors ⚫ Merge them back into a single coverage as bands ⚫ Allows rendering engine to work with them both as one FOSS4G 2018, August 29th/31st, Dar Es Salaam
  15. Native multi-coverage support ⚫ Satellite images often comes in one

    file per band ⚫ Requirement, as they have different resolutions ⚫ Grab the band name from file, build a “band specific” coverage ⚫ Use coverage views to merge them together in a virtual multiband coverage ⚫ Smarts to read only the desired files when doing band selection FOSS4G 2018, August 29th/31st, Dar Es Salaam
  16. Pluggability ⚫ Lots of extension points and pluggable bits ⚫

    PropertyCollectors: extract index attributes from granules ⚫ CoverageNameCollector: when you have a hash of files from multiple coverages ⚫ GranuleHandler: pre-process granules before getting them in the mosaic ⚫ GranuleAcceptor: decide if the granule is actually part of the mosaic ⚫ SubMosaicProducer: custom way to mosaic granules ⚫ GranuleCatalog: how you store the index ⚫ See more examples at the end of presentation FOSS4G 2018, August 29th/31st, Dar Es Salaam
  17. Documentation links ⚫ GeoServer reference docs: ⚫ http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html ⚫

    http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html ⚫ http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html ⚫ GeoSolutions training: ⚫ https://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/coveragev iew.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
  18. NetCDF and GRIB

  19. NetCDF format support ⚫ NetCDF support ⚫ Support COARDS/CF* conventions

    loosely ⚫ Expose NetCDF internal data as a set of 2D slices ⚫ Fast 2D (time, elevation) slice extraction ⚫ Supports NCML ⚫ GRIB supported by the same libraries, with same functionality FOSS4G 2018, August 29th/31st, Dar Es Salaam
  20. One or more variables ⚫ Polyphemus Sample Dataset ⚫ 1

    File → Multiple Coverages! FOSS4G 2018, August 29th/31st, Dar Es Salaam
  21. Multiple coverages per mosaic FOSS4G 2018, August 29th/31st, Dar Es

    Salaam
  22. NetCDF/GRIB Mosaic integration ⚫ ImageMosaic NetCDF and GRIB integration ⚫

    Allow the ImageMosaic to handle multiple files ⚫ Expose internal structure (times, elevations) ⚫ Make ImageMosaic handle slices of the file as granules ⚫ 5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N … FOSS4G 2018, August 29th/31st, Dar Es Salaam
  23. Documentation links ⚫ GeoServer documentation ⚫ http://docs.geoserver.org/latest/en/user/extensions/n etcdf/netcdf.html ⚫ http://docs.geoserver.org/latest/en/user/extensio

    ns/netcdf-out/index.html ⚫ GeoSolutions training: ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/mosaic_config/netc df_mosaic.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
  24. The services - visualize

  25. WMS-T support FOSS4G 2018, August 29th/31st, Dar Es Salaam

  26. Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?...

    &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z FOSS4G 2018, August 29th/31st, Dar Es Salaam
  27. Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

    A FOSS4G 2018, August 29th/31st, Dar Es Salaam
  28. Rendering transformations ⚫ Style based transformations ⚫ Applied on the

    fly, yet fast enough for interactive use ⚫ Pluggable: add your own contouring FOSS4G 2018, August 29th/31st, Dar Es Salaam
  29. Rendering transformations examples windbarbs currents FOSS4G 2018, August 29th/31st, Dar

    Es Salaam
  30. On the fly map algebra with Jiffle nir = src[7];

    vir = src[3]; dest = (nir-vir)/ (nir+vir); ⚫ Jiffle: powerful and fast map algebra language ⚫ Can be used as a rendering transformation FOSS4G 2018, August 29th/31st, Dar Es Salaam
  31. Documentation links ⚫ GeoServer documentation ⚫ http://docs.geoserver.org/latest/en/user/services/wm s/time.html ⚫ http://docs.geoserver.org/stable/en/user/data/webad

    min/layers.html#edit-layer-dimensions ⚫ http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html ⚫ GeoSolutions training: ⚫ https://geoserver.geo- solutions.it/edu/en/wps/rendering_tx.html ⚫ Jiffle documentation: ⚫ https://github.com/geosolutions-it/jai-ext/wiki/Jiffle FOSS4G 2018, August 29th/31st, Dar Es Salaam
  32. The services - download

  33. WCS 2.0 ⚫ Basics ⚫ Core service ⚫ KVP binding

    ⚫ XML binding ⚫ Common extensions ⚫ CRS ⚫ Scaling ⚫ Interpolation ⚫ Range subsetting ⚫ GeoTiff ⚫ GML ⚫ NetCDF ⚫ Add the output format extensions ⚫ GeoTIFF ⚫ GML Grid ⚫ NetCDF FOSS4G 2018, August 29th/31st, Dar Es Salaam
  34. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION

    details as extension CUSTOM FOSS4G 2018, August 29th/31st, Dar Es Salaam
  35. NetCDF WCS output ⚫ WCS can output NetCDF for any

    multidimensional source ⚫ Several customizations to control output and preserve info if the source is a NetCDF itself ⚫ Source has to be multidimensional (current restriction) FOSS4G 2018, August 29th/31st, Dar Es Salaam
  36. Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03-

    01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX FOSS4G 2018, August 29th/31st, Dar Es Salaam
  37. WCS – custom filtering and sorting &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

    A FOSS4G 2018, August 29th/31st, Dar Es Salaam
  38. WCS EO ⚫ Add support for WCS EO metadata in

    readers ⚫ Associate each file with EO metadata ⚫ Include such information in DescribeCoverage/DescribeEODataset ⚫ EODataset: multidimensional mosaic or NetCDF file ⚫ Describe the inner structure (granules listing) in DescribeEODataSet FOSS4G 2018, August 29th/31st, Dar Es Salaam
  39. OpenSearch for EO

  40. Putting it all together (and finding it) ⚫ Use case

    ⚫ Lots of satellite imagery ⚫ Different satellites ⚫ Different sensor ⚫ Deep time and space distribution ⚫ How to find the right image? ⚫ By time and space ⚫ By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO ⚫ “OpenSearch for EO“ to the rescue FOSS4G 2018, August 29th/31st, Dar Es Salaam
  41. General information schema Collection Product Granule Search Search Mosaic/OGC FOSS4G

    2018, August 29th/31st, Dar Es Salaam
  42. Two steps search ⚫ First find collection of similar products

    (by satellite/sensor), e.g. ⚫ Optical or SAR? ⚫ Supported resolutions ⚫ Availability of data in particular timeframe ⚫ Then search products in it, by ⚫ Time ⚫ Space ⚫ Cloud cover ⚫ … FOSS4G 2018, August 29th/31st, Dar Es Salaam
  43. Sample search ⚫ http://cloudsdi. geo-solutions.it/ ⚫ geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] ⚫

    Output is RSS with description, thumb/quicklook and links FOSS4G 2018, August 29th/31st, Dar Es Salaam
  44. Metadata and cross linking ⚫ Links to metadata: ⚫ ISO

    metadata for collection ⚫ GML O&M for products ⚫ Links to OGC services (cross linking) ⚫ WMS/WMTS to see collection/product ⚫ WFS to get “masks” (validity, sea, cloud, snow, …) ⚫ WCS to extract raster ⚫ Direct download to get original package FOSS4G 2018, August 29th/31st, Dar Es Salaam
  45. Tight integration with mosaic possible ⚫ The system creates one

    “feature type” per collection ⚫ Can be used as the index to a mosaic ⚫ One mosaic per collection ⚫ Filter using CQL and sort by attribute on the WMS/WCS service ⚫ Using all the attributes already searchable by OpenSearch FOSS4G 2018, August 29th/31st, Dar Es Salaam
  46. REST API to manage search index Swagger documentation FOSS4G 2018,

    August 29th/31st, Dar Es Salaam
  47. Real World Use cases

  48. Use Case 1 – Light Integration ⚫ GeoServer + MapStore

    + Legacy Ingestion ⚫ Maritime Data ⚫ SAR Time Series ⚫ Oil Spill + other vectors ⚫ Single Mosaic + CQL + Custom Access Manager ⚫ Integrated SSO with MapStore ⚫ Precooked Download of Scenes ⚫ Fine grain control over data access ⚫ Download ⚫ Visualization ⚫ Clever usage of Mosaic Indexes via WFS FOSS4G 2018, August 29th/31st, Dar Es Salaam
  49. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st,

    Dar Es Salaam
  50. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st,

    Dar Es Salaam
  51. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st,

    Dar Es Salaam
  52. Use Case 2 – Deep Integration ⚫ High Res Imagery

    ⚫ SSO with CAS ⚫ Legacy Ingestion FOSS4G 2018, August 29th/31st, Dar Es Salaam
  53. Use Case 2 ⚫ Database of satellite imagery, one satellite

    collect (strip) materialized as many files (granules) ⚫ Access filtering based on current user roles (custom data store implementation) ⚫ Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store FOSS4G 2018, August 29th/31st, Dar Es Salaam
  54. Use Case 3 – Let’s go Deeper! FOSS4G 2018, August

    29th/31st, Dar Es Salaam
  55. That’s all folks! Questions? info@geo-solutions.it FOSS4G 2018, August 29th/31st, Dar

    Es Salaam
  56. Assorted extras

  57. WMS-EO Extending LayerGroup concept Support same style on both raster

    and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups ⚫ Product layer tree ⚫ Mix of vector and raster info ⚫ Raw data, flags (clouds, snow), browse FOSS4G 2018, August 29th/31st, Dar Es Salaam
  58. WMS-EO ⚫ Helper UI to build the tree FOSS4G 2018,

    August 29th/31st, Dar Es Salaam