Serving earth observation data with GeoServer: addressing real world requirements with Open Source

Information on the latest developments in GeoServer for the support for Earth Observation data with support for the various OGC services and OpenSearch.

1. Serving EO Data with GeoServer Addressing real-world EO requirements With GeoServer Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
2. GeoSolutions ⚫ Founded in Italy in late 2006 ⚫ Expertise • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization ⚫ Supporting/Developing FOSS4G projects ⚫ GeoServer, MapStore ⚫ GeoNetwork, GeoNode, Ckan ⚫ Clients ⚫ Public Agencies ⚫ Private Companies ⚫ http://www.geo-solutions.it FOSS4G 2018, August 29th/31st, Dar Es Salaam
3. Reference scenario FOSS4G 2018, August 29th/31st, Dar Es Salaam
4. ImageMosaic – the basics
5. Terminology ⚫ Granule/Tile ⚫ The individual raster element composing the mosaic ⚫ (Granule) Index ⚫ The collection of metadata records describing the location, spatial coverage and other attributes of each single granule FOSS4G 2018, August 29th/31st, Dar Es Salaam
6. Complete freedom ⚫ Granules can: ⚫ Overlap as they please, can have different resolutions ⚫ Be in different file formats (faster if the format is uniform) ⚫ Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) ⚫ Be in different coordinate reference systems FOSS4G 2018, August 29th/31st, Dar Es Salaam
7. The mosaic index ⚫ The file lookup system ⚫ A table of granules, their footprint, location, and extra attributes ⚫ Currently supported/tested sources: ⚫ PostGIS, Oracle Shapefile, h (no attr indexing, no timestamp) ⚫ H2 (local embedded db) ⚫ Can be customized to support custom granule indexes (e.g. legacy catalog) FOSS4G 2018, August 29th/31st, Dar Es Salaam
8. The Dimensions ⚫ Maps to alphanumeric attributes in the index ⚫ TIME and ELEVATION receive special treatment for WMS and WCS ⚫ Custom/Additional dimensions ⚫ Everything besides TIME & ELEVATION ⚫ They can be dynamically discovered from GetCapabilities/DescribeCoverage FOSS4G 2018, August 29th/31st, Dar Es Salaam
9. Filling the index ⚫ Available options ⚫ Have GeoServer do it for you from the UI (one shot setup) ⚫ Use the REST API to add granules and make it populate the index ⚫ Directly write the index (common for moving window of data) ⚫ Use the OpenSearch for EO extension (later in the presentation) FOSS4G 2018, August 29th/31st, Dar Es Salaam
10. Documentation links ⚫ GeoServer reference docs: ⚫ http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html ⚫ http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html ⚫ http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml ⚫ GeoSolutions training: ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/imagemosaic/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/rest/index.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
11. ImageMosaic – the fun stuff
12. Filtering and sorting ⚫ The mosaic index can be filtered upon with CQL ⚫ Can be sorted on using WFS like syntax ⚫ More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D FOSS4G 2018, August 29th/31st, Dar Es Salaam
13. Masking Support ⚫ Mask allow finding valid data ⚫ Alternative to NODATA for lossy compressions ⚫ Vector and raster masks supported ⚫ Raster masks can be embedded (GDAL style) ⚫ Vector ones can be WKB or shapefiles ⚫ API to plug your own mask provider ⚫ Available for image mosaic and GDAL based data sources FOSS4G 2018, August 29th/31st, Dar Es Salaam
14. Coverage Views ⚫ NetCDF/Grib/Mosaic: one reader, many coverages ⚫ Sometimes the coverages are strictly related ⚫ E.g., U and V components of wind/current vectors ⚫ Merge them back into a single coverage as bands ⚫ Allows rendering engine to work with them both as one FOSS4G 2018, August 29th/31st, Dar Es Salaam
15. Native multi-coverage support ⚫ Satellite images often comes in one file per band ⚫ Requirement, as they have different resolutions ⚫ Grab the band name from file, build a “band specific” coverage ⚫ Use coverage views to merge them together in a virtual multiband coverage ⚫ Smarts to read only the desired files when doing band selection FOSS4G 2018, August 29th/31st, Dar Es Salaam
16. Pluggability ⚫ Lots of extension points and pluggable bits ⚫ PropertyCollectors: extract index attributes from granules ⚫ CoverageNameCollector: when you have a hash of files from multiple coverages ⚫ GranuleHandler: pre-process granules before getting them in the mosaic ⚫ GranuleAcceptor: decide if the granule is actually part of the mosaic ⚫ SubMosaicProducer: custom way to mosaic granules ⚫ GranuleCatalog: how you store the index ⚫ See more examples at the end of presentation FOSS4G 2018, August 29th/31st, Dar Es Salaam
17. Documentation links ⚫ GeoServer reference docs: ⚫ http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html ⚫ http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html ⚫ http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html ⚫ GeoSolutions training: ⚫ https://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/coveragev iew.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
18. NetCDF and GRIB
19. NetCDF format support ⚫ NetCDF support ⚫ Support COARDS/CF* conventions loosely ⚫ Expose NetCDF internal data as a set of 2D slices ⚫ Fast 2D (time, elevation) slice extraction ⚫ Supports NCML ⚫ GRIB supported by the same libraries, with same functionality FOSS4G 2018, August 29th/31st, Dar Es Salaam
20. One or more variables ⚫ Polyphemus Sample Dataset ⚫ 1 File → Multiple Coverages! FOSS4G 2018, August 29th/31st, Dar Es Salaam
21. Multiple coverages per mosaic FOSS4G 2018, August 29th/31st, Dar Es Salaam
22. NetCDF/GRIB Mosaic integration ⚫ ImageMosaic NetCDF and GRIB integration ⚫ Allow the ImageMosaic to handle multiple files ⚫ Expose internal structure (times, elevations) ⚫ Make ImageMosaic handle slices of the file as granules ⚫ 5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N … FOSS4G 2018, August 29th/31st, Dar Es Salaam
23. Documentation links ⚫ GeoServer documentation ⚫ http://docs.geoserver.org/latest/en/user/extensions/n etcdf/netcdf.html ⚫ http://docs.geoserver.org/latest/en/user/extensio ns/netcdf-out/index.html ⚫ GeoSolutions training: ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html ⚫ http://geoserver.geo- solutions.it/edu/en/multidim/mosaic_config/netc df_mosaic.html FOSS4G 2018, August 29th/31st, Dar Es Salaam
24. The services - visualize
25. WMS-T support FOSS4G 2018, August 29th/31st, Dar Es Salaam
26. Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?... &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z FOSS4G 2018, August 29th/31st, Dar Es Salaam
27. Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover A FOSS4G 2018, August 29th/31st, Dar Es Salaam
28. Rendering transformations ⚫ Style based transformations ⚫ Applied on the fly, yet fast enough for interactive use ⚫ Pluggable: add your own contouring FOSS4G 2018, August 29th/31st, Dar Es Salaam
29. Rendering transformations examples windbarbs currents FOSS4G 2018, August 29th/31st, Dar Es Salaam
30. On the fly map algebra with Jiffle nir = src[7]; vir = src[3]; dest = (nir-vir)/ (nir+vir); ⚫ Jiffle: powerful and fast map algebra language ⚫ Can be used as a rendering transformation FOSS4G 2018, August 29th/31st, Dar Es Salaam
31. Documentation links ⚫ GeoServer documentation ⚫ http://docs.geoserver.org/latest/en/user/services/wm s/time.html ⚫ http://docs.geoserver.org/stable/en/user/data/webad min/layers.html#edit-layer-dimensions ⚫ http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html ⚫ GeoSolutions training: ⚫ https://geoserver.geo- solutions.it/edu/en/wps/rendering_tx.html ⚫ Jiffle documentation: ⚫ https://github.com/geosolutions-it/jai-ext/wiki/Jiffle FOSS4G 2018, August 29th/31st, Dar Es Salaam
32. The services - download
33. WCS 2.0 ⚫ Basics ⚫ Core service ⚫ KVP binding ⚫ XML binding ⚫ Common extensions ⚫ CRS ⚫ Scaling ⚫ Interpolation ⚫ Range subsetting ⚫ GeoTiff ⚫ GML ⚫ NetCDF ⚫ Add the output format extensions ⚫ GeoTIFF ⚫ GML Grid ⚫ NetCDF FOSS4G 2018, August 29th/31st, Dar Es Salaam
34. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION details as extension CUSTOM FOSS4G 2018, August 29th/31st, Dar Es Salaam
35. NetCDF WCS output ⚫ WCS can output NetCDF for any multidimensional source ⚫ Several customizations to control output and preserve info if the source is a NetCDF itself ⚫ Source has to be multidimensional (current restriction) FOSS4G 2018, August 29th/31st, Dar Es Salaam
36. Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03- 01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX FOSS4G 2018, August 29th/31st, Dar Es Salaam
37. WCS – custom filtering and sorting &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover A FOSS4G 2018, August 29th/31st, Dar Es Salaam
38. WCS EO ⚫ Add support for WCS EO metadata in readers ⚫ Associate each file with EO metadata ⚫ Include such information in DescribeCoverage/DescribeEODataset ⚫ EODataset: multidimensional mosaic or NetCDF file ⚫ Describe the inner structure (granules listing) in DescribeEODataSet FOSS4G 2018, August 29th/31st, Dar Es Salaam
39. OpenSearch for EO
40. Putting it all together (and finding it) ⚫ Use case ⚫ Lots of satellite imagery ⚫ Different satellites ⚫ Different sensor ⚫ Deep time and space distribution ⚫ How to find the right image? ⚫ By time and space ⚫ By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO ⚫ “OpenSearch for EO“ to the rescue FOSS4G 2018, August 29th/31st, Dar Es Salaam
41. General information schema Collection Product Granule Search Search Mosaic/OGC FOSS4G 2018, August 29th/31st, Dar Es Salaam
42. Two steps search ⚫ First find collection of similar products (by satellite/sensor), e.g. ⚫ Optical or SAR? ⚫ Supported resolutions ⚫ Availability of data in particular timeframe ⚫ Then search products in it, by ⚫ Time ⚫ Space ⚫ Cloud cover ⚫ … FOSS4G 2018, August 29th/31st, Dar Es Salaam
43. Sample search ⚫ http://cloudsdi. geo-solutions.it/ ⚫ geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] ⚫ Output is RSS with description, thumb/quicklook and links FOSS4G 2018, August 29th/31st, Dar Es Salaam
44. Metadata and cross linking ⚫ Links to metadata: ⚫ ISO metadata for collection ⚫ GML O&M for products ⚫ Links to OGC services (cross linking) ⚫ WMS/WMTS to see collection/product ⚫ WFS to get “masks” (validity, sea, cloud, snow, …) ⚫ WCS to extract raster ⚫ Direct download to get original package FOSS4G 2018, August 29th/31st, Dar Es Salaam
45. Tight integration with mosaic possible ⚫ The system creates one “feature type” per collection ⚫ Can be used as the index to a mosaic ⚫ One mosaic per collection ⚫ Filter using CQL and sort by attribute on the WMS/WCS service ⚫ Using all the attributes already searchable by OpenSearch FOSS4G 2018, August 29th/31st, Dar Es Salaam
46. REST API to manage search index Swagger documentation FOSS4G 2018, August 29th/31st, Dar Es Salaam
47. Real World Use cases
48. Use Case 1 – Light Integration ⚫ GeoServer + MapStore + Legacy Ingestion ⚫ Maritime Data ⚫ SAR Time Series ⚫ Oil Spill + other vectors ⚫ Single Mosaic + CQL + Custom Access Manager ⚫ Integrated SSO with MapStore ⚫ Precooked Download of Scenes ⚫ Fine grain control over data access ⚫ Download ⚫ Visualization ⚫ Clever usage of Mosaic Indexes via WFS FOSS4G 2018, August 29th/31st, Dar Es Salaam
49. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st, Dar Es Salaam
50. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st, Dar Es Salaam
51. Use Case 1 – Light Integration FOSS4G 2018, August 29th/31st, Dar Es Salaam
52. Use Case 2 – Deep Integration ⚫ High Res Imagery ⚫ SSO with CAS ⚫ Legacy Ingestion FOSS4G 2018, August 29th/31st, Dar Es Salaam
53. Use Case 2 ⚫ Database of satellite imagery, one satellite collect (strip) materialized as many files (granules) ⚫ Access filtering based on current user roles (custom data store implementation) ⚫ Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store FOSS4G 2018, August 29th/31st, Dar Es Salaam
54. Use Case 3 – Let’s go Deeper! FOSS4G 2018, August 29th/31st, Dar Es Salaam
55. That’s all folks! Questions? info@geo-solutions.it FOSS4G 2018, August 29th/31st, Dar Es Salaam
56. Assorted extras
57. WMS-EO Extending LayerGroup concept Support same style on both raster and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups ⚫ Product layer tree ⚫ Mix of vector and raster info ⚫ Raw data, flags (clouds, snow), browse FOSS4G 2018, August 29th/31st, Dar Es Salaam
58. WMS-EO ⚫ Helper UI to build the tree FOSS4G 2018, August 29th/31st, Dar Es Salaam