Serving earth observation data with GeoServer: addressing real world requirements with Open Source
Information on the latest developments in GeoServer for the support for Earth Observation data with support for the various OGC services and OpenSearch.
Serving EO Data with GeoServer Addressing real-world EO requirements With GeoServer Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
Terminology ⚫ Granule/Tile ⚫ The individual raster element composing the mosaic ⚫ (Granule) Index ⚫ The collection of metadata records describing the location, spatial coverage and other attributes of each single granule FOSS4G 2018, August 29th/31st, Dar Es Salaam
Complete freedom ⚫ Granules can: ⚫ Overlap as they please, can have different resolutions ⚫ Be in different file formats (faster if the format is uniform) ⚫ Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) ⚫ Be in different coordinate reference systems FOSS4G 2018, August 29th/31st, Dar Es Salaam
The mosaic index ⚫ The file lookup system ⚫ A table of granules, their footprint, location, and extra attributes ⚫ Currently supported/tested sources: ⚫ PostGIS, Oracle Shapefile, h (no attr indexing, no timestamp) ⚫ H2 (local embedded db) ⚫ Can be customized to support custom granule indexes (e.g. legacy catalog) FOSS4G 2018, August 29th/31st, Dar Es Salaam
The Dimensions ⚫ Maps to alphanumeric attributes in the index ⚫ TIME and ELEVATION receive special treatment for WMS and WCS ⚫ Custom/Additional dimensions ⚫ Everything besides TIME & ELEVATION ⚫ They can be dynamically discovered from GetCapabilities/DescribeCoverage FOSS4G 2018, August 29th/31st, Dar Es Salaam
Filling the index ⚫ Available options ⚫ Have GeoServer do it for you from the UI (one shot setup) ⚫ Use the REST API to add granules and make it populate the index ⚫ Directly write the index (common for moving window of data) ⚫ Use the OpenSearch for EO extension (later in the presentation) FOSS4G 2018, August 29th/31st, Dar Es Salaam
Filtering and sorting ⚫ The mosaic index can be filtered upon with CQL ⚫ Can be sorted on using WFS like syntax ⚫ More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D FOSS4G 2018, August 29th/31st, Dar Es Salaam
Masking Support ⚫ Mask allow finding valid data ⚫ Alternative to NODATA for lossy compressions ⚫ Vector and raster masks supported ⚫ Raster masks can be embedded (GDAL style) ⚫ Vector ones can be WKB or shapefiles ⚫ API to plug your own mask provider ⚫ Available for image mosaic and GDAL based data sources FOSS4G 2018, August 29th/31st, Dar Es Salaam
Coverage Views ⚫ NetCDF/Grib/Mosaic: one reader, many coverages ⚫ Sometimes the coverages are strictly related ⚫ E.g., U and V components of wind/current vectors ⚫ Merge them back into a single coverage as bands ⚫ Allows rendering engine to work with them both as one FOSS4G 2018, August 29th/31st, Dar Es Salaam
Native multi-coverage support ⚫ Satellite images often comes in one file per band ⚫ Requirement, as they have different resolutions ⚫ Grab the band name from file, build a “band specific” coverage ⚫ Use coverage views to merge them together in a virtual multiband coverage ⚫ Smarts to read only the desired files when doing band selection FOSS4G 2018, August 29th/31st, Dar Es Salaam
Pluggability ⚫ Lots of extension points and pluggable bits ⚫ PropertyCollectors: extract index attributes from granules ⚫ CoverageNameCollector: when you have a hash of files from multiple coverages ⚫ GranuleHandler: pre-process granules before getting them in the mosaic ⚫ GranuleAcceptor: decide if the granule is actually part of the mosaic ⚫ SubMosaicProducer: custom way to mosaic granules ⚫ GranuleCatalog: how you store the index ⚫ See more examples at the end of presentation FOSS4G 2018, August 29th/31st, Dar Es Salaam
NetCDF format support ⚫ NetCDF support ⚫ Support COARDS/CF* conventions loosely ⚫ Expose NetCDF internal data as a set of 2D slices ⚫ Fast 2D (time, elevation) slice extraction ⚫ Supports NCML ⚫ GRIB supported by the same libraries, with same functionality FOSS4G 2018, August 29th/31st, Dar Es Salaam
NetCDF/GRIB Mosaic integration ⚫ ImageMosaic NetCDF and GRIB integration ⚫ Allow the ImageMosaic to handle multiple files ⚫ Expose internal structure (times, elevations) ⚫ Make ImageMosaic handle slices of the file as granules ⚫ 5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N … FOSS4G 2018, August 29th/31st, Dar Es Salaam
Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?... &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z FOSS4G 2018, August 29th/31st, Dar Es Salaam
Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover A FOSS4G 2018, August 29th/31st, Dar Es Salaam
Rendering transformations ⚫ Style based transformations ⚫ Applied on the fly, yet fast enough for interactive use ⚫ Pluggable: add your own contouring FOSS4G 2018, August 29th/31st, Dar Es Salaam
On the fly map algebra with Jiffle nir = src[7]; vir = src[3]; dest = (nir-vir)/ (nir+vir); ⚫ Jiffle: powerful and fast map algebra language ⚫ Can be used as a rendering transformation FOSS4G 2018, August 29th/31st, Dar Es Salaam
NetCDF WCS output ⚫ WCS can output NetCDF for any multidimensional source ⚫ Several customizations to control output and preserve info if the source is a NetCDF itself ⚫ Source has to be multidimensional (current restriction) FOSS4G 2018, August 29th/31st, Dar Es Salaam
Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03- 01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX FOSS4G 2018, August 29th/31st, Dar Es Salaam
WCS EO ⚫ Add support for WCS EO metadata in readers ⚫ Associate each file with EO metadata ⚫ Include such information in DescribeCoverage/DescribeEODataset ⚫ EODataset: multidimensional mosaic or NetCDF file ⚫ Describe the inner structure (granules listing) in DescribeEODataSet FOSS4G 2018, August 29th/31st, Dar Es Salaam
Putting it all together (and finding it) ⚫ Use case ⚫ Lots of satellite imagery ⚫ Different satellites ⚫ Different sensor ⚫ Deep time and space distribution ⚫ How to find the right image? ⚫ By time and space ⚫ By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO ⚫ “OpenSearch for EO“ to the rescue FOSS4G 2018, August 29th/31st, Dar Es Salaam
Two steps search ⚫ First find collection of similar products (by satellite/sensor), e.g. ⚫ Optical or SAR? ⚫ Supported resolutions ⚫ Availability of data in particular timeframe ⚫ Then search products in it, by ⚫ Time ⚫ Space ⚫ Cloud cover ⚫ … FOSS4G 2018, August 29th/31st, Dar Es Salaam
Sample search ⚫ http://cloudsdi. geo-solutions.it/ ⚫ geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] ⚫ Output is RSS with description, thumb/quicklook and links FOSS4G 2018, August 29th/31st, Dar Es Salaam
Metadata and cross linking ⚫ Links to metadata: ⚫ ISO metadata for collection ⚫ GML O&M for products ⚫ Links to OGC services (cross linking) ⚫ WMS/WMTS to see collection/product ⚫ WFS to get “masks” (validity, sea, cloud, snow, …) ⚫ WCS to extract raster ⚫ Direct download to get original package FOSS4G 2018, August 29th/31st, Dar Es Salaam
Tight integration with mosaic possible ⚫ The system creates one “feature type” per collection ⚫ Can be used as the index to a mosaic ⚫ One mosaic per collection ⚫ Filter using CQL and sort by attribute on the WMS/WCS service ⚫ Using all the attributes already searchable by OpenSearch FOSS4G 2018, August 29th/31st, Dar Es Salaam
Use Case 1 – Light Integration ⚫ GeoServer + MapStore + Legacy Ingestion ⚫ Maritime Data ⚫ SAR Time Series ⚫ Oil Spill + other vectors ⚫ Single Mosaic + CQL + Custom Access Manager ⚫ Integrated SSO with MapStore ⚫ Precooked Download of Scenes ⚫ Fine grain control over data access ⚫ Download ⚫ Visualization ⚫ Clever usage of Mosaic Indexes via WFS FOSS4G 2018, August 29th/31st, Dar Es Salaam
Use Case 2 ⚫ Database of satellite imagery, one satellite collect (strip) materialized as many files (granules) ⚫ Access filtering based on current user roles (custom data store implementation) ⚫ Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store FOSS4G 2018, August 29th/31st, Dar Es Salaam
WMS-EO Extending LayerGroup concept Support same style on both raster and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups ⚫ Product layer tree ⚫ Mix of vector and raster info ⚫ Raw data, flags (clouds, snow), browse FOSS4G 2018, August 29th/31st, Dar Es Salaam
simboss
sansantech
yoshiitaka
smt7174
k1low
kenchan
optimisuke
sugawani
yoshidashingo
nagix
mosa_siru
naohachi89
joker1007
aleyda
addyosmani
eileencodes
brad_frost
sachag
tammielis
lauravandoore
mza
geoffreycrofte
myddelton
jasonvnalue