$30 off During Our Annual Pro Sale. View Details »

Serving_earth_observation_data_with_GeoServer__COG__STAC__OpenSearch_and_more.pdf

 Serving_earth_observation_data_with_GeoServer__COG__STAC__OpenSearch_and_more.pdf

Simone Giannecchini
PRO

August 24, 2022
Tweet

More Decks by Simone Giannecchini

Other Decks in Technology

Transcript

  1. Serving earth observation data with GeoServer: COG, STAC, OpenSearch and

    more Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
  2. Affiliations We strongly support Open Source, it Is in our

    core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT
  3. GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions

    GeoNode • Offices in Italy & US, Global Clients/Team • 40+ collaborators, 30+ Engineers • Our products • Our Offer
  4. Reference scenario REP/COP InSitu Obs MetOc Models Remote Sensing

  5. Locating raster sources

  6. OpenSearch for EO (2017) • Use case • Lots of

    satellite imagery • Different satellites • Different sensor • Deep time and space distribution • How to find the right image? • By time and space • By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO • “OpenSearch for EO“ to the rescue
  7. General information schema Collection Product or Item Granule Search Search

    Mosaic/OGC
  8. Two steps search • First find collection of similar products

    (by satellite/sensor), e.g. • Optical or SAR? • Supported resolutions • Availability of data in particular timeframe • Then search products in it, by • Time • Space • Cloud cover • …
  9. Sample search • http://cloudsdi. geo-solutions.it/ • geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] •

    Output is RSS with description, thumb/quicklook and links • Since 2021, output can also be GeoJSON
  10. Metadata and cross linking • Links to metadata: • ISO

    metadata for collection • GML O&M for products • Links to OGC services (cross linking) • WMS/WMTS to see collection/product • WFS to get “masks” (validity, snow, cloud, …) • WCS to extract raster • Direct download to get original package
  11. STAC API (beta 2) - 2021 • Same use case

    as OpenSearch for EO, more modern take on architecture • Collections and items • Allows cross collection search. • Based on OGC API, compatible with OGC API - Features
  12. STAC API (beta 2) • Screenshots from DLR's EOC Geoservice

    (STAC support in development / Beta) • Customized database and templates Landing page
  13. STAC API (beta 2) Collections

  14. STAC API (beta 2) Single collection

  15. STAC API (beta 2) Items listing

  16. STAC API (beta 2) Single item

  17. Similar semantic, different names • We have 4 information models

    ◦ Database ◦ OpenSearch EO ▪ GeoRSS ▪ GeoJSON ◦ STAC • They express the same information with different names OpenSearch EO GeoRSS OpenSearch EO GeoJSON STAC collections/items Database
  18. Features templating to the rescue • “Features templating” community module

    to the rescue: ◦ The right “naming” is generated based on the output ◦ Public names are back-mapped to db names for filtering Database STAC Collections Items OpenSearch EO Search Templates
  19. Templating, JSON example { "type": "FeatureCollection", "features": [ { "stac_version":

    "$${env('stac_version', '1.0.0-rc2')}", "stac_extensions": [ // skip SAR sensors "$filter{eop:parentIdentifier LIKE 'SENTINEL%' or eop:parentIdentifier LIKE 'LANDSAT%'},https://stac-extensions.github.io/eo/v1.0.0/schema.json", "https://stac-extensions.github.io/view/v1.0.0/schema.json" ], "id": "${eop:identifier}", "type": "Feature", "geometry": "${footprint}", "bbox": [ "$${minx(footprint)}", "$${miny(footprint)}", "$${maxx(footprint)}", "$${maxy(footprint)}" ], "properties": { "start_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <> timeEnd},${timeStart}", "end_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <> timeEnd},${timeEnd}", "datetime": "$filter{timeStart is not null and (timeEnd is null or timeStart = timeEnd)},${timeStart}", ....
  20. REST API to manage search index Swagger documentation

  21. Accessing raster data

  22. Finding rasters • STAC has “assets” in Item pointing to

    the imagery files • OpenSearch can point to the raw package, and the “granules” table allow to get to the single images • Where can they be located? ◦ Local file system (classic approach) ◦ HTTP server ◦ S3 or other blob storage
  23. Reading from blobstore • Presently two choices: ◦ Old S3

    GeoTiff plugin (Boundless) ◦ COG plugin (Planet/GeoSolutions) • Old S3 plugin available for longer time, used by some production sites, but not maintained • COG plugin is new, and the way forward
  24. Old S3 GeoTIFF plugin 5MB • Split file in 5MB

    blocks (configurable) • Read the full block even a small part of it is used • Eventual local memory/disk cache • Works best for smaller files and strong locality • Can read any GeoTIFF 5MB 5MB 5MB
  25. COG plugin Hea der • Leverages COG structure • Reads

    header first • HTTP Range read of the tiles needed (coaleshe nearby tiles into single read) • Works against S3 and HTTP(s) • More blob storages incoming (Google Storage) • May get caching down the way Tile Tile Tile Tile Tile Tile R1 R2
  26. Mosaicking rasters

  27. Terminology • Granule/Tile • The individual raster element composing the

    mosaic • (Granule) Index • The collection of metadata records describing the location, spatial coverage and other attributes of each single granule
  28. Complete freedom • Granules can: • Overlap as they please,

    can have different resolutions • Be in different file formats (faster if the format is uniform) • Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) • Be in different coordinate reference systems
  29. The mosaic index • The file lookup system • A

    table of granules, their footprint, location, and extra attributes • Currently supported/tested sources: • PostGIS, Oracle Shapefile, h (no attr indexing, no timestamp) • H2 (local embedded db) • Can be customized to support custom granule indexes (e.g. legacy catalog)
  30. The Dimensions • Maps to alphanumeric attributes in the index

    • TIME and ELEVATION receive special treatment for WMS and WCS • Custom/Additional dimensions • Everything besides TIME & ELEVATION • They can be dynamically discovered from GetCapabilities/DescribeCoverage
  31. STAC/OpenSearch as an index • The OpenSearch/STAC database can be

    used as a mosaic index • SQL Views selecting a particular collection sub/set • REST API to automatically create them Collection Product or Item Granule
  32. External STAC API as index • Experimental modules on GitHub

    to create a mosaic out of a STAC API • Filter to establish which items take part • https://github.com/joshfix/imagemosaic-stac • Outdated code, but promising concept • Needs STAC API to settle down (beta/changes in filtering) STAC API ImageMosaic STAC GeoTools ImageMosaic
  33. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html •

    http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html • http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml • GeoSolutions training: • http://geoserver.geo- solutions.it/edu/en/multidim/imagemosaic/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/rest/index.html
  34. ImageMosaic – the fun stuff

  35. Filtering and sorting • The mosaic index can be filtered

    upon with CQL • Can be sorted on using WFS like syntax • More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D
  36. Masking Support • Mask allow finding valid data • Alternative

    to NODATA for lossy compressions • Vector and raster masks supported • Raster masks can be embedded (GDAL style) • Vector ones can be WKB or shapefiles • API to plug your own mask provider • Available for image mosaic and GDAL based data sources
  37. Coverage Views • NetCDF/Grib/Mosaic: one reader, many coverages • Sometimes

    the coverages are strictly related • E.g., U and V components of wind/current vectors • Merge them back into a single coverage as bands • Allows rendering engine to work with them both as one
  38. Native multi-coverage support • Satellite images often comes in one

    file per band • Requirement, as they have different resolutions • Grab the band name from file, build a “band specific” coverage • Use coverage views to merge them together in a virtual multiband coverage • Smarts to read only the desired files when doing band selection
  39. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html •

    http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html • http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html • GeoSolutions training: • https://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/coveragev iew.html
  40. The services - visualize

  41. WMS-T support

  42. Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?...

    &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z
  43. Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

  44. Rendering transformations • Style based transformations • Applied on the

    fly, yet fast enough for interactive use • Pluggable: add your own contouring
  45. Rendering transformations examples windbarbs currents

  46. On the fly map algebra with Jiffle nir = src[7];

    vir = src[3]; dest = (nir-vir)/ (nir+vir); • Jiffle: powerful and fast map algebra language • Can be used as a rendering transformation
  47. Documentation links • GeoServer documentation • http://docs.geoserver.org/latest/en/user/services/wm s/time.html • http://docs.geoserver.org/stable/en/user/data/webad

    min/layers.html#edit-layer-dimensions • http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html • GeoSolutions training: • https://geoserver.geo- solutions.it/edu/en/wps/rendering_tx.html • Jiffle documentation: • https://github.com/geosolutions-it/jai-ext/wiki/Jiffle
  48. Download via WCS

  49. WCS 2.0 • Basics • Core service • KVP binding

    • XML binding • Common extensions • CRS • Scaling • Interpolation • Range subsetting • GeoTiff • GML • NetCDF • Add the output format extensions • GeoTIFF • GML Grid • NetCDF
  50. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION

    details as extension CUSTOM
  51. WCS – custom filtering and sorting request=GetCoverage&... &CQL_FILTER=platform=SENTINEL2 &sortBy=time D

    &sortBy=cloudCover
  52. Download via WPS

  53. Asynchronous download • WCS 2.0 is a fine protocol •

    But it’s synchronous • Not well suited for large downloads • WPS execute can be asynchronous • WPS download plugin • Dedicated to large extractions
  54. WPS download - large raw download 25GB GeoTIFF 2 meters

    res Some time later
  55. WPS download - large rendered map 10GB NDVI GeoTIFF 2

    meters res Some time later
  56. WPS download - long animation 3GB .MP4 file Some time

    later
  57. That’s all folks! Questions? info@geo-solutions.it

  58. Want more? Bonus slides!!

  59. NetCDF and GRIB

  60. NetCDF format support • NetCDF support • Support COARDS/CF* conventions

    loosely • Expose NetCDF internal data as a set of 2D slices • Fast 2D (time, elevation) slice extraction • Supports NCML • GRIB supported by the same libraries, with same functionality
  61. One or more variables • Polyphemus Sample Dataset • 1

    File 🡪 Multiple Coverages!
  62. Multiple coverages per mosaic

  63. NetCDF/GRIB Mosaic integration • ImageMosaic NetCDF and GRIB integration •

    Allow the ImageMosaic to handle multiple files • Expose internal structure (times, elevations) • Make ImageMosaic handle slices of the file as granules • 5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N …
  64. NetCDF WCS output • WCS can output NetCDF for any

    multidimensional source • Several customizations to control output and preserve info if the source is a NetCDF itself • Source has to be multidimensional (current restriction)
  65. Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03-

    01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX
  66. Documentation links • GeoServer documentation • http://docs.geoserver.org/latest/en/user/extensions/n etcdf/netcdf.html • http://docs.geoserver.org/latest/en/user/extensio

    ns/netcdf-out/index.html • GeoSolutions training: • http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/mosaic_config/netc df_mosaic.html
  67. EO extensions

  68. WMS-EO Extending LayerGroup concept Support same style on both raster

    and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups • Product layer tree • Mix of vector and raster info • Raw data, flags (clouds, snow), browse
  69. WMS-EO • Helper UI to build the tree

  70. WCS EO • Add support for WCS EO metadata in

    readers • Associate each file with EO metadata • Include such information in DescribeCoverage/DescribeEODataset • EODataset: multidimensional mosaic or NetCDF file • Describe the inner structure (granules listing) in DescribeEODataSet
  71. Real World Use cases

  72. Use Case 1 – Light Integration • GeoServer + MapStore

    + Legacy Ingestion • Maritime Data • SAR Time Series • Oil Spill + other vectors • Single Mosaic + CQL + Custom Access Manager • Integrated SSO with MapStore • Precooked Download of Scenes • Fine grain control over data access • Download • Visualization • Clever usage of Mosaic Indexes via WFS
  73. Use Case 1 – Light Integration

  74. Use Case 1 – Light Integration

  75. Use Case 1 – Light Integration

  76. Use Case 2 – Deep Integration • High Res Imagery

    • SSO with CAS • Legacy Ingestion
  77. Use Case 2 • Database of satellite imagery, one satellite

    collect (strip) materialized as many files (granules) • Access filtering based on current user roles (custom data store implementation) • Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store
  78. Use Case 3 – Let’s go Deeper!