$30 off During Our Annual Pro Sale. View Details »

Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more (FOSS4G 2022 Edition)

Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more (FOSS4G 2022 Edition)

Never before have we had such a rich collection of satellite imagery available to both companies and the general public. Between missions such as Landsat 8 and Sentinels and the explosion of cubesats, as well as the free availability of worldwide data from the European Copernicus program and from Drones, a veritable flood of data is made available for everyday usage.
Managing, locating and displaying such a large volume of satellite images can be challenging. Join this presentation to learn how GeoServer can help with with that job, with real world examples, including:
- Indexing and locating images using The OpenSearch for EO and STAC protocols
- Managing large volumes of satellite images, in an efficient and cost effective way, using Cloud Optimized GeoTIFFs.
- Visualize mosaics of images, creating composite with the right set of views (filtering), in the desired stacking order (color on top, most recent on top, less cloudy on top, your choice)
- Perform both small and large extractions of imagery using the WCS and WPS protocols
- Generate and view time based animations of the above mosaics, in a period of interest
- Perform band algebra operations using Jiffle
Attend this talk to get a good update on the latest GeoServer capabilities in the Earth Observation field.

Simone Giannecchini
PRO

August 31, 2022
Tweet

More Decks by Simone Giannecchini

Other Decks in Technology

Transcript

  1. Serving earth observation data with GeoServer: COG, STAC, OpenSearch and

    more Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
  2. GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions

    GeoNode • Offices in Italy & US, Global Clients/Team • 40+ collaborators, 30+ Engineers • Our products • Our Offer
  3. Affiliations We strongly support Open Source, it Is in our

    core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT
  4. Reference scenario REP/COP InSitu Obs MetOc Models Remote Sensing

  5. Locating raster sources

  6. OpenSearch for EO (2017) • Use case • Lots of

    satellite imagery • Different satellites • Different sensor • Deep time and space distribution • How to find the right image? • By time and space • By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO • “OpenSearch for EO“ to the rescue
  7. General information schema Collection Product or Item Granule Search Search

    Mosaic/OGC
  8. Two steps search • First find collection of similar products

    (by satellite/sensor), e.g. • Optical or SAR? • Supported resolutions • Availability of data in particular timeframe • Then search products in it, by • Time • Space • Cloud cover • …
  9. Sample search • http://cloudsdi. geo-solutions.it/ • geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] •

    Output is RSS with description, thumb/quicklook and links • Since 2021, output can also be GeoJSON
  10. Metadata and cross linking • Links to metadata: • ISO

    metadata for collection • GML O&M for products • Links to OGC services (cross linking) • WMS/WMTS to see collection/product • WFS to get “masks” (validity, snow, cloud, …) • WCS to extract raster • Direct download to get original package
  11. STAC API (RC1) • Same use case as OpenSearch for

    EO • Based on OGC API, JSON based • Collections and items • Cross collection search at /search ◦ filter, sort, field selection
  12. STAC API (RC1) • Screenshots from DLR's EOC Geoservice (STAC

    support in development / Beta) • Customized database, HTML and JSON templates Landing page
  13. STAC API (RC1) - Collections Collections

  14. STAC API (RC1) - Collection

  15. STAC API (RC1) - items

  16. STAC API (RC1) - item

  17. REST API to manage search index Swagger documentation

  18. Accessing raster data

  19. Finding rasters • STAC has “assets” in Item pointing to

    the imagery files • OpenSearch can point to the raw package, and the “granules” table allow to get to the single images • Where can they be located? ◦ Local file system (classic approach) ◦ HTTP server ◦ S3 or other blob storage
  20. Reading from blobstore • COG plugin ◦ Leverages COG structure

    • But also, old S3 GeoTIFF plugin (Boundless) ◦ Reads any GeoTIFF (non COG) from S3 ◦ Reads in 5MB blocks, caches them ◦ Deprecated soon
  21. COG plugin Hea der • Leverages COG structure • Reads

    header first • HTTP Range read of the tiles needed (coaleshe nearby tiles into single read) • Works against S3, HTTP(s) and Google Storage • More blob storages incoming (Azure blob) • May get caching down the way (interested?) Tile Tile Tile Tile Tile Tile R1 R2
  22. Mosaicking rasters

  23. Terminology • Granule/Tile • The individual raster element composing the

    mosaic • (Granule) Index • The collection of metadata records describing the location, spatial coverage and other attributes of each single granule
  24. Complete freedom • Granules can: • Overlap as they please,

    can have different resolutions • Be in different file formats (faster if the format is uniform) • Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) • Be in different coordinate reference systems
  25. The mosaic index • The file lookup system • A

    table of granules, their footprint, location, and extra attributes • Currently supported/tested sources: • PostGIS, Oracle, SQL Server • Shapefile (no attr indexing, no timestamp) • Other GeoTools supported stores • Can be customized to support custom granule indexes (e.g. legacy catalog)
  26. The Dimensions • Maps to alphanumeric attributes in the index

    • TIME and ELEVATION receive special treatment for WMS and WCS • Custom/Additional dimensions • Everything besides TIME & ELEVATION • They can be dynamically discovered from GetCapabilities/DescribeCoverage
  27. STAC/OpenSearch as an index ・ The OpenSearch/STAC database can be

    used as a mosaic index ・ SQL Views selecting a particular collection sub/set ・ REST API to automatically create them Collection Product or Item Granule
  28. External STAC API as index ・ Community module for STAC

    API store landing just landed on 2.22.x ・ Adapting mosaic code to use STAC items as we speak STAC API STAC store GeoTools ImageMosaic COG
  29. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html •

    http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html • http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml • GeoSolutions training: • http://geoserver.geo-solutions.it/edu/en/multidim/imagemo saic/index.html • http://geoserver.geo-solutions.it/edu/en/multidim/accessing _multidim/index.html • http://geoserver.geo-solutions.it/edu/en/multidim/rest/index .html
  30. ImageMosaic – the fun stuff

  31. Filtering and sorting • The mosaic index can be filtered

    upon with CQL • Can be sorted on using WFS like syntax • More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D
  32. Coverage Views • NetCDF/Grib/Mosaic: one reader, many coverages • Sometimes

    the coverages are strictly related • E.g., U and V components of wind/current vectors • Merge them back into a single coverage as bands • Allows rendering engine to work with them both as one
  33. Native multi-coverage support • Satellite images often comes in one

    file per band • Requirement, as they have different resolutions • Grab the band name from file, build a “band specific” coverage • Use coverage views to merge them together in a virtual multiband coverage • Smarts to read only the desired files when doing band selection
  34. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html •

    http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html • http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html • GeoSolutions training: • https://geoserver.geo-solutions.it/edu/en/multidim/accessin g_multidim/coverageview.html
  35. The services - visualize

  36. WMS-T (time) support

  37. Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?...

    &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z
  38. Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

  39. Rendering transformations • Style based transformations • Applied on the

    fly, yet fast enough for interactive use • Pluggable: add your own contouring
  40. Rendering transformations examples windbarbs currents

  41. On the fly map algebra with Jiffle nir = src[7];

    vir = src[3]; dest = (nir-vir)/ (nir+vir); • Jiffle: powerful and fast map algebra language • Can be used as a rendering transformation
  42. Documentation links • GeoServer documentation • http://docs.geoserver.org/latest/en/user/services/wms /time.html • http://docs.geoserver.org/stable/en/user/data/webad

    min/layers.html#edit-layer-dimensions • http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html • GeoSolutions training: • https://geoserver.geo-solutions.it/edu/en/wps/renderi ng_tx.html • Jiffle documentation: • https://github.com/geosolutions-it/jai-ext/wiki/Jiffle
  43. Download via WCS

  44. WCS 2.0 • Basics • Core service • KVP binding

    • XML binding • Common extensions • CRS • Scaling • Interpolation • Range subsetting • GeoTiff • GML • NetCDF • Add the output format extensions • GeoTIFF • GML Grid • NetCDF
  45. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION

    details as extension CUSTOM
  46. WCS – custom filtering and sorting request=GetCoverage&... &CQL_FILTER=platform=SENTINEL2 &sortBy=time D

    &sortBy=cloudCover
  47. Download via WPS

  48. Asynchronous download • WCS 2.0 is a fine protocol •

    But it’s synchronous • Not well suited for large downloads • WPS execute can be asynchronous • WPS download plugin • Dedicated to large extractions
  49. WPS download - large raw download 25GB GeoTIFF 2 meters

    res Some time later
  50. WPS download - large rendered map 10GB NDVI GeoTIFF 2

    meters res Some time later
  51. WPS download - long animation 3GB .MP4 file Some time

    later
  52. That’s all folks! Questions? info@geo-solutions.it