Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more

Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more

Never before have we had such a rich collection of satellite imagery available to both companies and the general public. Between missions such as Landsat 8 and Sentinels and the explosion of cubesats, as well as the free availability of worldwide data from the European Copernicus program and from Drones, a veritable flood of data is made available for everyday usage.
Managing, locating and displaying such a large volume of satellite images can be challenging. Join this presentation to learn how GeoServer can help with with that job, with real world examples, including:

- Indexing and locating images using The OpenSearch for EO and STAC protocols
- Managing large volumes of satellite images, in an efficient and cost effective way, using Cloud Optimized GeoTIFFs.
- Visualize mosaics of images, creating composite with the right set of views (filtering), in the desired stacking order (color on top, most recent on top, less cloudy on top, your choice)
- Perform both small and large extractions of imagery using the WCS and WPS protocols
- Generate and view time based animations of the above mosaics, in a period of interest
- Perform band algebra operations using Jiffle

This talk will give a good update on the latest GeoServer capabilities in the Earth Observation field.

Simone Giannecchini

November 09, 2023
Tweet

More Decks by Simone Giannecchini

Other Decks in Technology

Transcript

  1. Serving earth observation data
    with GeoServer:
    COG, STAC, OpenSearch
    and more
    Ing. Simone Giannecchini
    Ing. Daniele Romagnoli
    Ing. Andrea Aime
    GeoSolutions

    View full-size slide

  2. GeoSolutions
    Enterprise Support
    Services
    Deployment
    Subscription
    Professional
    Training
    Customized
    Solutions
    GeoNode
    • Offices in Italy & US, Global Clients/Team
    • 30+ collaborators, 25+ Engineers
    • Our products
    • Our Offer

    View full-size slide

  3. Affiliations
    We strongly support Open
    Source, it Is in our core
    We actively participate in
    OGC working groups and
    get funded to advance new
    open standards
    We support standards
    critical to GEOINT

    View full-size slide

  4. Reference scenario
    REP/COP
    InSitu
    Obs
    MetOc
    Models
    Remote
    Sensing

    View full-size slide

  5. Locating raster sources

    View full-size slide

  6. STAC API (RC1)
    ● Same use case as
    OpenSearch for EO
    ● Based on OGC API,
    JSON based
    ● Collections and items
    ● Cross collection
    search at /search
    ○ filter, sort, field
    selection

    View full-size slide

  7. General information schema
    Collection
    Product or
    Item
    Asset
    Search
    Search
    Mosaic/OGC

    View full-size slide

  8. STAC API (RC1)
    ● Screenshots from DLR's EOC Geoservice (STAC
    support in development / Beta)
    ● Customized database, HTML and JSON templates
    Landing page

    View full-size slide

  9. STAC API (RC1) - Collections
    Collections

    View full-size slide

  10. STAC API (RC1) - Collection

    View full-size slide

  11. STAC API (RC1) - items

    View full-size slide

  12. STAC API (RC1) - item

    View full-size slide

  13. OpenSearch for EO (2017)

    Earlier OGC specification
    to search for satellite
    imagery

    Based on OpenSearch
    and OpenSearch Geo
    and Time
    OpenSearch
    OGC OpenSearch Geo and
    Time
    OGC OpenSearch for EO

    View full-size slide

  14. Sample search

    http://cloudsdi.
    geo-solutions.it/

    geoserver/oseo/search?
    parentId=SENTINEL2
    &cloudCover=30]

    Output is RSS with
    description,
    thumb/quicklook and
    links
    ● Since 2021, output can
    also be GeoJSON

    View full-size slide

  15. Metadata and cross linking

    Links to metadata:

    ISO metadata for collection

    GML O&M for products

    Links to OGC services (cross linking)

    WMS/WMTS to see collection/product

    WFS to get “masks” (validity, snow, cloud, …)

    WCS to extract raster

    Direct download to get original package

    View full-size slide

  16. REST API to manage search index
    Swagger documentation

    View full-size slide

  17. Accessing raster data

    View full-size slide

  18. Finding rasters
    ● STAC has “assets” in Item pointing to the
    imagery files
    ● OpenSearch can point to the raw package
    ● Where can they be located?
    ○ Local file system (classic approach)
    ○ HTTP server
    ○ S3 or other blob storage

    View full-size slide

  19. Reading from blobstore
    ● COG plugin
    ○ Leverages COG structure
    ● But also, old S3 GeoTIFF plugin (Boundless)
    ○ Reads any GeoTIFF (non COG) from S3
    ○ Reads in 5MB blocks, caches them
    ○ Deprecated soon

    View full-size slide

  20. COG plugin
    Hea
    der
    ● Leverages COG structure
    ● Reads header first
    ● HTTP Range read of the tiles needed (coalesce
    nearby tiles into single read)
    ● Works against HTTP(s), S3, Azure and Google
    Storage
    ● May get caching down the way (interested?)
    Tile Tile Tile Tile Tile Tile
    R1 R2

    View full-size slide

  21. Mosaicking rasters

    View full-size slide

  22. Mosaic Terminology

    Granule/Tile

    The individual raster element composing the
    mosaic (asset in STAC)

    (Granule) Index

    The collection of metadata records describing
    the location, spatial coverage and other
    attributes of each single granule

    View full-size slide

  23. Complete freedom

    Granules can:

    Overlap as they please, can have different
    resolutions

    Be in different file formats (faster if the format is
    uniform)

    Have different color models, RGB, gray, paletted
    (needs JAI-EXT enabled)

    Be in different coordinate reference systems

    View full-size slide

  24. The mosaic index

    The granule lookup system

    A “table” of granules, their footprint, location,
    and extra attributes

    Currently supported/tested sources:

    PostGIS, Oracle, SQL Server, Shapefile

    STAC API (local or remote)

    Can be customized to support custom
    granule indexes (e.g. legacy catalog)

    View full-size slide

  25. The Dimensions

    Maps to alphanumeric attributes in the index

    TIME and ELEVATION receive special
    treatment for WMS and WCS

    Custom/Additional dimensions

    Everything besides TIME & ELEVATION

    They can be dynamically discovered
    from GetCapabilities/DescribeCoverage

    View full-size slide

  26. Internal STAC as an index
    ・ The OpenSearch/STAC database can be
    used as a mosaic index
    ・ SQL Views selecting a particular collection
    sub/set
    ・ REST API to automatically create them
    Collection
    Product or
    Item
    Granule

    View full-size slide

  27. External STAC API as index
    ・ Community module for STAC API store
    ・ Integrates well with image mosaic
    ・ Tricky issues: each system uses different
    auth mechanisms to
    actually download data
    STAC
    API
    STAC
    store
    GeoTools
    ImageMosaic
    COG

    View full-size slide

  28. Documentation links

    GeoServer reference docs:

    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/index.html

    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/tutorial.html

    http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y
    aml

    GeoSolutions training:

    http://geoserver.geo-solutions.it/edu/en/multidim/imagemo
    saic/index.html

    http://geoserver.geo-solutions.it/edu/en/multidim/accessing
    _multidim/index.html

    http://geoserver.geo-solutions.it/edu/en/multidim/rest/index
    .html

    View full-size slide

  29. ImageMosaic – the fun stuff

    View full-size slide

  30. Filtering and sorting

    The mosaic index can be filtered upon with CQL

    Can be sorted on using WFS like syntax

    More possibilities than just Time and Elevation
    SENSOR = SAR AND SATELLITE=XYZ
    SORTBY = TIME D

    View full-size slide

  31. Coverage Views

    NetCDF/Grib/Mosaic: one reader, many coverages

    Sometimes the coverages are strictly related

    E.g., U and V components of wind/current vectors

    Merge them back into a single coverage as bands

    Allows rendering engine to work with them both as one

    View full-size slide

  32. Native multi-coverage support

    Satellite images often
    comes in one file per band

    Requirement, as they have
    different resolutions

    Grab the band name from
    file, build a “band specific”
    coverage

    Use coverage views to
    merge them together in a
    virtual multiband coverage

    Smarts to read only the
    desired files when doing
    band selection

    View full-size slide

  33. Hyperspectral imagery support
    ● Hyperspectral images:
    those having hundreds of
    bands
    ● Typically stored in
    band-interleaved structure
    (one data bank for all pixels
    in the same tile)
    Rows
    Cols
    Bands

    View full-size slide

  34. Documentation links

    GeoServer reference docs:

    http://docs.geoserver.org/latest/en/user/data/raster/image
    mosaic/index.html

    http://docs.geoserver.org/latest/en/user/data/raster/covera
    geview.html

    http://docs.geoserver.org/latest/en/user/tutorials/imagemos
    aic_footprint/imagemosaic_footprint.html

    GeoSolutions training:

    https://geoserver.geo-solutions.it/edu/en/multidim/accessin
    g_multidim/coverageview.html

    View full-size slide

  35. The services - visualize

    View full-size slide

  36. WMS-T (time) support

    View full-size slide

  37. Filtering via WMS-T support
    TIME, ELEVATION & Custom dimensions
    http://localhost:8080/geoserver/geosolutions/wms?...
    &time=2013-03-1T00:00:00.000Z
    &elevation=35.0
    &DIM_FILEDATE=2013-03-01T00:00:00.000Z
    &DIM_UPDATED=2013-04-08T08:18:41.597Z

    View full-size slide

  38. Filtering and sorting via vendor options
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover

    View full-size slide

  39. Rendering transformations

    Style based transformations

    Applied on the fly, yet fast enough for
    interactive use

    Pluggable: add your own
    contouring

    View full-size slide

  40. Rendering transformations examples
    windbarbs
    currents

    View full-size slide

  41. On the fly map algebra with Jiffle
    nir = src[7];
    vir = src[3];
    dest = (nir-vir)/
    (nir+vir);

    Jiffle: powerful and fast
    map algebra language

    Can be used as a
    rendering transformation

    View full-size slide

  42. Documentation links

    GeoServer documentation

    http://docs.geoserver.org/latest/en/user/services/wms
    /time.html

    http://docs.geoserver.org/stable/en/user/data/webad
    min/layers.html#edit-layer-dimensions

    http://docs.geoserver.org/latest/en/user/styling/sld/ex
    tensions/rendering-transform.html

    GeoSolutions training:

    https://geoserver.geo-solutions.it/edu/en/wps/renderi
    ng_tx.html

    Jiffle documentation:

    https://github.com/geosolutions-it/jai-ext/wiki/Jiffle

    View full-size slide

  43. Download via WCS

    View full-size slide

  44. WCS 2.0

    Basics

    Core service

    KVP binding

    XML binding

    Common extensions

    CRS

    Scaling

    Interpolation

    Range subsetting

    GeoTiff

    GML

    NetCDF

    Add the output format extensions

    GeoTIFF

    GML Grid

    NetCDF

    View full-size slide

  45. WCS 2.0 DescribeCoverage
    4D BBOX
    TIME details
    as extension
    ELEVATION
    details as
    extension
    CUSTOM

    View full-size slide

  46. WCS – custom filtering and sorting
    request=GetCoverage&...
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover

    View full-size slide

  47. Download via WPS

    View full-size slide

  48. Asynchronous download
    ● WCS 2.0 is a fine protocol
    ● But it’s synchronous
    ● Not well suited for large downloads
    ● WPS execute can be
    asynchronous
    ● WPS download
    plugin
    ● Dedicated to large
    extractions

    View full-size slide

  49. WPS download - large raw download
    25GB
    GeoTIFF
    2 meters res
    Some time later

    View full-size slide

  50. WPS download - large rendered map
    10GB
    NDVI
    GeoTIFF
    2 meters res
    Some time later

    View full-size slide

  51. WPS download - long animation
    3GB
    .MP4 file
    Some time later

    View full-size slide

  52. That’s all folks!
    Questions?
    [email protected]

    View full-size slide