Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serving_earth_observation_data_with_GeoServer__COG__STAC__OpenSearch_and_more.pdf

 Serving_earth_observation_data_with_GeoServer__COG__STAC__OpenSearch_and_more.pdf

Simone Giannecchini
PRO

August 24, 2022
Tweet

More Decks by Simone Giannecchini

Other Decks in Technology

Transcript

  1. Serving earth observation data
    with GeoServer:
    COG, STAC, OpenSearch
    and more
    Ing. Simone Giannecchini
    Ing. Daniele Romagnoli
    Ing. Andrea Aime
    GeoSolutions

    View Slide

  2. Affiliations
    We strongly support Open
    Source, it Is in our core
    We actively participate in OGC
    working groups and get funded
    to advance new open
    standards
    We support standards critical to
    GEOINT

    View Slide

  3. GeoSolutions
    Enterprise Support
    Services
    Deployment
    Subscription
    Professional
    Training
    Customized
    Solutions
    GeoNode
    • Offices in Italy & US, Global Clients/Team
    • 40+ collaborators, 30+ Engineers
    • Our products
    • Our Offer

    View Slide

  4. Reference scenario
    REP/COP
    InSitu
    Obs
    MetOc
    Models
    Remote
    Sensing

    View Slide

  5. Locating raster sources

    View Slide

  6. OpenSearch for EO (2017)

    Use case

    Lots of satellite imagery

    Different satellites

    Different sensor

    Deep time and space
    distribution

    How to find the right
    image?

    By time and space

    By attributes (cloud cover,
    snow cover, off nadir, ….)
    OpenSearch
    OGC OpenSearch Geo and
    Time
    OGC OpenSearch for EO

    “OpenSearch for EO“ to the rescue

    View Slide

  7. General information schema
    Collection
    Product or
    Item
    Granule
    Search
    Search
    Mosaic/OGC

    View Slide

  8. Two steps search

    First find collection of
    similar products (by
    satellite/sensor), e.g.

    Optical or SAR?

    Supported resolutions

    Availability of data in
    particular timeframe

    Then search products in it,
    by

    Time

    Space

    Cloud cover


    View Slide

  9. Sample search

    http://cloudsdi.
    geo-solutions.it/

    geoserver/oseo/search?
    parentId=SENTINEL2
    &cloudCover=30]

    Output is RSS with
    description,
    thumb/quicklook and
    links
    ● Since 2021, output can
    also be GeoJSON

    View Slide

  10. Metadata and cross linking

    Links to metadata:

    ISO metadata for collection

    GML O&M for products

    Links to OGC services (cross linking)

    WMS/WMTS to see collection/product

    WFS to get “masks” (validity, snow, cloud, …)

    WCS to extract raster

    Direct download to get original package

    View Slide

  11. STAC API (beta 2) - 2021
    ● Same use case as
    OpenSearch for EO,
    more modern take on
    architecture
    ● Collections and items
    ● Allows cross
    collection search.
    ● Based on OGC API,
    compatible with
    OGC API - Features

    View Slide

  12. STAC API (beta 2)
    ● Screenshots from DLR's EOC Geoservice (STAC
    support in development / Beta)
    ● Customized database and templates
    Landing page

    View Slide

  13. STAC API (beta 2)
    Collections

    View Slide

  14. STAC API (beta 2)
    Single collection

    View Slide

  15. STAC API (beta 2)
    Items listing

    View Slide

  16. STAC API (beta 2)
    Single item

    View Slide

  17. Similar semantic, different names
    ● We have 4 information models
    ○ Database
    ○ OpenSearch EO
    ■ GeoRSS
    ■ GeoJSON
    ○ STAC
    ● They express the same information with
    different names
    OpenSearch EO
    GeoRSS
    OpenSearch EO
    GeoJSON
    STAC
    collections/items
    Database

    View Slide

  18. Features templating to the rescue
    ● “Features templating” community module to
    the rescue:
    ○ The right “naming” is generated based on
    the output
    ○ Public names are back-mapped to db
    names for filtering
    Database
    STAC
    Collections
    Items
    OpenSearch
    EO Search
    Templates

    View Slide

  19. Templating, JSON example
    {
    "type": "FeatureCollection",
    "features": [
    {
    "stac_version": "$${env('stac_version', '1.0.0-rc2')}",
    "stac_extensions": [
    // skip SAR sensors
    "$filter{eop:parentIdentifier LIKE 'SENTINEL%' or eop:parentIdentifier LIKE
    'LANDSAT%'},https://stac-extensions.github.io/eo/v1.0.0/schema.json",
    "https://stac-extensions.github.io/view/v1.0.0/schema.json"
    ],
    "id": "${eop:identifier}",
    "type": "Feature",
    "geometry": "${footprint}",
    "bbox": [
    "$${minx(footprint)}",
    "$${miny(footprint)}",
    "$${maxx(footprint)}",
    "$${maxy(footprint)}"
    ],
    "properties": {
    "start_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <>
    timeEnd},${timeStart}",
    "end_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <>
    timeEnd},${timeEnd}",
    "datetime": "$filter{timeStart is not null and (timeEnd is null or timeStart =
    timeEnd)},${timeStart}",
    ....

    View Slide

  20. REST API to manage search index
    Swagger documentation

    View Slide

  21. Accessing raster data

    View Slide

  22. Finding rasters
    ● STAC has “assets” in Item pointing to the
    imagery files
    ● OpenSearch can point to the raw package,
    and the “granules” table allow to get to the
    single images
    ● Where can they be located?
    ○ Local file system (classic approach)
    ○ HTTP server
    ○ S3 or other blob storage

    View Slide

  23. Reading from blobstore
    ● Presently two choices:
    ○ Old S3 GeoTiff plugin (Boundless)
    ○ COG plugin (Planet/GeoSolutions)
    ● Old S3 plugin available for longer time, used
    by some production sites, but not
    maintained
    ● COG plugin is new, and the way forward

    View Slide

  24. Old S3 GeoTIFF plugin
    5MB
    ● Split file in 5MB blocks (configurable)
    ● Read the full block even a small part of it is
    used
    ● Eventual local memory/disk cache
    ● Works best for smaller files and strong
    locality
    ● Can read any GeoTIFF
    5MB 5MB 5MB

    View Slide

  25. COG plugin
    Hea
    der
    ● Leverages COG structure
    ● Reads header first
    ● HTTP Range read of the tiles needed (coaleshe
    nearby tiles into single read)
    ● Works against S3 and HTTP(s)
    ● More blob storages incoming (Google Storage)
    ● May get caching down the way
    Tile Tile Tile Tile Tile Tile
    R1 R2

    View Slide

  26. Mosaicking rasters

    View Slide

  27. Terminology

    Granule/Tile

    The individual raster element composing the
    mosaic

    (Granule) Index

    The collection of metadata records describing
    the location, spatial coverage and other
    attributes of each single granule

    View Slide

  28. Complete freedom

    Granules can:

    Overlap as they please, can have different
    resolutions

    Be in different file formats (faster if the format is
    uniform)

    Have different color models, RGB, gray, paletted
    (needs JAI-EXT enabled)

    Be in different coordinate reference systems

    View Slide

  29. The mosaic index

    The file lookup system

    A table of granules, their footprint, location,
    and extra attributes

    Currently supported/tested sources:

    PostGIS, Oracle Shapefile, h (no attr
    indexing, no timestamp)

    H2 (local embedded db)

    Can be customized to support custom
    granule indexes (e.g. legacy catalog)

    View Slide

  30. The Dimensions

    Maps to alphanumeric attributes in the index

    TIME and ELEVATION receive special
    treatment for WMS and WCS

    Custom/Additional dimensions

    Everything besides TIME & ELEVATION

    They can be dynamically discovered
    from GetCapabilities/DescribeCoverage

    View Slide

  31. STAC/OpenSearch as an index

    The OpenSearch/STAC database can be
    used as a mosaic index
    ● SQL Views selecting a particular collection
    sub/set
    ● REST API to automatically create them
    Collection
    Product or
    Item
    Granule

    View Slide

  32. External STAC API as index
    ● Experimental modules on GitHub to create a
    mosaic out of a STAC API
    ● Filter to establish which items take part
    ● https://github.com/joshfix/imagemosaic-stac
    ● Outdated code, but promising concept
    ● Needs STAC API to settle down
    (beta/changes in filtering)
    STAC API
    ImageMosaic
    STAC
    GeoTools
    ImageMosaic

    View Slide

  33. Documentation links

    GeoServer reference docs:

    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/index.html

    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/tutorial.html

    http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y
    aml

    GeoSolutions training:

    http://geoserver.geo-
    solutions.it/edu/en/multidim/imagemosaic/index.html

    http://geoserver.geo-
    solutions.it/edu/en/multidim/accessing_multidim/index.html

    http://geoserver.geo-
    solutions.it/edu/en/multidim/rest/index.html

    View Slide

  34. ImageMosaic – the fun stuff

    View Slide

  35. Filtering and sorting

    The mosaic index can be filtered upon with CQL

    Can be sorted on using WFS like syntax

    More possibilities than just Time and Elevation
    SENSOR = SAR AND SATELLITE=XYZ
    SORTBY = TIME D

    View Slide

  36. Masking Support

    Mask allow finding valid data

    Alternative to NODATA for lossy compressions

    Vector and raster masks supported

    Raster masks can be embedded (GDAL style)

    Vector ones can be WKB or shapefiles

    API to plug your own mask provider

    Available for image mosaic and GDAL based data
    sources

    View Slide

  37. Coverage Views

    NetCDF/Grib/Mosaic: one reader, many coverages

    Sometimes the coverages are strictly related

    E.g., U and V components of wind/current vectors

    Merge them back into a single coverage as bands

    Allows rendering engine to work with them both as one

    View Slide

  38. Native multi-coverage support

    Satellite images often
    comes in one file per band

    Requirement, as they have
    different resolutions

    Grab the band name from
    file, build a “band specific”
    coverage

    Use coverage views to
    merge them together in a
    virtual multiband coverage

    Smarts to read only the
    desired files when doing
    band selection

    View Slide

  39. Documentation links

    GeoServer reference docs:

    http://docs.geoserver.org/latest/en/user/data/raster/image
    mosaic/index.html

    http://docs.geoserver.org/latest/en/user/data/raster/covera
    geview.html

    http://docs.geoserver.org/latest/en/user/tutorials/imagemos
    aic_footprint/imagemosaic_footprint.html

    GeoSolutions training:

    https://geoserver.geo-
    solutions.it/edu/en/multidim/accessing_multidim/coveragev
    iew.html

    View Slide

  40. The services - visualize

    View Slide

  41. WMS-T support

    View Slide

  42. Filtering via WMS-T support
    TIME, ELEVATION & Custom dimensions
    http://localhost:8080/geoserver/geosolutions/wms?...
    &time=2013-03-1T00:00:00.000Z
    &elevation=35.0
    &DIM_FILEDATE=2013-03-01T00:00:00.000Z
    &DIM_UPDATED=2013-04-08T08:18:41.597Z

    View Slide

  43. Filtering and sorting via vendor options
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover

    View Slide

  44. Rendering transformations

    Style based transformations

    Applied on the fly, yet fast enough for
    interactive use

    Pluggable: add your own
    contouring

    View Slide

  45. Rendering transformations examples
    windbarbs
    currents

    View Slide

  46. On the fly map algebra with Jiffle
    nir = src[7];
    vir = src[3];
    dest = (nir-vir)/
    (nir+vir);

    Jiffle: powerful and fast
    map algebra language

    Can be used as a
    rendering transformation

    View Slide

  47. Documentation links

    GeoServer documentation

    http://docs.geoserver.org/latest/en/user/services/wm
    s/time.html

    http://docs.geoserver.org/stable/en/user/data/webad
    min/layers.html#edit-layer-dimensions

    http://docs.geoserver.org/latest/en/user/styling/sld/ex
    tensions/rendering-transform.html

    GeoSolutions training:

    https://geoserver.geo-
    solutions.it/edu/en/wps/rendering_tx.html

    Jiffle documentation:

    https://github.com/geosolutions-it/jai-ext/wiki/Jiffle

    View Slide

  48. Download via WCS

    View Slide

  49. WCS 2.0

    Basics

    Core service

    KVP binding

    XML binding

    Common extensions

    CRS

    Scaling

    Interpolation

    Range subsetting

    GeoTiff

    GML

    NetCDF

    Add the output format extensions

    GeoTIFF

    GML Grid

    NetCDF

    View Slide

  50. WCS 2.0 DescribeCoverage
    4D BBOX
    TIME details
    as extension
    ELEVATION
    details as
    extension
    CUSTOM

    View Slide

  51. WCS – custom filtering and sorting
    request=GetCoverage&...
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover

    View Slide

  52. Download via WPS

    View Slide

  53. Asynchronous download
    ● WCS 2.0 is a fine protocol
    ● But it’s synchronous
    ● Not well suited for large downloads
    ● WPS execute can be
    asynchronous
    ● WPS download
    plugin
    ● Dedicated to large
    extractions

    View Slide

  54. WPS download - large raw download
    25GB
    GeoTIFF
    2 meters res
    Some time later

    View Slide

  55. WPS download - large rendered map
    10GB
    NDVI
    GeoTIFF
    2 meters res
    Some time later

    View Slide

  56. WPS download - long animation
    3GB
    .MP4 file
    Some time later

    View Slide

  57. That’s all folks!
    Questions?
    [email protected]

    View Slide

  58. Want more? Bonus slides!!

    View Slide

  59. NetCDF and GRIB

    View Slide

  60. NetCDF format support

    NetCDF support

    Support COARDS/CF* conventions loosely

    Expose NetCDF internal data as a set of 2D slices

    Fast 2D (time, elevation) slice extraction

    Supports NCML

    GRIB supported by the same libraries, with same
    functionality

    View Slide

  61. One or more variables

    Polyphemus Sample Dataset

    1 File 🡪 Multiple Coverages!

    View Slide

  62. Multiple coverages per mosaic

    View Slide

  63. NetCDF/GRIB Mosaic integration

    ImageMosaic NetCDF and GRIB integration

    Allow the ImageMosaic to handle multiple files

    Expose internal structure (times, elevations)

    Make ImageMosaic handle slices of the file as granules

    5-dim mosaic (space/time/runtime other dimensions)
    Mosaic
    NetCDF 1 NetCDF 2 NetCDF … N

    View Slide

  64. NetCDF WCS output

    WCS can output NetCDF for any multidimensional source

    Several customizations to control output and preserve info
    if the source is a NetCDF itself

    Source has to be multidimensional (current restriction)

    View Slide

  65. Download ND NetCDF
    http://localhost:8080/geoserver/wcs?request=GetCoverage
    &service=WCS&version=2.0.1&coverageId=geosolutions__NO2
    &Format=application/x-netcdf
    &subset=Long(5,20)
    &subset=Lat(40,50)
    &subset=elevation(300,1250)
    &subset=time("2013-03-01T10:00:00.000Z","2013-03-
    01T22:00:00.000Z")
    • Can also handle custom
    dimensions
    • Described in DescribeCoverage
    in a vendor metadata section
    4D
    BBOX

    View Slide

  66. Documentation links

    GeoServer documentation

    http://docs.geoserver.org/latest/en/user/extensions/n
    etcdf/netcdf.html

    http://docs.geoserver.org/latest/en/user/extensio
    ns/netcdf-out/index.html

    GeoSolutions training:

    http://geoserver.geo-
    solutions.it/edu/en/multidim/netcdf/index.html

    http://geoserver.geo-
    solutions.it/edu/en/multidim/netcdf/index.html

    http://geoserver.geo-
    solutions.it/edu/en/multidim/mosaic_config/netc
    df_mosaic.html

    View Slide

  67. EO extensions

    View Slide

  68. WMS-EO
    Extending LayerGroup
    concept
    Support same style on
    both raster and vector
    data
    Support custom
    dimensions
    Alter map on the fly to
    support band
    combination
    Wizard to configure EO
    layer groups

    Product layer tree

    Mix of vector and raster info

    Raw data, flags (clouds,
    snow), browse

    View Slide

  69. WMS-EO

    Helper UI to build the tree

    View Slide

  70. WCS EO

    Add support for WCS EO metadata in readers

    Associate each file with EO metadata

    Include such information in
    DescribeCoverage/DescribeEODataset

    EODataset: multidimensional mosaic or NetCDF file

    Describe the inner structure (granules listing) in
    DescribeEODataSet

    View Slide

  71. Real World Use cases

    View Slide

  72. Use Case 1 – Light Integration

    GeoServer + MapStore + Legacy Ingestion

    Maritime Data

    SAR Time Series

    Oil Spill + other vectors

    Single Mosaic + CQL + Custom Access
    Manager

    Integrated SSO with MapStore

    Precooked Download of Scenes

    Fine grain control over data access

    Download

    Visualization

    Clever usage of Mosaic Indexes via WFS

    View Slide

  73. Use Case 1 – Light Integration

    View Slide

  74. Use Case 1 – Light Integration

    View Slide

  75. Use Case 1 – Light Integration

    View Slide

  76. Use Case 2 – Deep Integration

    High Res Imagery

    SSO with CAS

    Legacy Ingestion

    View Slide

  77. Use Case 2

    Database of satellite imagery, one satellite collect
    (strip) materialized as many files (granules)

    Access filtering based on current user roles (custom
    data store implementation)

    Custom stacking order (SORT BY) also user based if
    not already provided in the request
    Mosaic
    POSTGIS
    Custom store wrapper
    Postgis store

    View Slide

  78. Use Case 3 – Let’s go Deeper!

    View Slide