Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more (FOSS4G 2022 Edition)

Never before have we had such a rich collection of satellite imagery available to both companies and the general public. Between missions such as Landsat 8 and Sentinels and the explosion of cubesats, as well as the free availability of worldwide data from the European Copernicus program and from Drones, a veritable flood of data is made available for everyday usage.
Managing, locating and displaying such a large volume of satellite images can be challenging. Join this presentation to learn how GeoServer can help with with that job, with real world examples, including:
- Indexing and locating images using The OpenSearch for EO and STAC protocols
- Managing large volumes of satellite images, in an efficient and cost effective way, using Cloud Optimized GeoTIFFs.
- Visualize mosaics of images, creating composite with the right set of views (filtering), in the desired stacking order (color on top, most recent on top, less cloudy on top, your choice)
- Perform both small and large extractions of imagery using the WCS and WPS protocols
- Generate and view time based animations of the above mosaics, in a period of interest
- Perform band algebra operations using Jiffle
Attend this talk to get a good update on the latest GeoServer capabilities in the Earth Observation field.

Transcript

1. Serving earth observation data
   with GeoServer:
   COG, STAC, OpenSearch
   and more
   Ing. Simone Giannecchini
   Ing. Daniele Romagnoli
   Ing. Andrea Aime
   GeoSolutions
   

   View Slide

2. GeoSolutions
   Enterprise Support
   Services
   Deployment
   Subscription
   Professional
   Training
   Customized
   Solutions
   GeoNode
   • Offices in Italy & US, Global Clients/Team
   • 40+ collaborators, 30+ Engineers
   • Our products
   • Our Offer
   

   View Slide

3. Affiliations
   We strongly support Open
   Source, it Is in our core
   We actively participate in OGC
   working groups and get funded
   to advance new open
   standards
   We support standards critical to
   GEOINT
   

   View Slide

4. Reference scenario
   REP/COP
   InSitu
   Obs
   MetOc
   Models
   Remote
   Sensing
   

   View Slide

5. Locating raster sources
   

   View Slide

6. OpenSearch for EO (2017)
   ●
   Use case
   ●
   Lots of satellite imagery
   ●
   Different satellites
   ●
   Different sensor
   ●
   Deep time and space
   distribution
   ●
   How to find the right
   image?
   ●
   By time and space
   ●
   By attributes (cloud cover,
   snow cover, off nadir, ….)
   OpenSearch
   OGC OpenSearch Geo and
   Time
   OGC OpenSearch for EO
   ●
   “OpenSearch for EO“ to the rescue
   

   View Slide

7. General information schema
   Collection
   Product or
   Item
   Granule
   Search
   Search
   Mosaic/OGC
   

   View Slide

8. Two steps search
   ●
   First find collection of
   similar products (by
   satellite/sensor), e.g.
   ●
   Optical or SAR?
   ●
   Supported resolutions
   ●
   Availability of data in
   particular timeframe
   ●
   Then search products in it,
   by
   ●
   Time
   ●
   Space
   ●
   Cloud cover
   ●
   …
   

   View Slide

9. Sample search
   ●
   http://cloudsdi.
   geo-solutions.it/
   ●
   geoserver/oseo/search?
   parentId=SENTINEL2
   &cloudCover=30]
   ●
   Output is RSS with
   description,
   thumb/quicklook and
   links
   ● Since 2021, output can
   also be GeoJSON
   

   View Slide

10. Metadata and cross linking
    ●
    Links to metadata:
    ●
    ISO metadata for collection
    ●
    GML O&M for products
    ●
    Links to OGC services (cross linking)
    ●
    WMS/WMTS to see collection/product
    ●
    WFS to get “masks” (validity, snow, cloud, …)
    ●
    WCS to extract raster
    ●
    Direct download to get original package
    

    View Slide

11. STAC API (RC1)
    ● Same use case as
    OpenSearch for EO
    ● Based on OGC API,
    JSON based
    ● Collections and items
    ● Cross collection
    search at /search
    ○ filter, sort, field
    selection
    

    View Slide

12. STAC API (RC1)
    ● Screenshots from DLR's EOC Geoservice (STAC
    support in development / Beta)
    ● Customized database, HTML and JSON templates
    Landing page
    

    View Slide

13. STAC API (RC1) - Collections
    Collections
    

    View Slide

14. STAC API (RC1) - Collection
    

    View Slide

15. STAC API (RC1) - items
    

    View Slide

16. STAC API (RC1) - item
    

    View Slide

17. REST API to manage search index
    Swagger documentation
    

    View Slide

18. Accessing raster data
    

    View Slide

19. Finding rasters
    ● STAC has “assets” in Item pointing to the
    imagery files
    ● OpenSearch can point to the raw package,
    and the “granules” table allow to get to the
    single images
    ● Where can they be located?
    ○ Local file system (classic approach)
    ○ HTTP server
    ○ S3 or other blob storage
    

    View Slide

20. Reading from blobstore
    ● COG plugin
    ○ Leverages COG structure
    ● But also, old S3 GeoTIFF plugin (Boundless)
    ○ Reads any GeoTIFF (non COG) from S3
    ○ Reads in 5MB blocks, caches them
    ○ Deprecated soon
    

    View Slide

21. COG plugin
    Hea
    der
    ● Leverages COG structure
    ● Reads header first
    ● HTTP Range read of the tiles needed (coaleshe
    nearby tiles into single read)
    ● Works against S3, HTTP(s) and Google Storage
    ● More blob storages incoming (Azure blob)
    ● May get caching down the way (interested?)
    Tile Tile Tile Tile Tile Tile
    R1 R2
    

    View Slide

22. Mosaicking rasters
    

    View Slide

23. Terminology
    ●
    Granule/Tile
    ●
    The individual raster element composing the
    mosaic
    ●
    (Granule) Index
    ●
    The collection of metadata records describing
    the location, spatial coverage and other
    attributes of each single granule
    

    View Slide

24. Complete freedom
    ●
    Granules can:
    ●
    Overlap as they please, can have different
    resolutions
    ●
    Be in different file formats (faster if the format is
    uniform)
    ●
    Have different color models, RGB, gray, paletted
    (needs JAI-EXT enabled)
    ●
    Be in different coordinate reference systems
    

    View Slide

25. The mosaic index
    ●
    The file lookup system
    ●
    A table of granules, their footprint, location,
    and extra attributes
    ●
    Currently supported/tested sources:
    ●
    PostGIS, Oracle, SQL Server
    ●
    Shapefile (no attr indexing, no timestamp)
    ●
    Other GeoTools supported stores
    ●
    Can be customized to support custom
    granule indexes (e.g. legacy catalog)
    

    View Slide

26. The Dimensions
    ●
    Maps to alphanumeric attributes in the index
    ●
    TIME and ELEVATION receive special
    treatment for WMS and WCS
    ●
    Custom/Additional dimensions
    ●
    Everything besides TIME & ELEVATION
    ●
    They can be dynamically discovered
    from GetCapabilities/DescribeCoverage
    

    View Slide

27. STAC/OpenSearch as an index
    ・ The OpenSearch/STAC database can be
    used as a mosaic index
    ・ SQL Views selecting a particular collection
    sub/set
    ・ REST API to automatically create them
    Collection
    Product or
    Item
    Granule
    

    View Slide

28. External STAC API as index
    ・ Community module for STAC API store
    landing just landed on 2.22.x
    ・ Adapting mosaic code to use STAC items as
    we speak
    STAC
    API
    STAC
    store
    GeoTools
    ImageMosaic
    COG
    

    View Slide

29. Documentation links
    ●
    GeoServer reference docs:
    ●
    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/index.html
    ●
    http://docs.geoserver.org/stable/en/user/data/raster/image
    mosaic/tutorial.html
    ●
    http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y
    aml
    ●
    GeoSolutions training:
    ●
    http://geoserver.geo-solutions.it/edu/en/multidim/imagemo
    saic/index.html
    ●
    http://geoserver.geo-solutions.it/edu/en/multidim/accessing
    _multidim/index.html
    ●
    http://geoserver.geo-solutions.it/edu/en/multidim/rest/index
    .html
    

    View Slide

30. ImageMosaic – the fun stuff
    

    View Slide

31. Filtering and sorting
    ●
    The mosaic index can be filtered upon with CQL
    ●
    Can be sorted on using WFS like syntax
    ●
    More possibilities than just Time and Elevation
    SENSOR = SAR AND SATELLITE=XYZ
    SORTBY = TIME D
    

    View Slide

32. Coverage Views
    ●
    NetCDF/Grib/Mosaic: one reader, many coverages
    ●
    Sometimes the coverages are strictly related
    ●
    E.g., U and V components of wind/current vectors
    ●
    Merge them back into a single coverage as bands
    ●
    Allows rendering engine to work with them both as one
    

    View Slide

33. Native multi-coverage support
    ●
    Satellite images often
    comes in one file per band
    ●
    Requirement, as they have
    different resolutions
    ●
    Grab the band name from
    file, build a “band specific”
    coverage
    ●
    Use coverage views to
    merge them together in a
    virtual multiband coverage
    ●
    Smarts to read only the
    desired files when doing
    band selection
    

    View Slide

34. Documentation links
    ●
    GeoServer reference docs:
    ●
    http://docs.geoserver.org/latest/en/user/data/raster/image
    mosaic/index.html
    ●
    http://docs.geoserver.org/latest/en/user/data/raster/covera
    geview.html
    ●
    http://docs.geoserver.org/latest/en/user/tutorials/imagemos
    aic_footprint/imagemosaic_footprint.html
    ●
    GeoSolutions training:
    ●
    https://geoserver.geo-solutions.it/edu/en/multidim/accessin
    g_multidim/coverageview.html
    

    View Slide

35. The services - visualize
    

    View Slide

36. WMS-T (time) support
    

    View Slide

37. Filtering via WMS-T support
    TIME, ELEVATION & Custom dimensions
    http://localhost:8080/geoserver/geosolutions/wms?...
    &time=2013-03-1T00:00:00.000Z
    &elevation=35.0
    &DIM_FILEDATE=2013-03-01T00:00:00.000Z
    &DIM_UPDATED=2013-04-08T08:18:41.597Z
    

    View Slide

38. Filtering and sorting via vendor options
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover
    

    View Slide

39. Rendering transformations
    ●
    Style based transformations
    ●
    Applied on the fly, yet fast enough for
    interactive use
    ●
    Pluggable: add your own
    contouring
    

    View Slide

40. Rendering transformations examples
    windbarbs
    currents
    

    View Slide

41. On the fly map algebra with Jiffle
    nir = src[7];
    vir = src[3];
    dest = (nir-vir)/
    (nir+vir);
    ●
    Jiffle: powerful and fast
    map algebra language
    ●
    Can be used as a
    rendering transformation
    

    View Slide

42. Documentation links
    ●
    GeoServer documentation
    ●
    http://docs.geoserver.org/latest/en/user/services/wms
    /time.html
    ●
    http://docs.geoserver.org/stable/en/user/data/webad
    min/layers.html#edit-layer-dimensions
    ●
    http://docs.geoserver.org/latest/en/user/styling/sld/ex
    tensions/rendering-transform.html
    ●
    GeoSolutions training:
    ●
    https://geoserver.geo-solutions.it/edu/en/wps/renderi
    ng_tx.html
    ●
    Jiffle documentation:
    ●
    https://github.com/geosolutions-it/jai-ext/wiki/Jiffle
    

    View Slide

43. Download via WCS
    

    View Slide

44. WCS 2.0
    ●
    Basics
    ●
    Core service
    ●
    KVP binding
    ●
    XML binding
    ●
    Common extensions
    ●
    CRS
    ●
    Scaling
    ●
    Interpolation
    ●
    Range subsetting
    ●
    GeoTiff
    ●
    GML
    ●
    NetCDF
    ●
    Add the output format extensions
    ●
    GeoTIFF
    ●
    GML Grid
    ●
    NetCDF
    

    View Slide

45. WCS 2.0 DescribeCoverage
    4D BBOX
    TIME details
    as extension
    ELEVATION
    details as
    extension
    CUSTOM
    

    View Slide

46. WCS – custom filtering and sorting
    request=GetCoverage&...
    &CQL_FILTER=platform=SENTINEL2
    &sortBy=time D
    &sortBy=cloudCover
    

    View Slide

47. Download via WPS
    

    View Slide

48. Asynchronous download
    ● WCS 2.0 is a fine protocol
    ● But it’s synchronous
    ● Not well suited for large downloads
    ● WPS execute can be
    asynchronous
    ● WPS download
    plugin
    ● Dedicated to large
    extractions
    

    View Slide

49. WPS download - large raw download
    25GB
    GeoTIFF
    2 meters res
    Some time later
    

    View Slide

50. WPS download - large rendered map
    10GB
    NDVI
    GeoTIFF
    2 meters res
    Some time later
    

    View Slide

51. WPS download - long animation
    3GB
    .MP4 file
    Some time later
    

    View Slide

52. That’s all folks!
    Questions?
    info@geo-solutions.it
    

    View Slide