Tweet
Tweet

Slide 1

Slide 1 text

Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions

Slide 2

Slide 2 text

GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode • Offices in Italy & US, Global Clients/Team • 30+ collaborators, 25+ Engineers • Our products • Our Offer

Slide 3

Slide 3 text

Affiliations We strongly support Open Source, it Is in our core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT

Slide 4

Slide 4 text

Reference scenario REP/COP InSitu Obs MetOc Models Remote Sensing

Slide 5

Slide 5 text

Locating raster sources

Slide 6

Slide 6 text

STAC API (RC1) ● Same use case as OpenSearch for EO ● Based on OGC API, JSON based ● Collections and items ● Cross collection search at /search ○ filter, sort, field selection

Slide 7

Slide 7 text

General information schema Collection Product or Item Asset Search Search Mosaic/OGC

Slide 8

Slide 8 text

STAC API (RC1) ● Screenshots from DLR's EOC Geoservice (STAC support in development / Beta) ● Customized database, HTML and JSON templates Landing page

Slide 9

Slide 9 text

STAC API (RC1) - Collections Collections

Slide 10

Slide 10 text

STAC API (RC1) - Collection

Slide 11

Slide 11 text

STAC API (RC1) - items

Slide 12

Slide 12 text

STAC API (RC1) - item

Slide 13

Slide 13 text

OpenSearch for EO (2017) ● Earlier OGC specification to search for satellite imagery ● Based on OpenSearch and OpenSearch Geo and Time OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO

Slide 14

Slide 14 text

Sample search ● http://cloudsdi. geo-solutions.it/ ● geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] ● Output is RSS with description, thumb/quicklook and links ● Since 2021, output can also be GeoJSON

Slide 15

Slide 15 text

Metadata and cross linking ● Links to metadata: ● ISO metadata for collection ● GML O&M for products ● Links to OGC services (cross linking) ● WMS/WMTS to see collection/product ● WFS to get “masks” (validity, snow, cloud, …) ● WCS to extract raster ● Direct download to get original package

Slide 16

Slide 16 text

REST API to manage search index Swagger documentation

Slide 17

Slide 17 text

Accessing raster data

Slide 18

Slide 18 text

Finding rasters ● STAC has “assets” in Item pointing to the imagery files ● OpenSearch can point to the raw package ● Where can they be located? ○ Local file system (classic approach) ○ HTTP server ○ S3 or other blob storage

Slide 19

Slide 19 text

Reading from blobstore ● COG plugin ○ Leverages COG structure ● But also, old S3 GeoTIFF plugin (Boundless) ○ Reads any GeoTIFF (non COG) from S3 ○ Reads in 5MB blocks, caches them ○ Deprecated soon

Slide 20

Slide 20 text

COG plugin Hea der ● Leverages COG structure ● Reads header first ● HTTP Range read of the tiles needed (coalesce nearby tiles into single read) ● Works against HTTP(s), S3, Azure and Google Storage ● May get caching down the way (interested?) Tile Tile Tile Tile Tile Tile R1 R2

Slide 21

Slide 21 text

Mosaicking rasters

Slide 22

Slide 22 text

Mosaic Terminology ● Granule/Tile ● The individual raster element composing the mosaic (asset in STAC) ● (Granule) Index ● The collection of metadata records describing the location, spatial coverage and other attributes of each single granule

Slide 23

Slide 23 text

Complete freedom ● Granules can: ● Overlap as they please, can have different resolutions ● Be in different file formats (faster if the format is uniform) ● Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) ● Be in different coordinate reference systems

Slide 24

Slide 24 text

The mosaic index ● The granule lookup system ● A “table” of granules, their footprint, location, and extra attributes ● Currently supported/tested sources: ● PostGIS, Oracle, SQL Server, Shapefile ● STAC API (local or remote) ● Can be customized to support custom granule indexes (e.g. legacy catalog)

Slide 25

Slide 25 text

The Dimensions ● Maps to alphanumeric attributes in the index ● TIME and ELEVATION receive special treatment for WMS and WCS ● Custom/Additional dimensions ● Everything besides TIME & ELEVATION ● They can be dynamically discovered from GetCapabilities/DescribeCoverage

Slide 26

Slide 26 text

Internal STAC as an index ・ The OpenSearch/STAC database can be used as a mosaic index ・ SQL Views selecting a particular collection sub/set ・ REST API to automatically create them Collection Product or Item Granule

Slide 27

Slide 27 text

External STAC API as index ・ Community module for STAC API store ・ Integrates well with image mosaic ・ Tricky issues: each system uses different auth mechanisms to actually download data STAC API STAC store GeoTools ImageMosaic COG

Slide 28

Slide 28 text

Documentation links ● GeoServer reference docs: ● http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html ● http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html ● http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml ● GeoSolutions training: ● http://geoserver.geo-solutions.it/edu/en/multidim/imagemo saic/index.html ● http://geoserver.geo-solutions.it/edu/en/multidim/accessing _multidim/index.html ● http://geoserver.geo-solutions.it/edu/en/multidim/rest/index .html

Slide 29

Slide 29 text

ImageMosaic – the fun stuff

Slide 30

Slide 30 text

Filtering and sorting ● The mosaic index can be filtered upon with CQL ● Can be sorted on using WFS like syntax ● More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D

Slide 31

Slide 31 text

Coverage Views ● NetCDF/Grib/Mosaic: one reader, many coverages ● Sometimes the coverages are strictly related ● E.g., U and V components of wind/current vectors ● Merge them back into a single coverage as bands ● Allows rendering engine to work with them both as one

Slide 32

Slide 32 text

Native multi-coverage support ● Satellite images often comes in one file per band ● Requirement, as they have different resolutions ● Grab the band name from file, build a “band specific” coverage ● Use coverage views to merge them together in a virtual multiband coverage ● Smarts to read only the desired files when doing band selection

Slide 33

Slide 33 text

Hyperspectral imagery support ● Hyperspectral images: those having hundreds of bands ● Typically stored in band-interleaved structure (one data bank for all pixels in the same tile) Rows Cols Bands

Slide 34

Slide 34 text

Documentation links ● GeoServer reference docs: ● http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html ● http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html ● http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html ● GeoSolutions training: ● https://geoserver.geo-solutions.it/edu/en/multidim/accessin g_multidim/coverageview.html

Slide 35

Slide 35 text

The services - visualize

Slide 36

Slide 36 text

WMS-T (time) support

Slide 37

Slide 37 text

Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?... &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z

Slide 38

Slide 38 text

Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

Slide 39

Slide 39 text

Rendering transformations ● Style based transformations ● Applied on the fly, yet fast enough for interactive use ● Pluggable: add your own contouring

Slide 40

Slide 40 text

Rendering transformations examples windbarbs currents

Slide 41

Slide 41 text

On the fly map algebra with Jiffle nir = src[7]; vir = src[3]; dest = (nir-vir)/ (nir+vir); ● Jiffle: powerful and fast map algebra language ● Can be used as a rendering transformation

Slide 42

Slide 42 text

Documentation links ● GeoServer documentation ● http://docs.geoserver.org/latest/en/user/services/wms /time.html ● http://docs.geoserver.org/stable/en/user/data/webad min/layers.html#edit-layer-dimensions ● http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html ● GeoSolutions training: ● https://geoserver.geo-solutions.it/edu/en/wps/renderi ng_tx.html ● Jiffle documentation: ● https://github.com/geosolutions-it/jai-ext/wiki/Jiffle

Slide 43

Slide 43 text

Download via WCS

Slide 44

Slide 44 text

WCS 2.0 ● Basics ● Core service ● KVP binding ● XML binding ● Common extensions ● CRS ● Scaling ● Interpolation ● Range subsetting ● GeoTiff ● GML ● NetCDF ● Add the output format extensions ● GeoTIFF ● GML Grid ● NetCDF

Slide 45

Slide 45 text

WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION details as extension CUSTOM

Slide 46

Slide 46 text

WCS – custom filtering and sorting request=GetCoverage&... &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover

Slide 47

Slide 47 text

Download via WPS

Slide 48

Slide 48 text

Asynchronous download ● WCS 2.0 is a fine protocol ● But it’s synchronous ● Not well suited for large downloads ● WPS execute can be asynchronous ● WPS download plugin ● Dedicated to large extractions

Slide 49

Slide 49 text

WPS download - large raw download 25GB GeoTIFF 2 meters res Some time later

Slide 50

Slide 50 text

WPS download - large rendered map 10GB NDVI GeoTIFF 2 meters res Some time later

Slide 51

Slide 51 text

WPS download - long animation 3GB .MP4 file Some time later

Slide 52

Slide 52 text

That’s all folks! Questions? info@geo-solutions.it