Serving_earth_observation_data_with_GeoServer__COG__STAC__OpenSearch_and_more.pdf

Transcript

1. Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
2. Affiliations We strongly support Open Source, it Is in our core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT
3. GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode • Offices in Italy & US, Global Clients/Team • 40+ collaborators, 30+ Engineers • Our products • Our Offer
4. Reference scenario REP/COP InSitu Obs MetOc Models Remote Sensing
5. Locating raster sources
6. OpenSearch for EO (2017) • Use case • Lots of satellite imagery • Different satellites • Different sensor • Deep time and space distribution • How to find the right image? • By time and space • By attributes (cloud cover, snow cover, off nadir, ….) OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO • “OpenSearch for EO“ to the rescue
7. General information schema Collection Product or Item Granule Search Search Mosaic/OGC
8. Two steps search • First find collection of similar products (by satellite/sensor), e.g. • Optical or SAR? • Supported resolutions • Availability of data in particular timeframe • Then search products in it, by • Time • Space • Cloud cover • …
9. Sample search • http://cloudsdi. geo-solutions.it/ • geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] • Output is RSS with description, thumb/quicklook and links • Since 2021, output can also be GeoJSON
10. Metadata and cross linking • Links to metadata: • ISO metadata for collection • GML O&M for products • Links to OGC services (cross linking) • WMS/WMTS to see collection/product • WFS to get “masks” (validity, snow, cloud, …) • WCS to extract raster • Direct download to get original package
11. STAC API (beta 2) - 2021 • Same use case as OpenSearch for EO, more modern take on architecture • Collections and items • Allows cross collection search. • Based on OGC API, compatible with OGC API - Features
12. STAC API (beta 2) • Screenshots from DLR's EOC Geoservice (STAC support in development / Beta) • Customized database and templates Landing page
13. STAC API (beta 2) Collections
14. STAC API (beta 2) Single collection
15. STAC API (beta 2) Items listing
16. STAC API (beta 2) Single item
17. Similar semantic, different names • We have 4 information models ◦ Database ◦ OpenSearch EO ▪ GeoRSS ▪ GeoJSON ◦ STAC • They express the same information with different names OpenSearch EO GeoRSS OpenSearch EO GeoJSON STAC collections/items Database
18. Features templating to the rescue • “Features templating” community module to the rescue: ◦ The right “naming” is generated based on the output ◦ Public names are back-mapped to db names for filtering Database STAC Collections Items OpenSearch EO Search Templates
19. Templating, JSON example { "type": "FeatureCollection", "features": [ { "stac_version": "$${env('stac_version', '1.0.0-rc2')}", "stac_extensions": [ // skip SAR sensors "$filter{eop:parentIdentifier LIKE 'SENTINEL%' or eop:parentIdentifier LIKE 'LANDSAT%'},https://stac-extensions.github.io/eo/v1.0.0/schema.json", "https://stac-extensions.github.io/view/v1.0.0/schema.json" ], "id": "${eop:identifier}", "type": "Feature", "geometry": "${footprint}", "bbox": [ "$${minx(footprint)}", "$${miny(footprint)}", "$${maxx(footprint)}", "$${maxy(footprint)}" ], "properties": { "start_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <> timeEnd},${timeStart}", "end_datetime": "$filter{timeStart is not null and timeEnd is not null and timeStart <> timeEnd},${timeEnd}", "datetime": "$filter{timeStart is not null and (timeEnd is null or timeStart = timeEnd)},${timeStart}", ....
20. REST API to manage search index Swagger documentation
21. Accessing raster data
22. Finding rasters • STAC has “assets” in Item pointing to the imagery files • OpenSearch can point to the raw package, and the “granules” table allow to get to the single images • Where can they be located? ◦ Local file system (classic approach) ◦ HTTP server ◦ S3 or other blob storage
23. Reading from blobstore • Presently two choices: ◦ Old S3 GeoTiff plugin (Boundless) ◦ COG plugin (Planet/GeoSolutions) • Old S3 plugin available for longer time, used by some production sites, but not maintained • COG plugin is new, and the way forward
24. Old S3 GeoTIFF plugin 5MB • Split file in 5MB blocks (configurable) • Read the full block even a small part of it is used • Eventual local memory/disk cache • Works best for smaller files and strong locality • Can read any GeoTIFF 5MB 5MB 5MB
25. COG plugin Hea der • Leverages COG structure • Reads header first • HTTP Range read of the tiles needed (coaleshe nearby tiles into single read) • Works against S3 and HTTP(s) • More blob storages incoming (Google Storage) • May get caching down the way Tile Tile Tile Tile Tile Tile R1 R2
26. Mosaicking rasters
27. Terminology • Granule/Tile • The individual raster element composing the mosaic • (Granule) Index • The collection of metadata records describing the location, spatial coverage and other attributes of each single granule
28. Complete freedom • Granules can: • Overlap as they please, can have different resolutions • Be in different file formats (faster if the format is uniform) • Have different color models, RGB, gray, paletted (needs JAI-EXT enabled) • Be in different coordinate reference systems
29. The mosaic index • The file lookup system • A table of granules, their footprint, location, and extra attributes • Currently supported/tested sources: • PostGIS, Oracle Shapefile, h (no attr indexing, no timestamp) • H2 (local embedded db) • Can be customized to support custom granule indexes (e.g. legacy catalog)
30. The Dimensions • Maps to alphanumeric attributes in the index • TIME and ELEVATION receive special treatment for WMS and WCS • Custom/Additional dimensions • Everything besides TIME & ELEVATION • They can be dynamically discovered from GetCapabilities/DescribeCoverage
31. STAC/OpenSearch as an index • The OpenSearch/STAC database can be used as a mosaic index • SQL Views selecting a particular collection sub/set • REST API to automatically create them Collection Product or Item Granule
32. External STAC API as index • Experimental modules on GitHub to create a mosaic out of a STAC API • Filter to establish which items take part • https://github.com/joshfix/imagemosaic-stac • Outdated code, but promising concept • Needs STAC API to settle down (beta/changes in filtering) STAC API ImageMosaic STAC GeoTools ImageMosaic
33. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html • http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html • http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml • GeoSolutions training: • http://geoserver.geo- solutions.it/edu/en/multidim/imagemosaic/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/rest/index.html
34. ImageMosaic – the fun stuff
35. Filtering and sorting • The mosaic index can be filtered upon with CQL • Can be sorted on using WFS like syntax • More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ SORTBY = TIME D
36. Masking Support • Mask allow finding valid data • Alternative to NODATA for lossy compressions • Vector and raster masks supported • Raster masks can be embedded (GDAL style) • Vector ones can be WKB or shapefiles • API to plug your own mask provider • Available for image mosaic and GDAL based data sources
37. Coverage Views • NetCDF/Grib/Mosaic: one reader, many coverages • Sometimes the coverages are strictly related • E.g., U and V components of wind/current vectors • Merge them back into a single coverage as bands • Allows rendering engine to work with them both as one
38. Native multi-coverage support • Satellite images often comes in one file per band • Requirement, as they have different resolutions • Grab the band name from file, build a “band specific” coverage • Use coverage views to merge them together in a virtual multiband coverage • Smarts to read only the desired files when doing band selection
39. Documentation links • GeoServer reference docs: • http://docs.geoserver.org/latest/en/user/data/raster/image mosaic/index.html • http://docs.geoserver.org/latest/en/user/data/raster/covera geview.html • http://docs.geoserver.org/latest/en/user/tutorials/imagemos aic_footprint/imagemosaic_footprint.html • GeoSolutions training: • https://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/coveragev iew.html
40. The services - visualize
41. WMS-T support
42. Filtering via WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?... &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z
43. Filtering and sorting via vendor options &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover
44. Rendering transformations • Style based transformations • Applied on the fly, yet fast enough for interactive use • Pluggable: add your own contouring
45. Rendering transformations examples windbarbs currents
46. On the fly map algebra with Jiffle nir = src[7]; vir = src[3]; dest = (nir-vir)/ (nir+vir); • Jiffle: powerful and fast map algebra language • Can be used as a rendering transformation
47. Documentation links • GeoServer documentation • http://docs.geoserver.org/latest/en/user/services/wm s/time.html • http://docs.geoserver.org/stable/en/user/data/webad min/layers.html#edit-layer-dimensions • http://docs.geoserver.org/latest/en/user/styling/sld/ex tensions/rendering-transform.html • GeoSolutions training: • https://geoserver.geo- solutions.it/edu/en/wps/rendering_tx.html • Jiffle documentation: • https://github.com/geosolutions-it/jai-ext/wiki/Jiffle
48. Download via WCS
49. WCS 2.0 • Basics • Core service • KVP binding • XML binding • Common extensions • CRS • Scaling • Interpolation • Range subsetting • GeoTiff • GML • NetCDF • Add the output format extensions • GeoTIFF • GML Grid • NetCDF
50. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION details as extension CUSTOM
51. WCS – custom filtering and sorting request=GetCoverage&... &CQL_FILTER=platform=SENTINEL2 &sortBy=time D &sortBy=cloudCover
52. Download via WPS
53. Asynchronous download • WCS 2.0 is a fine protocol • But it’s synchronous • Not well suited for large downloads • WPS execute can be asynchronous • WPS download plugin • Dedicated to large extractions
54. WPS download - large raw download 25GB GeoTIFF 2 meters res Some time later
55. WPS download - large rendered map 10GB NDVI GeoTIFF 2 meters res Some time later
56. WPS download - long animation 3GB .MP4 file Some time later
57. That’s all folks! Questions? info@geo-solutions.it
58. Want more? Bonus slides!!
59. NetCDF and GRIB
60. NetCDF format support • NetCDF support • Support COARDS/CF* conventions loosely • Expose NetCDF internal data as a set of 2D slices • Fast 2D (time, elevation) slice extraction • Supports NCML • GRIB supported by the same libraries, with same functionality
61. One or more variables • Polyphemus Sample Dataset • 1 File 🡪 Multiple Coverages!
62. Multiple coverages per mosaic
63. NetCDF/GRIB Mosaic integration • ImageMosaic NetCDF and GRIB integration • Allow the ImageMosaic to handle multiple files • Expose internal structure (times, elevations) • Make ImageMosaic handle slices of the file as granules • 5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N …
64. NetCDF WCS output • WCS can output NetCDF for any multidimensional source • Several customizations to control output and preserve info if the source is a NetCDF itself • Source has to be multidimensional (current restriction)
65. Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03- 01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX
66. Documentation links • GeoServer documentation • http://docs.geoserver.org/latest/en/user/extensions/n etcdf/netcdf.html • http://docs.geoserver.org/latest/en/user/extensio ns/netcdf-out/index.html • GeoSolutions training: • http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html • http://geoserver.geo- solutions.it/edu/en/multidim/mosaic_config/netc df_mosaic.html
67. EO extensions
68. WMS-EO Extending LayerGroup concept Support same style on both raster and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups • Product layer tree • Mix of vector and raster info • Raw data, flags (clouds, snow), browse
69. WMS-EO • Helper UI to build the tree
70. WCS EO • Add support for WCS EO metadata in readers • Associate each file with EO metadata • Include such information in DescribeCoverage/DescribeEODataset • EODataset: multidimensional mosaic or NetCDF file • Describe the inner structure (granules listing) in DescribeEODataSet
71. Real World Use cases
72. Use Case 1 – Light Integration • GeoServer + MapStore + Legacy Ingestion • Maritime Data • SAR Time Series • Oil Spill + other vectors • Single Mosaic + CQL + Custom Access Manager • Integrated SSO with MapStore • Precooked Download of Scenes • Fine grain control over data access • Download • Visualization • Clever usage of Mosaic Indexes via WFS
73. Use Case 1 – Light Integration
74. Use Case 1 – Light Integration
75. Use Case 1 – Light Integration
76. Use Case 2 – Deep Integration • High Res Imagery • SSO with CAS • Legacy Ingestion
77. Use Case 2 • Database of satellite imagery, one satellite collect (strip) materialized as many files (granules) • Access filtering based on current user roles (custom data store implementation) • Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store
78. Use Case 3 – Let’s go Deeper!