Publishing INSPIRE datasets in GeoServer made easy with Smart Data Loader and Features Templating (FOSS4G 2022 Edition)

GeoServer is a well-established multiplatform, open-source geospatial server providing a variety of OGC services, including WMS (view services), WFS and WCS (download services) as well as WPS (spatial data processing services). Among the open-source GIS web servers, GeoServer is well known for the ease of setup, the web console helping the administrator to configure data and services, the variety of OGC services available out of the box, as well as the rich set of data sources that it can connect to (open source, such as PostGIS as well as proprietaries, such as ArcSDE, Oracle or ECW rasters). GeoServer also provides several OGC APIs, including the OGC API - Features which recently attracted the interest of the INSPIRE community.
As far as the INSPIRE scenario is concerned GeoServer has extensive support for implementing view and download services thanks to its core capabilities but also to a number of free and open-source extensions; undoubtedly the most well-known (and dreaded) extension is App-Schema which can be used to publish complex data models (with nested properties and multiple-cardinality relationships) and implement sophisticated download services for vector data. Based on the feedback of App-Schema users collected over the years, a new generation of open-source mapping extensions have been implemented in GeoServer: Smart Data Loader and Features Templating, these extensions are built on top of App-Schema and ease the mapping of the data models by allowing us to act directly on the domain model and target output schema using a what you see is what you get approach.
This presentation will introduce the new GeoServer Smart Data Loader and Features Templating extensions, covering in detail ongoing and planned work on GeoServer. We will also provide an overview about how those extensions are serving as a foundation for new approaches to publish complex data: publishing data models directly from MongoDB, embracing the NoSQL nature of it, and support new output formats like JSON-LD which allows us to embed well known semantics in our data. Eventually, real-world use-cases from the organizations that have selected GeoServer and GeoSolutions to support their use cases will be introduced to provide the attendees with references and lessons learned that could put them on the right path when adopting GeoServer.

Transcript

1. Nuno Oliveira Marco Volpini GeoSolutions Publishing INSPIRE datasets in GeoServer made easy with Smart Data Loader and Features Templating
2. GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode • Offices in Italy & US, Global Clients/Team • 40+ collaborators, 30+ Engineers • Our products • Our Offer
3. Affiliations We strongly support Open Source, it Is in our core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT
4. Where are we?
5. • Available at http://www.geoserver.org • GeoSpatial enterprise gateway • Java Enterprise • Management and Dissemination of raster and vector data • Standards compliant • OGC WCS 1.0, 1.1.1 (RI), 2.0.1 🡪 Download Services • OGC WFS 1.0, 1.1 (RI), 2.0.0 🡪 Download Services • OGC WMS 1.1.1, 1.3.0 🡪 View Services • OGC WPS 1.0.0 🡪 Transformation Services • OGC CSW 2.0.2 🡪 Discovery Services • OGC APIs: • Features, Images, Styles and Tiles INSPIRE support in GeoServer
6. INSPIRE support in GeoServer • INSPIRE Compliance we have (capabilities) full internationalization! • Available specific INSPIRE extension • Free and Open Source • New GUI Sections on services • Inject new elements in GetCapabilities • Download Services • OGC WFS 2.0 and GML 3.2.1 • OGC WCS 2.0.1 Enhancements in progress • OGC API - Features GeoJSON (and GML) • View Services • OGC WMS 1.1.1, 1.3 • OGC WMTS 1.0.0 • Transformation Service • OGC WPS 1.0.0, gs:Reproject process
7. GeoServer App-Schema • Free and open source extension: • PostgreSQL, Oracle, Solr and MongoDB • Support for complex features: 1 2 3 4 5
8. GeoServer App-Schema • No special schema requirement on the data sources! App-Schema produces the expected GML output by transforming data on the fly!
9. HALE App-Schema plugin • Defining App-schema mappings manually can be complex! • HALE App-Schema Plugin to the rescue! • Create HALE alignment visually • Create GeoServer App-Schema mapping from it • Upload it automatically through GeoServer REST API Automatic export to GeoServer!
10. HALE App-Schema plugin • Available at (compatible with HALE 3.5.x): • http://hale-geoserver.geo-solutions.it/index.html • Open Source visual editor for schema mappings! • Main features: • Import source schema(s) and data • Shapefile, databases, XSD / XML, etc… • Import target schema(s) • XSD, databases, INSPIRE schemas • Transformations are: • Graphically designed • Interactively executed • Interactively viewed
11. New mappings approaches
12. Our use case WARNING: Over Simplified Example!
13. Separation of concerns Data Modeling Data Storage
14. Separation of concerns Data Modeling Data Storage Publishing
15. Separation of concerns Data Modeling Data Storage Data Mapping Publishing
16. Separation of concerns Data Modeling Data Storage Data Mapping Building Publishing
17. Separation of concerns Data Modeling Data Storage Data Mapping Querying Building Publishing
18. Separation of concerns Data Modeling Data Storage Data Mapping Querying Building Publishing Currently the target GML schema leads the dance!
19. Currently GML leads the dance!
20. Currently GML leads the dance! Currently the target GML schema leads the dance! "The entry ticket is too high, which reduces the uptake!" & | "but not all data is a flat table how do we do?”
21. Smart data loader jumps in! Data Modeling Data Storage Data Mapping Querying Building Publishing Currently the target GML schema leads the dance! Smart Data Loader
22. Features templating jumps in! Data Modeling Data Storage Data Mapping Querying Building Publishing Currently the target GML schema leads the dance! Smart Data Loader Features Templating (WYSIWYG) Features Templating (WYSIWYG) Features Templating (WYSIWYG)
23. Flow of data overview Domain Model OGC Services and OGC APIs GML Output Format GeoJSON Output Format Others Output Format Data Smart Data Loader Memory Features Stream Features Templating HERE HERE
24. Smart data loader • Free and Open Source GeoServer extension. • Will Automatically: ◦ Walk the Data Source and produces a Domain Model. ◦ Generate an internal target GML Schema and an internal App-Schema Mapping. ◦ Build an in-memory Stream of Complex Features. • The target Output Format and target schema are not important at this stage!
25. Smart data loader demo
26. Features templating • Free and Open Source GeoServer extension! • What You See Is What You Get (WYSIWYG) approach: ◦ Define how your features should appear with imperative instructions. ◦ Query the dataset using the output format structure thanks to backwards mapping capability. • Integrated with both OGC API - Features, WFS and WMS. • Efficient Acts as a mapping over an in-memory stream of features!
27. Our use case (reminder) WARNING: Over Simplified Example!
28. Stations GeoJSON template 1. { 2. "type":"FeatureCollection", 3. "features":[ 4. {"$source":"stations"}, 5. { 6. "Identifier":"${id}", 7. "geometry":"${position}", 8. "properties":{ 9. "Name":"${common_name}", 10. "Code":"$${strConcat('STATION-', property('code'))}", 11. "Location":"$${toWKT(property('position'))}", 12. "Observations":[ 13. {"$source":"observations"}, 14. { 15. "Timestamp":"${time}", 16. "Value":"$${strConcat(strConcat(property('value'), 17. 'property('parameters.param_unit'))}", 18. "Type":"${parameters.param_name}" 19. } 20. ] 21. } 22. } 23. ] 24. }
29. Stations GeoJSON template 1. { 2. "type":"FeatureCollection", 3. "features":[ 4. {"$source":"stations"}, 5. { 6. "Identifier":"${id}", 7. "geometry":"${position}", 8. "properties":{ 9. "Name":"${common_name}", 10. "Code":"$${strConcat('STATION-', property('code'))}", 11. "Location":"$${toWKT(property('position'))}", 12. "Observations":[ 13. {"$source":"observations"}, 14. { 15. "Timestamp":"${time}", 16. "Value":"$${strConcat( 17. strConcat(property('value'), 18. 'property('parameters.param_unit'))}", 19. "Type":"${parameters.param_name}" 20. } 21. ] 22. } 23. } 24. ] 25. } 1. { 2. "type":"FeatureCollection", 3. "features":[ 4. { 5. "Identifier":7, 6. "geometry":{ 7. "type":"Point", 8. "coordinates":[11.34, 44.5] 9. }, 10. "properties":{ 11. "Name":"Bologna", 12. "Code":"STATION-BOL", 13. "Location":"POINT (11.34 44.5)", 14. "Observations":[ 15. { 16. "Timestamp":"2016/28...", 17. "Value":"35.0 C", 18. "Type":"temperature" 19. }, 20. { 21. "Timestamp":"2016/29...", 22. "Value":"80.0 Km/h", 23. "Type":"wind speed" 24. } 25. ] 26. } 27. } 28. (...)
30. Features templating demo
31. Creating the templates
32. Validating the templates
33. Content negotiation
34. Use cases highlights
35. BRGM - Boreholes use case • Demo with almost 1 million boreholes: • Available through: • WFS 2.0.0 • WFS 3.0.0 • OGC API – Features • Simple Features (SF-0) and Complex Features (SF-1)!
36. BRGM - Boreholes use case • Templated JSON-LD output! • What You See Is What You Get!
37. BRGM - Boreholes use case • Filtering using CQL is supported: • http://.../geoserver/ogc/features/collections/eposb:Borehole/items?f=applica tion/ld+json&limit=50&filter=features.gsmlp:boreholeLength_m.om:amount> 83&filter-lang=cql-text Filtering using the JSON-LD properties!
38. NPRA – DATEX II use case • Complex GeoJSON served using WMS and WFS! • Norwegian Public Road Administration (NPRA). • DATEX II (3.0), traffic related data: We are using GeoServer MongoDB Schemaless!
39. The End Questions? nuno.oliveira@geosolutionsgroup.com marco.volpini@geosolutionsgroup.com info@geosolutionsgroup.com