Publishing INSPIRE datasets in GeoServer made easy with Smart Data Loader and Features Templating (FOSS4G 2022 Edition)

GeoServer is a well-established multiplatform, open-source geospatial server providing a variety of OGC services, including WMS (view services), WFS and WCS (download services) as well as WPS (spatial data processing services). Among the open-source GIS web servers, GeoServer is well known for the ease of setup, the web console helping the administrator to configure data and services, the variety of OGC services available out of the box, as well as the rich set of data sources that it can connect to (open source, such as PostGIS as well as proprietaries, such as ArcSDE, Oracle or ECW rasters). GeoServer also provides several OGC APIs, including the OGC API - Features which recently attracted the interest of the INSPIRE community.
As far as the INSPIRE scenario is concerned GeoServer has extensive support for implementing view and download services thanks to its core capabilities but also to a number of free and open-source extensions; undoubtedly the most well-known (and dreaded) extension is App-Schema which can be used to publish complex data models (with nested properties and multiple-cardinality relationships) and implement sophisticated download services for vector data. Based on the feedback of App-Schema users collected over the years, a new generation of open-source mapping extensions have been implemented in GeoServer: Smart Data Loader and Features Templating, these extensions are built on top of App-Schema and ease the mapping of the data models by allowing us to act directly on the domain model and target output schema using a what you see is what you get approach.
This presentation will introduce the new GeoServer Smart Data Loader and Features Templating extensions, covering in detail ongoing and planned work on GeoServer. We will also provide an overview about how those extensions are serving as a foundation for new approaches to publish complex data: publishing data models directly from MongoDB, embracing the NoSQL nature of it, and support new output formats like JSON-LD which allows us to embed well known semantics in our data. Eventually, real-world use-cases from the organizations that have selected GeoServer and GeoSolutions to support their use cases will be introduced to provide the attendees with references and lessons learned that could put them on the right path when adopting GeoServer.

Transcript

1. Nuno Oliveira
   Marco Volpini
   GeoSolutions
   Publishing INSPIRE datasets
   in GeoServer made easy with
   Smart Data Loader and
   Features Templating
   

   View Slide

2. GeoSolutions
   Enterprise Support
   Services
   Deployment
   Subscription
   Professional
   Training
   Customized
   Solutions
   GeoNode
   • Offices in Italy & US, Global Clients/Team
   • 40+ collaborators, 30+ Engineers
   • Our products
   • Our Offer
   

   View Slide

3. Affiliations
   We strongly support Open
   Source, it Is in our core
   We actively participate in
   OGC working groups and
   get funded to advance new
   open standards
   We support standards
   critical to GEOINT
   

   View Slide

4. Where are we?
   

   View Slide

5. ● Available at http://www.geoserver.org
   ● GeoSpatial enterprise gateway
   ●
   Java Enterprise
   ●
   Management and Dissemination of raster and vector data
   ● Standards compliant
   ●
   OGC WCS 1.0, 1.1.1 (RI), 2.0.1 🡪 Download Services
   ●
   OGC WFS 1.0, 1.1 (RI), 2.0.0 🡪 Download Services
   ●
   OGC WMS 1.1.1, 1.3.0 🡪 View Services
   ●
   OGC WPS 1.0.0 🡪 Transformation Services
   ●
   OGC CSW 2.0.2 🡪 Discovery Services
   ● OGC APIs:
   ● Features, Images, Styles and Tiles
   INSPIRE support in GeoServer
   

   View Slide

6. INSPIRE support in GeoServer
   ● INSPIRE Compliance we have (capabilities) full internationalization!
   ● Available specific INSPIRE extension
   ● Free and Open Source
   ● New GUI Sections on services
   ● Inject new elements in GetCapabilities
   ● Download Services
   ● OGC WFS 2.0 and GML 3.2.1
   ● OGC WCS 2.0.1 Enhancements in progress
   ● OGC API - Features GeoJSON (and GML)
   ● View Services
   ● OGC WMS 1.1.1, 1.3
   ● OGC WMTS 1.0.0
   ● Transformation Service
   ● OGC WPS 1.0.0, gs:Reproject process
   

   View Slide

7. GeoServer App-Schema
   ● Free and open source extension:
   ● PostgreSQL, Oracle, Solr and MongoDB
   ● Support for complex features:
   1
   2
   3
   4
   5
   

   View Slide

8. GeoServer App-Schema
   ● No special schema requirement on the data sources!
   App-Schema produces the
   expected GML output by
   transforming data on the fly!
   

   View Slide

9. HALE App-Schema plugin
   ● Defining App-schema mappings manually can be complex!
   ● HALE App-Schema Plugin to the rescue!
   ● Create HALE alignment visually
   ● Create GeoServer App-Schema mapping from it
   ● Upload it automatically through GeoServer REST API
   Automatic
   export to
   GeoServer!
   

   View Slide

10. HALE App-Schema plugin
    ● Available at (compatible with HALE 3.5.x):
    ●
    http://hale-geoserver.geo-solutions.it/index.html
    ● Open Source visual editor for schema mappings!
    ● Main features:
    ● Import source schema(s) and data
    ●
    Shapefile, databases, XSD / XML, etc…
    ● Import target schema(s)
    ●
    XSD, databases, INSPIRE schemas
    ● Transformations are:
    ●
    Graphically designed
    ●
    Interactively executed
    ●
    Interactively viewed
    

    View Slide

11. New mappings approaches
    

    View Slide

12. Our use case
    WARNING: Over Simplified Example!
    

    View Slide

13. Separation of concerns
    Data
    Modeling
    Data
    Storage
    

    View Slide

14. Separation of concerns
    Data
    Modeling
    Data
    Storage
    Publishing
    

    View Slide

15. Separation of concerns
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Publishing
    

    View Slide

16. Separation of concerns
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Building Publishing
    

    View Slide

17. Separation of concerns
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Querying Building Publishing
    

    View Slide

18. Separation of concerns
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Querying Building Publishing
    Currently the target GML
    schema leads the dance!
    

    View Slide

19. Currently GML leads the dance!
    

    View Slide

20. Currently GML leads the dance!
    Currently the target GML
    schema leads the dance!
    "The entry ticket is too high, which reduces
    the uptake!" & | "but not all data is a flat
    table how do we do?”
    

    View Slide

21. Smart data loader jumps in!
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Querying Building Publishing
    Currently the
    target GML
    schema leads
    the dance!
    Smart Data
    Loader
    

    View Slide

22. Features templating jumps in!
    Data
    Modeling
    Data
    Storage
    Data
    Mapping
    Querying Building Publishing
    Currently the
    target GML
    schema leads
    the dance!
    Smart Data
    Loader
    Features
    Templating
    (WYSIWYG)
    Features
    Templating
    (WYSIWYG)
    Features
    Templating
    (WYSIWYG)
    

    View Slide

23. Flow of data overview
    Domain
    Model
    OGC
    Services
    and
    OGC APIs
    GML
    Output Format
    GeoJSON
    Output Format
    Others
    Output Format
    Data
    Smart Data Loader
    Memory
    Features
    Stream
    Features
    Templating
    HERE
    HERE
    

    View Slide

24. Smart data loader
    ● Free and Open Source GeoServer
    extension.
    ● Will Automatically:
    ○ Walk the Data Source and produces a
    Domain Model.
    ○ Generate an internal target GML
    Schema and an internal App-Schema
    Mapping.
    ○ Build an in-memory Stream of Complex
    Features.
    ● The target Output Format and target
    schema are not important at this stage!
    

    View Slide

25. Smart data loader demo
    

    View Slide

26. Features templating
    ● Free and Open Source GeoServer extension!
    ● What You See Is What You Get (WYSIWYG) approach:
    ○ Define how your features should appear with imperative
    instructions.
    ○ Query the dataset using the output format structure thanks to
    backwards mapping capability.
    ● Integrated with both OGC API - Features, WFS and
    WMS.
    ● Efficient Acts as a mapping over an in-memory
    stream of features!
    

    View Slide

27. Our use case (reminder)
    WARNING: Over Simplified Example!
    

    View Slide

28. Stations GeoJSON template
    1. {
    2. "type":"FeatureCollection",
    3. "features":[
    4. {"$source":"stations"},
    5. {
    6. "Identifier":"${id}",
    7. "geometry":"${position}",
    8. "properties":{
    9. "Name":"${common_name}",
    10. "Code":"$${strConcat('STATION-', property('code'))}",
    11. "Location":"$${toWKT(property('position'))}",
    12. "Observations":[
    13. {"$source":"observations"},
    14. {
    15. "Timestamp":"${time}",
    16. "Value":"$${strConcat(strConcat(property('value'),
    17. 'property('parameters.param_unit'))}",
    18. "Type":"${parameters.param_name}"
    19. }
    20. ]
    21. }
    22. }
    23. ]
    24. }
    

    View Slide

29. Stations GeoJSON template
    1. {
    2. "type":"FeatureCollection",
    3. "features":[
    4. {"$source":"stations"},
    5. {
    6. "Identifier":"${id}",
    7. "geometry":"${position}",
    8. "properties":{
    9. "Name":"${common_name}",
    10. "Code":"$${strConcat('STATION-', property('code'))}",
    11. "Location":"$${toWKT(property('position'))}",
    12. "Observations":[
    13. {"$source":"observations"},
    14. {
    15. "Timestamp":"${time}",
    16. "Value":"$${strConcat(
    17. strConcat(property('value'),
    18. 'property('parameters.param_unit'))}",
    19. "Type":"${parameters.param_name}"
    20. }
    21. ]
    22. }
    23. }
    24. ]
    25. }
    1. {
    2. "type":"FeatureCollection",
    3. "features":[
    4. {
    5. "Identifier":7,
    6. "geometry":{
    7. "type":"Point",
    8. "coordinates":[11.34, 44.5]
    9. },
    10. "properties":{
    11. "Name":"Bologna",
    12. "Code":"STATION-BOL",
    13. "Location":"POINT (11.34
    44.5)",
    14. "Observations":[
    15. {
    16. "Timestamp":"2016/28...",
    17. "Value":"35.0 C",
    18. "Type":"temperature"
    19. },
    20. {
    21. "Timestamp":"2016/29...",
    22. "Value":"80.0 Km/h",
    23. "Type":"wind speed"
    24. }
    25. ]
    26. }
    27. }
    28. (...)
    

    View Slide

30. Features templating demo
    

    View Slide

31. Creating the templates
    

    View Slide

32. Validating the templates
    

    View Slide

33. Content negotiation
    

    View Slide

34. Use cases highlights
    

    View Slide

35. BRGM - Boreholes use case
    ● Demo with almost 1 million boreholes:
    ● Available through:
    ● WFS 2.0.0
    ● WFS 3.0.0
    ● OGC API – Features
    ● Simple Features
    (SF-0) and Complex
    Features (SF-1)!
    

    View Slide

36. BRGM - Boreholes use case
    ● Templated
    JSON-LD
    output!
    ● What You
    See Is What
    You Get!
    

    View Slide

37. BRGM - Boreholes use case
    ● Filtering using CQL is supported:
    ● http://.../geoserver/ogc/features/collections/eposb:Borehole/items?f=applica
    tion/ld+json&limit=50&filter=features.gsmlp:boreholeLength_m.om:amount>
    83&filter-lang=cql-text
    Filtering using the
    JSON-LD
    properties!
    

    View Slide

38. NPRA – DATEX II use case
    ● Complex GeoJSON served using WMS
    and WFS!
    ● Norwegian Public Road Administration (NPRA).
    ● DATEX II (3.0), traffic related data:
    We are using
    GeoServer
    MongoDB
    Schemaless!
    

    View Slide

39. The End
    Questions?
    nuno.oliveira@geosolutionsgroup.com
    marco.volpini@geosolutionsgroup.com
    info@geosolutionsgroup.com
    

    View Slide