State of GeoServer Q1 2021, 2.18 and beyond

Webinar from GeoSolutions on GeoServer 2.18 and the coming 2.19: Improvements on raster analysis, GeoPackage metadata and styles, S3 COG support and more. We will have a 30 minutes presentation and 30 minutes of Q&A with our lead GeoServer developer, Andrea Aime.

Transcript

1. State of GeoServer Q1 2021,
   2.18 and beyond
   Feb 3, 2021
   Luis E. Bermudez, Ph.D., Andrea Aime
   Virtual
   

   View Slide

2. Agenda and Q&A
   • 11:00 - 11:10 Introduction
   – Luis Bermudez
   • 11:10 - 12:00 State of
   GeoServer – Andrea Aime
   • 12:00 Poll
   • 12:00 - 12:30 AAA (Ask
   Anything to Andrea)
   

   View Slide

3. Robust Geospatial
   Server
   • GeoServer is the lead open source server to publish
   geospatial data at scale.
   • GeoServer is a robust solution for management and
   disseminating geospatial data according to the OGC,
   ISO and INSPIRE standards.
   • Organizations use GeoServer to serve petabytes of
   data, from point data (e.g. real-time world-wide ship
   positions) to massive earth observation data.
   

   View Slide

4. GeoServer
   WFS
   1.0, 1.1,
   2.0
   WMS
   1.1.1
   1.3.0
   PostGIS
   Oracle
   H2
   DB2
   SQL
   Server
   MySql
   Spatialite
   GeoCouch
   Shapefile
   GeoPackage
   ----------
   ----------
   ---------
   ----------
   ----------
   ----------
   ---------
   ----------
   ----------
   ----------
   ---------
   ----------
   ArcSDE
   WFS
   PNG, GIF
   JPEG
   TIFF, GeoTIFF
   SVG, PDF
   KML/KMZ
   Shapefile
   GML2
   GML3
   GeoRSS
   GeoJSON
   CSV/XLS
   Raw vector
   data
   Servers
   Styled
   maps
   DBMS
   Vector files
   WCS
   1.0,1.1.1
   2.0.1
   GeoTIFF
   WMS
   ArcGrid
   GTopo30
   Img+world
   Mosaic
   MrSID
   JPEG 2000
   ECW,Pyramid
   Oracle
   GeoRaster,
   PostGis Raster
   NetCDF
   GeoPackage
   Raster files
   Raw raster
   data GeoTIFF
   ArcGrid
   GTopo30
   Img+World
   Google
   WPS
   1.0.0
   Vector
   Tiles
   Vector tiles
   format
   MapBox Vector
   (MVT)
   GeoJSON
   TopoJSON
   OGC
   Open
   APIs
   Mostly from
   directory Others
   GeoWebCache
   Authentication ( OKTA, LDAP,
   CAS, WSO2, OAUTH..)
   GeoFence
   Active Clustering (Hazelcast)
   …
   

   View Slide

5. World leader in the development and maintenance of
   open source enterprise software for geospatial
   information management
   

   View Slide

6. UN FAO (CIOK, FIGIS,
   NRL, FORESTRY,
   ESTG), UN WFP,
   World Bank, DLR,
   EUMETSAT, JRC,
   ARPAT, NATO CMRE,
   UNESCO, IGAD,
   UNEP, ...
   BAYER, BASF,
   DigitalGlobe, MDA,
   TOPCON, SwissRE, e-
   GEOS, Halliburton,
   ..
   More than 200 clients since 2006
   

   View Slide

7. GeoSolutions Support
   • Advice regarding a geospatial challenge
   • Architect, help configure and deploy clusters of GeoServers
   in the cloud
   • Automate deployments from different environments
   • Performance monitoring
   • Develop custom application (server or client slide)
   • Optimize ingesting and publishing geospatial data
   • Training
   • Other technical support
   

   View Slide

8. Enterprise Support Services
   

   View Slide

9. Subscription Services
   Example Premium Plan
   4 hours – response time
   16 # of incidents per year
   20 Additional Support hours
   24/7 support
   Oracle, SQL Server and PostGIS
   Support
   

   View Slide

10. Training
    

    View Slide

11. Customized Solutions
    

    View Slide

12. A Solution for Data
    Management
    Identify how and where
    products, services, and
    data are utilized
    The monitor extension tracks
    requests made against a
    GeoServer instance. With the
    extension request data can be
    persisted to a database, used
    to generate simple reports ,
    and routed to a customized
    request audit log.
    10 Most Accessed Layers 10 Most Active Users
    10 Most Active IPs Request types
    

    View Slide

13. GeoSolutions Contact Information
    Luis E. Bermudez, Ph.D.
    CEO
    GeoSolutions USA
    https://www.geosolutionsgroup.com/contacts/
    +1 (301) 337-7055
    luis.bermudez@geosolutionsgroup.com
    https://www.linkedin.com/in/bermudez
    https://www.geosolutionsgroup.com
    https://twitter.com/geosolutions_it
    

    View Slide

14. Andrea Aime
    GeoSolutions
    State of GeoServer
    Q1 2021
    

    View Slide

15. GeoServer 2.18.x
    

    View Slide

16. GeoServer 2.18.x
    l
    Releases already performed:
    l
    2.18.0, released September 2020
    l
    2.18.1, released November 2020
    l
    2.18.2, released January 2021
    l
    Planned releases:
    l
    2.18.3, April 2021
    l
    2.18.4, June 2021
    l
    2.18.5, August 2021
    

    View Slide

17. GeoServer 2.18.0 - extensions
    

    View Slide

18. GeoServer 2.18.0 - community
    https://build.geoserver.org/geoserver/2.18.x/community-latest/
    

    View Slide

19. Small GUI improvements
    l
    "Apply" button on most pages
    l
    Better visual separation for "cancel"
    l
    Save/Apply/Cancel bar always visible
    

    View Slide

20. Jiffle now supports multiband
    l
    GeoServer uses "Jiffle" as its map algebra
    language
    l
    In 2.18.0 improved to support multi-band output
    l
    Very simple before/after/after-before comparison
    script, generating a 3-band image from two
    separate one-banded images:
    before = src[0];
    after = src[1];
    dest[0] = before;
    dest[1] = after;
    dest[2] = after - before;
    

    View Slide

21. Ship density in the Black Sea
    T1
    T2
    T1/T2/DIFFERENCE
    

    View Slide

22. Projection subsytem improvements
    l
    Goode's interrupted homolosine
    l
    Full "advanced projection handling" support
    l
    Polygon sliced automatically at interruptions,
    dynamic densification
    

    View Slide

23. Projection subsytem improvements
    l
    Azimuthal equidistant support improved
    l
    Set of improvements to support Equi7Grid
    (method to store raster data with low distortion)
    

    View Slide

24. Projection subsytem improvements
    l
    Azimuthal equidistant polar area improvements
    l
    Example source
    is vector, 3857
    (hence hole)
    

    View Slide

25. Features template community module
    l
    Old "JSON-LD" community module
    l
    Can now generate GeoJSON too
    Simple features
    Complex features JSON-LD
    GeoJSON
    JSON template
    

    View Slide

26. GeoPackage improvements
    l
    GeoServer "gs-geopkg" community module
    contains a WPS process that can produce
    GeoPackages
    l
    During OGC Testbed 16 it has been improved
    significantly:
    l
    Embeddig metadata
    l
    Embedding styles
    l
    Increased writing speed (2x or more)
    l
    Enumeration support
    l
    Generalized tables
    l
    GeoHash sorting
    

    View Slide

27. GeoPackage metadata embedding
    l
    Metadata links target embedded in generated
    GeoPackage
    

    View Slide

28. GeoPackage, storing styles
    l
    OS Mastermap Topography
    l
    Comes with 6 layers, and 4 different ways to
    present them ("layer groups")
    Backdrop Light
    Outdoor Standard
    

    View Slide

29. GeoPackage, storing styles
    l
    GeoPackage "portrayal" extension stores:
    l
    Style bodies (eventually in multiple languages, e.g.,
    MBStyles and SLD)
    l
    Style icons (fully self contained!)
    l
    Semantic annotation extension
    l
    Links styles to tables (tags both with the same
    annotation)
    l
    Metadata extension
    l
    Stores "operational pictures", stacks of layers and the
    associated styles
    l
    Stored as OWS context documents in the metadata
    table
    

    View Slide

30. GeoPackage writing speed
    l
    Writing OS Master Topography, full UK
    l
    Six tables, 500 million records!
    l
    300GB as stored in PostgreSQL (roughly the size of a
    world wide OSM database!)
    Before TB16 After TB16
    Cut it off at 24H, did not finish Terminates in 9 hours
    

    View Slide

31. GeoPackage enumerations
    l
    Added support for the GeoPackage "schema
    extension" (one of the core extensions)
    l
    Ability to map integer numbers to value
    (enumeration)
    l
    Lots of enumerated fields in Mastermap
    

    View Slide

32. GeoPackage enumerations
    l
    Part of the standard
    l
    Yet generally not well
    supported in clients
    

    View Slide

33. GeoPackage, arrays
    l
    Storing them as JSON arrays
    l
    Using the JSON1 SQLite extension for querying
    l
    Eventually enumerated
    l
    TB16 specific extension, not part of the std
    

    View Slide

34. GeoPackage, GeoHash sorting
    l
    Ability to write records sorted by Geohash
    l
    Up to 100x speed on Mastermap topography
    (topographicline table, 330 million records)
    https://postgis.net/workshops/postgis-intro/clusterindex.html
    Not an extension, just
    improved physical layout!
    

    View Slide

35. GeoPackage, generalized tables
    l
    Speed up multi-scale representation
    l
    OS ZoomStack, free 10GB GeoPackage download
    

    View Slide

36. GeoPackage, generalized tables
    l
    Speed up multi-scale representation
    l
    OSM style, secondary tables with simpler
    geometries and less records
    Table name Record count
    waterlines 2.431.848,00
    waterlines_g1 164.208,00
    waterlines_g2 12.752,00
    waterlines_g3 2.481,00
    Up to 100x speedup
    (depending on dataset)
    

    View Slide

37. GeoPackage, more info
    l
    Get more information about this work!
    l
    10 minutes video presentation
    l
    Testbed 16 GeoPackage Engineering Report
    

    View Slide

38. GeoServer 2.19.x
    

    View Slide

39. Modules: going up, going down
    Extension
    Community
    GWC S3
    WPS
    download
    WPS
    JDBC
    Params
    extractor
    WMTS
    multidim
    COG
    ArcSDE Script
    DGGS
    S3
    GeoTiff
    Imagemosaic
    JDBC
    

    View Slide

40. GWC – S3
    l
    GWC blobstore
    l
    Stores GWC tiles on S3.
    l
    Cheaper alternative to local file system for cloud
    deploys.
    GWC
    GWC
    GWC
    WMTS
    OGC API - Tiles
    

    View Slide

41. WPS – Download
    l
    Set of WPS processes to replace WFS/WCS/WMS for large download
    workloads
    l
    Asynch request support
    l
    gs:Download à Vector and raster large download
    l
    gs:DownloadMap à Large map download (with eventual external layers)
    l
    gs:DownloadAnimation à MP4 generator based on maps and time series
    Execute wps:Download (asynch)
    GetStatus
    Download sesult
    

    View Slide

42. WPS - JDBC
    l
    WPS status storage
    l
    Stores status of async requests in a database of choice
    l
    Allows clustered usage of WPS async requests
    LOAD BALANCER
    

    View Slide

43. Params-extractor
    l
    Tool to smuggle vendor params into capabilities URLs
    l
    Makes un-aware shrink wrapped GIS software use
    vendor parameters (e.g., CQL_FILTER, VIEW_PARAMS,
    ENV, …)
    /geoserver/tiger/wms/H11
    ?SERVICE=WMS
    &VERSION=1.1.1
    &REQUEST=GetMap
    &…
    /geoserver/tiger/wms
    ?SERVICE=WMS
    &VERSION=1.1.1
    &REQUEST=GetMap
    &CQL_FILTER=CFCC%3D%27H11%27
    &…
    

    View Slide

44. WMTS multidim
    l
    Extensions to the WMTS protocol developed in
    Testbed12
    l
    Extra requests allowing to explore dimensions
    (time, elevation and others)
    l
    DescribeDomains: list of dimensions and their
    description
    l
    GetDomainValues: list values in a dimension,
    eventually filtering on other dimension ranges (e.g.,
    "what are the times available in this bbox"?)
    l
    GetFeature: get tuples of all actual dimension value
    combinations
    l
    GetHistogram: produce histogram of values along a
    given dimension, with filtering
    

    View Slide

45. WMTS multidim
    l
    MapStore Timeline plugin, powered by WMTS
    multidim
    Animation
    controls
    Histogram
    Filtering by
    current view
    

    View Slide

46. COG replaces S3-GeoTiff
    l
    The COG module provides improved support for
    Cloud Optimized GeoTiffs
    l
    Works both on HTTP servers and S3
    l
    Leverages the COG structure to minimize the
    download
    l
    Integration with REST mosaic harvest operations in
    GeoServer
    l
    S3-GeoTIFF instead saved chunks of the remote
    TIFF on disk, without really leveraging its structure
    

    View Slide

47. DGGS support
    l
    Set of community modules, under the GeoServer
    OGC API umbrella, to support Discrete Global Grid
    Systems (hierarchical, equal-ish area tassellations
    of the word)
    

    View Slide

48. DGGS geometry store
    l
    No data per se, but allows to explore the DGGS and
    learn its structure
    rHealPix H3
    

    View Slide

49. DGGS ClickHouse
    l
    Allows for actual data
    storage
    l
    Uses ClickHouse OLAP
    database
    l
    DGGS zones (aka cells) are
    stored by cell identifier
    l
    Fast lookup and
    aggregation using
    multiple cores/nodes
    

    View Slide

50. DGGS API
    l
    Grab data, explore
    relationships between
    cells
    l
    E.g., children/parents,
    neighbors
    Neighbors
    Grand-children of a cell
    

    View Slide

51. DGGS API
    l
    Map polygons to set of DGGS cells at a given
    resolution
    

    View Slide

52. DGGS for analysis
    l
    Multi-resolution database
    l
    Perform aggregations at
    the desired zoom level
    l
    Choose between speed
    and accuracy
    

    View Slide

53. DGGS DAPA API
    l
    Data Access and Processing API
    l
    Multi-aggregations on a given layer, based on
    space-time filter
    l
    Aggregate on time only or space only: series of value
    l
    Aggregate on both time and space: single values
    l
    DGGS implementation allows to choose desired
    resolution
    l
    Fast
    ClickHouse
    aggregation
    l
    Space/time aggregation
    l
    16 attributes
    l
    8.9 million records aggregated in
    less than a second
    l
    On spinning disks (RAID)
    

    View Slide

54. DGGS: learn more
    l
    Dedicated 10 minutes presentation
    l
    OGC Testbed 16 - DGGS AND DGGS API
    engineering report
    l
    OGC Testbed-16: Data Access and Processing
    Engineering Report
    

    View Slide

55. Modules demoted and removed
    l
    The script modules allowed to script certain
    GeoServer plugins in Python, Javascript, Groovy
    l
    The module fell un-maintained for years, we
    eventually had to drop it
    l
    ArcSDE is no longer supported by ESRI, and the
    module was also un-maintained
    l
    Imagemosaic-JDBC allows to read rasters from
    databases. No longer maintained, dropped to
    "community" status for the time being.
    

    View Slide

56. That’s all!
    https://www.geosolutionsgroup.com/contacts/
    

    View Slide