State of GeoServer, 2.22 Release

In this presentation, our GeoServer Lead Andrea Aime walks us through the key points of the GeoServer release 2.22 covering also what to expect for the next release.

1. State of GeoServer 2.22
2. Jody Garnett jody.garnett@geocat.net @jodygarnett GeoCat Government Geographic Data publishing Andrea Aime Technical Lead andrea.aime@geo-solutions.it @geowolf GeoSolutions Innovative, robust, cost-effective solutions leveraging best-of-breed Open Source products. This presentation is brought to you by
3. GeoServer at a Glance Java Web Application to share and edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WMTS – tiles TMS – tiles WMS-C – tiles Extension/community protocols WPS – process CSW – search OpenSearch for Earth Observation – search OGC API - prototype json + rest standards
4. GeoServer Community 2022 Update
5. Core committers are nominated by their peers and trusted with the ability to approve pull-requests. We also support community commit access by request (often to work on a specific research and development topic): • 22 committers New committer this week: Joseph Miller! Project Steering Committee serves to guide the project drawing from the various groups with a stake in the success of the project. • Alessio Fabiani • Andrea Aime • Ian Turton • Jody Garnett • Jukka Rahkonen • Kevin Smith • Nuno Oliveira • Simone Giannecchini • Torben Barsballe GeoServer Team
6. GeoServer maintains a list service providers: • Core Contributors Ongoing commitment to the project devoting resources to security fixes, releases and maintenance activities. • Experienced Providers Successfully contribute functionality to the project on behalf of their customers. • Additional Services Providers Provide training, setup and integration support and assistance using GeoServer. (request: please take part in community) GeoServer Service Providers Policy change recognize participation: Core contributors directly contribute to project sustainability and are recognized for: • core, committer, support Experienced providers participate in aspects of the project as community members: • development, coding, documentation, outreach, translation, training, service, product
7. • Infrastructure ◦ geoserver.org (GitHub Pages) geoserver.org/blog ◦ github.com/geoserver (GitHub) ◦ blog.geoserver.org (github) ◦ repo.osgeo.org (OSGeo) ◦ domain names (OSGeo) ◦ downloads (SourceForge) ◦ email (SourceForge) ◦ build.geoserver.org (GeoSolutions) ◦ web map (GeoSolutions) • Transition from Boundless → OSGeo ◦ docs.geoserver.org (s3) GeoServer Infrastructure
8. Incoming: • gt-json-core and gt-json-store • gwc-mbtiles • LibDeflate • VSI (Virtual File system GDAL reader) Outgoing: • gt-geojson (soon) • epsg-oracle • Ingres datastore • geogig • importer-bdb • nsg-wfs-profiles • qos • wfs3 Community modules GeoServer maintains a “community space” for experiments and new developers to work. • Developers propose a community module idea and request commit access • We make sure they compile! • May be proposed as an extension when both the author and functionality is ready Spot a community module of interest, reach out to see how you can help. Graduating • csw-iso and metadata • kml-ppio Downgraded • Teradata store
9. GeoServer Releases 2022/2023 Update
10. stable maintenance development May 2022 November 2022 March 2023 2.21.x 2.22.x Releases covered by this presentation 2.23.x September 2023 Java 11 minimum! YOU ARE HERE
11. Are you using a older version? Upgrade! • User list and stack exchange typically covers only supported versions ◦ In September 2022 “supported” means 2.21.x and 2.22.x ◦ In March 2023 “supported” will mean 2.22.x and 2.23.x • Security fixes added to supported versions only • Please upgrade your GeoServer installations! Photo by SpaceX on Unsplash
12. Upgrade? What’s in it for me? • Much! • Check what’s new in 2.21, 2.22 and incoming in 2.23 • Check the bottom of each slide to see who sponsored a certain feature, who implemented it, and what version contains it • This icon marks activities done without any sponsoring Version Author Sponsor ♥
13. Distribution and runtime
14. Official Docker Image • Breakout session after FOSS4G ◦ GSIP 192 - Official Docker Image ◦ https://github.com/geoserver/docker • Proposal 1: Luca Pasquali ◦ custom_build.sh ◦ option to use local geoserver.war ◦ download and install extensions ◦ option to include local data dir • Proposal 2: Nils Bühner ◦ enable / disable CORS ◦ Install extensions from external folder ◦ Install libs from external folder • Bolsena 2022 ◦ Reboot at Bolsena code sprint ◦ Combine both proposal to some extent ◦ First official release with 2.21.1 2.21.1 Nils Bühner Luca Pasquali ♥ $ docker pull docker.osgeo.org/geoserver:2.21.1 $ docker run -it -p8080:8080 \ docker.osgeo.org/geoserver:2.21.1
15. Windows Installer • Last released for GeoServer 2.14.3 ◦ “Lost” in infrastructure vandalism • Manual process has been in dev guide (for organizations requiring installer) • GSIP-198: small contract with GeoCat ◦ New NSIS installer ◦ New Service wrapper ◦ Automation for build.geoserver.org 2.21 Sander Schaminee GeoCat GeoServer OSGeo
16. Java 11 • GeoServer 2.23.0 will _require_ Java 11 • Java 17 supported in an experimental way 2.23 Community GeoServer
17. Mapping Data making a scene!
18. Layer group styles • A layer group style is a style name offering a different list of layers and styles to build the group • Use cases: ◦ Different representation of same layers ◦ Simplified representation of same map (less layers) • Example here: Ordnance Survey MasterMap (6 layers) in 4 alternative styles 2.21 Marco Volpini GeoSolutions SWAM
19. Symbology providers control • Point Vector symbology in SLD comes from “mark factories” • Control order of execution and availability in styles • For complex point maps, big speed improvements by putting the most used factories first 2.21 Fernando Mino EMSA Above: EMSA, real time tracking of 150k ships in the EU area
20. Improved mosaicking performance - Trying to mosaic several hundreds of images in the same output? - The index can have millions mind, we’re talking hundreds images to build a single GetMap output - Performance and memory usage for this use case have been improved by a couple of orders of magnitude (GetMap built with 300 images) 2.22 Andrea Aime GeoSolutions USGS
21. Improved hyperspectral performance - Hyperspectral images: those having hundreds of bands - Typically stored in band-interleaved structure (one data bank for all pixels in the same tile) - GS is now orders of magnitude faster at reading them 2.22 Andrea Aime GeoSolutions DLR Rows Cols Bands
22. New Natural Earth 1:50m demo layers - Simple map based on 1:50m data sets - Included in the built-in data directory 2.22/various Jody/Andrea GeoCat/GeoSolutions - For full layers support up to 1:10m, check this out: https://github.com/geosolutions-it/ne-styles
23. Data sources and formats News and Updates
24. Feature Type Customisation • Rename attributes • Change attribute order • Change attribute type • Generate attributes using expressions 2.21 Andrea Aime (GeoSolutions) Canton of Solothurn
25. • GeoServer now works and is tested with ◦ GDAL 3.x Series ◦ GDAL 3.2 and 3.4 tested ◦ (GDAL 3.5 build was not yet stable) • Primarily update to ImageIO-EXT GDAL 3.x Support 2.21 David Blasby GeoCat Rijkswaterstaat
26. COG for Google Cloud Storage - Cloud Optimized GeoTIFF native support for Google Cloud Storage - Allows access to authenticated sources - Native access, improved performance 2.21 Andrea Aime GeoSolutions Farmers Edge
27. COG for Azure - Cloud Optimized GeoTIFF native support for Azure blob storage - Allows access to authenticated sources - Native access, improved performance 2.22 Daniele Romagnoli GeoSolutions TOTAL
28. libdeflate support for GeoTIFF - libdeflate is a “Heavily optimized library for DEFLATE/zlib/gzip compression and decompression” - Currently used to decompress/compress GeoTIFF files - Community module adds out of the box support for Linux (for other OS, add native lib) - PNG encoding support in the roadmap 2.21 Andrea Aime GeoSolutions DLR Zlib Implementation CPU Time Original 1m 17s Intel 1m 15s CloudFlare 1m 4s Libdeflate 0m 43s http://www.htslib.org/benchmarks/zlib.html
29. STAC Datastore and Mosaic • Pull STAC items from a STAC API • Render footprints on WMS • Mosaic images on the fly • Community module! 2.22 Andrea Aime GeoSolutions DLR STAC API STAC datastore COGs Image mosaic World Settlements Footprints, STAC API + COG
30. Vector mosaicking store • Useful to handle thousands of files with similar structure • Database used only to index them • Much cheaper than storing everything in a database (on cloud) • Not suitable if on the fly aggregations are needed • Great if the end user application mostly pulls a few files (one?) at a time 2.22 Joseph Miller GeoSolutions TROO Community module!
31. Services News and Updates
32. GeoPackage output extension (for WMS and WFS) GeoPackage output for: • WFS • WMS - produces a rendered tileset Aside: • WPS download process can produce very simple geopackage output similar to WMS and WFS above • geopackage community module still a work-in-progress supporting experimental geopkg extensions like generalization 2.21 David Blasby GeoCat Rijkswaterstaat wfs?version=2.0.0&request=GetFeature& typeNames=topp:states&outputFormat=geopkg wms?version=1.1.0&request=GetMap&layers=...& styles=&bbox=...&srs=EPSG:4326&& width=1536&height=984&format=geopkg
33. CSW-ISO module graduation • The CSW-ISO module allows to generate ISO metadata records around GeoServer layer and services. This is a rare come back from community (used to be extension already) 2.22 @fileIdentifier.CharacterString=prefixedName identificationInfo.AbstractMD_Identification.citation.CI_Citation.title.CharacterString=title identificationInfo.AbstractMD_Identification.descriptiveKeywords.MD_Keywords.keyword.CharacterString=keywords identificationInfo.AbstractMD_Identification.abstract.CharacterString=abstract $dateStamp.Date= if_then_else ( isNull("metadata.date") , 'Unknown', "metadata.date") <gmd:MD_Metadata> <gmd:fileIdentifier> <gco:CharacterString>CoverageInfoImpl--4a9eec43:132d48aac79:-8000</gco:CharacterString> </gmd:fileIdentifier> <gmd:dateStamp> <gco:Date>Unknown</gco:Date> </gmd:dateStamp> <gmd:identificationInfo> <gmd:MD_DataIdentification> <gmd:extent> … Niels Charlier Scitus -
34. Metadata module graduation • The metadata module adds an editor for extra metadata information (to be used in CSW-ISO) The editor structure can be customized using YAML configs 2.22 Niels Charlier Scitus - attributes: - key: metadata-identifier fieldType: UUID - key: metadata-datestamp label: Date fieldType: DATETIME - key: data-language fieldType: DROPDOWN values: - dut - eng - fre - ger - key: topic-category fieldType: SUGGESTBOX occurrence: REPEAT values: - farming - …
35. KML output for WPS • The KML-PPIO (Process-Parameter-IO) module graduated, part of the WPS package now • Adds KML output to every process generating a feature collection • Better alignment for WFS vs WPS Download Process 2.21 Alessio Fabiani GeoSolutions GeoSolutions
36. MBTiles as GWC layer • Use a pre-seeded MBTiles file as a GWC layer • Serve the contents over all GWC tile protocols • Core in GWC, community module in GS 2.20 Daniele Romagnoli GeoSolutions DLR MBTiles
37. Descriptions in DescribeFeatureType • If a database has column described with comments, they can be advertised in the WFS XML schema returned by DescribeFeatureType • They will be soon also editable from GUI as well (for all vector data sources) • PR in review today! 2.22.3 Joseph Miller GeoSolutions GeoScience Australia
38. Configuration and Setup
39. New Welcome Page Layout GSIP-202 Welcome Page Layout - Available now in 2.22-M0 Milestone - Inspired by ogc-api community module - Use title and description to make a heading for each service - List capabilities and tools for each section - Change workspace - A welcome page for each workspace - Click to explore virtual web services - Change layer/group - A welcome page specific to a given layer/group - Layer specific services 2.22 Jody Garnett GeoCat GeoServer Enterprise GeoCat
40. Speed-up for many layers/many rules servers • The new home-page was very slow on some server • Reference server, 20k layers, 40k security rules! • Made GUI pay more attention to large setups, fall back on simple text boxes • Sped up security filtering in home page, capabilities, preview page by 2 orders of magnitude (yes, 200 times faster) 2.22 Andrea Aime GeoSolutioons Undisclosed customer Graceful degradation
41. Language Chooser and INSPIRE language support • Language chooser added • Internationalization updated (thanks to Alexandre Gacon and others) 2.21 Andrea Aime ♥
42. Language specific service/layer metadata • Abstract, title • Contact information • Service descriptions • In multiple languages • Support INSPIRE “LANGUAGE” request parameter (e.g. GetMap with language specific labels) • Drive styles using the “language” function 2.20 Marco Volpini Provincia di Bolzano
43. Style format in styles page • Playing around with multiple styling languages? • SLD 1.0, SLD 1.1, CSS, YSLD, MBStyles, oh my • The styles page now indicates the format for each style 2.22 Mohammad Mohiuddin Ahmed
44. Server Status Cleanup • Status tab now uses units ◦ GB, threads, milliseconds, … • Resource cache count ◦ Indicates resources in “ObjectPool” cache ◦ Provides feedback for Clear button • Documentation updated ◦ Clarified what the “clear” button actually clears (e.g. use clear to reload user projections) 2.21.1 Jody Garnett GeoCat Live
45. Server Status JVM Console • User interface access and download ◦ Java memory use ◦ Java thread use • Troubleshooting docs updated 2.21.1 Nuno Oliveira GeoSolutioons
46. Reset of single store/layer via REST API • Currently one can do a system wide “reset” • Clears up feature type caches, connection pools and so on, from the whole GeoServer • Add the ability to perform reset of caches/pools on a single store and on a single layer 2.22 Andrea Aime GeoSolutions GeoSolutions
47. Logging
48. Log4j 2 Upgrade 2.21 Jody Garnett (GeoCat) GeoServer OSGeo • Data directory updated in place ◦ DEFAULT_LOGGING.properties → DEFAULT_LOGGING.xml ◦ Custom logging profiles left alone • Built-in logging profiles ◦ Built-in logging profiles update as needed ◦ Extensions include built-in logging profiles • Thanks sponsors ◦ Opengeogroep.nl ◦ www.terrestris.de ◦ How2map.com ◦ www.geonovation.nl
49. Log4j 2 Upgrade • There is a new CONFIG logging level ◦ Reports loading / saving config changes ◦ Similar to INFO (reports normal operation) • Log4J 2 Technical details ◦ Lots of flexibility and capabilities ◦ Config first focus for administrators ◦ Program change to configuration (for log location) was not advisable ◦ Had to stop and ask log4j team for help 07 003 12:03:22 CONFIG [org.geoserver] - GeoServer configuration lock is enabled 07 003 12:03:22 CONFIG [org.geoserver] - Loading catalog... 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'pophatch' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'simple_roads' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'burg' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'giant_polygon' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'poi' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'population' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'polygon' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'raster' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'simple_streams' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'restricted' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'point' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'tiger_roads' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'grass' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'green' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'poly_landmarks' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'generic' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'capitals' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'line' 07 003 12:03:22 CONFIG [org.geoserver] - Loaded style 'cite_lakes' 2.21 Jody Garnett (GeoCat) GeoServer OSGeo
50. Logging REST API • REST API ◦ Logging profile ◦ Log file location ◦ Console logging • Tips: ◦ rest/resources to manage logging profiles ◦ Log4J watches config file for changes • Logging REST API ◦ logging profile and request logging {"logging": { "level": "PRODUCTION_LOGGING", "location": "logs/geoserver.log", "stdOutLogging": true } } /geoserver/rest/logging 2.20 Yalın Eren Deliorman (GeoSolutions) Eumetsat
51. Request Logging • Global Settings for Request Logging ◦ Previously required hand-edit web.xml • Request Logging ◦ Log Requests ◦ Log Request Headers ◦ Log Request Bodies ▪ text content logged ▪ number of characters buffered • Logged as INFO (normal operation) ◦ DEFAULT_LOGGING shows INFO 2.21 Joseph Miller GeoSolutions Eumetsat 07 027 12:27:39 INFO [geoserver.filters] - 127.0.0.1 "POST /geoserver/wfs" "Java/1.8.0_322" "" "application/xml" Headers: Cache-Control: no-cache Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 User-Agent: Java/1.8.0_322 Connection: keep-alive Host: localhost:8080 Pragma: no-cache Content-Length: 1242 Content-Type: application/xml request-size: 1242 body: <wfs:GetFeature service="WFS" version="1.1.0" xmlns:topp="http://www.openplans.org/topp" xmlns:wfs="http://www.opengis.net/wfs" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd"> <wfs:Query typeName="topp:states"> <ogc:Filter> <ogc:FeatureId fid="states.3"/> </ogc:Filter> </wfs:Query> </wfs:GetFeature>
52. Security Vulnerabilities
53. • Keep exploit details out of issue report • Mark the issue as a vulnerability. • Be prepared to work with Project Steering Committee (PSC) on a solution • Keep in mind PSC members are volunteers and an extensive fix may require fundraising / resources If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org. In case you stumble into a vulnerability: Responsible Disclosure
54. • Log4Shell ◦ Published immediate assessment (that GeoServer was not vulnerable) ◦ Released GeoServer w/ patched log4k 1.2 (with no JMS Appender) ◦ Instructions for patching prior versions ◦ Range of community response from demands to offering funding • Spring4Shell ◦ Could not find exploit ◦ Mitigation: Java 8 ◦ GeoServer Release with Spring update • RCE in Jiffle ◦ CVE-2022-24847 ← our first ever CVE! ▪ Mitigation: janino.jar Security Vulnerabilities • RCE in JNDI lookups ◦ Similar to Log4Shell ◦ Made safe lookup methods in GeoTools • Log4J 2 Upgrade ◦ Did a small fundraising activity ◦ Public proposal process ◦ Funds raised (and effort to raise funds) not sufficient for these activities ◦ GeoServer 2.21 Log4j 2 Update • 2023 SQL injection ◦ Through filters. Been there since 2005, pretty hard to spot. ◦ No mitigation, upgrade! • Website: tag “security vulnerability” posts GeoServer PSC 🔥🔥🔥 Various versions
55. Community Building
56. Developer Participation Concerns: • Project care and feeding relies on a low number of people Actions: • Reach out to “downstream projects” to test release candidates • Revamp service providers page to highlight participation and experience (encourage service providers to participate) • Exploring small contracts, and “cost recovery” code-sprint model
57. Looking Ahead Community modules RnD seeking participation and funding
58. • Supports efficient access to Cloud Optimized GeoTiff (COG) • HTTP/S3/Google storage • Azure support incoming • Minimizes reads on blob storage Cloud Optimized GeoTIFF community module Planet GeoSolutions Header Tile Tile Tile Tile Tile Tile Read 1 Read 2 Josh Fix Daniele Romagnoli • Contact GeoSolutions to sponsor: ◦ More blob storage options ◦ Caching ◦ Other improvements Community
59. • New OGC Standards ◦ Self describing OpenAPI ◦ REST / JSON ◦ Building on from W3C collaboration and STAC / WFS3 progress • OGC API - Features moving towards CITE/INSPIRE compliance • Contact Andrea Aime if you are interested in helping with development or funding! OGC API community module Andrea Aime GeoSolutions GeoSolutions/OGC API Common Maps API Coverages API Implementing (as part of TB14/tb15) Looking for volunteers/sponsors Process API Records API Features API Tiles API Styles API STAC API Community
60. OGC API - Features • Has been around for a while as a community module • Core specification (space/time filtering) and CRS support finalized • Advanced filtering (CQL2) still in progress • Fully customizable appearance • Moving towards graduation of the core portion 2.23 Andrea Aime GeoSolutions Geonovum Various others
61. GeoPackage community module improvements • core: geopackage a data format • geopkg output extension: WFS and WMS • geopkg community module: WPS • Reading grids other than png/jpeg tiles: TBD • Extending GeoPackage - yes you can ◦ Include in-line linked metadata ◦ Include styles ◦ Included generalized tables (store pre-simplified geometries) • Contact Andrea Aime if you are interested in helping with time or funding! Community Andrea Aime GeoSolutions GeoSolutions/OGC
62. Webp output format • Output format for WMS • Well supported across modern browsers • Compared to PNG ◦ Up to 2 times smaller file ◦ Up to 3 times slower to produce on topp:states (this can be improved) • The actual speed difference depends on the map production time (less noticeable on more complex maps with rich symbology) • For now, best suited for tile caches Community OpenDEM ♥
63. • The community really does not really have a “road map” • The core developers are mostly employed in companies providing commercial services for GeoServer, or using it in some hosting solution → customer driven (no other significant source of funding) • Other developers pop up occasionally to provide improvements, fixes and new features they need Whatever else you want to push for Community
64. Thanks! info@geosolutionsgroup.com