State of GeoServer 2.16 - FOSS4G 2019

State of GeoServer provides an update on our community and reviews the new and noteworthy features for 2019. GeoServer is a web service for publishing your geospatial data. using industry standards for vector, raster and mapping.
We have an active community and a lot to cover for 2.14 and 2.15 release, as well what is cooking in September’s 2.16 release.

Transcript

1. State of GeoServer 2.16
2. Jody Garnett jody.garnett@geocat.net @jodygarnett GeoCat Government Geographic Data publishing Andrea Aime Technical Lead andrea.aime@geo-solutions.it @geowolf GeoSolutions Innovative, robust, cost-effective solutions leveraging best-of-breed Open Source products. This presentation is brought to you by
3. GeoServer at a Glance Java Web Application to share and edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WMTS – tiles TMS – tiles WMS-C – tiles Extension/community protocols WPS – process CSW – search OpenSearch for Earth Observation – search OGC API - prototype json + rest standards
4. GeoServer 2019 What’s in a year?
5. stable maintenance development September 2018 March 2019 September 2019 March 2020 2.14.x (Java 8) 2.15.x (Java 8/11) 2.16.x (Java 8/11) Releases covered by this presentation
6. Are you using a older version? Upgrade! • User list and stack exchange typically covers only supported versions ◦ Today it means 2.15.x and 2.14.x ◦ In September 2019 it will mean 2.15.x and 2.16.x • Security fixes added to supported versions only • Please upgrade your GeoServer installations!
7. Upgrade? What’s in it for me? • Much! • Let’s check what’s new in 2.14, 2.15 and in the upcoming 2.16 releases • Check the bottom of each slide to see who sponsored a certain feature, who implemented it, and what version contains it • This icon marks activities done without any sponsoring Version Author Sponsor ♥
8. Project Steering Committee Updates The PSC is made up of individuals who are intended to represent the various communities which have a stake in GeoServer. Turnover expected to accommodate people only able to become active on the project in intervals. Thanks • Ben Caradoc-Davies
9. Committers Developers who earned the trust and respect of the team are nominated for “committer status”. At a technical level committers have access to the code repository and the ability to make changes directly and review merge in the work of others. Welcome: • Steve Ikeoka • Fernando Mino
10. Community modules Incoming: • api - Andrea Aime (GeoSolutions) • geostyler - Andreas Schmitz (Terrestris) • Gwc-azure - Andrea Aime (GeoSolutions) • mapml - Chris Hodgson (Refractions) • ogr - Andrea Aime (GeoSolutions) • saml - Mauro Bartolomeoli (GeoSolutions) Outgoing: • authkey → graduated! • Status monitoring → graduated! • geogig • monitor-hibernate • rest-ext • rest-upload • spatialite-store → replaced GeoServer maintains a “community space” for experiments and new developers to work. • Developers can propose a new community module and receive commit access to work on their idea. • We make sure they compile! • May be proposed as an extension when both the author and functionality is ready Spot a community module of interest, reach out to see how you can help.
11. Infrastructure challenges We suffered some vandalism of our Jenkins build server in Q2 2019 impacting the team. • Monthly releases (restored) • Windows installers • Documentation (restored) • Nightly builds (restored) • Nightly CITE Tests • Online Tests ◦ AppSchema (restored) ◦ MySQL ◦ Oracle ◦ PostGIS (restored) ◦ SQLServer Thanks to Planet Federal staff and Torben for rebuilding Jenkins:
12. Service Providers Update GeoServer PSC maintains a list of recognized service providers • Core Contributors Ongoing commitment to the project devoting resources to security fixes, releases and maintenance activities. • Experienced Providers Successfully contribute functionality to the project on behalf of their customers. • Additional Services Welcome: • GeoCat BV Thanks: • Boundless • Envitia • LISAsoft
13. Vector data sources Precisely placed and attributed!
14. MongoDB Graduated to extension status! • Makes use of the supported GeoTools MongoDB DataStore • Publish information stored in the MongoDB Document Database, via a mapping configuration 2.14 Nuno Oliveira GeoSolutions NPRA
15. Support for more PostgreSQL data types • Array support (2.14.x), read, write, limited filtering • Geometry XYZM support (2.14.x) • TWKB Support (PostGIS 2.2.0+ and GeoServer 2.16.x) • Better performance for OR-ed filters that can be expressed as an “in (x,y,z)” clause (2.16.x) • “Nearest” search (as a NEAREST function in CQL) (2.16.x) • Full binary transfer when prepared statements are enabled • SSL control (can be disabled, significant performance speedup) * GeoSolutions/Andrea Aime/Others Various sponsors ♥
16. Spatialite native datastore gone • The spatialite datastore has been removed • We asked around and waited a year for a new maintainer to show up • But you can still use Spatialite with GeoServer. How? Ready for next slide? 2.16
17. OGR data store revived • The OGR vector data store has been revived and upgraded to GDAL 2.x • Now uses publicly available packages, no need for special builds • Tested also on Windows, can leverage the native FileGBD library too • And it can open Spatialite! • Only in 2.16.x 2.16 Andrea Aime GeoSolutions GeoSolutions
18. Raster data sources Pixels, pixels, billions of them!
19. S3 GeoTIFF Community Module + Mosaic Support for GeoTiffs hosted on Amazon S3 • In 2.14.x, can be coupled with image mosaic to support mosaics of AWS images (experimental!) 2.14 Devon / Niels Boundless Boundless s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC8001 0022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2
20. GDAL data sources • The imageio-ext coverage readers have been upgraded to GDAL 2.x • Now uses publicly available packages, no need for special builds • Only in 2.16.x! 2.16 Andrea Aime GeoSolutions GeoSolutions
21. Web Map Service Got maps?
22. Nearest match in time dimension • Time dimension now supports “nearest match” • Map will return the closest time to the one selected • Possibility to set a search “radius” • The actual time used is returned in a HTTP response header 2.14 Andrea Aime GeoSolutions EUMETSAT
23. GetLegendGraphics application/json output: • Description of style at current scale • Use to create your own legend on the client side WMS GetLegendGraphics JSON ouput {"Legend": [{ "layerName": "states", "title": "USA Population", "rules": [ { "title": "< 2M", "filter": "[PERSONS < '2000000']", "symbolizers": [{"Polygon": { "fill": "#4DFF4D", "fill-opacity": "0.7" }}] }, ] }]} 2.15 Ian Turton GeoSolutions
24. Dynamic densification of long lines on reprojection 2.16 Mauro Bartolomeoli GeoSolutions Before: used only available vertices, sharp turns After: dynamically adds vertices to create a soft curve
25. EPSG database upgraded to 9.6 2.16 Andrea Aime • 1000 new coordinate reference codes system added • In particular sponsored to recognize the new GDA2020 Australian datum and associated NTv2 grid shift files • But useful for everybody else! Rant: ◦ Asked on user list for testers: nobody answered! ◦ Look for “GeoServer master upgraded to EPSG 9.6, anyone wants to try it out?” GeoScience Australia
26. Speedups for complex styles 2.16 Andrea Aime • Do you have a style with hundreds of rules • With complex filtering conditions • You know, something like OSM bright for example? • Then rejoice, GeoServer 2.16.x can process those rules and find the right symbolizer quite a bit quicker Don’t miss my talk Friday, 12:00, “Standing up a OSM clone with GeoServer and CSS” ♥
27. Web Feature Service To the source of vector data!
28. Almost WFS 2.0 certified compliant • Lot of work went into making GeoServer pass the WFS 2.0 certification on simple features • 48 issues fixed • Almost there, still a couple of minor fixes to do (but so many other urgent things to work on…) 2.14 Andrea Aime GeoSolutions OGC and GeoSolutions
29. <gs:geometry> <gml:LineString srsName="urn:ogc:def:crs:EPSG::4326" srsDimension="4" gml:id="postgres_lines_zm.1.geometry"> <gml:posList> 1 -2 10 0 1 3 15 1 5 3 20 2 5 -2 25 3</gml:posList> </gml:LineString> </gs:geometry> Nuno / GeoSolutions Jody / Boundless Encode coordinates measures - Enable to encode coordinate measures (“M”) in WFS output formats that support measures. - PostGIS is supported today! WFS Measure Support Community 2.14
30. App-schema improvements • Improvements to connection pool usage (only one connection used per request now) • Some improvements helping while using MongoDB as a source for app-schema records • HALE mapping support being extended to support MongoDB • Support for SOLR as a data source for MongoDB, as well as secondary index for full text search engines 2.14 Nuno Oliveira GeoSolutions NPRA
31. Complex JSON • GeoJSON out of complex features looked odd, out of place • Improvements to make it look more natural while retaining the same information • E.g., types, identities, avoid container property/element value alternation, XML attributes prefixed by @ 2.16 Andrea Aime GeoSolutions BRGM { "type": "FeatureCollection", "features": [ { "type": "Feature", "id": "0001000001", "geometry": { "type": "Point", "coordinates": [51.0684, 1.4298] }, "properties": { "@featureType": "Borehole", "identifier": { "value": "BSS000AAAA", "@codeSpace": "http://www.ietf.org/rfc/rfc2616" }, "bholeHeadworks": [ { "type": "Feature", "geometry": { "type": "Point", "coordinates": [51.0684, 1.4298] }, "properties": { "@featureType": "BoreCollar", "collarElevation": { "value": -32, "@srsName": "http://www.opengis.net/def/crs/EPSG/0/5720", "@srsDimension": "1", "@uomLabels": "m" } } } ],
32. WFS 3.0 community module • OpenAPI based • GeoJSON first • Developed along with the spec (still in DRAFT!) • Some experimental extensions like tiling with Mapbox vector tiles 2.15 Andrea Aime GeoSolutions GeoSolutions/OGC
33. OGC API services Following new trends
34. • WFS3 Hackaton, 6 and 7 March 2018, Fort Collins (CO) ◦ OpenAPI based ◦ JSON first ◦ Schemaless ◦ Small core • OGC API Hackaton, June 20th, 2019, London (UK) ◦ Expand what was learned to all other specs ◦ Features API, Maps and Tiles API, Coverages API, Processes API OGC API, bit of history 2.16/2.17 • Testbed 14, 2018 ◦ CITE tests for WFS3 ◦ Experiment with WFS3 extensions • Vector tiles pilot, 2018/2019 ◦ Use vector tiles on top of WFS3 and WMTS 1.0 ◦ Styles and tiles extensions for WFS3 ◦ Client side rendering • Testbed 15, 2019 ◦ Tiles API, Image API (to update image mosaics), Delta Tile API (to fetch only tiles that changed) ◦ Styles API, finding, editing, managing client and server side styles
35. OGC API and new “ogcapi” community module Andrea Aime GeoSolutions GeoSolutions/OGC API common 2.16/2.17 Features API Tiles API Styles API Maps API Coverages API Process API Implementing now or in the short term (as part of TB14/tb15) Looking for volunteers/sponsors
36. Warning: shameless plug! • By all means attend this presentation! ◦ GeoServer WFS3: introduction to the RESTful, schemaless, JSON first download service, by Andrea Aime ◦ 3:30-3:50 Wednesday August 28th (today!) ◦ Fortuna East Want to know more? 2.16/2.17
37. Web Coverage Service To the source of raster data, too!
38. Tiles My alphabet: X,Y and Z
39. • Allows to store tiles into Azure blob, targeting a specific container • Warning, unlike S3, Azure lacks mass delete operations, each tile/blob need to be deleted with a separate call (GWC does that concurrently) Azure blobstore (community module) 2.16 Andrea Aime GeoSolutions SwissRE
40. Styling Your cartographic tailor
41. • Allow using SVG as shapes for marks • Filling and stroking can be controlled from SLD SVGs as fillable/strokable marks 2.14 Andrea Aime GeoSolutions <PointSymbolizer> <Graphic> <Mark> <WellKnownName>file://firestation.svg</WellKnownName> <Fill> <CssParameter name="fill">#FF0000</CssParameter> </Fill> </Mark> <Size>32</Size> </Graphic> </PointSymbolizer> ♥
42. • Disables cascading • Overrides still allowed by nesting rules • Best approach for anyone confused by cascading • Useful for large sheets too (faster to translate, easier to control) CSS: flat @mode with nested rules 2.14 Andrea Aime GeoSolutions @mode "Flat"; [type = 'Capital'],[mode = 'highlight'] { mark: symbol(circle); :mark { fill: white; stroke: black; size: 12px; } } [test = 'Local'] { mark: symbol(square); :mark { fill: red; size: 3px; } } Will not try to mix them ♥
43. QGIS SLD Export Improvements GeoServer PSC combined funding from OSGeo:UK local chapter, our OSGeo budget, and personal donations: - QGIS Export of RasterSymbolizer - Section on Generating SLD styles with QGIS added to GeoServer user manual 2.15 Luigi Pirelli GeoSolutions GeoServer PSC
44. SLDService Extension to Generate Thematic maps Extension provides a REST endpoint using data analysis to generate a thematic map as an SLD document: • Unique/Equal interval/Quantiles Graduated to supported, but also new functionality: • Equal area classification • Standard deviation filtering • Classification of raster data also! 2.16 GeoSolutions
45. SLDService Extension to Generate Thematic maps curl -v -u admin:geoserver -XGET http://localhost:8080/geoserver/rest/sldservice/states/classify.xml ?attribute=PERSONS &ramp=CUSTOM &method=quantile &intervals=3 &startColor=0xFF0000 &endColor=0x0000FF &open=true
46. • ChannelSelection allows for env variables now • Build pseudo-color styles with a default band layout • Change the selected bands in GetMap request • No need to prepare tens of different styles • What if you want to do band math? You wait just 1 minute! Dynamic channel selection 2.14 Andrea Aime GeoSolutions DLR ...&env=r:4;g:3,b:2 ...&env=r:8;g:11,b:4 Pseudo color Vegetation analysis
47. Web Processing Service When the going gets tough the processes get going
48. Unleash map algebra with Jiffle • Powerful and fast map algebra language • Can be used as a rendering transformation • Example, computing NDVI on the fly: 2.14 Andrea Aime GeoSolutions Bayer nir = src[7]; vir = src[3]; dest = (nir-vir)/(nir+vir); Sentinel 2 Jiffle script On the fly NDVI index display
49. WPS GetExecutions vendor operation WPS Asynchronous processes run in the background and provide a URL that clients can poll to get a progress report (and eventual result). • GetExecutions operation lists running processes • Users can review their running processes • Administrators can list all running processes 2.15 Alessio Fabiani GeoSolutions DLR
50. WPS Dismiss vendor operation WPS Asynchronous processes run in the background and provide a URL that clients can poll to get a progress report (and eventual result). • Dismiss operation cancels a running process • Why vendor? Because it’s added it to WPS 1.0 instead of implementing WPS 2.0 2.15 Alessio Fabiani GeoSolutions DLR
51. Configuration and management
52. Full-screen style editor Andrea & Torben 2.14 ♥
53. External graphic chooser in style editor Niels - • Security fixes being added to supported versions only • Please upgrade your GeoServer installations! 2.14
54. Color picker in style editor Andrea Aime You know…. just in case you don’t remember the hex code of the color you want! 2.14 ♥
55. Style Editor SLD Auto-Complete Andrea Aime Torben Basballe To help make editing easier, now you do not have to have the SLD 1.0 specification handy 2.14 Keyboard shortcut: Control-Space ♥
56. Enable Services Per Layer One of our most requested features! • The layer publish tab allows you to control exactly what services are enabled on a layer by layer basis! 2.15 Andrea Aime GeoSolutions DLR
57. Status monitoring extension 2.16 Nuno Oliveira GeoSolutions GeoSolutions • Graduated up from community in 2.16.x • Access to system information like: ◦ CPU load ◦ physical memory ◦ Swap ◦ Free disk ◦ Network usage • Quite handy in production
58. Security Stop or I will stay stop again!
59. GeoFence graduated to extension The use of an stand alone GeoFence server has been a fixture of professional GeoServer deployments offering far greater control then the “out of the box” GeoServer security system. GeoServer 2.15 packages up GeoFence as an extension you can drop into your GeoServer installation offering: • layer by layer restrictions • override layer details to filter content and adjust styling used • limit access to a geographic extent 2.15 Alessio Fabiani GeoSolutions GeoSolutions
60. Authentication key graduated to extension • Helps matching a key in the URL to a user • Allows application with no auth support to access secured resources • Visible in the URL, use only via HTTPS • Could be eventually copied from URl bar, token should be short lived (keys can be managed via REST API) • MapStore uses it for a simple SSO, see: https://mapstore.readthedocs.io/en/latest/d eveloper-guide/integrations/users/geoserv er/ 2.16 Alessio Fabiani GeoSolutions GeoSolutions
61. Assorted changes • Hierarchical LDAP user/group support (2.16.x, 2.15.x) 2.16 Alessio Fabiani GeoSolutions GeoSolutions
62. GeoServer Internals
63. Java 11 Support GeoServer works with Java 8: • Use OpenJDK (Oracle JDK support ended) • Tomcat 8 or greater recommended GeoServer works with Java 11: • Use OpenJDK (from Linux distribution or AdoptOpenJDK) • Tomcat 9 or greater required Thanks to code-sprint sponsors! 2.15 GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering
64. Java 11 Support About Oracle and Java: • Donated “professional features” to OpenJDK to make it the lead platform (flight recorder, java mission control, …) • Oracle OpenJDK every 6-months • Oracle JDK w/ paid commercial license OracleJDK Java 8 has ended but • Java 8 will continue to be available! • RedHat committed to backporting fixes • AdoptOpenJDK makes binaries available for linux, windows, mac Thanks to code sprint participants! GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering 2.15
65. Java 11 Support - Code Sprint • Victoria: David Vick, Devon Tucker, Jim Hughes, Jody Garnett, Kevin Smith, Torben Barsballe • Italy: Andrea Aime, Antonello Andrea • World: Ian Turton, Landon Blake, Brad Hards GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering 2.15
66. • No longer just a recommendation, these are on by default! • pure java image operations that respect NODATA pixels and vector footprints JAI-EXT Operations on by default 2.15 GeoSolutions GeoSolutions
67. Codebase Quality Assurance and automation Thanks to Andrea Aime for fixing all existing failures and automating checks (one vacation + months of weekends to complete): • PMD source code analysis • Google’s Error Prone Java 11 byte code analysis • Spotbugs byte code analysis • Checkstyle verify javadocs and headers • Removed all deprecated method calls Automated PR checks prevent new failures from being committed. Andrea Aime 2.15 ♥
68. Lots of security fixes • Several security fixes are going into the latest round of releases (2.14.5, 2.15.3, 2.16.0) • Thanks to Gremwell, Leidos, Reggie Dodds • Rant: ◦ Unfortunately, this work is 100% un-funded ◦ Often not possible to act quickly or at all, especially if several reports show up together ◦ Remember the GPL licence: “THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.” GeoSolutions - Planet - B 2.15/2.16.x ♥
69. Details only a developer can love: • Latest JTS release from LocationTech • Upgrade NetCDF library used • Latest JAI-EXT library Update to latest libraries
70. Help us help you
71. User List Participation • Answering users questions relies on a low number of people Action: • We are sending reminder on effective use of the user list on subscription and at the bottom of each mail • Community building activity - remind developers about the definition of “supported” (not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)
72. • Keep exploit details out of issue report • Mark the issue as a vulnerability. • Be prepared to work with Project Steering Committee (PSC) on a solution • Keep in mind PSC members are volunteers and an extensive fix may require fundraising / resources If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org. In case you stumble into a vulnerability: Responsible Disclosure
73. Community Building
74. Developer Participation • Project care and feeding relies on a low number of people Actions: • Recruiting with FOSS4G Developer Workshops • Encourage Service Providers to participate in monthly bug stomps • Reach out to “downstream projects” to test release candidates
75. • Bug stomp introduced in 2016 ◦ Relying on volunteer time not working out ◦ allowed developers to reserve time • Moved to fixed schedule in 2017 ◦ Arranging day each month too random • Last Friday of each month ◦ Check blog post for tips on participation You can join and help, too! Standing “last friday of the month” bug stomp
76. Looking Ahead
77. CITE Conformance Tests OSGeo priority for 2019: • PSC requested small budget for activity • Phase I: CITE Test Automation RFP • Phase II: Code Sprint to pass tests Going beyond just the “marketing” sticker: • build.geoserver.org uses CITE tests from 2013 on each “nightly” build • Updating these “black box” tests are the priority as they contribute confidence in each release we make 2.17 Community OSGeo
78. Thanks!
79. Question and Answer Q: What happened to Boundless? • Boundless has been acquired by Planet, Inc