Upgrade to Pro — share decks privately, control downloads, hide ads and more …

State of GeoServer 2.16 - FOSS4G 2019

State of GeoServer 2.16 - FOSS4G 2019

State of GeoServer provides an update on our community and reviews the new and noteworthy features for 2019. GeoServer is a web service for publishing your geospatial data. using industry standards for vector, raster and mapping.
We have an active community and a lot to cover for 2.14 and 2.15 release, as well what is cooking in September’s 2.16 release.

Simone Giannecchini

August 28, 2019
Tweet

More Decks by Simone Giannecchini

Other Decks in Programming

Transcript

  1. Jody Garnett [email protected] @jodygarnett GeoCat Government Geographic Data publishing Andrea

    Aime Technical Lead [email protected] @geowolf GeoSolutions Innovative, robust, cost-effective solutions leveraging best-of-breed Open Source products. This presentation is brought to you by
  2. GeoServer at a Glance Java Web Application to share and

    edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WMTS – tiles TMS – tiles WMS-C – tiles Extension/community protocols WPS – process CSW – search OpenSearch for Earth Observation – search OGC API - prototype json + rest standards
  3. stable maintenance development September 2018 March 2019 September 2019 March

    2020 2.14.x (Java 8) 2.15.x (Java 8/11) 2.16.x (Java 8/11) Releases covered by this presentation
  4. Are you using a older version? Upgrade! • User list

    and stack exchange typically covers only supported versions ◦ Today it means 2.15.x and 2.14.x ◦ In September 2019 it will mean 2.15.x and 2.16.x • Security fixes added to supported versions only • Please upgrade your GeoServer installations!
  5. Upgrade? What’s in it for me? • Much! • Let’s

    check what’s new in 2.14, 2.15 and in the upcoming 2.16 releases • Check the bottom of each slide to see who sponsored a certain feature, who implemented it, and what version contains it • This icon marks activities done without any sponsoring Version Author Sponsor ♥
  6. Project Steering Committee Updates The PSC is made up of

    individuals who are intended to represent the various communities which have a stake in GeoServer. Turnover expected to accommodate people only able to become active on the project in intervals. Thanks • Ben Caradoc-Davies
  7. Committers Developers who earned the trust and respect of the

    team are nominated for “committer status”. At a technical level committers have access to the code repository and the ability to make changes directly and review merge in the work of others. Welcome: • Steve Ikeoka • Fernando Mino
  8. Community modules Incoming: • api - Andrea Aime (GeoSolutions) •

    geostyler - Andreas Schmitz (Terrestris) • Gwc-azure - Andrea Aime (GeoSolutions) • mapml - Chris Hodgson (Refractions) • ogr - Andrea Aime (GeoSolutions) • saml - Mauro Bartolomeoli (GeoSolutions) Outgoing: • authkey → graduated! • Status monitoring → graduated! • geogig • monitor-hibernate • rest-ext • rest-upload • spatialite-store → replaced GeoServer maintains a “community space” for experiments and new developers to work. • Developers can propose a new community module and receive commit access to work on their idea. • We make sure they compile! • May be proposed as an extension when both the author and functionality is ready Spot a community module of interest, reach out to see how you can help.
  9. Infrastructure challenges We suffered some vandalism of our Jenkins build

    server in Q2 2019 impacting the team. • Monthly releases (restored) • Windows installers • Documentation (restored) • Nightly builds (restored) • Nightly CITE Tests • Online Tests ◦ AppSchema (restored) ◦ MySQL ◦ Oracle ◦ PostGIS (restored) ◦ SQLServer Thanks to Planet Federal staff and Torben for rebuilding Jenkins:
  10. Service Providers Update GeoServer PSC maintains a list of recognized

    service providers • Core Contributors Ongoing commitment to the project devoting resources to security fixes, releases and maintenance activities. • Experienced Providers Successfully contribute functionality to the project on behalf of their customers. • Additional Services Welcome: • GeoCat BV Thanks: • Boundless • Envitia • LISAsoft
  11. MongoDB Graduated to extension status! • Makes use of the

    supported GeoTools MongoDB DataStore • Publish information stored in the MongoDB Document Database, via a mapping configuration 2.14 Nuno Oliveira GeoSolutions NPRA
  12. Support for more PostgreSQL data types • Array support (2.14.x),

    read, write, limited filtering • Geometry XYZM support (2.14.x) • TWKB Support (PostGIS 2.2.0+ and GeoServer 2.16.x) • Better performance for OR-ed filters that can be expressed as an “in (x,y,z)” clause (2.16.x) • “Nearest” search (as a NEAREST function in CQL) (2.16.x) • Full binary transfer when prepared statements are enabled • SSL control (can be disabled, significant performance speedup) * GeoSolutions/Andrea Aime/Others Various sponsors ♥
  13. Spatialite native datastore gone • The spatialite datastore has been

    removed • We asked around and waited a year for a new maintainer to show up • But you can still use Spatialite with GeoServer. How? Ready for next slide? 2.16
  14. OGR data store revived • The OGR vector data store

    has been revived and upgraded to GDAL 2.x • Now uses publicly available packages, no need for special builds • Tested also on Windows, can leverage the native FileGBD library too • And it can open Spatialite! • Only in 2.16.x 2.16 Andrea Aime GeoSolutions GeoSolutions
  15. S3 GeoTIFF Community Module + Mosaic Support for GeoTiffs hosted

    on Amazon S3 • In 2.14.x, can be coupled with image mosaic to support mosaics of AWS images (experimental!) 2.14 Devon / Niels Boundless Boundless s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC8001 0022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2
  16. GDAL data sources • The imageio-ext coverage readers have been

    upgraded to GDAL 2.x • Now uses publicly available packages, no need for special builds • Only in 2.16.x! 2.16 Andrea Aime GeoSolutions GeoSolutions
  17. Nearest match in time dimension • Time dimension now supports

    “nearest match” • Map will return the closest time to the one selected • Possibility to set a search “radius” • The actual time used is returned in a HTTP response header 2.14 Andrea Aime GeoSolutions EUMETSAT
  18. GetLegendGraphics application/json output: • Description of style at current scale

    • Use to create your own legend on the client side WMS GetLegendGraphics JSON ouput {"Legend": [{ "layerName": "states", "title": "USA Population", "rules": [ { "title": "< 2M", "filter": "[PERSONS < '2000000']", "symbolizers": [{"Polygon": { "fill": "#4DFF4D", "fill-opacity": "0.7" }}] }, ] }]} 2.15 Ian Turton GeoSolutions
  19. Dynamic densification of long lines on reprojection 2.16 Mauro Bartolomeoli

    GeoSolutions Before: used only available vertices, sharp turns After: dynamically adds vertices to create a soft curve
  20. EPSG database upgraded to 9.6 2.16 Andrea Aime • 1000

    new coordinate reference codes system added • In particular sponsored to recognize the new GDA2020 Australian datum and associated NTv2 grid shift files • But useful for everybody else! Rant: ◦ Asked on user list for testers: nobody answered! ◦ Look for “GeoServer master upgraded to EPSG 9.6, anyone wants to try it out?” GeoScience Australia
  21. Speedups for complex styles 2.16 Andrea Aime • Do you

    have a style with hundreds of rules • With complex filtering conditions • You know, something like OSM bright for example? • Then rejoice, GeoServer 2.16.x can process those rules and find the right symbolizer quite a bit quicker Don’t miss my talk Friday, 12:00, “Standing up a OSM clone with GeoServer and CSS” ♥
  22. Almost WFS 2.0 certified compliant • Lot of work went

    into making GeoServer pass the WFS 2.0 certification on simple features • 48 issues fixed • Almost there, still a couple of minor fixes to do (but so many other urgent things to work on…) 2.14 Andrea Aime GeoSolutions OGC and GeoSolutions
  23. <gs:geometry> <gml:LineString srsName="urn:ogc:def:crs:EPSG::4326" srsDimension="4" gml:id="postgres_lines_zm.1.geometry"> <gml:posList> 1 -2 10 0

    1 3 15 1 5 3 20 2 5 -2 25 3</gml:posList> </gml:LineString> </gs:geometry> Nuno / GeoSolutions Jody / Boundless Encode coordinates measures - Enable to encode coordinate measures (“M”) in WFS output formats that support measures. - PostGIS is supported today! WFS Measure Support Community 2.14
  24. App-schema improvements • Improvements to connection pool usage (only one

    connection used per request now) • Some improvements helping while using MongoDB as a source for app-schema records • HALE mapping support being extended to support MongoDB • Support for SOLR as a data source for MongoDB, as well as secondary index for full text search engines 2.14 Nuno Oliveira GeoSolutions NPRA
  25. Complex JSON • GeoJSON out of complex features looked odd,

    out of place • Improvements to make it look more natural while retaining the same information • E.g., types, identities, avoid container property/element value alternation, XML attributes prefixed by @ 2.16 Andrea Aime GeoSolutions BRGM { "type": "FeatureCollection", "features": [ { "type": "Feature", "id": "0001000001", "geometry": { "type": "Point", "coordinates": [51.0684, 1.4298] }, "properties": { "@featureType": "Borehole", "identifier": { "value": "BSS000AAAA", "@codeSpace": "http://www.ietf.org/rfc/rfc2616" }, "bholeHeadworks": [ { "type": "Feature", "geometry": { "type": "Point", "coordinates": [51.0684, 1.4298] }, "properties": { "@featureType": "BoreCollar", "collarElevation": { "value": -32, "@srsName": "http://www.opengis.net/def/crs/EPSG/0/5720", "@srsDimension": "1", "@uomLabels": "m" } } } ],
  26. WFS 3.0 community module • OpenAPI based • GeoJSON first

    • Developed along with the spec (still in DRAFT!) • Some experimental extensions like tiling with Mapbox vector tiles 2.15 Andrea Aime GeoSolutions GeoSolutions/OGC
  27. • WFS3 Hackaton, 6 and 7 March 2018, Fort Collins

    (CO) ◦ OpenAPI based ◦ JSON first ◦ Schemaless ◦ Small core • OGC API Hackaton, June 20th, 2019, London (UK) ◦ Expand what was learned to all other specs ◦ Features API, Maps and Tiles API, Coverages API, Processes API OGC API, bit of history 2.16/2.17 • Testbed 14, 2018 ◦ CITE tests for WFS3 ◦ Experiment with WFS3 extensions • Vector tiles pilot, 2018/2019 ◦ Use vector tiles on top of WFS3 and WMTS 1.0 ◦ Styles and tiles extensions for WFS3 ◦ Client side rendering • Testbed 15, 2019 ◦ Tiles API, Image API (to update image mosaics), Delta Tile API (to fetch only tiles that changed) ◦ Styles API, finding, editing, managing client and server side styles
  28. OGC API and new “ogcapi” community module Andrea Aime GeoSolutions

    GeoSolutions/OGC API common 2.16/2.17 Features API Tiles API Styles API Maps API Coverages API Process API Implementing now or in the short term (as part of TB14/tb15) Looking for volunteers/sponsors
  29. Warning: shameless plug! • By all means attend this presentation!

    ◦ GeoServer WFS3: introduction to the RESTful, schemaless, JSON first download service, by Andrea Aime ◦ 3:30-3:50 Wednesday August 28th (today!) ◦ Fortuna East Want to know more? 2.16/2.17
  30. • Allows to store tiles into Azure blob, targeting a

    specific container • Warning, unlike S3, Azure lacks mass delete operations, each tile/blob need to be deleted with a separate call (GWC does that concurrently) Azure blobstore (community module) 2.16 Andrea Aime GeoSolutions SwissRE
  31. • Allow using SVG as shapes for marks • Filling

    and stroking can be controlled from SLD SVGs as fillable/strokable marks 2.14 Andrea Aime GeoSolutions <PointSymbolizer> <Graphic> <Mark> <WellKnownName>file://firestation.svg</WellKnownName> <Fill> <CssParameter name="fill">#FF0000</CssParameter> </Fill> </Mark> <Size>32</Size> </Graphic> </PointSymbolizer> ♥
  32. • Disables cascading • Overrides still allowed by nesting rules

    • Best approach for anyone confused by cascading • Useful for large sheets too (faster to translate, easier to control) CSS: flat @mode with nested rules 2.14 Andrea Aime GeoSolutions @mode "Flat"; [type = 'Capital'],[mode = 'highlight'] { mark: symbol(circle); :mark { fill: white; stroke: black; size: 12px; } } [test = 'Local'] { mark: symbol(square); :mark { fill: red; size: 3px; } } Will not try to mix them ♥
  33. QGIS SLD Export Improvements GeoServer PSC combined funding from OSGeo:UK

    local chapter, our OSGeo budget, and personal donations: - QGIS Export of RasterSymbolizer - Section on Generating SLD styles with QGIS added to GeoServer user manual 2.15 Luigi Pirelli GeoSolutions GeoServer PSC
  34. SLDService Extension to Generate Thematic maps Extension provides a REST

    endpoint using data analysis to generate a thematic map as an SLD document: • Unique/Equal interval/Quantiles Graduated to supported, but also new functionality: • Equal area classification • Standard deviation filtering • Classification of raster data also! 2.16 GeoSolutions
  35. SLDService Extension to Generate Thematic maps curl -v -u admin:geoserver

    -XGET http://localhost:8080/geoserver/rest/sldservice/states/classify.xml ?attribute=PERSONS &ramp=CUSTOM &method=quantile &intervals=3 &startColor=0xFF0000 &endColor=0x0000FF &open=true
  36. • ChannelSelection allows for env variables now • Build pseudo-color

    styles with a default band layout • Change the selected bands in GetMap request • No need to prepare tens of different styles • What if you want to do band math? You wait just 1 minute! Dynamic channel selection 2.14 Andrea Aime GeoSolutions DLR ...&env=r:4;g:3,b:2 ...&env=r:8;g:11,b:4 Pseudo color Vegetation analysis
  37. Unleash map algebra with Jiffle • Powerful and fast map

    algebra language • Can be used as a rendering transformation • Example, computing NDVI on the fly: 2.14 Andrea Aime GeoSolutions Bayer nir = src[7]; vir = src[3]; dest = (nir-vir)/(nir+vir); Sentinel 2 Jiffle script On the fly NDVI index display
  38. WPS GetExecutions vendor operation WPS Asynchronous processes run in the

    background and provide a URL that clients can poll to get a progress report (and eventual result). • GetExecutions operation lists running processes • Users can review their running processes • Administrators can list all running processes 2.15 Alessio Fabiani GeoSolutions DLR
  39. WPS Dismiss vendor operation WPS Asynchronous processes run in the

    background and provide a URL that clients can poll to get a progress report (and eventual result). • Dismiss operation cancels a running process • Why vendor? Because it’s added it to WPS 1.0 instead of implementing WPS 2.0 2.15 Alessio Fabiani GeoSolutions DLR
  40. External graphic chooser in style editor Niels - • Security

    fixes being added to supported versions only • Please upgrade your GeoServer installations! 2.14
  41. Color picker in style editor Andrea Aime You know…. just

    in case you don’t remember the hex code of the color you want! 2.14 ♥
  42. Style Editor SLD Auto-Complete Andrea Aime Torben Basballe To help

    make editing easier, now you do not have to have the SLD 1.0 specification handy 2.14 Keyboard shortcut: Control-Space ♥
  43. Enable Services Per Layer One of our most requested features!

    • The layer publish tab allows you to control exactly what services are enabled on a layer by layer basis! 2.15 Andrea Aime GeoSolutions DLR
  44. Status monitoring extension 2.16 Nuno Oliveira GeoSolutions GeoSolutions • Graduated

    up from community in 2.16.x • Access to system information like: ◦ CPU load ◦ physical memory ◦ Swap ◦ Free disk ◦ Network usage • Quite handy in production
  45. GeoFence graduated to extension The use of an stand alone

    GeoFence server has been a fixture of professional GeoServer deployments offering far greater control then the “out of the box” GeoServer security system. GeoServer 2.15 packages up GeoFence as an extension you can drop into your GeoServer installation offering: • layer by layer restrictions • override layer details to filter content and adjust styling used • limit access to a geographic extent 2.15 Alessio Fabiani GeoSolutions GeoSolutions
  46. Authentication key graduated to extension • Helps matching a key

    in the URL to a user • Allows application with no auth support to access secured resources • Visible in the URL, use only via HTTPS • Could be eventually copied from URl bar, token should be short lived (keys can be managed via REST API) • MapStore uses it for a simple SSO, see: https://mapstore.readthedocs.io/en/latest/d eveloper-guide/integrations/users/geoserv er/ 2.16 Alessio Fabiani GeoSolutions GeoSolutions
  47. Java 11 Support GeoServer works with Java 8: • Use

    OpenJDK (Oracle JDK support ended) • Tomcat 8 or greater recommended GeoServer works with Java 11: • Use OpenJDK (from Linux distribution or AdoptOpenJDK) • Tomcat 9 or greater required Thanks to code-sprint sponsors! 2.15 GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering
  48. Java 11 Support About Oracle and Java: • Donated “professional

    features” to OpenJDK to make it the lead platform (flight recorder, java mission control, …) • Oracle OpenJDK every 6-months • Oracle JDK w/ paid commercial license OracleJDK Java 8 has ended but • Java 8 will continue to be available! • RedHat committed to backporting fixes • AdoptOpenJDK makes binaries available for linux, windows, mac Thanks to code sprint participants! GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering 2.15
  49. Java 11 Support - Code Sprint • Victoria: David Vick,

    Devon Tucker, Jim Hughes, Jody Garnett, Kevin Smith, Torben Barsballe • Italy: Andrea Aime, Antonello Andrea • World: Ian Turton, Landon Blake, Brad Hards GeoSolutions, Boundless, ASTUN Technology, CCRi, HydroloGIS Environmental Engineering 2.15
  50. • No longer just a recommendation, these are on by

    default! • pure java image operations that respect NODATA pixels and vector footprints JAI-EXT Operations on by default 2.15 GeoSolutions GeoSolutions
  51. Codebase Quality Assurance and automation Thanks to Andrea Aime for

    fixing all existing failures and automating checks (one vacation + months of weekends to complete): • PMD source code analysis • Google’s Error Prone Java 11 byte code analysis • Spotbugs byte code analysis • Checkstyle verify javadocs and headers • Removed all deprecated method calls Automated PR checks prevent new failures from being committed. Andrea Aime 2.15 ♥
  52. Lots of security fixes • Several security fixes are going

    into the latest round of releases (2.14.5, 2.15.3, 2.16.0) • Thanks to Gremwell, Leidos, Reggie Dodds • Rant: ◦ Unfortunately, this work is 100% un-funded ◦ Often not possible to act quickly or at all, especially if several reports show up together ◦ Remember the GPL licence: “THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.” GeoSolutions - Planet - B 2.15/2.16.x ♥
  53. Details only a developer can love: • Latest JTS release

    from LocationTech • Upgrade NetCDF library used • Latest JAI-EXT library Update to latest libraries
  54. User List Participation • Answering users questions relies on a

    low number of people Action: • We are sending reminder on effective use of the user list on subscription and at the bottom of each mail • Community building activity - remind developers about the definition of “supported” (not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)
  55. • Keep exploit details out of issue report • Mark

    the issue as a vulnerability. • Be prepared to work with Project Steering Committee (PSC) on a solution • Keep in mind PSC members are volunteers and an extensive fix may require fundraising / resources If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at [email protected]. In case you stumble into a vulnerability: Responsible Disclosure
  56. Developer Participation • Project care and feeding relies on a

    low number of people Actions: • Recruiting with FOSS4G Developer Workshops • Encourage Service Providers to participate in monthly bug stomps • Reach out to “downstream projects” to test release candidates
  57. • Bug stomp introduced in 2016 ◦ Relying on volunteer

    time not working out ◦ allowed developers to reserve time • Moved to fixed schedule in 2017 ◦ Arranging day each month too random • Last Friday of each month ◦ Check blog post for tips on participation You can join and help, too! Standing “last friday of the month” bug stomp
  58. CITE Conformance Tests OSGeo priority for 2019: • PSC requested

    small budget for activity • Phase I: CITE Test Automation RFP • Phase II: Code Sprint to pass tests Going beyond just the “marketing” sticker: • build.geoserver.org uses CITE tests from 2013 on each “nightly” build • Updating these “black box” tests are the priority as they contribute confidence in each release we make 2.17 Community OSGeo