Simone Giannecchini Andrea Aime GeoSolutions Supporting precision farming with GeoServer Past experiences and way forward 

GeoSolutions Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode • Offices in Italy & US, Global Clients/Team • 40+ collaborators, 30+ Engineers • Our products • Our Offer 

Affiliations We strongly support Open Source, it Is in our core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT 

Setting the context 

GeoServer ● GeoSpatial enterprise gateway − Modular & extensible − Management & dissemination of raster and vector data ● Cloud/ Big Data friendly − Cluster deployments in AWS, Azure, On Premise − Powering petabytes-size data (EO, MetOcean, etc.) − Serving up to 10k+ requests per second ● Standards compliant − OGC WCS 1.0, 1.1.1 (RI), 2.0.1 − OGC WFS 1.0, 1.1 (RI), 2.0 − OGC WMS 1.1.1, 1.3 − OGC WPS 1.0.0 − OGC CSW 2.0.2 ● Google Earth/Maps support ● License is GPL v2.0 More Information 

GeoServer GeoServer WFS WMS PostGIS Oracle H2 DB2 SQL Server GeoPackag e MySql Spatialite Elastic MongoDB Shapefile ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- WFS WMS WMTS PNG, GIF JPEG TIFF, GeoTIFF SVG, PDF KML/KMZ Shapefile GML2 GML3 GeoRSS GeoJSON CSV/XLS GeoPackage Raw vector data Servers Styled maps DBMS Vector files WCS GeoTIFF WMS ArcGrid Img+world Mosaic MrSID JPEG 2000 ECW,Pyrami d, NetCDF,... Raster files Raw raster data GeoTIFF ArcGrid GTopo30 Img+World WMTS, TMS KML superoverlays Google maps tiles OGC tiles OSGEO tiles KML WPS CSW ESRI REST 

Key Concepts • Supporting farming decision makers • with consuming various types of data • EO Data • Drone Imagery • Field Sensors (e.g. Meteo Stations) • Vehicles Positions & Sensors • Meteorological Models • More (not related to our work) • from different deployments • (Private|Public) Cloud • On Prem 

Key Concepts • via Heterogeneous types of platforms • All-in-one DAAS solution to farmers • EO focused platform • IOT Focused platform for vehicle data • IOT Focused platform for field sensors • Various combinations of the above • setup by entities with diff bkgs & objectives • Large Corp. focusing solely on Precision Farming • Subsidiaries|Branches of large Corp. focusing solely on Precision Farming • Startups focusing solely on Precision Farming 

Key Concepts • Challenging environments to support • Lots of different data • Continuously ingested • & processed into new data, alerts, charts, etc.. • We are going to address some challenges & Scenarios • Serving EO Data • Serving Drone Data • Serving IOT Like Data • Deploying & Operating GeoServer 

Serving EO Data 

Serving EO Data • Typical EO scenario • Multispectral data (Sentinel, Landsat, Planet, etc…) • Hyperspectral data (Hyperion, EnMap, etc..) • Rarely SAR data • Focus on • RGB data → mostly pure visualization • Indexes → mostly band algebra like NDVI and friends • Deep time series for comparison and analysis • Continuous ingestion is crucial 

Serving EO Data Hyperion 4 years MSAVI Sentinel 2 Time Series Multiple Indexes with Sentinel, Landsat and Planet Data 

• Poor Data Preprocessing|Optimization • GeoServer is slow at serving index XXX out of my uncompressed, untiled, multiband geotiff, why? • Rule #1 → preprocess|optimize data at ingestion • Rule #2 → preprocess|optimize data at ingestion (not a typo, just reinforcing) • Hint #1 → GDAL is your friend|swiss knife • Hint #2 → Automate! Airflow, Lambda, etc.. • Suggestions • GeoTiff is usually the best format (COG later on) and it is free! • Compression, Tiling and Overviews are crucial → spend some early CPU cycle to get performance later on • Do not fear lossy compression when possible • Do not be afraid of sacrificing space for performance if needed Typical Mistakes and how to avoid them 

Typical Mistakes and how to avoid them • Resources on data preprocessing • Our training material • Best Practices for Optimizing Performance with GeoServer (deck, video) • A DevOps perspective on GeoServer: Deployment planning guidelines, (video, deck, FOSS4G2021) • Precision Farming with GeoServer, GeoSolutions Recommendations-v01.01 (beware, long document!) • GeoServer on steroids presentations from FOSS4G • A ton of GDAL related material you can find on the web! 

• Poor data organization in GeoServer • My GeoServer does not scale as I expected, despite proper data preprocessing. It takes minutes to (re)start, why? • Fact #1 → if you have thousands of layers, you need to reorganize data quickly • Fact #2 → if you have millions of layers, you are doomed! • Suggestions • Conceptually organize EO data by common sensor, data type, etc.. • Map this to multidimensional ImageMosaic → break the 1 scene 1 layer circle • Use REST API to automate data management → no need to reload configuration, can use simple passive clustering • 99% of the time all you need are a handful (<100) of multidimensional ImageMosaic layers! Typical Mistakes and how to avoid them 

• Resources • A DevOps perspective on GeoServer: Deployment planning guidelines, (video, deck) • Precision Farming with GeoServer, GeoSolutions Recommendations-v01.01 (beware, long document!) Typical Mistakes and how to avoid them 

• Poor caching • My GeoServer does not scale as I expected, despite proper data preprocessing and organization, why? • Rule number 1 → preprocess|optimize data at ingestion (not a typo, just reinforcing) • Rule number 2 → introduce tile caching • Rule number 3 → introduce HTTP caching (if possible) • Suggestions • Once data is optimized and all, focus on caching, not before! • Tile Caching comes at a cost, make sure you can reuse tiles multiple times otherwise avoid caching • HTTP Caching is super important but also tricky, evaluate carefully the impact before adopting • Tile|HTTP Caching works nicely with URLs whose content does not change over time Typical Mistakes and how to avoid them 

• Resources • A DevOps perspective on GeoServer: Deployment planning guidelines, (video, deck) • Precision Farming with GeoServer, GeoSolutions Recommendations-v01.01 (beware, long document!) Typical Mistakes and how to avoid them 

More Goodies: COG • Cloud Optimize Geotiff • EFS costs are killing us, what can we do? • COG is (usually) the answer • GeoServer supports COG as single GeoTiff or ImageMosaic • We support S3, Google Cloud Storage and (soon) Azure • Missing bits • Local caching • Support for Binary Masks • Before you ask • Object Storage gives you better scalability at a fraction of the price → you might sacrifice some raw speed! 

More Goodies: COG • Resources • Modern Cloud Geospatial Architectures Survey • Best Practices for Optimizing Performance with GeoServer (deck, video) • Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more • GeoServer docs • Many other resources on the web 

More Goodies: ImageMosaic & Time Series ● About data ● Data is added or removed to GS ● Data is rarely updated ● E.g. data from EO, MetOc, Drones, IOT Sensors ● Refrain from creating millions of layers, it won’t scale 🡪 organize your data properly ● Use ImageMosaic for rasters if possible ● Use TIME and other dimensions ● Use CQL_Filter ● Keep # of layers low if not constant ● Manager data via REST API 🡪 no configuration reload ● Use single tables for vector if possible ● Use TIME and other dimensions ● Use CQL_Filter or Parametric SQL View ● Keep # of layers low if not constant as you ingest data 

More Goodies: ImageMosaic & Time Series • Resources • Best Practices for Optimizing Performance with GeoServer (deck, video) • Serving earth observation data with GeoServer: COG, STAC, OpenSearch and more • Our training material • Precision Farming with GeoServer, GeoSolutions Recommendations-v01.01 (beware, long document!) • GeoServer on steroids presentations from FOSS4G 

More Goodies: Map Algebra with Jiffle • Map algebra for GeoServer • Simple expressions, but also decisions, loops, arrays, variables • Reduce time to publishing • Compute products on the fly • Add new products without writing code • Docs • Use it in • Straight WPS calculation • As part of styles • Large computed downloads (also WPS) 

More Goodies: Map Algebra with Jiffle • Normalized Differential Vegetation Index (Hyperion) coverage script nir = src[46]; vir = src[31]; dest = (nir - vir) / (nir + vir); 1.0 …. 

Serving Drone Data 

Serving Drone Data • Typical Drone scenario • Multispectral data • Hyperspectral data • Downstream Products → vector and raster • Focus on • RGB data → mostly pure visualization • Downstream Products → analysis & decision making • Time series for comparison and analysis • Ad-hoc or Periodic ingestion • Heavy automated processing • Data size can be challenging 

Serving Drone Data • Most (all?) the previous recommendations are valid • Optimize data and product before serving • Use COG • Use Tile Caching • ImageMosaic is still crucial • Organize conceptually your data: same data type, same CRS, etc.. • Use ImageMosaic with dimensions: acquisition time, flightUUID, anything • Decouple # of layers from # of acquisitions • Caching is crucial 

Serving Drone Data • What about vector products? • Many products are vector datasets • We need to be able to related them to raw raster data and rater products • SQL Views & Table Partitioning to the rescue • Create a single table in Postgis • add pivot attributes to filter later like like acquisition time, flightUUID, anything • use SQL views in GeoServer to decouple # of layers from # of acquisitions • ingest directly in the database, no changes to the GeoServer configuration! 

Serving Drone Data • Mosaicking datastore in under development! • Ability to create a seamless vector store in GeoServer • which can index products stored separately (flatgeobuf?) • and works similarly to ImageMosaic to support dimensions like acquisition time, flightUUID, anything • to decouple # of layers from # of acquisitions • Simplify ingestion & reduce cost • This solution does not fit all use cases • Secondary store might not support advanced filtering 

Serving Drone Data 

Serving IOT Like Data 

Serving IOT Data • Typical Scenario • Vehicle telemetries • Vehicle sensors data • Seeds planting information • Huge amounts of points & trucks • Focus on • RGB data → mostly pure visualization • Continuous ingestion 

Serving IOT Data • SQL Views & Table Partitioning to the rescue • Create a single table in Postgis • add pivot attributes to filter later like like acquisition time, flightUUID, anything • use SQL views in GeoServer to decouple # of layers from # of acquisitions • ingest directly in the database, no changes to the GeoServer configuration! • Potential issues • Operating a large Postgis cluster • Cost (AWS, Azure) 

Serving IOT Data • ElasticSearch to the rescue • Manage billions of points, shard data and scale queries linearly • Add extra hardware as needed, automatic resharding • Very fast aggregation of point to build summary views (GeoHash gridding, Tile gridding) • Potential issues • Operating ES • Cost (AWS, Azure) • Mosaicking Datastore? 

Serving IOT Data • Rasterization to speed up serving? • Rasterize common visualizations • Use vector data visualization for other • Caching is crucial 

Serving IOT Data • SLD Service • Generate classified styles based on data • Methods: equalInterval, uniqueInterval, quantile, jenks, equalArea, standardDeviation • Provide custom colors • Clip by standard deviation • Works both on raster and vector • Work on large rasters by statistical sampling MapStore driving SLDService 

Deploying & Operating GeoServer 

Deploying & Operating GeoServer → Resources • A DevOps perspective on GeoServer material • A DevOps perspective on GeoServer: Deployment planning guidelines, (video, deck, FOSS4G21) • GeoServer on Kubernetes: Set up and operate a GeoServer Cluster in K8s, (video, deck) • A DevOps perspective on GeoServer: Monitoring, Metering, Logging and Troubleshooting, (video, deck) • Our Training material • Advanced GeoServer Configuration • Enterprise Set-up Recommendations • More material • Best Practices for Optimizing Performance with GeoServer (deck, video) • Precision Farming with GeoServer, GeoSolutions Recommendations-v01.01 (beware, long document!) • GeoServer on steroids presentations from FOSS4G 

Cloud & GeoServer ● GeoServer is not cloud-native ● It was born when cloud meant this 🡪 ● We can’t depend on any cloud provider ● GeoServer is cloud-ready ● It is known to run in AWS, Azure, GCP, OpenShift, IBM Cloud, etc.. ● It is known to run in K8s, Rancher ● It can autoscale (CPU is the resource to look at) ● It can use Object Storage (Tile Cache, COG, etc..) ● Prefers compute intensive instances ● Likes Containers ● Likes Automation! (Azure Pipelines, Jenkins, etc..) 

GeoServer Facts & Myths ● GeoServer Data Directory ● Where GeoServer stores configuration in files ● No automatic way to pick up config changes from files ● Data can live in it, but we do not recommend it in enterprise set ups ● Manually messing with the configuration files is dangerous ● Memory-bound configuration ● GeoServer loads data configuration in memory at startup (configuration not data itself) ● GeoServer exposes GUI and REST endpoints to reload config when needed ● Configuration reloading does not break OGC services ● Configuration reloading blocks GUI and REST API 

GeoServer Facts & Myths ● GeoServer Needs a lot of memory ● With properly configured data and styles the bottleneck is usually the CPU not the memory ● Our reference dimensioning is 4CPU, 2 to 4 GB of HEAP ● Do you have 1M+ layers? If no, 2GB is enough ● Do you generate large PDF prints of PNG maps? If no, 2GB is enough ● Do you have 8 or more CPUs? If no, 2GB is enough ● GeoServer is slow ● Are you serving a 1TB striped Bigtiff with no overviews? Are you visualizing 10M points from an Oracle table? ● You have deployed a single GeoServer instance with 2 CPUs, no caching and you expect it to handle 200 req/sec? ● 99% of performance issues is either data not optimized or simply trying to render too much data 

GeoServer Facts & Myths ● Serving a large number of layers ● Large usually mean 50k or more ● Start up times / Reload times can grow (e.g. DBMS tables) ● Heap Memory usage might grow ● GetCapabilities documents become slow and hard to parse for clients (e.g. bloated 100MB+ files) ● Partitioning with Virtual Services can help ● Sharding on different instances can help 

Clustering ● Scalability + High Availability ● Scaling GeoServer ● Scaling up to 64 cores has been proven in the past ● Scaling out has been done in K8s, AWS, Azure, GCP, etc… ● Our Recommendations ● Scale out instances with 4CPU + 8GB Ram ● Scale up instances if really needed ● Autoscaling based on CPU usage is expected under load instances with 4CPU + 8GB Ram ● Autoscaling based on RAM might be the effect of an issue or a big ● Clustering Paradigms ● Passive Clustering 🡪 GS instances ignore each other ● Active Clustering 🡪 GS instances talk to each other 

Preferred Clustering Layout Backoffice - Production ● Backoffice instance is for administration ● Changes via GUI or via REST Interface ● Can do Active/Passive ● Productions instances are for data serving ● No config changes ● Can scale horizontally (autoscaling) ● Data is centralized and shared between instances ● Configuration changes requires reload ● It can cover most use cases 

Clustering - Takeaways ● GS stores its config in files in the data directory ● GS load its config in memory at startup ● GS does not automatically pick up config changes from the data directory ● GS startup/reload times can be long with 10k+ layers ● GS continuously write to log files ● GS TileCache can work in clustering ● 99% of cases clustering with backoffice-production is the way to go! 

Final Checklist ✔ Study/Analyze your data ✔ Study/Analyze your users/scenario ✔ Study/Analyze the deployment environment ✔ Study/Analyze GeoServer strengths and limitations wrt to the above (we can help here ) ✔ Prepare a deployment plan ✔ Repeat 🡪 perfection comes from practice! • Do your homework or suffer forever! 

Conclusions 

Conclusions • Key element to focus on • data optimization • data modeling • GeoServer configuration • Favor parametric stores in GeoServer • ImageMosaic • SQL Views • After data optimization → plan caching • Favor simpler deployment → easier to scale • Automate as much as possible 

The End Questions? [email protected] [email protected] [email protected]