Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GeoServer on Kubernetes: Set up and operate a G...

GeoServer on Kubernetes: Set up and operate a GeoServer Cluster in K8s

In this presentation, our Lead DevOps Alessandro Parma, together with our Director Simone Giannecchini, will walk us through the key points for setting up your own GeoServer cluster in K8s.

Simone Giannecchini

July 29, 2021
Tweet

More Decks by Simone Giannecchini

Other Decks in Technology

Transcript

  1. GeoServer on Kubernetes: Set up and Operate a GeoServer Cluster

    in K8s Alessandro Parma Simone Giannecchini Luis E. Bermudez GeoSolutions
  2. Contents ⚫ About us ⚫ What is GeoServer? ⚫ What

    is Kubernetes? ⚫ Key Concepts ⚫ Running GeoServer on Kubernetes ⚫ Conclusions 29th of July 2021- Online
  3. GeoSolutions ⚫ Founded in 2006, offices in Italy & US

    ⚫ Our core products ⚫ Our offer Enterprise Support Services Deployment Subscription Professional Training Customized Solutions GeoNode 29th of June 2021- Online
  4. Trusted by more than 200 clients • UN FAO (CIOK,

    FIGIS, NRL, FORESTRY, ESTG), UN WFP, World Bank, DLR, EUMETSAT, JRC, ARPAT, NATO CMRE, UNESCO, IGAD, UNEP, etc.. • BAYER, BASF, DigitalGlobe, MDA, TOPCON, SwissRE, e-GEOS, Halliburton, etc.. 29th of July 2021- Online
  5. Industries Smart Cities Space MetOcean Defense Natural Resources OpenData Utilities

    Research Emergency Response Government 29th of July 2021- Online
  6. Associations We strongly support Open Source, it Is in our

    core We actively participate in OGC working groups and get funded to advance new open standards We support standards critical to GEOINT 29th of July 2021- Online
  7. Our Distinctive Traits ⚫ Lead Developers of GeoNode, GeoServer, MapStore

    and GeoNetwork ⚫ Vast experience with Raster Serving ⚫ Designed and developed JAI-Ext ⚫ Designed and developed ImageIO-Ext ⚫ Design and Developed most raster code in GeoTools/GeoServer ⚫ Vast Experience with Vector Data Serving ⚫ WFS, WMS, Vector Tiles with OGV ⚫ Extensive Experience with Spatial DBMS ⚫ Oracle, SQL Server, Postgis, MongoDB, etc.. ⚫ Extensive Experience with creating webgis applications ⚫ OpenLayers, Leaflet, Cesium, MapboxGL ⚫ Ext-JS, JQuery, Bootstrap, Angular, React, Redux ⚫ Extensive Experience with OGC Protocols ⚫ Extensive Experience in Performance and Scalability (Big Data and Cloud) ⚫ Unparalleled multi-industry experience 29th of July 2021- Online
  8. Team – Key Members 30+ Staff Members, 25+ Software Engineers

    ⚫ Andrea Aime: GeoServer Project Steering Committee, GeoTools PMC, JAI-Tools Lead, ImageIO-Ext committer ⚫ Simone Giannecchini: Founder, GeoServer PSC, GeoTools PMC, ImageIO-Ext Architect, JAI-Tools founder ⚫ Luis E. Bermudez: US CEO since 2020. 2010-2020 OGC Exec, Director of the Innovation and Compliance Programs. ⚫ Giovanni Allegri: Senior Project Manager, QGIS, GeoNode ⚫ Alessio Fabiani: Founder, GeoServer PSC, GeoTools Committer, MapStore Committer ⚫ Emanuele Tajariol: GeoServer Committer, GeoTools Committer, GeoNetwork PSC ⚫ Mauro Bartolomeoli: GeoServer Committer, GeoTools Comitter, GeoBatch Committer, MapStore Architect ⚫ Lorenzo Natali: MapStore2 Technical Lead 29th of July 2021- Online
  9. We are hiring! ⚫ You are fond on Open Source

    ⚫ If you are fond on GeoServer|GeoNode|MapStore|QGIS ⚫ If you like an international environment (both clients and colleagues) ⚫ If you like a challenging position ⚫ Send us your resume → multiple positions: ⚫ DevOps Engineer ⚫ Senior Frontend SW Engineer ⚫ Java SW Engineer ⚫ Support Engineer ⚫ Python SW Engineer 29th of July 2021- Online
  10. What is GeoServer? ⚫ GeoSpatial enterprise gateway • Java Enterprise

    • Management and Dissemination of raster and vector data ⚫ Standards compliant • OGC WCS 1.0, 1.1.1 (RI), 2.0 • OGC WFS 1.0, 1.1 (RI), 2.0 • OGC WMS 1.1.1, 1.3.0 • OGC WPS 1.0.0 • OGC CSW 2.0.1 (ebRIM) ⚫ Google Earth/Maps support • KML, GeoSearch, etc.. 29th of July 2021- Online
  11. What is GeoServer? GeoServer WFS WMS PostGIS Oracle H2 DB2

    SQL Server GeoPackage MySql Spatialite Elastic MongoDB Shapefile ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- WFS PNG, GIF JPEG TIFF, GeoTIFF SVG, PDF KML/KMZ Shapefile GML2 GML3 GeoRSS GeoJSON CSV/XLS GeoPackage Raw vector data Servers Styled maps DBMS Vector files WCS GeoTIFF WMS ArcGrid Img+world Mosaic MrSID JPEG 2000 ECW,Pyramid, Oracle GeoRaster, PostGis Raster, NetCDF Raster files Raw raster data GeoTIFF ArcGrid GTopo30 Img+World WMTS, TMS KML superoverlays Google maps tiles OGC tiles OSGEO tiles KML WPS CSW ESRI REST 29th of July 2021- Online
  12. What is Kubernetes ⚫ Kubernetes is an open source platform

    to manage containerized workloads and services ⚫ Developed by Google ⚫ Based on their experience with Borg ⚫ Manages a cluster of nodes running Containerized applications ⚫ Helps you manage and scale your applications 29th of June 2021- Online
  13. What is a Container ⚫ What is a container then?

    ⚫ Type of virtualization that happens at the operating system level ⚫ Applications can run in an isolated user spaces called containers ⚫ Implemented at the kernel level, multiple containers share the same OS kernel 29th of July 2021- Online
  14. Containers vs VMs ⚫ How does it compare to VMs?

    ⚫ VMs ⚫ Each individual VM has its own full OS ⚫ Hypervisor abstracting and managing resources ⚫ Typically, multiple applications running in the same VM ⚫ Looks pretty much like a dedicated machine ⚫ Pretty much transparent for Devs and Ops (with extra flexibility) 29th of July 2021- Online
  15. Containers vs VMs ⚫ How does it compare to VMs?

    ⚫ Containers ⚫ Common, shared, OS kernel ⚫ Multiple containers (possibly from multiple projects) on the same machine ⚫ The OS provides isolation and security ⚫ Has its own idiosyncrasies, tools, skillset.. ⚫ Usually run one application per container ⚫ The application is bundled with its dependencies and runs in its own isolated user space (the “image”) 29th of July 2021- Online
  16. Why is it relevant? ⚫ Traditional deployments ⚫ No resource

    boundaries → some applications starve for resources ⚫ Can’t easily reallocate resources after the initial setup ⚫ Virtual Machines ⚫ Multiple VMs on the same server → better resource utilization ⚫ Better isolation ⚫ Each VM has a copy of the OS 29th of July 2021- Online
  17. Why is it relevant? ⚫ Containers ⚫ shared kernel with

    isolated userspace ⚫ each container has its own filesystem, a share of CPU cores ⚫ decoupled from they underlying infrastructure → portable across distributions and cloud providers ⚫ … 29th of July 2021- Online
  18. Why is it relevant? ⚫ … ⚫ fast image creation

    and easy rollback compared to VMs → Good fit for frequent deployments and CI/CD ⚫ separation of concerns between Devs and Ops ⚫ consistency across development in multiple environments 29th of July 2021- Online
  19. Why Kubernetes? ⚫ To manage the containers that run your

    applications in production with no downtime ⚫ Takes care of running your application containers on a distributed system ⚫ Takes care of scaling (both the application and the nodes cluster) and failover 29th of June 2021- Online
  20. Why Kubernetes? ⚫ Also provides ⚫ Service discovery and load

    balancing ⚫ Storage Orchestration: mount storage of choice) ⚫ Automated rollouts and rollbacks. You describe the desired state for your containers ⚫ Self Healing: restart failing containers, hide them until they are ready state, .. ⚫ Configuration management and Secrets management: change config without rebuilding the application 29th of July 2021- Online
  21. How does it compare to.. ⚫ There are other orchestrators

    and tools available to manage containers ⚫ Docker Compose ⚫ allows you to define services as collections but that is pretty much it ⚫ Docker Swarm ⚫ gives you to work on a distributed environment ⚫ services definition and commands are somewhat similar to compose ⚫ not as sophisticated (and complex! as K8s) 29th of July 2021- Online
  22. How does it compare to.. ⚫ Rancher ⚫ Started before

    K8s became so popular ⚫ Had it’s own “engine” to run containers (“Cattle”) ⚫ Rancher 2 is a different, it is used to deploy and manage K8s cluster 29th of July 2021- Online
  23. Resources ⚫ What is Kubernetes ⚫ Borg: The Predecessor to

    Kubernetes ⚫ Containerization ⚫ What is a Container ⚫ Docker Compose ⚫ Docker Swarm ⚫ Rancher 29th of July 2021- Online
  24. Kubernetes Key concepts ⚫ Kubernetes manages a set of cluster

    nodes (“workers”) running containerized applications ⚫ The Control Pane manages the pods and the worker nodes ⚫ API Server ⚫ Scheduler ⚫ Controller Managers 29th of July 2021- Online
  25. Kubernetes Key concepts ⚫ The Worker Nodes host our pods

    ⚫ Kubelet - makes sure that containers are running in a Pod ⚫ Proxy – communication with pods in and out of the cluster ⚫ Container Runtime - software that is responsible for running containers (docker, containerd, …) 29th of July 2021- Online
  26. Pods ⚫ A Pod is basically a group of Docker

    containers with shared namespaces and filesystem volumes ⚫ You can have more that one container running in a pod. They will be co-located and tightly coupled ⚫ Example: content serving container with a sidecar container that is updating the content ⚫ Pods can be scaled horizontally to run multiple instances of the same applications i.e. multiple replicas of the pod 29th of July 2021- Online
  27. Controllers ⚫ Controllers for a given resource manages the state

    of the resource ⚫ Replicas ⚫ Rollout ⚫ Healing ⚫ .. ⚫ Node failure → creates a replacement pod ⚫ Deployment ⚫ Statefulset ⚫ DaemonSet 29th of July 2021- Online
  28. Set up K8s cluster ⚫ Minikube to quickly spin up

    a local Kubernetes Cluster ⚫ Compatible with multiple Oses ⚫ Cloud Service all major cloud providers offer Kubernetes as a managed service ⚫ AWS ⚫ Google Cloud ⚫ Azure ⚫ Digital Ocean 29th of July 2021- Online
  29. Containerize GeoServer ⚫ We’ll need a docker image GeoServer ⚫

    Many Images available on the Hub ⚫ GeoSolutions one here with sources here ⚫ Community image is a WIP ⚫ Most of the work already done ⚫ Based on Tomcat images ⚫ Built periodically by Docker Hub ⚫ Readily usable, just change the webapp 29th of July 2021- Online
  30. Example K8s deployment ⚫ Passive Cluster with Backoffice instance ⚫

    GeoServer is defined as a StatefulSets ⚫ Dedicated resources for backoffice and slave ⚫ Secrets for sensitive data ⚫ Master password ⚫ Database connection details ⚫ Volumes ⚫ Abstracting the underlying storage ⚫ Provide different StorageClasses 29th of July 2021- Online
  31. GeoServer StatefulSet ⚫ GeoServer StatefulSet ⚫ Statefulset contains the Pod

    definition ⚫ Sets the number of replicas ⚫ Health checks ⚫ References to Volumes ⚫ ConfigMap ⚫ Secrets ⚫ Environment Variables ⚫ … 29th of July 2021- Online
  32. Shared Volumes ⚫ Shared Volumes ⚫ Accessible by multiple nodes

    ⚫ GeoServer datadirs ⚫ Cached tiles ⚫ Spatial Data 29th of July 2021- Online
  33. Non-Shared Volumes ⚫ Local Storage ⚫ Visible only to pods

    running on the same node ⚫ Fast read writes for ephemeral data ⚫ Be prepared to loose the data at any time ⚫ Good fit for Logs and Audit files 29th of July 2021- Online
  34. Other Resources ⚫ Kubernetes Jobs and DeamonSets to run tasks

    in your cluster (Cleanup logs and audit files) 29th of July 2021- Online
  35. Scaling your Pods ⚫ Manually scaling your StatefulSets ⚫ Use

    K8s Horizontal Pod Autoscaler and Scale based on metrics like CPU 29th of July 2021- Online
  36. Conclusions ⚫ Containers are changing the way we deploy applications

    ⚫ You can run GeoServer in Kubernetes ⚫ There are readily available images for GeoServer on the Docker Hub ⚫ A K8s deployment of a GeoServer cluster requires you to define many resources Statefulsets, Volumes, Secrets, .. ⚫ Choose the appropriate StorageClass for each Volume 29th of July 2021- Online