Monitoring in Kubernetes with Prometheus and Grafana

Monitoring in Kubernetes with Prometheus and Grafana

Kubernetes makes it easy and reliable to deploy and run your services. But in order to optimize performance and scale them, you need to know more than just that they are running. In this talk you will learn how to set up a powerful monitoring infrastructure on Kubernetes with Prometheus and Grafana which lets you measure and analyze the latency, availability and resource usage of your complete system as well as individual services and requests.

Ded87c77266697ee6981c2277bb97633?s=128

Bastian Hofmann

July 21, 2020
Tweet

Transcript

  1. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 1

    © Copyright 2020 Rancher Labs. All Rights Reserved. 1 Monitoring in Kubernetes with Prometheus and Grafana BASTIAN HOFMANN Field Engineer - DACH
  2. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 2

    Rancher Technical Overview Rancher Technical Overview © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 5 Rancher’s recipe for production quality Kubernetes at scale Shared Tooling & Services Security & Authentication Simplified Cluster Operations & Infrastructure Management Policy management Pod & network security policies CIS benchmark monitoring RBAC policies Configuration enforcement Visibility & diagnostics Centralized audit Monitoring & alerting Kubernetes version management Node pool management Cluster provisioning Amazon EKS Azure AKS Google GKE Cloud Datacenter Dev Branch Edge Secure Application Deployment Routing Autoscaling Metrics Load Balancing Canary Git Deployments Beta © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 1 Rancher enables production quality Kubernetes operations everywhere Amazon EKS Azure AKS Google GKE Containerized App 1 Containerized App 2 Containerized App 3 Cloud Datacenter Dev Branch Edge Central Management Shared Tooling & Services Security & Authentication Simplified Cluster Operations & Infrastructure Management Policy management Pod & network security policies CIS benchmark monitoring RBAC policies Configuration enforcement Visibility & diagnostics Centralized audit Monitoring & alerting Kubernetes version management Node pool management Cluster provisioning Infrastructure agnostic
  3. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 3

    Agenda • How to set up Prometheus and Grafana • How to get and visualize metrics from Kubernetes • How to get and visualize metrics from your own applications • How to add alerts • How to collect logs from your applications • How Service Meshes help with traffic observability
  4. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 4

    4 Demos, Demos, Demos
  5. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 5

    5 First, we need a Kubernetes Cluster
  6. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 6

    6 Demo
  7. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 7

    7 Extending Kubernetes
  8. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 8

    Standardization of compute, network and service discovery
  9. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 9

    Extensible API
  10. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 10

    Easy integration of additional tools
  11. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 11

    Custom Resource Definitions • Extend the Kubernetes API with additional Resource Definitions • Certificate • MySQLCluster • Prometheus • … • Deploy a controller into the cluster that listens on creation, change and deletion of these resources and perform the necessary actions
  12. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 12

  13. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 13

    13 Prometheus-operator
  14. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 14

    Manages and Configures Prometheus, Alertmanager (and Grafana)
  15. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 15

    CustomResourceDefinitions • Workload Management • Prometheus • Alertmanager • Prometheus Configuration • ServiceMonitor • PodMonitor • Rule
  16. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 16

  17. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 17

    17 Demo
  18. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 18

    18 Monitoring external resources
  19. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 19

    Blackbox-monitor
  20. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 20

    20 Demo
  21. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 21

    21 Central log management
  22. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 22

    Several solutions • Commercial • OpenSource • ElasticSearch – FluentD/FluentBit/Logstash – Kibana • Loki – Promtail – Grafana
  23. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 23

    23 Demo
  24. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 24

    24 Service Meshes
  25. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 25

    Service Meshes • Observability • Traffic Control • Traffic Security • Connectivity • On top of the Kubernetes Pod network
  26. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 26

    Service Meshes • Istio • Linkerd • Maesh • Kuma • …
  27. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 27

  28. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 28

    28 Service Mesh Features
  29. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 29

    Traffic Security • Automatic mutual TLS encryption of traffic • Automatic certificate management • Optional certificate-based authentication of traffic
  30. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 30

  31. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 31

    Traffic Control • Advanced traffic management between services in the mesh • Blue/Green Deployments • Canary Deployments • A/B Testing • Fault injection • Circuit breakers
  32. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 32

  33. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 33

    Multi-cluster Service Mesh • Connect separate Kubernetes cluster together • Securely handle traffic between clusters
  34. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 34

    Observability • Monitor all TCP connections, HTTP and GRCP requests • Bytes in/out • Amount of requests • Success rate • Response time • Visualize service communication
  35. None
  36. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 36

    Tracing • Trace a single incoming request through all services • Enhance with application spans through Open Tracing
  37. None
  38. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 38

    38 Demo
  39. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 39

    39 Thank you Bastian Hofmann Field Engineer, DACH * bastian@rancher.com