Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring in Kubernetes with Prometheus and Grafana

Bastian Hofmann
July 21, 2020
Programming
0
220

Monitoring in Kubernetes with Prometheus and Grafana

Kubernetes makes it easy and reliable to deploy and run your services. But in order to optimize performance and scale them, you need to know more than just that they are running. In this talk you will learn how to set up a powerful monitoring infrastructure on Kubernetes with Prometheus and Grafana which lets you measure and analyze the latency, availability and resource usage of your complete system as well as individual services and requests.

Bastian Hofmann

July 21, 2020
Tweet

More Decks by Bastian Hofmann

See All by Bastian Hofmann
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
74
Highly available cross-region deployments with Kubernetes
bastianhofmann
1
100
From source to Kubernetes in 30 minutes
bastianhofmann
0
100
Introduction to Kubernetes
bastianhofmann
1
140
CI/CD with Kubernetes
bastianhofmann
0
120
Creating a fast Kubernetes Development Workflow
bastianhofmann
1
230
Deploying your first Micro-Service application to Kubernetes
bastianhofmann
2
140
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
140
Dive-In-Workshop: Kubernetes
bastianhofmann
0
420

Other Decks in Programming

See All in Programming
Evolving a microservice architecture: how to right-size your services
cer
PRO
0
700
OpenSearchを使って ごちクルの検索画面を爆速にする
konpay
0
150
Parsing RBS
soutaro
0
490
Multiverse Ruby
shioyama
1
2.5k
LlamaIndex で Text-to-SQL 100 本ノック！
os1ma
2
770
Firebase A/B TestingとRemote Config
を最大限に活用する方法
mukky620
1
340
The future vision of Ruby Parser
yui_knk
3
3.4k
Adding custom rule for Rubocop in the 2 month of employment
little_rubyist
0
1k
PO,SMに送るテスト自動化の8原則に5箇条を添えて / scrumniigata2023
pineapplecandy
2
790
サーバーサイド開発にありがたい GitHub Copilot / ChatGPT
masatoshiitoh
0
500
Composing a Design System
stewemetal
0
170
Devoxx UK - Java Life is Short
kdubois
0
180

Featured

See All Featured
Web development in the modern age
philhawksworth
197
9.7k
A Modern Web Designer's Workflow
chriscoyier
689
190k
Designing for humans not robots
tammielis
245
24k
A designer walks into a library…
pauljervisheath
199
17k
Visualization
eitanlees
130
12k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
13
1.4k
The Web Native Designer (August 2011)
paulrobertlloyd
78
2.3k
Writing Fast Ruby
sferik
613
58k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
225
50k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
658
120k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
25
5.3k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
33
21k

Transcript

  1. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 1
    © Copyright 2020 Rancher Labs. All Rights Reserved. 1
    Monitoring in Kubernetes
    with Prometheus and Grafana
    BASTIAN HOFMANN
    Field Engineer - DACH

    View Slide

  2. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 2
    Rancher Technical Overview
    Rancher Technical Overview
    © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 5
    Rancher’s recipe for production quality Kubernetes at scale
    Shared Tooling & Services
    Security & Authentication
    Simplified Cluster Operations & Infrastructure Management
    Policy management
    Pod & network
    security policies
    CIS benchmark
    monitoring
    RBAC
    policies
    Configuration
    enforcement
    Visibility &
    diagnostics
    Centralized
    audit
    Monitoring
    & alerting
    Kubernetes version
    management
    Node pool
    management
    Cluster
    provisioning
    Amazon
    EKS
    Azure
    AKS
    Google
    GKE
    Cloud
    Datacenter Dev Branch Edge
    Secure Application Deployment
    Routing Autoscaling
    Metrics
    Load Balancing Canary Git Deployments
    Beta
    © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 1
    Rancher enables production quality Kubernetes
    operations everywhere
    Amazon
    EKS
    Azure
    AKS
    Google
    GKE
    Containerized App 1 Containerized App 2 Containerized App 3
    Cloud
    Datacenter Dev Branch Edge
    Central
    Management
    Shared Tooling & Services
    Security & Authentication
    Simplified Cluster Operations & Infrastructure Management
    Policy management
    Pod & network
    security policies
    CIS benchmark
    monitoring
    RBAC
    policies
    Configuration
    enforcement
    Visibility &
    diagnostics
    Centralized
    audit
    Monitoring
    & alerting
    Kubernetes version
    management
    Node pool
    management
    Cluster
    provisioning
    Infrastructure
    agnostic

    View Slide

  3. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 3
    Agenda
    • How to set up Prometheus and Grafana
    • How to get and visualize metrics from Kubernetes
    • How to get and visualize metrics from your own applications
    • How to add alerts
    • How to collect logs from your applications
    • How Service Meshes help with traffic observability

    View Slide

  4. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 4 4
    Demos, Demos, Demos

    View Slide

  5. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 5 5
    First, we need a Kubernetes Cluster

    View Slide

  6. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 6 6
    Demo

    View Slide

  7. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 7 7
    Extending Kubernetes

    View Slide

  8. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 8
    Standardization of
    compute, network and service discovery

    View Slide

  9. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 9
    Extensible API

    View Slide

  10. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 10
    Easy integration of additional tools

    View Slide

  11. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 11
    Custom Resource Definitions
    • Extend the Kubernetes API with additional Resource Definitions
    • Certificate
    • MySQLCluster
    • Prometheus
    • …
    • Deploy a controller into the cluster that listens on creation, change and
    deletion of these resources and perform the necessary actions

    View Slide

  12. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 12

    View Slide

  13. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 13 13
    Prometheus-operator

    View Slide

  14. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 14
    Manages and Configures
    Prometheus, Alertmanager (and Grafana)

    View Slide

  15. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 15
    CustomResourceDefinitions
    • Workload Management
    • Prometheus
    • Alertmanager
    • Prometheus Configuration
    • ServiceMonitor
    • PodMonitor
    • Rule

    View Slide

  16. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 16

    View Slide

  17. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 17 17
    Demo

    View Slide

  18. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 18 18
    Monitoring external resources

    View Slide

  19. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 19
    Blackbox-monitor

    View Slide

  20. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 20 20
    Demo

    View Slide

  21. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 21 21
    Central log management

    View Slide

  22. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 22
    Several solutions
    • Commercial
    • OpenSource
    • ElasticSearch – FluentD/FluentBit/Logstash – Kibana
    • Loki – Promtail – Grafana

    View Slide

  23. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 23 23
    Demo

    View Slide

  24. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 24 24
    Service Meshes

    View Slide

  25. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 25
    Service Meshes
    • Observability
    • Traffic Control
    • Traffic Security
    • Connectivity
    • On top of the Kubernetes Pod network

    View Slide

  26. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 26
    Service Meshes
    • Istio
    • Linkerd
    • Maesh
    • Kuma
    • …

    View Slide

  27. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 27

    View Slide

  28. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 28 28
    Service Mesh Features

    View Slide

  29. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 29
    Traffic Security
    • Automatic mutual TLS encryption of traffic
    • Automatic certificate management
    • Optional certificate-based authentication of traffic

    View Slide

  30. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 30

    View Slide

  31. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 31
    Traffic Control
    • Advanced traffic management between services in the mesh
    • Blue/Green Deployments
    • Canary Deployments
    • A/B Testing
    • Fault injection
    • Circuit breakers

    View Slide

  32. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 32

    View Slide

  33. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 33
    Multi-cluster Service Mesh
    • Connect separate Kubernetes cluster together
    • Securely handle traffic between clusters

    View Slide

  34. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 34
    Observability
    • Monitor all TCP connections, HTTP and GRCP requests
    • Bytes in/out
    • Amount of requests
    • Success rate
    • Response time
    • Visualize service communication

    View Slide

  35. View Slide

  36. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 36
    Tracing
    • Trace a single incoming request through all services
    • Enhance with application spans through Open Tracing

    View Slide

  37. View Slide

  38. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 38 38
    Demo

    View Slide

  39. © Copyright 2020 Rancher Labs. All Rights Reserved. Confidential 39 39
    Thank you
    Bastian Hofmann
    Field Engineer, DACH
    * [email protected]

    View Slide