Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Kubernetes Clusters with Prometheus

Monitoring Kubernetes Clusters with Prometheus

Fabian Reinartz

November 10, 2016
Tweet

More Decks by Fabian Reinartz

Other Decks in Technology

Transcript

  1. Monitoring Challenges • A lot of targets to monitor •

    Targets constantly change • Need high-level overview (by namespace, service, …) • Need drill-down for investigation (down to pod and below)
  2. Prometheus Recap • Pull-based monitoring system • Multi-dimensional data model

    • Handles millions of time series per instance http_requests_total{path="/home", status="200", method="GET"} 9523 http_requests_total{path="/home", status="500", method="GET"} 233 http_requests_total{path="/settings", status="200", method="GET"} 512 http_requests_total{path="/settings", status="200", method="POST"} 68
  3. Powerful Querying histogram_quantile(0.99, sum by(path, le) (rate(request_latency_seconds_bucket[5m])) ) 99th percentile

    latency on API server operations per resource? {path="/status"} 0.012 {path="/"} 0.43 {path="/api/v1/topics/:topic"} 1.31 {path="/api/v1/topics} 0.192
  4. Kubernetes Integration • Always sync monitoring targets with Kubernetes API

    • Use meta information to enrich metrics scrape metrics API Server sync monitoring targets
  5. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet

    etcd0 etcd1 etcd2 ... pods kube-state-metrics
  6. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet

    etcd0 etcd1 etcd2 ... pods kube-state-metrics
  7. Metrics of Kubernetes cAdvisor API server Kubelet Node node exporter

    etcd0 etcd1 etcd2 ... pods kube-state-metrics
  8. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet

    etcd0 etcd1 etcd2 ... pods kube-state-metrics
  9. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet

    etcd0 etcd1 etcd2 ... pods kube-state-metrics
  10. Metrics of Kubernetes Node node exporter API server Kubelet etcd0

    cAdvisor etcd1 etcd2 ... pods kube-state-metrics
  11. Monitoring Challenges • A lot of targets to monitor ✓

    • Targets constantly change ✓ • Need high-level overview (by namespace, service, …) ✓ • Need drill-down for investigation (down to pod and below) ✓ • AND: Make monitoring trivial to deploy & operate
  12. Managed Deployments apiVersion: prometheus.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-k8s spec:

    replicas: 2 version: v1.3.0 Prometheus TPR defines desired Prometheus setup Operator deploys and manages Prometheus instances Operator deploy & manage Prometheus Server watch
  13. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:

    tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declarative definition of how to monitor a group of services Loosely coupled via labels Part of your cluster’s API
  14. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:

    tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Select applicable services by their labels
  15. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:

    tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declare where these services expose metrics
  16. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:

    tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Prometheus deployments include ServiceMonitors by their labels
  17. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-frontend Spec:

    version: v1.3.0 serviceMonitors: - selector: matchLabels: tier: frontend Prometheus deployments include ServiceMonitors by their labels
  18. Service 1 Service 2 Service 3 Service 4 Service 5

    ServiceMonitor 1 ServiceMonitor 2 Prometheus Operator deploy & manage Prometheus Server watch
  19. [email protected] @fabxc QUESTIONS? Thanks! We’re hiring: coreos.com/careers Let’s talk! #prometheus

    on Freenode More events: coreos.com/community LONGER CHAT? also in Berlin!