Paris container Day 2017 - End-to-end monitoring with the Prometheus Operator - Max Inden, CoreOS

Transcript

1. End-to-end Monitoring with the Prometheus Operator By @mxinden
2. Max Inden Test-Engineer at CoreOS @mxinden Max.Inden@CoreOS.com
3. Secure, simplify and automate container infrastructure
4. Secure, simplify and automate container infrastructure
5. Secure, simplify and automate container infrastructure
6. Secure, simplify and automate container infrastructure
8. Why Monitoring?
9. Why Monitoring? Alerting
10. Why Monitoring? Long-term trends Alerting
11. What is Prometheus? • Open Source Monitoring • Built by Soundcloud • Inspired by borgmon •
12. What is Prometheus? • Pull-based •
13. What is Prometheus? • Pull-based • Multi-Dimensional •
14. What is Prometheus? • Pull-based • Multi-Dimensional • Metrics, not logging, not tracing •
15. What is Prometheus? • Pull-based • Multi-Dimensional • Metrics, not logging, not tracing • No magic! •
16. Target Target Target
17. Target /metrics Target /metrics Target /metrics
18. Prometheus Target /metrics Target /metrics Target /metrics
19. Prometheus Target /metrics Target /metrics Target /metrics 15s
20. Target /metrics # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",path="/status"} 8
21. Target /metrics # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",path="/status"} 8 Metric name
22. Target /metrics # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",path="/status"} 8 Label
23. Target /metrics # HELP http_requests_total Total number of HTTP requests made. # TYPE http_requests_total counter http_requests_total{code="200",path="/status"} 8 Value
24. Prometheus Target /metrics Target /metrics Target /metrics
25. Prometheus Target /metrics Target /metrics Target /metrics PromQL
26. Current percentage of HTTP errors across all service instances?
27. Current percentage of HTTP errors across all service instances? sum by(path) rate(http_requests_total{status="500"}[5m])) / sum by(path) rate(http_requests_total[5m]))
28. Current percentage of HTTP errors across all service instances? {path="/status"} 0.0039 {path="/"} 0.0011 {path="/api/v1/topics/:topic"} 0.087 {path="/api/v1/topics} 0.0342 sum by(path) rate(http_requests_total{status="500"}[5m])) / sum by(path) rate(http_requests_total[5m]))
29. Prometheus Target /metrics Target /metrics Target /metrics PromQL
30. Prometheus Target /metrics Target /metrics Target /metrics PromQL Web UI Dashboard
31. Prometheus Target /metrics Target /metrics Target /metrics
32. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition
33. ALERT DiskWillFillIn4Hours IF predict_linear(node_filesystem_free[1h], 4*3600) < 0 Is any disk about to run full within 4 hours? 0 now -1h +4h
34. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition
35. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition 1m
36. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition 1m
37. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition Alertmanager 1m
38. Alertmanager Deduplicates Alert Alert Alert Alert Alert Alert Alert
39. Alertmanager Deduplicates Alert Alert Alert Alert Alert Alert Alert Groups Alert Alert Alert Alert Alert Alert
40. Alertmanager Deduplicates Alert Alert Alert Alert Alert Alert Alert Groups Alert Alert Alert Alert Alert Alert Routes Alert Alert Alert Alert Alert Team A Team B Team C
41. Alertmanager Deduplicates Alert Alert Alert Alert Alert Alert Alert Groups Alert Alert Alert Alert Alert Alert Routes Alert Alert Alert Alert Alert Team A Team B Team C
42. Prometheus Target /metrics Target /metrics Target /metrics Alertmanager
43. Prometheus Target /metrics Target /metrics Target /metrics Alertmanager
45. Monitoring
46. Application Cluster Monitoring
47. Cluster Monitoring
48. What is Kubernetes? Platform for running containerized applications
49. What is Kubernetes? Announced 2014 by Google Influenced by Borg & Omega v1.01 in July 2015 Kubernetes joins the CNCF
50. Master
51. Master API-Server etcd Controller-Manager Scheduler Kube-DNS ...
52. Master API-Server etcd Controller-Manager Scheduler Kube-DNS ... Worker
53. Master API-Server etcd Controller-Manager Scheduler Kube-DNS ... Worker Kubelet Kube-Proxy ...
54. Application Monitoring
55. Location User AppX
56. Location User AppX User AppX Location
57. Location User AppX User AppX Location Service Service Service
58. Location User AppX User AppX Location Service Service Service Prometheus
59. Location User AppX User AppX Location Service Service Service Prometheus ?
60. K8s-API-Server Location User AppX User AppX Location Service Service Service Prometheus
61. Location User AppX User AppX Location Service Service Service Prometheus K8s-API-Server
62. Service Discovery • Static target list • DNS discovery • Kubernetes discovery • ...
64. Master API-Server etcd Controller-Manager Scheduler Kube-DNS ... Worker Kubelet Kube-Proxy ... Location User AppX User AppX Location Service Service Service Prometheus K8s-API-Server Application-Monitoring Cluster-Monitoring
65. Problem Prometheus is stateful and difficult to configure!
66. Introducing the Prometheus Operator
67. What is a K8s Operator?
68. What is a K8s Operator? Application specific operational knowledge
69. What is a K8s Operator?
70. What is a K8s Operator? </>
71. What is a K8s Operator? </>
72. What is a K8s Operator? </> Operator
74. Prometheus Operator • Kubernetes native configuration • Automated management and upgrades of Prometheus & Alertmanager
75. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app spec: ...
76. apiVersion: monitoring.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-k8s spec: ...
77. Kube-Prometheus Single command to install: • Prometheus & Alertmanager Cluster • Alerting rules • Dashboarding
78. Demo
79. Recap
80. What is Prometheus? • Pull-based • Multi-Dimensional • Metrics, not logging, not tracing • No magic! •
81. Prometheus Target /metrics Target /metrics Target /metrics 15s
82. Prometheus Target /metrics Target /metrics Target /metrics Alert Definition Alertmanager 1m
83. Prometheus-Operator & Kube-Prometheus </> Operator
84. Where to go from here? Prometheus.io /coreos/prometheus-operator
85. San Francisco, New York & Berlin We are hiring!
86. Max Inden Test-Engineer at CoreOS @mxinden Max.Inden@CoreOS.com