Hynek Schlawack - Get Instrumented: How Prometheus Can Unify Your Metrics

To get real time insight into your running applications you need to instrument them and collect metrics: count events, measure times, expose numbers. Sadly this important aspect of development was a patchwork of half-integrated solutions for years. Prometheus changed that and this talk will walk you through instrumenting your apps and servers, building dashboards, and monitoring using metrics.

https://us.pycon.org/2016/schedule/presentation/1601/

Transcript

1. Hynek Schlawack Get Instrumented How Prometheus Can Unify Your Metrics
2. Goals
3. Goals
4. Goals
5. Goals
6. Goals
7. Service Level
8. Service Level Indicator
9. Service Level Indicator Objective
10. Service Level Indicator Objective (Agreement)
11. Metrics
12. Metrics avg latency 0.3 0.5 0.8 1.1 2.6
13. Metrics 12:00 12:01 12:02 12:03 12:04 avg latency 0.3 0.5 0.8 1.1 2.6
14. Metrics 12:00 12:01 12:02 12:03 12:04 avg latency 0.3 0.5 0.8 1.1 2.6 server load 0.3 1.0 2.3 3.5 5.2
16. Instrument
17. Instrument
18. Instrument
19. Instrument
20. Instrument
23. Metric Types
24. Metric Types ❖counter
25. Metric Types ❖counter ❖gauge
26. Metric Types ❖counter ❖gauge ❖summary
27. Metric Types ❖counter ❖gauge ❖summary ❖histogram
28. Metric Types ❖counter ❖gauge ❖summary ❖histogram ❖ buckets (1s, 0.5s, 0.25, …)
29. Averages
30. ❖ avg(request time) ≠ avg(UX) Averages
31. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 Averages
32. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 Averages
33. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 Averages
34. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 ❖ median({1, 1, 1, 1, 10}) = 1 Averages
35. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 ❖ median({1, 1, 1, 1, 10}) = 1 Averages
36. ❖ avg(request time) ≠ avg(UX) ❖ avg({1, 1, 1, 1, 10}) = 2.8 ❖ median({1, 1, 1, 1, 10}) = 1 ❖ median({1, 1, 100_000}) = 1 Averages
37. Percentiles
38. Percentiles nth percentile P of a data set = P ≥ n% of values
40. 50th percentile = 1 ms
41. 50th percentile = 1 ms 50% of requests done by 1 ms
42. Percentiles
43. Percentiles P {1, 1, 100_000} 50th 1
44. Percentiles P {1, 1, 100_000} 50th 1 95th 90_000
48. Naming
49. Naming backend1_app_http_reqs_msgs_post backend1_app_http_reqs_msgs_get …
50. Naming backend1_app_http_reqs_msgs_post backend1_app_http_reqs_msgs_get … app_http_reqs_total
51. Naming backend1_app_http_reqs_msgs_post backend1_app_http_reqs_msgs_get … app_http_reqs_total
52. Naming backend1_app_http_reqs_msgs_post backend1_app_http_reqs_msgs_get … app_http_reqs_total
53. Naming backend1_app_http_reqs_msgs_post backend1_app_http_reqs_msgs_get … app_http_reqs_total{meth="POST", path="/msgs", backend="1"} app_http_reqs_total{meth="GET", path="/msgs", backend="1"} … app_http_reqs_total
56. 1. resolution = scraping interval
57. 1. resolution = scraping interval 2. missing scrapes = less resolution
58. Pull: Problems ❖ short lived jobs
60. Pull: Problems ❖ short lived jobs ❖ target discovery
61. Configuration scrape_configs: - job_name: 'prometheus' target_groups: - targets: - 'localhost:9090'
62. Configuration scrape_configs: - job_name: 'prometheus' target_groups: - targets: - 'localhost:9090'
63. Configuration scrape_configs: - job_name: 'prometheus' target_groups: - targets: - 'localhost:9090'
64. Configuration scrape_configs: - job_name: 'prometheus' target_groups: - targets: - 'localhost:9090' {instance="localhost:9090",job="prometheus"}
66. Pull: Problems ❖ target discovery ❖ short lived jobs ❖ Heroku/NATed systems
67. Pull: Advantages
68. Pull: Advantages ❖ multiple Prometheis easy
69. Pull: Advantages ❖ multiple Prometheis easy ❖ outage detection
70. Pull: Advantages ❖ multiple Prometheis easy ❖ outage detection ❖ predictable, no self-DoS
71. Pull: Advantages ❖ multiple Prometheis easy ❖ outage detection ❖ predictable, no self-DoS ❖ easy to instrument 3rd parties
72. Metrics Format # HELP req_seconds Time spent \ processing a request in seconds. # TYPE req_seconds histogram req_seconds_count 390.0 req_seconds_sum 177.0319407
73. Metrics Format # HELP req_seconds Time spent \ processing a request in seconds. # TYPE req_seconds histogram req_seconds_count 390.0 req_seconds_sum 177.0319407
74. Metrics Format # HELP req_seconds Time spent \ processing a request in seconds. # TYPE req_seconds histogram req_seconds_count 390.0 req_seconds_sum 177.0319407
75. Metrics Format # HELP req_seconds Time spent \ processing a request in seconds. # TYPE req_seconds histogram req_seconds_count 390.0 req_seconds_sum 177.0319407
76. Metrics Format # HELP req_seconds Time spent \ processing a request in seconds. # TYPE req_seconds histogram req_seconds_count 390.0 req_seconds_sum 177.0319407
77. Percentiles req_seconds_bucket{le="0.05"} 0.0 req_seconds_bucket{le="0.25"} 1.0 req_seconds_bucket{le="0.5"} 273.0 req_seconds_bucket{le="0.75"} 369.0 req_seconds_bucket{le="1.0"} 388.0 req_seconds_bucket{le="2.0"} 390.0 req_seconds_bucket{le="+Inf"} 390.0
78. Percentiles req_seconds_bucket{le="0.05"} 0.0 req_seconds_bucket{le="0.25"} 1.0 req_seconds_bucket{le="0.5"} 273.0 req_seconds_bucket{le="0.75"} 369.0 req_seconds_bucket{le="1.0"} 388.0 req_seconds_bucket{le="2.0"} 390.0 req_seconds_bucket{le="+Inf"} 390.0
79. Percentiles req_seconds_bucket{le="0.05"} 0.0 req_seconds_bucket{le="0.25"} 1.0 req_seconds_bucket{le="0.5"} 273.0 req_seconds_bucket{le="0.75"} 369.0 req_seconds_bucket{le="1.0"} 388.0 req_seconds_bucket{le="2.0"} 390.0 req_seconds_bucket{le="+Inf"} 390.0
81. Aggregation
82. Aggregation sum( rate( req_seconds_count[1m] ) )
83. Aggregation sum( rate( req_seconds_count[1m] ) )
84. Aggregation sum( rate( req_seconds_count[1m] ) )
85. Aggregation sum( rate( req_seconds_count[1m] ) )
86. Aggregation sum( rate( req_seconds_count{dc="west"}[1m] ) )
87. Aggregation sum( rate( req_seconds_count[1m] ) ) by (dc)
88. Percentiles histogram_quantile( 0.9, rate( req_seconds_bucket[10m] ))
89. Percentiles histogram_quantile( 0.9, rate( req_seconds_bucket[10m] ))
90. Percentiles histogram_quantile( 0.9, rate( req_seconds_bucket[10m] ))
91. Percentiles histogram_quantile( 0.9, rate( req_seconds_bucket[10m] ))
92. Percentiles histogram_quantile( 0.9, rate( req_seconds_bucket[10m] ))
95. Internal
96. Internal ❖ great for ad-hoc
97. Internal ❖ great for ad-hoc ❖ 1 expr per graph
98. Internal ❖ great for ad-hoc ❖ 1 expr per graph ❖ templating
99. PromDash
100. PromDash ❖ best integration
101. PromDash ❖ best integration ❖ former official
102. PromDash ❖ best integration ❖ former official ❖ now deprecated ❖ don’t bother
103. Grafana
104. Grafana ❖ pretty & powerful
105. Grafana ❖ pretty & powerful ❖ many integrations
106. Grafana ❖ pretty & powerful ❖ many integrations ❖ mix and match!
107. Grafana ❖ pretty & powerful ❖ many integrations ❖ mix and match! ❖ use this!
109. Alerts & Scrying
110. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
111. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
112. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
113. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
114. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
115. Alerts & Scrying ALERT DiskWillFillIn4Hours IF predict_linear( node_filesystem_free[1h], 4*3600) < 0 FOR 5m
119. Environment
121. Apache nginx Django PostgreSQL MySQL MongoDB CouchDB redis Varnish etcd Kubernetes Consul collectd HAProxy statsd graphite InfluxDB SNMP
122. Apache nginx Django PostgreSQL MySQL MongoDB CouchDB redis Varnish etcd Kubernetes Consul collectd HAProxy statsd graphite InfluxDB SNMP
123. node_exporter
124. node_exporter cAdvisor
125. System Insight
126. System Insight ❖ load
127. System Insight ❖ load ❖ procs
128. System Insight ❖ load ❖ procs ❖ memory
129. System Insight ❖ load ❖ procs ❖ memory ❖ network
130. System Insight ❖ load ❖ procs ❖ memory ❖ network ❖ disk
131. System Insight ❖ load ❖ procs ❖ memory ❖ network ❖ disk ❖ I/O
132. mtail
133. mtail ❖ follow (log) files
134. mtail ❖ follow (log) files ❖ extract metrics using regex
135. mtail ❖ follow (log) files ❖ extract metrics using regex ❖ can be better than direct
136. Moar
137. Moar ❖ Edges: web servers/HAProxy
138. Moar ❖ Edges: web servers/HAProxy ❖ black box
139. Moar ❖ Edges: web servers/HAProxy ❖ black box ❖ databases
140. Moar ❖ Edges: web servers/HAProxy ❖ black box ❖ databases ❖ network
141. So Far
142. So Far ❖ system stats
143. So Far ❖ system stats ❖ outside look
144. So Far ❖ system stats ❖ outside look ❖ 3rd party components
145. Code
146. cat-or.not
147. cat-or.not ❖ HTTP service
148. cat-or.not ❖ HTTP service ❖ upload picture
149. cat-or.not ❖ HTTP service ❖ upload picture ❖ meow!/nope meow!
150. from flask import Flask, g, request from cat_or_not import is_cat app = Flask(__name__) @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) return ("meow!" if is_cat(request.files["pic"]) else "nope!") if __name__ == "__main__": app.run()
151. from flask import Flask, g, request from cat_or_not import is_cat app = Flask(__name__) @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) return ("meow!" if is_cat(request.files["pic"]) else "nope!") if __name__ == "__main__": app.run()
152. from flask import Flask, g, request from cat_or_not import is_cat app = Flask(__name__) @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) return ("meow!" if is_cat(request.files["pic"]) else "nope!") if __name__ == "__main__": app.run()
153. pip install prometheus_client
154. from prometheus_client import \ start_http_server # … if __name__ == "__main__": start_http_server(8000) app.run()
155. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
156. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
157. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
158. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
159. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
160. process_virtual_memory_bytes 156393472.0 process_resident_memory_bytes 20480000.0 process_start_time_seconds 1460214325.21 process_cpu_seconds_total 0.169999999998 process_open_fds 8.0 process_max_fds 1024.0
162. from prometheus_client import \ Histogram, Gauge REQUEST_TIME = Histogram( "cat_or_not_request_seconds", "Time spent in HTTP requests.")
163. from prometheus_client import \ Histogram, Gauge REQUEST_TIME = Histogram( "cat_or_not_request_seconds", "Time spent in HTTP requests.") ANALYZE_TIME = Histogram( "cat_or_not_analyze_seconds", "Time spent analyzing pictures.")
164. from prometheus_client import \ Histogram, Gauge REQUEST_TIME = Histogram( "cat_or_not_request_seconds", "Time spent in HTTP requests.") ANALYZE_TIME = Histogram( "cat_or_not_analyze_seconds", "Time spent analyzing pictures.") IN_PROGRESS = Gauge( "cat_or_not_in_progress_requests", "Number of requests in progress.")
165. @IN_PROGRESS.track_inprogress() @REQUEST_TIME.time() @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) with ANALYZE_TIME.time(): result = is_cat( request.files["pic"].stream) return "meow!" if result else "nope!"
166. @IN_PROGRESS.track_inprogress() @REQUEST_TIME.time() @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) with ANALYZE_TIME.time(): result = is_cat( request.files["pic"].stream) return "meow!" if result else "nope!"
167. AUTH_TIME = Histogram("auth_seconds", "Time spent authenticating.") AUTH_ERRS = Counter("auth_errors_total", "Errors while authing.") AUTH_WRONG_CREDS = Counter("auth_wrong_creds_total", "Wrong credentials.") class Auth: # ... @AUTH_TIME.time() def auth(self, request): while True: try: return self._auth(request) except WrongCredsError: AUTH_WRONG_CREDS.inc() raise except Exception: AUTH_ERRS.inc()
168. AUTH_TIME = Histogram("auth_seconds", "Time spent authenticating.") AUTH_ERRS = Counter("auth_errors_total", "Errors while authing.") AUTH_WRONG_CREDS = Counter("auth_wrong_creds_total", "Wrong credentials.") class Auth: # ... @AUTH_TIME.time() def auth(self, request): while True: try: return self._auth(request) except WrongCredsError: AUTH_WRONG_CREDS.inc() raise except Exception: AUTH_ERRS.inc()
169. AUTH_TIME = Histogram("auth_seconds", "Time spent authenticating.") AUTH_ERRS = Counter("auth_errors_total", "Errors while authing.") AUTH_WRONG_CREDS = Counter("auth_wrong_creds_total", "Wrong credentials.") class Auth: # ... @AUTH_TIME.time() def auth(self, request): while True: try: return self._auth(request) except WrongCredsError: AUTH_WRONG_CREDS.inc() raise except Exception: AUTH_ERRS.inc()
170. AUTH_TIME = Histogram("auth_seconds", "Time spent authenticating.") AUTH_ERRS = Counter("auth_errors_total", "Errors while authing.") AUTH_WRONG_CREDS = Counter("auth_wrong_creds_total", "Wrong credentials.") class Auth: # ... @AUTH_TIME.time() def auth(self, request): while True: try: return self._auth(request) except WrongCredsError: AUTH_WRONG_CREDS.inc() raise except Exception: AUTH_ERRS.inc()
171. @app.route("/analyze", methods=["POST"]) def analyze(): g.auth.check(request) with ANALYZE_TIME.time(): result = is_cat( request.files["pic"].stream) return "meow!" if result else "nope!"
172. pip install prometheus_async
173. Wrapper from prometheus_async.aio import time @time(REQUEST_TIME) async def view(request): # ...
174. Goodies
175. Goodies ❖ aiohttp-based metrics export
176. Goodies ❖ aiohttp-based metrics export ❖ also in thread!
177. Goodies ❖ aiohttp-based metrics export ❖ also in thread! ❖ Consul Agent integration
178. Wrap Up
179. Wrap Up
180. Wrap Up ✓
181. Wrap Up ✓ ✓
182. Wrap Up ✓ ✓ ✓
183. ox.cx/p @hynek vrmd.de