Prometheus & Time-Series monitoring

Talk given at Chadevs Meetup - http://www.meetup.com/chadevs/events/231946415/
https://www.youtube.com/watch?v=lHTCeRKq0lc

Chattanooga, TN

1. Prometheus & Time-Series Monitoring by Micah Hausler
2. some background
3. four golden signals latency traffic errors saturation
4. latency
5. traffic
6. errors
7. saturation
8. a typical monitoring setup your server monitoring system agent your server agent your server agent
9. there are some drawbacks metric data in multiple formats additional processes running on hosts report failure is not a signal
10. what is an alternative?
11. Prometheus architecture server Prometheus :443/metrics server server :80/metrics :8080/metrics
12. a single metric $ curl http://webserver:80/metrics http_requests 37

13. time-series arena (Beyer, Jones, and Petoff, p111)
14. play out the math
15. a single metric 1 data point = 24 bytes 1 min scrape interval 12 hours of data
16. a single metric 1 data point = 24 bytes 1 min scrape interval 12 hours of data 12h * 60m/h * 24b = 16.88 kb
17. a single metric 1 data point = 24 bytes 1 min scrape interval 12 hours of data 12h * 60m/h * 24b = 16.88 kb 17 * 2^30 = 17gb
18. (17 * 2^30) / (12*60*24) = 1,056,343 17gb / 16kb = 1 million time-series
19. (Beyer, Jones, and Petoff, p111)
20. (Beyer, Jones, and Petoff, p111)
22. multi-dimension metrics $ curl http://webserver.host{0...4}:80/metrics {var=http_requests,job=webserver,instance=host0:80,service=web,zone=us-west} 10 {var=http_requests,job=webserver,instance=host1:80,service=web,zone=us-west} 9 {var=http_requests,job=webserver,instance=host2:80,service=web,zone=us-west} 11 {var=http_requests,job=webserver,instance=host3:80,service=web,zone=us-west} 0 {var=http_requests,job=webserver,instance=host4:80,service=web,zone=us-west} 10 

23. each query is a vector
24. metric types • counter - a number that only increases • gauge - dynamic changing number • histogram - samples observations, counts in buckets - useful for things like Apdex • summary - samples observations, calculates quantiles over a sliding time window
25. histogram # HELP state_store_seconds Registry write latency in seconds # TYPE state_store_seconds histogram state_store_seconds_bucket{le="0.005"} 0 state_store_seconds_bucket{le="0.01"} 0 state_store_seconds_bucket{le="0.025"} 0 state_store_seconds_bucket{le="0.05"} 0 state_store_seconds_bucket{le="0.1"} 0 state_store_seconds_bucket{le="0.25"} 0 state_store_seconds_bucket{le="0.5"} 0 state_store_seconds_bucket{le="1"} 0 state_store_seconds_bucket{le="2.5"} 0 state_store_seconds_bucket{le="5"} 0 state_store_seconds_bucket{le="10"} 0 state_store_seconds_bucket{le="+Inf"} 0 state_store_seconds_sum 0 state_store_seconds_count 0
26. summary # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0.00019505300000000002 go_gc_duration_seconds{quantile="0.25"} 0.000299606 go_gc_duration_seconds{quantile="0.5"} 0.00039634 go_gc_duration_seconds{quantile="0.75"} 0.000436071 go_gc_duration_seconds{quantile="1"} 0.000758476 go_gc_duration_seconds_sum 240.533560116 go_gc_duration_seconds_count 660994
27. demo time.
28. high-availability & federation
29. questions?
30. links. • https://prometheus.io/ • https://landing.google.com/sre/book.html • https://github.com/micahhausler/chadev-prometheus-demo
32. Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media, 2016. Print. bibliography
33. Micah Hausler @micahhausler micah@skuid.com thanks! skuid is hiring • Site Reliability Engineers • DevOps Engineers • NodeJS Backend Engineers • Frontend Engineers • Quality Engineers • System Engineers