Generating SLO rules for Prometheus

Generating SLO rules for Prometheus

We want to use SLOs in Prometheus and a lot of the alerting rules and recording rules are quite some repetition, so we can use Jsonnet to generate Prometheus files.

915d80f0d9b6678fad4d1ab36dfc8960?s=128

Matthias Loibl

November 07, 2019
Tweet

Transcript

  1. SLO libsonnet Matthias Loibl [@metalmatze] Software Engineer, Red Hat

  2. SLOs Service Level Objectives

  3. SLIs Service Level Indicators

  4. SLIs rate(http_requests_total{code=~”5..”}[5m])

  5. SLOs rate(http_requests_total{code=~”5..”}[5m]) > 0.01

  6. Multi-Burn-Rate Alerts

  7. alert: ErrorBudgetBurn expr: ( status_class_5xx:http_requests_total:ratio_rate1h{job="prometheus"} > (14.4*0.001000) and status_class_5xx:http_requests_total:ratio_rate5m{job="prometheus"} >

    (14.4*0.001000) ) or ( status_class_5xx:http_requests_total:ratio_rate6h{job="prometheus"} > (6*0.001000) and status_class_5xx:http_requests_total:ratio_rate30m{job="prometheus"} > (6*0.001000) )
  8. record: status_class_5xx:http_requests_total:ratio_rate5m expr: sum(status_class:http_requests_total:rate5m{job="prometheus",status_class="5xx"}) / sum(status_class:http_requests_total:rate5m{job="prometheus"})

  9. record: status_class:http_requests_total:rate5m expr: sum by (status_class) ( label_replace( rate(http_requests_total{job="prometheus"}[5m] ),

    "status_class", "${1}xx", "code", "([0-9])..") )
  10. Jsonnet

  11. SLO-libsonnet github.com/metalmatze/slo-libsonnet

  12. local slo = import '../slo-libsonnet/slo.libsonnet'; { local errorburnrate = slo.errorburn({

    metric: http_requests_total, selectors: ['job="prometheus"'], errorBudget: 1-0.99, }), rules: errorburnrate.recordingrules + errorburnrate.alerts, }
  13. Demo

  14. HTML 43.4% Dart (no JS) 23.0% Go 21.3% Jsonnet 7.1%

  15. Demo

  16. Ideas

  17. Feedback! ...and help :D

  18. Thanks to Björn Rabenstein & Aditya Konarde

  19. promtools.matthiasloibl.com github.com/metalmatze/slo-libsonnet github.com/metalmatze/slo-libsonnet-web