Share
Embed
Anomaly Detection using Prometheus
I discuss some techniques for doing anomaly detection using prometheus queries and recording rules.
Andrew Newdigate
June 05, 2019
Transcript
32 slides
-
@suprememoocow Why is anomaly detection useful? • Diagnosing incidents /
Improving MTTD • Detecting application performance regressions • Abuse • Security issues -
@suprememoocow Getting Started - example metric http_requests_total{ job="apiserver", method="GET", controller="ProjectsController",
status_code="200", environment="prod" } -
@suprememoocow Aggregating your metrics # Too much aggregation! - record:
job:http_requests:rate5m expr: sum(rate(http_requests_total[5m])) # No labels # --> job:http_requests:rate5m{} 12389 -
@suprememoocow Aggregating your metrics - too little - record: instance_method_controller:http_requests:rate5m
expr: sum without(status_code) (rate(http_requests_total[5m])) # --> instance_method_controller:http_requests:rate5m{ # instance="api-01:8080", # job="apiserver", # method="GET", # controller="ProjectsController", # environment="prod"} 21321 # --> instance_method_controller:http_requests:rate5m{ # instance="api-01:8080", # job="apiserver", # method="POST", # controller="ProjectsController", # environment="prod"} 2133 # ... 400 more series ... -
@suprememoocow Aggregating your metrics - just right - record: job:http_requests:rate5m
expr: sum on(job, environment) (rate(http_requests_total[5m])) # --> job:http_requests:rate5m{job="apiserver", environment="prod"} 21321 # --> job:http_requests:rate5m{job="gitserver", environment="prod"} 2212 # --> job:http_requests:rate5m{job="webserver", environment="prod"} 53091 -
@suprememoocow Anomaly detection using z-scores # Long-term average value for
the series - record: job:http_requests:rate5m:avg_over_time_1w expr: avg_over_time(job:http_requests:rate5m[1w]) # Long-term standard deviation for the series - record: job:http_requests:rate5m:stddev_over_time_1w expr: stddev_over_time(job:http_requests:rate5m[1w]) -
@suprememoocow Anomaly detection using z-scores # Z-Score for aggregation (
job:http_requests:rate5m - job:http_requests:rate5m:avg_over_time_1w ) / job:http_requests:rate5m:stddev_over_time_1w -
@suprememoocow Visualizing z-scores GitLab.com Pages service RPS over 48 hours,
with ±3 z-score region in green -
@suprememoocow Visualizing z-scores GitLab.com Pages service RPS over 48 hours,
with ±3 z-score region in green job:http_requests:rate5m ZSCORE: 0 job:http_requests:rate5m:avg_over_time_1w ZSCORE: 3 job:http_requests:rate5m:avg_over_time_1w + 3 * job:http_requests:rate5m:stddev_over_time_1w ZSCORE: -3 job:http_requests:rate5m:avg_over_time_1w - 3 * job:http_requests:rate5m:stddev_over_time_1w -
@suprememoocow Basic normal distribution test ( max_over_time(job:http_requests:rate5m[1w]) - avg_over_time(job:http_requests:rate5m[1w]) )
/ stddev_over_time(job:http_requests:rate5m[1w]) # --> {job="apiserver", environment="prod"} 4.01 # --> {job="gitserver", environment="prod"} 3.96 # --> {job="webserver", environment="prod"} 2.96 ( min_over_time(job:http_requests:rate5m[1w]) - avg_over_time(job:http_requests:rate5m[1w]) ) / stddev_over_time(job:http_requests:rate5m[1w]) # --> {job="apiserver", environment="prod"} -3.8 # --> {job="gitserver", environment="prod"} -4.1 # --> {job="webserver", environment="prod"} -3.2 -
@suprememoocow Leveraging seasonality with offset Gitaly service RPS (blue), with
7-day rolling average (yellow), over two week period -
@suprememoocow Seasonality with Prometheus, v1 - record: job:http_requests:rate5m_prediction expr: >
job:http_requests:rate5m offset 1w # Value from last period + job:http_requests:rate5m:avg_over_time_1w # Add 1w growth trend - job:http_requests:rate5m:avg_over_time_1w offset 1w -
@suprememoocow Seasonality with Prometheus, v2 - record: job:http_requests:rate5m_prediction expr: >
avg_over_time(job:http_requests:rate5m[4h] offset 166h) # Rounded value from last period + job:http_requests:rate5m:avg_over_time_1w # Add 1w growth trend - job:http_requests:rate5m:avg_over_time_1w offset 1w -
@suprememoocow Seasonal predictions Gitaly service RPS (yellow) vs prediction (dotted
red), over two weeks. -
@suprememoocow Avoiding last weeks anomalies... Gitaly service RPS over two
week period, from Monday 29 April 2019 1 May 2019, International Labour Day: no work today... -
@suprememoocow Avoiding last weeks anomalies... Gitaly service RPS (yellow), with
prediction (blue), over two week period, from Monday 29 April 2019 -
@suprememoocow Use multiple predictions... avg_over_time(job:http_requests:rate5m[4h] offset 166h) # 1 week
- 2 hours + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 1w avg_over_time(job:http_requests:rate5m[4h] offset 334h) # 2 weeks - 2 hours + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 2w avg_over_time(job:http_requests:rate5m[4h] offset 502h) # 3 weeks - 2 hours + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 3w -
@suprememoocow Three predictions from three Wednesdays 3 predictions vs actual
Gitaly RPS, Wednesday 8 May (1 week following Intl Labour Day) -
@suprememoocow - record: job:http_requests:rate5m_prediction expr: > quantile(0.5, label_replace( avg_over_time(job:http_requests:rate5m[4h] offset
166h) + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 1w , "offset", "1w", "", "") or label_replace( avg_over_time(job:http_requests:rate5m[4h] offset 334h) + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 2w , "offset", "2w", "", "") or label_replace( avg_over_time(job:http_requests:rate5m[4h] offset 502h) + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 3w , "offset", "3w", "", "") ) without (offset) -
@suprememoocow One prediction from three weeks… Median predictions vs actual
Gitaly RPS, Wednesday 8 May (1 week following Intl Labour Day) -
@suprememoocow Visualising seasonal anomalies Predicted normal range ± 1.5σ for
Gitaly Service https://dashboards.gitlab.com/d/26q8nTzZz/service-platform-metrics?var-type=gitaly&orgId=1 -
@suprememoocow Alerting - alert: RequestRateOutsideNormalRange expr: > abs( ( job:http_requests:rate5m
- job:http_requests:rate5m_prediction ) / job:http_requests:rate5m:stddev_over_time_1w ) > 2 for: 10m labels: severity: warning annotations: summary: Requests for job {{ $labels.job }} are outside of expected operating parameters -
@suprememoocow Using z-scores for anomaly triage GitLab.com Triage with RPS
(left) and RPS z-scores (right) https://dashboards.gitlab.com/d/XufqmIGWk/platform-triage -
@suprememoocow Conclusion • Anomaly detection is possible in Prometheus •
The right level of aggregation is the key to anomaly detection • Z-scores will only work with normally distributed data • Seasonal metrics are good for anomaly detection -
@suprememoocow Thank You! • GitLab Public Grafana: http://dashboards.gitlab.com • GitLab
Prometheus Recording Rules for Anomaly Detection https://gitlab.com/gitlab-com/runbooks • #talk-andrew-newdigate • All the code snippets: https://gitlab.com/snippets/1863717