Monitoring with Prometheus & Thanos

Agenda Why Monitoring Monitoring Stack Why Thanos Thanos Components Managing Overload

Why SRE's Care About Monitoring?

How we monitor Monitoring Stack Prometheus, Operator: we k8s Thanos, Grafana Legacy Graphite & StatsD Prometheus on EC2

Deployment Prometheus operator helm chart: Prometheus, Alertmanager, Grafana servers Host node_exporter kube-state-metrics Default set of Alerts and Dashboards The helm chart IS NOT kube-prometheus

Configuration kube-prometheus/values.yaml prometheus: ingress: ... prometheusSpec: replicas: 2 retention: 48h storageSpec: ... alertmanager: ingress: ... alertmanagerSpec: ... config: route: routes: ... receivers: ...

Scraping targets kube-prometheus/values.yaml prometheus: prometheusSpec: serviceMonitorNamespaceSelector: any: true # scrape any ServiceMonitor which has metadata labels `prometheus=my-prometheus` serviceMonitorSelector: matchExpressions: - key: prometheus operator: In values: - my-prometheus ...

Scraping targets kubectl get servicemonitor random-cvs-k8s-sm -o yaml kind: ServiceMonitor metadata: labels: prometheus: my-prometheus name: random-svc-k8s-sm spec: namespaceSelector: matchNames: - tribe-name selector: matchLabels: app: random-svc-k8s

Namespaced Prometheus HA Prometheus pods per namespace Only for product engineering teams Reduce coupling between teams

Provisioning cluster-namespaces/charts/helmfile.yaml - name: {{ requiredEnv "NS_NAME" }}-monitoring namespace: monitoring chart: ./ns-monitoring version: 0.1.0 values: - ../namespaces/{{ requiredEnv "NS_NAME" }}/{{ requiredEnv "ENV_NAME" }}/values.yaml - ns-monitoring/values-{{ requiredEnv "ENV_NAME" }}.yaml.gotmpl

Configuration cluster-namespaces/charts/ns-monitoring/values-live.yaml.gotmpl prometheusSpec: matchExpressions: - key: app.kubernetes.io/instance operator: In values: - {{ requiredEnv "NS_NAME" }}-init ruleNamespaceSelector: matchExpressions: - key: app.kubernetes.io/instance operator: In values: - {{ requiredEnv "NS_NAME" }}-init serviceMonitorSelector: matchExpressions: ...

Listing Running Prometheus kgp -l app=prometheus -n monitoring | awk '{print $1}' prometheus-consumer-core-monitoring-prometheus-0 prometheus-consumer-core-monitoring-prometheus-1 prometheus-conversions-monitoring-prometheus-0 prometheus-conversions-monitoring-prometheus-1 prometheus-crm-data-monitoring-prometheus-0 prometheus-crm-data-monitoring-prometheus-1 prometheus-discovery-monitoring-prometheus-0 prometheus-discovery-monitoring-prometheus-1 prometheus-payments-monitoring-prometheus-0 prometheus-payments-monitoring-prometheus-1 prometheus-platform-monitoring-prometheus-0 prometheus-platform-monitoring-prometheus-1 ...

http://grafana.live-k8s.hellofresh.io/d/prometheus/prometheus? orgId=1&refresh=1m

Limitations Alerting on longer periods are not possible, retention is too low

Wrapping up HA Prometheus per namespace are automatically provisioned, altho they all run at monitoring namespace. Platform on-call team are paged if smth goes wrong. Shared responsibility between Cloud Runtime and SRE squads. Namespaced Prometheus pods have 12h of retention.

Routing Alerts charts/external/kube-prometheus/values-live-eks.yaml alertmanager: config: route: receiver: 'default-receiver' routes: - match: tribe: platform receiver: 'platform-opsgenie' # we need to do the step above for every tribe - match: tribe: page receiver: 'opsgenie'

Why Thanos? Prometheus federation is hard Global View Long term storage (LTS) Downsampling

Prometheus Federation Global Prometheus that pulls aggregated metrics from datacenter Prometheus servers.

No content

Prometheus Federation prometheus.yaml scrape_configs: - job_name: 'federate' honor_labels: true metrics_path: '/federate' params: 'match[]': - '{__name__=~"job:.*"}' static_configs: - targets: - 'kube-prometheus:9090' - 'platform-prometheus:9090' - 'consumer-core-prometheus:9090'

Thanos Architecture

Thanos Querier

Thanos Sidecar

Thanos Store

Thanos Compactor

Thanos Ruler

HA Ruler Two different deployments rule: replicas: 1 persistence: enabled: true size: 10Gi storageClass: default ruleReplica: replicas: 1 ...

Deployment Models Federation, Global View Monitoring Cluster

Federation with Thanos

No content

Managing Overload High cardinality metrics are likely to be the primary cause of the performance problems.

TSDB (Time Series Database) Briefly

Detecting a Problem prometheus_tsdb_head_series metric tells the total number of time series, check if it has increased

Detecting a Problem prometheus_tsdb_head_samples_appended_total tells the ingestion rate

Detecting a Problem Find expensive metrics topk(10, count by (name, job) ({name=~".+"}))

Reduce Load Dropping labels kubectl edit servicemonitors service-monitor-name endpoints: ... metricRelabelings: - action: labeldrop regex: (id|path|status)

Reduce Load Dropping metrics endpoints: ... metricRelabelings: - action: drop regex: component_(expensive_metric_foo|expensive_metric_bar) sourceLabels: - __name__

Reduce Load Increase scrape_interval and evaluation_interval On service monitor kubectl edit servicemonitors service-monitor-name endpoints: interval: 45s ... PS: Keep in mind that it’s not practical to increase these beyond 2 minutes

Reduce Load On Prometheus operator globally prometheusSpec: scrapeInterval: 45s evaluationInterval: 45s ...

Sample Limit On service monitor endpoints: sampleLimit: 150000 On thanos store helm chart config containers: - name: thanos-store args: - "--store.grpc.series-sample-limit={{ .Values.store.sampleLimit }}"

Thank you