Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lightning Talk Prometheus

Lightning Talk Prometheus

Lightning talk I gave internally at Skyscrapers about Prometheus.

Mattias Gees

March 01, 2018
Tweet

More Decks by Mattias Gees

Other Decks in Programming

Transcript

  1. /metrics # HELP go_gc_duration_seconds A summary of the GC invocation

    durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
  2. Exporters • Converts application metrics to prometheus formatted Metrics •

    3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
  3. Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti

    UniFi exporter • RabbitMQ exporter • Apache exporter
  4. Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS

    ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
  5. Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics

    scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
  6. Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels:

    severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
  7. Alertmanager Route route: receiver: opsgenie group_by: - job routes: -

    receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
  8. Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true

    api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'