Slide 1

Slide 1 text

Kubernetes' Metrics API Past, Present and Future of the Prometheus Adapter

Slide 2

Slide 2 text

whoami - Sergiusz Urbaniak Software Engineer at CoreOS/RedHat ● Worked on ○ Mesos (Kubernetes scheduler integration) ○ rkt/rktnetes ○ minikube Linux ○ Kubernetes CRI, OCI ○ Tectonic Installer ● Now working on all things Prometheus in Kubernetes

Slide 3

Slide 3 text

whoami - Matthias Loibl

Slide 4

Slide 4 text

whoami - Matthias Loibl Software Engineer at RedHat Prev at Loodse & JustWatch All about: Go, Prometheus, Kubernetes, Drone and much more Creator of gopass Organizer of the Berlin Prometheus MeetUp

Slide 5

Slide 5 text

Agenda ● History ● Current state ● Resource metrics ● Custom metrics ● HPA example ● Prometheus Adapter Future ● How to get involved

Slide 6

Slide 6 text

History: Metrics in Kubernetes ~v1.1 Scheduling Decision Metrics Node Pod CPU Based on currently measured node capacity Scheduled Resource Requests Memory

Slide 7

Slide 7 text

Architecture - ~ Kubernetes v1.1 https://kubernetes.io/blog/2015/05/resource-usage-monitoring-kubernetes/

Slide 8

Slide 8 text

Problems with Heapster ● Push based model ● Sink plugins results in a vendor dump ● Opinionated tooling ● No abstraction ● What if I want to use Prometheus instead? https://brancz.com/2018/01/05/prometheus-vs-heapster-vs-kubernetes-metrics-apis/

Slide 9

Slide 9 text

Wait .... What about Prometheus ???

Slide 10

Slide 10 text

Goals ● Decouple scheduling and scaling decisions from Heapster ● Introduce an abstract API scheme ● Let different vendors implement that scheme ○ Prometheus ○ DataDog ○ Cloud Providers: Azure, AWS, GCP, etc

Slide 11

Slide 11 text

Meet the Metrics APIs ● Resource Metrics API ○ CPU & memory per node & pod ○ Opinionated ○ Concrete, pre-defined metrics ● Custom Metrics API ○ Relate to in-cluster objects, pods, etc. ○ Meta-model, abstract metrics ● External Metrics API ○ Relate to external non-cluster objects, i.e. Load Balancers, Infra components ○ Meta-model, abstract metrics

Slide 12

Slide 12 text

Architecture

Slide 13

Slide 13 text

Implementations API metrics-server k8s- prometheus- adapter azure- metrics- adapter custom- metrics- stackdriver- adapter metrics. k8s.io X X custom.metrics. k8s.io X X X external.metrics. k8s.io (WIP) X X

Slide 14

Slide 14 text

Resource Metrics

Slide 15

Slide 15 text

Discovering Metrics APIs $ kubectl get apiservices v1beta1.metrics.k8s.io NAME SERVICE AVAILABLE AGE v1beta1.metrics.k8s.io monitoring/prometheus-adapter True 4h

Slide 16

Slide 16 text

Discovering Metrics APIs $ kubectl api-resources | grep metrics.k8s.io NAME SHORTNAMES APIGROUP NAMESPACED KIND nodes metrics.k8s.io false NodeMetrics pods metrics.k8s.io true PodMetrics

Slide 17

Slide 17 text

Resource Metrics API /apis/metrics.k8s.io $ kubectl get pods.metrics.k8s.io grafana-5c5d49b4c4-m25pc -o yaml kind: PodMetrics apiVersion: metrics.k8s.io/v1beta1 metadata: creationTimestamp: 2018-11-13T12:44:37Z name: grafana-5c5d49b4c4-m25pc namespace: monitoring containers: - name: grafana usage: cpu: 10m memory: 44444Ki - name: "" usage: cpu: 14m memory: 45780Ki timestamp: 2018-11-13T12:44:37Z window: 1m0s

Slide 18

Slide 18 text

Example $ kubectl top pod NAME CPU(cores) MEMORY(bytes) alertmanager-main-0 15m 27Mi alertmanager-main-1 9m 26Mi alertmanager-main-2 11m 26Mi grafana-5c5d49b4c4-m25pc 29m 76Mi kube-state-metrics-668bd94f74-xpfjf 9m 107Mi node-exporter-rfmjb 12m 55Mi prometheus-adapter-6fbffbd98c-k6gc4 4m 26Mi prometheus-k8s-0 41m 307Mi prometheus-k8s-1 64m 315Mi prometheus-operator-88fcf6d95-4phlm 34m 54Mi

Slide 19

Slide 19 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: cpu: containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

Slide 20

Slide 20 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: cpu: containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>) nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)

Slide 21

Slide 21 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: cpu: containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>) nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>) resources: overrides: node: resource: node namespace: resource: namespace pod_name: resource: pod

Slide 22

Slide 22 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: cpu: containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>) nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>) resources: overrides: node: resource: node namespace: resource: namespace pod_name: resource: pod containerLabel: container_name

Slide 23

Slide 23 text

Association $ curl -s 'http://localhost:9090/api/v1/series?match[]=container_cpu_usage_seconds_total' \ | jq . { "status": "success", "data": [ { "__name__": "container_cpu_usage_seconds_total", "container_name": "POD", ===> Associates with container "cpu": "total", "endpoint": "https-metrics", "id": "/kubepods/besteffort/pod446c9ff5-2781-...", "image": "k8s.gcr.io/pause:3.1", "instance": "192.168.122.2:10250", "job": "kubelet", "name": "k8s_POD_kube-proxy-kh87t...", "namespace": "kube-system", ===> Associates with namespace[.core] "node": "minikube", ===> Associates with node[.core] "pod_name": "kube-proxy-kh87t", ===> Associates with pod[.core] "service": "kubelet" },

Slide 24

Slide 24 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: memory: containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>) nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>) ...

Slide 25

Slide 25 text

Conversion Kubernetes <-> Prometheus $ kubectl get configmap adapter-config -o jsonpath='{.data.config\.yaml}' resourceRules: cpu: containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>) nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>) ... memory: ... window: 1m

Slide 26

Slide 26 text

Custom Metrics

Slide 27

Slide 27 text

/apis/custom-metrics/v1beta1 Scale based on different Kubernetes types: ● Pods ● Services ● Jobs ● ... Custom Metrics API

Slide 28

Slide 28 text

Custom Metrics API Service $ kubectl get apiservices v1beta1.custom.metrics.k8s.io NAME SERVICE AVAILABLE AGE v1beta1.custom.metrics.k8s.io custom-metrics/custom-metrics-apiserver True 3h

Slide 29

Slide 29 text

Discovering registered custom metrics $ kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1' | jq . { "kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "custom.metrics.k8s.io/v1beta1", "resources": [ ... { "name": "pods/http_requests_per_second", "singularName": "", "namespaced": true, "kind": "MetricValueList", "verbs": [ "get" ] } ] }

Slide 30

Slide 30 text

Conversion Kubernetes <-> Prometheus ● Discovery of metrics in Prometheus ● Association with Kubernetes resources ● Naming of metrics ● Querying of metric values in Prometheus

Slide 31

Slide 31 text

Discovery $ kubectl -n custom-metrics get configmaps adapter-config -o yaml apiVersion: v1 kind: ConfigMap data: config.yaml: | rules: # 1. Discovery: match all http_requests_total metrics, having a (k8s) namespace set - seriesQuery: 'http_requests_total{namespace!=""}'

Slide 32

Slide 32 text

Discovery $ curl -s 'http://localhost:9090/api/v1/series?match[]=http_requests_total\{namespace!=""\}' \ | jq . { "status": "success", "data": [ { "__name__": "http_requests_total", "endpoint": "http", "instance": "172.17.0.14:8080", "job": "podinfo", "namespace": "default", "pod": "podinfo-67c9fd95d-fqk4g", "service": "podinfo", "status": "200" }, ...

Slide 33

Slide 33 text

Association $ kubectl -n custom-metrics get configmaps adapter-config -o yaml apiVersion: v1 kind: ConfigMap data: config.yaml: | rules: # 1. Discovery: match all http_requests_total metrics, having a (k8s) namespace set - seriesQuery: 'http_requests_total{namespace!=""}' # 2. Association: Match k8s resources to discovered metrics resources: template: <<.Resource>>

Slide 34

Slide 34 text

Association $ curl -s 'http://localhost:9090/api/v1/series?match[]=http_requests_total\{namespace!=""\}' \ | jq . { "status": "success", "data": [ { "__name__": "http_requests_total", "endpoint": "http", "instance": "172.17.0.14:8080", "job": "podinfo", ===> Associates with jobs.batch "namespace": "default", ===> Associates with namespace[.core] "pod": "podinfo-67c9fd95d-fqk4g", ===> Associates with pod[.core] "service": "podinfo", ===> Associates with service[.core] "status": "200" }, ...

Slide 35

Slide 35 text

Naming $ kubectl -n custom-metrics get configmaps adapter-config -o yaml apiVersion: v1 kind: ConfigMap data: config.yaml: | rules: # 1. Discovery: match all http_requests_total metrics, having a (k8s) namespace set - seriesQuery: 'http_requests_total{namespace!=""}' # 2. Association: Match k8s resources to discovered metrics resources: template: <<.Resource>> # 3. Naming: Convert `http_requests_total` to `http_requests_total_per_second` name: matches: "^(.*)_total" as: "${1}_per_second"

Slide 36

Slide 36 text

Naming $ curl -s 'http://localhost:9090/api/v1/series?match[]=http_requests_total\{namespace!=""\}' \ | jq . { "status": "success", "data": [ { "__name__": "http_requests_total", ===> Renames to http_requests_total_seconds "endpoint": "http", "instance": "172.17.0.14:8080", "job": "podinfo", "namespace": "default", "pod": "podinfo-67c9fd95d-fqk4g", "service": "podinfo", "status": "200" }, ...

Slide 37

Slide 37 text

Querying $ kubectl -n custom-metrics get configmaps adapter-config -o yaml apiVersion: v1 kind: ConfigMap data: config.yaml: | rules: # 1. Discovery: match all http_requests_total metrics, having a (k8s) namespace set - seriesQuery: 'http_requests_total{namespace!=""}' # 2. Association: Match k8s resources to discovered metrics resources: template: <<.Resource>> # 3. Naming: Convert `http_requests_total` to `http_requests_total_per_second` name: matches: "^(.*)_total" as: "${1}_per_second" # 4. Querying: Execute a per-second rate query metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Slide 38

Slide 38 text

Querying $ kubectl get --raw \ '/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/podinfo-67c9fd95d-fqk4g/http_requests_per_second' \ | jq . { "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/podinfo-67c9fd95d-fqk4g/http_requests_per_second" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-67c9fd95d-fqk4g", "apiVersion": "/v1" }, "metricName": "http_requests_per_second", "timestamp": "2019-02-02T11:52:35Z", "value": "5133m" } ] }

Slide 39

Slide 39 text

Configurable queries

Slide 40

Slide 40 text

Prometheus Adapter - Future ● Merge external metrics support ● Move to kubernetes organization ● Tackle complex configuration ● Tackle scalability issues due to discovery overhead ● Planned refactoring potentially based on CRDs

Slide 41

Slide 41 text

Sample HPA configuration kind: HorizontalPodAutoscaler apiVersion: autoscaling/v2beta1 metadata: name: podinfo namespace: default spec: minReplicas: 1 maxReplicas: 10 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: podinfo metrics: # use a "Pods" metric, which takes the average of the # given metric across all pods controlled by the autoscaling target - type: Pods pods: metricName: http_requests_per_second # target 1000 milli-requests per second = 1 req/second, targetAverageValue: 1000m

Slide 42

Slide 42 text

User - QuickStart ● kube-prometheus ships with Prometheus Adapter and resource metrics enabled. ● Upstream Prometheus adapter includes custom metrics sample deployment. In both cases: $ kubectl apply -f manifests/

Slide 43

Slide 43 text

How to get involved?

Slide 44

Slide 44 text

Developer - Kubernetes SIGs Community activity is organized into Special Interest Groups (SIGs) The topics in this talk are related to SIG-instrumentation mailing list and bi-weekly meetings SIG-autoscaling mailing list and bi-weekly meetings https://github.com/kubernetes/community/blob/master/sig-list.md

Slide 45

Slide 45 text

Thank you! Questions?

Slide 46

Slide 46 text

Deleted Scenes Slides

Slide 47

Slide 47 text

Sample load with 5 requests/second $ kubectl -n default get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 966m/1 1 10 6 9m52s $ kubectl -n default get pod NAME READY STATUS RESTARTS AGE podinfo-67c9fd95d-4x5gc 1/1 Running 0 6m54s podinfo-67c9fd95d-bfv64 1/1 Running 0 7m39s podinfo-67c9fd95d-kqx99 1/1 Running 0 7m9s podinfo-67c9fd95d-pg5h5 1/1 Running 0 5m39s podinfo-67c9fd95d-rm4wc 1/1 Running 0 7m54s podinfo-67c9fd95d-ttqpn 1/1 Running 0 9m55s

Slide 48

Slide 48 text

Custom metrics vs. External metrics ? type MetricValue struct { metav1.TypeMeta # Kubernetes resource reference DescribedObject ObjectReference # Metric reference Metric MetricIdentifier Timestamp metav1.Time WindowSeconds *int64 Value resource.Quantity } type ExternalMetricValue struct { metav1.TypeMeta # External metric reference MetricName string `json:"metricName"` MetricLabels map[string]string Timestamp metav1.Time WindowSeconds *int64 Value resource.Quantity } kubernetes-1.13.2/pkg/apis/custom_metrics/types.go kubernetes-1.13.2/pkg/apis/external_metrics/types.go