Setting up Monitoring for Kubernetes Prathamesh Sonpatki Last9.io 1 

2 

Monitoring is crucial - Prometheus is King of Kubernetes Monitoring 🔥 - Kube-Prometheus-Stack - Container level monitoring via cAdvisor - Cluster level monitoring via Kube State Metrics 3 

Story of optimizing cAdvisor metrics 4 

cAdvisor - https://github.com/google/cadvisor - Analyzes resource usage and performance of running containers - Metrics for specific hardware and software components such as disk, CPU, memory, network, process, TCP, and much more. 5 

kubelet-hosted cAdvisor - https://github.com/prometheus-community/helm-charts/blob/main/chart s/kube-prometheus-stack - Standard deployment using Helm 6 

Everything is Good 7 

Everything is Good.. Right? 8 

But wait… - 9 

But wait… - 91K samples per minute 😥 😨 - 21 nodes, 600 pods, 125 containers - ~ 4B per month only for cAdvisor metrics - 80% of metrics are unused! 10 

The ratio of metric samples scanned/evaluated over those ingested is never 1:1. 11 

Let’s take an action - We don’t need all of the - Accelerator - Disk - diskIO - Network - .. - TCP - … - Let’s disable these metrics with `/-disable_metrics` 12 

But wait… - We don’t need all of the - Accelerator - Disk - diskIO - Network - .. - TCP - … - Let’s disable these metrics with `/-disable_metrics` 13 

Alternate Strategy - Disable this kubelet-hosted cAdvisor. 14 

Alternate Strategy - Use alternate helm chart - 15 https://github.com/ckotzbauer/helm-charts/tree/main/charts/cadvisor 

Alternate Strategy - 65% of savings in samples collected! 16 https://github.com/ckotzbauer/helm-charts/tree/main/charts/cadvisor 

Prathamesh Sonpatki Last9.io Srestories.dev o11y.wiki 17