Monitoring Kubernetes at Scale

Kubernetes is changing the game in the data centre, but also in the monitoring and troubleshooting landscape. Static tools and vertically scalable TSDBs are no longer fit for the job. Large-scale dynamic infrastructures require scalable dynamic monitoring.
This talk presents how the Elastic Stack collects logs, metrics, and APM traces from the applications running in Kubernetes:
– Collect application logs, metrics and enhance them with Kubernetes metadata
– Collect application metrics from Prometheus endpoints
– Collect Kubernetes metrics
– Collect application performance traces (APM)
– Autodiscover new pods and monitor them based on their type
– Control the monitoring via Kubernetes annotations
– Use Kibana as a single looking glass to visualize the collected data

Transcript

1. Monica Sarbu Team Lead, Beats and Logstash Monitoring Kubernetes at Scale
2. Monitoring Kubernetes with Elastic Stack
3. Elastic Stack
4. A single Beat for each use case Beats family libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat
5. Growing Beats community !5 50M Cumulative downloads 3 Years
6. !6
7. Monitoring challenges
8. !8 With containers architecture, everything is a moving target
9. !9 High level overview
10. !10
11. !11 Monitor all the things! ✓ Monitor services running in Kubernetes ✓ Collect application logs, metrics, traces
12. Monitoring with Beats
13. !13 Monitor Kubernetes cluster Via the Kubernetes module in Metricbeat Node metrics, from kubelet ✓ pod ✓ node ✓ system ✓ container ✓ volume State metrics, from kube-state-metrics ✓ state_container ✓ state_deployment ✓ state_node ✓ state_pod ✓ state_replicated ✓ state_statefulset Kubernetes events ✓ event Kubernetes API server ✓ apiserver
14. !14 Monitor services running inside Kubernetes Metricbeat Filebeat Node n Logs Metrics Nginx
15. !15 Collect service logs filebeat.prospectors: - type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch: 9200\r\n","stream":"stdout","time":"2018-02-11T23:29:19. 236692181Z"} via Docker input in Filebeat
16. !16 Metadata processors Enrich events with useful metadata to correlate logs, metrics & traces • cloud.availability_zone • cloud.region • cloud.instance_id • cloud.machine_type • cloud.project_id • cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata
17. !17 Metadata processors Example { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "project_id": "carlosperez-163008", "provider": "gce" } }, }
18. !18 Understand the service logs Via Filebeat modules • Apache • Auditd • Icinga • IIS • Kafka • Logstash • MySQL • Nginx • Osquery • Postgresql • Redis • System • Traffic ✓Filebeat modules - Out of the box - Pipeline configuration to parse logs in Elasticsearch Ingest Node - Dashboards included - Come with ML definitions
19. !19 Collect service metrics Via Metricbeat modules Data stores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike • Graphite Queues • Kafka • Redis • RabbitMQ Caches • Memcached Custom apps • JMX/Jolokia • PHP-FPM • Golang Web servers • Apache • Nginx Other • HAProxy • Zookeeper
20. !20 Collect APM traces With Elastic APM, previously Opbeat ✓ apm-server based on libbeat ✓Collects traces from agents ✓ Benefits from metadata processors ✓Agents: Python, Node.js, Ruby(Beta), RUM(Beta), Java (Alpha), Go (wip) ✓ Apache 2 license
21. !21 Monitor applications instrumented with Prometheus Via the Prometheus module in Metricbeat Node 1 Metricbeat Node 2 Metricbeat Node n Metricbeat App App App pull pull pull
22. Deployment strategies
23. !23 Kubernetes deployment Deploy Metricbeat and Filebeat as DaemonSets Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet
24. !24 Sample manifests files To Deploy Filebeat and Metricbeat as DaemonSets $ curl -L -O https:// raw.githubusercontent.com/elastic/beats/ master/deploy/kubernetes/filebeat- kubernetes.yaml $ kubectl create -f filebeat-kubernetes.yaml
25. Dynamic configuration
26. !26 Autodiscover (new in 6.2) Watch Kubernetes events and react to changes filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: kube-system config: - type: docker containers.ids: - ${data.kubernetes.container.id} exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart
27. !27 Hints based autodiscover (new in 6.3) Invert control of monitoring settings annotations: co.elastic.logs/module: nginx co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error co.elastic.metrics/module: nginx co.elastic.metrics/hosts: “${data.host}:8080" filebeat.autodiscovery: providers: - type: kubernetes hints.enabled: true
28. Curated visualizations
29. !29 High level overview Group by namespace, service
30. Questions?