Monica Sarbu Team Lead, Beats and Logstash Monitoring Kubernetes at Scale 

Monitoring Kubernetes with Elastic Stack 

Elastic Stack 

A single Beat for each use case Beats family libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat 

Growing Beats community !5 50M Cumulative downloads 3 Years 

!6 

Monitoring challenges 

!8 With containers architecture, everything is a moving target 

!9 High level overview 

!10 

!11 Monitor all the things! ✓ Monitor services running in Kubernetes ✓ Collect application logs, metrics, traces 

Monitoring with Beats 

!13 Monitor Kubernetes cluster Via the Kubernetes module in Metricbeat Node metrics, from kubelet ✓ pod ✓ node ✓ system ✓ container ✓ volume State metrics, from kube-state-metrics ✓ state_container ✓ state_deployment ✓ state_node ✓ state_pod ✓ state_replicated ✓ state_statefulset Kubernetes events ✓ event Kubernetes API server ✓ apiserver 

!14 Monitor services running inside Kubernetes Metricbeat Filebeat Node n Logs Metrics Nginx 

!15 Collect service logs filebeat.prospectors: - type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch: 9200\r\n","stream":"stdout","time":"2018-02-11T23:29:19. 236692181Z"} via Docker input in Filebeat 

!16 Metadata processors Enrich events with useful metadata to correlate logs, metrics & traces • cloud.availability_zone • cloud.region • cloud.instance_id • cloud.machine_type • cloud.project_id • cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata 

!17 Metadata processors Example { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "project_id": "carlosperez-163008", "provider": "gce" } }, } 

!18 Understand the service logs Via Filebeat modules • Apache • Auditd • Icinga • IIS • Kafka • Logstash • MySQL • Nginx • Osquery • Postgresql • Redis • System • Traffic ✓Filebeat modules - Out of the box - Pipeline configuration to parse logs in Elasticsearch Ingest Node - Dashboards included - Come with ML definitions 

!19 Collect service metrics Via Metricbeat modules Data stores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike • Graphite Queues • Kafka • Redis • RabbitMQ Caches • Memcached Custom apps • JMX/Jolokia • PHP-FPM • Golang Web servers • Apache • Nginx Other • HAProxy • Zookeeper 

!20 Collect APM traces With Elastic APM, previously Opbeat ✓ apm-server based on libbeat ✓Collects traces from agents ✓ Benefits from metadata processors ✓Agents: Python, Node.js, Ruby(Beta), RUM(Beta), Java (Alpha), Go (wip) ✓ Apache 2 license 

!21 Monitor applications instrumented with Prometheus Via the Prometheus module in Metricbeat Node 1 Metricbeat Node 2 Metricbeat Node n Metricbeat App App App pull pull pull 

Deployment strategies 

!23 Kubernetes deployment Deploy Metricbeat and Filebeat as DaemonSets Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet 

!24 Sample manifests files To Deploy Filebeat and Metricbeat as DaemonSets $ curl -L -O https:// raw.githubusercontent.com/elastic/beats/ master/deploy/kubernetes/filebeat- kubernetes.yaml $ kubectl create -f filebeat-kubernetes.yaml 

Dynamic configuration 

!26 Autodiscover (new in 6.2) Watch Kubernetes events and react to changes filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: kube-system config: - type: docker containers.ids: - ${data.kubernetes.container.id} exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart 

!27 Hints based autodiscover (new in 6.3) Invert control of monitoring settings annotations: co.elastic.logs/module: nginx co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error co.elastic.metrics/module: nginx co.elastic.metrics/hosts: “${data.host}:8080" filebeat.autodiscovery: providers: - type: kubernetes hints.enabled: true 

Curated visualizations 

!29 High level overview Group by namespace, service 

Questions?