Infrastructure Engineer, Elastic leothrix tylerjl Kubernetes, Docker, and Containers at Elastic: Tyler Langlois Monitoring, Logging, and More Software Engineer, Beats, Elastic exekias Carlos Pérez-Aradros

About us Tyler Langlois Infrastructure Engineer Elastic Carlos Pérez-Aradros Software Engineer Elastic

No content

4 4 We need specific tools to track things down With containers architecture everything is a moving target

Beats containers toolbox 5 Modules: System, Docker and Kubernetes 1 Metadata processors 3 Autodiscover 4 Docker logs input 2

Metricbeat modules 6 Monitor Docker & Kubernetes

filebeat.prospectors: - type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","t ime":"2018-02-11T23:29:19.236692181Z"} Docker logs input Retrieve logs from Docker containers

Metadata processors 8 Enrich events with useful metadata to correlate logs, metrics & traces ●cloud.availability_zone ●cloud.region ●cloud.instance_id ●cloud.machine_type ●cloud.project_id ●cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata

9 Example Metadata processors { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "provider": "gce" } }, }

10 add_kubernetes_metadata Metadata processors API Server add_kubernetes_metadata pod watcher Pod start/stop events Docker Logs Cont. ID Metadata update 418a913c7076 ……………… e4e4c23175cd ……………… ef997054d40c ……………… c626cfdf38614 ……………… e5563a7cb80e ……………… 73de79be045c ……………... Elasticsearch Parse Enrich

11 Watch Docker events and react to changes Autodiscover metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379"

12 Watch Docker events and react to changes Autodiscover Events API Container start/stop events Beats { "host": "10.4.15.9", "port": 2379, "docker": { "container": { "id": "13a2...d716" "name": "etcd", "image": "quay.io/coreos/etcd:v3.0.0", "labels": { "io.kubernetes.pod.name": "etcd-4dk4c", "io.kubernetes.pod.namespace": "kube- system" ... } } } } 1. autodiscover event - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379" config template 3. var expansion 2. match condition 4. launch module - module: etcd metricsets: ["leader", "self", "store"] hosts: "10.4.15.9:2379"

13 Watch Kubernetes events and react to changes Autodiscover: Kubernetes provider filebeat.autodiscover: providers: - type: kubernetes include_annotations: ["harvest"] templates: - condition: contains: kubernetes.annotations.harvest: true config: - type: docker containers.ids: - ${data.kubernetes.container.id}

Deployment Strategies

Docker deployment 15 volume mounts volume mounts Kibana Elasticsearch docker host Log files (/var/lib/docker/containers) Web Apps Services Docker API ... Networking Filebeat Metricbeat /proc filesystem

Kubernetes deployment (cluster) 16 Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet

Kubernetes @ Elastic

18 Background What we monitor • Migration to container scheduler runtimes over traditional Cloud instances • Dogfooding! • We use GCP GKE • Stock monitoring options work, but we wanted to leverage our Stack • All container logs and metrics • Host metrics • Beats + k8s metadata Beats + Kubernetes in Practice Deployment

What are we interested in monitoring? 19 Some of our apps and use cases Kubernetes itself (events, unavailable pods, etc.) Why is $x burning its allocation of CPU resources? What is our current cluster capacity, and how much room to grow? Logs and metrics for dev apps, ES, Vault, Logstash, etc. “How much network traffic is our nginx ingress controller serving?”

20 • We use our stock Kubernetes manifests with slight changes for our environment (helm) • RBAC and default configs all work well out-of-the box • Just pay attention where to send logs and metrics (internal vs. external Elasticsearch) This is a sample image Kubernetes Definitions Setup go.es.io/beats-k8s

• Coupled with emptyDir registry paths, Filebeat is performant and handles Pod restarts well • Defining pipelines at config-time is useful for extra parsing without changing container configs • Very favorable performance versus stock GKE fluentd log shippers (more on that later) Deployment Observations 21 From the field

The Payoff

Demo Time!

24 Service: “beats- demo” LoadBalancer Deployment: “beats-demo“ Demo scenario: http://beats-demo.elstc.co Pod Pod ... Deployment: “mysql“ Logs Metrics Pod

25 (Live!) Dashboards Example Dashboards

Which Nodes and Namespaces Are Busy?

Measuring an Individual Pod

Comparing Resource Utilization Across Pods

Comparing Resources for Multiple Pods

Overall Resources with Log Volume

Visualizing Node Utilization vs. Capacity

Visualizing Node Utilization vs. Capacity

• Use the pre-existing k8s deployment manifests to run Beats in k8s • Gets ahead of questions like Metricbeat’s ability to introspect host node metrics, Docker socket access, etc. • Pour everything into Elasticsearch and repurpose your data • Other engineers have built monitoring tools atop the metrics we’re already collecting to alert on unavailable pods with Watcher, for example (one label = monitored app) • Democratizing metrics and logs in Elasticsearch has permitted anyone to create app dashboards for their own purposes (self-serve monitoring, alerting, and more) • Textual search alongside metrics is powerful (aggregations, regexes, etc.) Lessons about Beats and Kubernetes 33

34 More Questions? Visit us at the AMA

www.elastic.c o

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 36 Please attribute Elastic with a link to elastic.co