Slide 1

Slide 1 text

Infrastructure Engineer, Elastic leothrix tylerjl Kubernetes, Docker, and Containers at Elastic: Tyler Langlois Monitoring, Logging, and More Software Engineer, Beats, Elastic exekias Carlos Pérez-Aradros

Slide 2

Slide 2 text

About us Tyler Langlois Infrastructure Engineer Elastic Carlos Pérez-Aradros Software Engineer Elastic

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

4 4 We need specific tools to track things down With containers architecture everything is a moving target

Slide 5

Slide 5 text

Beats containers toolbox 5 Modules: System, Docker and Kubernetes 1 Metadata processors 3 Autodiscover 4 Docker logs input 2

Slide 6

Slide 6 text

Metricbeat modules 6 Monitor Docker & Kubernetes

Slide 7

Slide 7 text

filebeat.prospectors: - type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","t ime":"2018-02-11T23:29:19.236692181Z"} Docker logs input Retrieve logs from Docker containers

Slide 8

Slide 8 text

Metadata processors 8 Enrich events with useful metadata to correlate logs, metrics & traces ●cloud.availability_zone ●cloud.region ●cloud.instance_id ●cloud.machine_type ●cloud.project_id ●cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata

Slide 9

Slide 9 text

9 Example Metadata processors { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "provider": "gce" } }, }

Slide 10

Slide 10 text

10 add_kubernetes_metadata Metadata processors API Server add_kubernetes_metadata pod watcher Pod start/stop events Docker Logs Cont. ID Metadata update 418a913c7076 ……………… e4e4c23175cd ……………… ef997054d40c ……………… c626cfdf38614 ……………… e5563a7cb80e ……………… 73de79be045c ……………... Elasticsearch Parse Enrich

Slide 11

Slide 11 text

11 Watch Docker events and react to changes Autodiscover metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379"

Slide 12

Slide 12 text

12 Watch Docker events and react to changes Autodiscover Events API Container start/stop events Beats { "host": "10.4.15.9", "port": 2379, "docker": { "container": { "id": "13a2...d716" "name": "etcd", "image": "quay.io/coreos/etcd:v3.0.0", "labels": { "io.kubernetes.pod.name": "etcd-4dk4c", "io.kubernetes.pod.namespace": "kube- system" ... } } } } 1. autodiscover event - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379" config template 3. var expansion 2. match condition 4. launch module - module: etcd metricsets: ["leader", "self", "store"] hosts: "10.4.15.9:2379"

Slide 13

Slide 13 text

13 Watch Kubernetes events and react to changes Autodiscover: Kubernetes provider filebeat.autodiscover: providers: - type: kubernetes include_annotations: ["harvest"] templates: - condition: contains: kubernetes.annotations.harvest: true config: - type: docker containers.ids: - ${data.kubernetes.container.id}

Slide 14

Slide 14 text

Deployment Strategies

Slide 15

Slide 15 text

Docker deployment 15 volume mounts volume mounts Kibana Elasticsearch docker host Log files (/var/lib/docker/containers) Web Apps Services Docker API ... Networking Filebeat Metricbeat /proc filesystem

Slide 16

Slide 16 text

Kubernetes deployment (cluster) 16 Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet

Slide 17

Slide 17 text

Kubernetes @ Elastic

Slide 18

Slide 18 text

18 Background What we monitor • Migration to container scheduler runtimes over traditional Cloud instances • Dogfooding! • We use GCP GKE • Stock monitoring options work, but we wanted to leverage our Stack • All container logs and metrics • Host metrics • Beats + k8s metadata Beats + Kubernetes in Practice Deployment

Slide 19

Slide 19 text

What are we interested in monitoring? 19 Some of our apps and use cases Kubernetes itself (events, unavailable pods, etc.) Why is $x burning its allocation of CPU resources? What is our current cluster capacity, and how much room to grow? Logs and metrics for dev apps, ES, Vault, Logstash, etc. “How much network traffic is our nginx ingress controller serving?”

Slide 20

Slide 20 text

20 • We use our stock Kubernetes manifests with slight changes for our environment (helm) • RBAC and default configs all work well out-of-the box • Just pay attention where to send logs and metrics (internal vs. external Elasticsearch) This is a sample image Kubernetes Definitions Setup go.es.io/beats-k8s

Slide 21

Slide 21 text

• Coupled with emptyDir registry paths, Filebeat is performant and handles Pod restarts well • Defining pipelines at config-time is useful for extra parsing without changing container configs • Very favorable performance versus stock GKE fluentd log shippers (more on that later) Deployment Observations 21 From the field

Slide 22

Slide 22 text

The Payoff

Slide 23

Slide 23 text

Demo Time!

Slide 24

Slide 24 text

24 Service: “beats- demo” LoadBalancer Deployment: “beats-demo“ Demo scenario: http://beats-demo.elstc.co Pod Pod ... Deployment: “mysql“ Logs Metrics Pod

Slide 25

Slide 25 text

25 (Live!) Dashboards Example Dashboards

Slide 26

Slide 26 text

Which Nodes and Namespaces Are Busy?

Slide 27

Slide 27 text

Measuring an Individual Pod

Slide 28

Slide 28 text

Comparing Resource Utilization Across Pods

Slide 29

Slide 29 text

Comparing Resources for Multiple Pods

Slide 30

Slide 30 text

Overall Resources with Log Volume

Slide 31

Slide 31 text

Visualizing Node Utilization vs. Capacity

Slide 32

Slide 32 text

Visualizing Node Utilization vs. Capacity

Slide 33

Slide 33 text

• Use the pre-existing k8s deployment manifests to run Beats in k8s • Gets ahead of questions like Metricbeat’s ability to introspect host node metrics, Docker socket access, etc. • Pour everything into Elasticsearch and repurpose your data • Other engineers have built monitoring tools atop the metrics we’re already collecting to alert on unavailable pods with Watcher, for example (one label = monitored app) • Democratizing metrics and logs in Elasticsearch has permitted anyone to create app dashboards for their own purposes (self-serve monitoring, alerting, and more) • Textual search alongside metrics is powerful (aggregations, regexes, etc.) Lessons about Beats and Kubernetes 33

Slide 34

Slide 34 text

34 More Questions? Visit us at the AMA

Slide 35

Slide 35 text

www.elastic.c o

Slide 36

Slide 36 text

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 36 Please attribute Elastic with a link to elastic.co