Monitor your containers with the Elastic Stack

Containers as well as orchestration systems like Kubernetes are quickly gaining popularity as the prefered tools for deploying and running microservices. While being easier to deploy and isolate, containerized applications are creating new challenges for the logging and monitoring systems.

One popular solution for logging and monitoring is the Elastic Stack composed of Elasticsearch, Logstash, Kibana, and Beats. This talk shows you how to use the Elastic Stack, and in particular the Beats lightweight shippers, to collect logs and metrics from your containers.

The session includes details about how to:
- fetch the logs of the containers with Filebeat
- collect container metrics with Metricbeat
- monitor the network traffic exchanged between containers with Packetbeat
- automatically discover metadata from Docker containers
- visualize the collected data with predefined Kibana dashboards scaling Logstash deployments

Transcript

1. Monitor your containers with the Elastic Stack Monica Sarbu
2. 2 Monica Sarbu Team lead, Beats team Email: monica@elastic.co Twitter: @monicasarbu
3. @monicasarbu Monitor your containers 3 Apache logs
4. @monicasarbu Monitor your containers 4 memory % CPU % Apache logs
5. @monicasarbu Monitor your containers 5 Apache metrics memory % CPU % Apache logs
6. @monicasarbu Monitor your containers 6 Apache metrics memory % CPU % HTTP transactions Apache logs
7. @monicasarbu Multiple data types, one storage 7 Apache metrics memory % CPU % HTTP transactions Apache logs
8. @monicasarbu Scalable from day 1 8
9. Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
10. 10 Elastic Stack Kibana Elasticsearch Beats Logstash
11. @monicasarbu The Beats 11 30+ other community Beats shipping
12. Filebeat 12
13. tail -f
14. tail -f over the Network
15. tail -f over the Network with extra powers http://www.clipartpanda.com/clipart_images/witches-clip-art-6144127
16. Multiline JSON logs Filtering
17. Send raw log lines 17 { message: “55.3.244.1 GET /index.html 15824 0.043” }
18. @monicasarbu Parse log lines by defining grok patterns 18 I N G E S T or
19. @monicasarbu Grok patterns 19 %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} { message: “55.3.244.1 GET /index.html 15824 0.043”, … }
20. @monicasarbu After parsing 20 { message: “55.3.244.1 GET /index.html 15824 0.043” client: “55.3.244.1”, method: “GET”, request: “/index.html” bytes: 15824, duration: 0.043 … }
21. Handle back-pressure
22. @monicasarbu Why back-pressure is key? 22
23. @monicasarbu Synchronous sending 23 batch of messages ack stream of log lines read read acked registry file
24. Filebeat adapts its speed automatically to as much as the next stage can process
25. @monicasarbu When next stage is down … • Filebeat patiently waits • Log lines are not lost • It doesn’t allocate memory • It doesn’t buffer log lines on disk 25
26. At-Least-Once delivery
27. #velo @monicasarbu No such think as “exactly once” 27
28. #velo @monicasarbu 28 batch of messages ack batch of messages same batch of messages ack duplicates! 28
29. Filebeat Collect container logs 29
30. @monicasarbu Docker logging drivers 30 https://docs.docker.com/engine/admin/logging/overview/
31. @monicasarbu 001 Gelf driver + Logstash Pros: • logs send directly to Logstash 31 Cons: • UDP based, no delivery guarantees, no congestion control
32. @monicasarbu 010 json-file driver + Filebeat Pros: • Simple to setup as it’s the default driver • Easy to add container metadata (name, labels, etc.) • `docker logs` works 32 Cons: • json-file driver can slow down Docker container
33. @monicasarbu 011 Syslog driver + Syslog server + Filebeat Pros: • Good control over the path where the files are written, rotation strategies, etc. 33 Cons: • you need to manage the syslog server • metadata is serialized as string, needs to be de- serialized again • multiline is difficult because data from containers can be mixed
34. @monicasarbu 100 Journald driver + Filebeat Pros: • journald is often already available • convenient support for container metadata (name, labels, etc.) • `docker logs` works 34 Cons: • Filebeat doesn’t yet support journald • You can use the community Beat, Journalbeat
35. @monicasarbu 101 Shared volume + Filebeat Pros: • If your app can rotate it’s own logs, it’s very easy to setup • Scales well 35 Cons: • Difficult to pass container metadata (name, labels, etc.)
36. Conclusion “At-least-once” guarantees and handle back-pressure: • json driver + Filebeat • Syslog driver + Filebeat • Shared volume + Filebeat • Journald driver + Filebeat (in the future) 36 No guarantees: • Gelf driver + Logstash • Fluentd + Logstash
37. Metricbeat 37 new in 5.0
38. @monicasarbu One Metricbeat module for each service 38 + Add your own
39. @monicasarbu Metricbeat system module 39 CPU Mem diskIO filesystem processes load network cores
40. Metricbeat Collect container metrics 40
41. @monicasarbu Querying the Docker API • CPU and memory • Docker container information • network (in/out bytes, dropped) • diskIO (reads/writes) • status of containers (# of stopped, running, etc) 41
42. @monicasarbu Docker module • Get container metrics by querying the Docker API • Has access to container names and labels • Easy to setup 42 available in 5.1.1
43. @monicasarbu Reading cgroup data from /proc/ • Doesn’t require access to the Docker API (can be a security issue) • Works for any container runtime (Docker, rkt, runC, LXD, etc.) • Cannot get the container name and labels only the container ID 43
44. @monicasarbu System module + cgroup data • if cgroup option is enabled (by default is disabled) • Automatically enhances process data with cgroup information 44
45. @monicasarbu Run as a container 45 App1 App2 App3 Host
46. 46 Elasticsearch as time series DB
47. #velo @monicasarbu Elasticsearch BKD trees 47 • Added for Geo-points • faster to index • faster to query • more disk-efficient • more memory efficient
48. @monicasarbu 0 10000 20000 30000 40000 50000 60000 70000 80000 float half float scaled float (factor = 4000) scaled float (factor = 100) On Disk Usage in kb Points disk usage (kb) docs_values disk usage (kb) Float values 48 • half floats • scaled floats (using a scaling factor) - great for things like percentage points
49. #velo @monicasarbu Why Elasticsearch for time series • Horizontal scalability. Mature and battle tested cluster support. • Flexible aggregations (incl moving averages & Holt Winters) • One system for both logs and metrics • Timelion UI, Grafana • Great ecosystem: e.g. alerting tools 49
50. Packetbeat 50
51. @monicasarbu How Packetbeat works 51 1 2 3 4 capture network traffic decodes network traffic correlates request & response into transactions send transactions to Elasticsearch
52. @monicasarbu Supported traffic decoders 52 + Add your own http:// Thrift DNS ICMP AMQP
53. @monicasarbu Unknown traffic, use flows •Look into data for which we don’t understand the application layer protocol •TLS •Protocols we don’t yet support •Get data about IP / TCP / UDP layers •number of packets & bytes •retransmissions •inter-arrival time 53
54. @monicasarbu Monitor traffic exchanged by containers 54 App1 Host App2 App3 Packetbeat traffic exchanged between your containers
55. 55 Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana
56. Thank you • github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode 56