From ELK to Elastic: Modern logging and monitoring

My presentation from Velocity Amsterdam 2016.

Transcript

1. From ELK to the Elastic stack Tudor Golubenco / @tudor_g 1 Modern logging and monitoring
2. 2 Dev working on beats
3. The “ELK” stack 3 Elasticsearch Logstash Kibana
4. 4 And then the Beats happened
5. 5 Beats Logstash
6. • Core and main plugins re- written in Java • performance improvements • Persistent Queue (WIP) • no drops when killed • Monitoring APIs • no more a black box 6 Logstash new features
7. BELK? KELB? ELKB? ELK-Bee? 7
8. 8 Say “Heya” to the Elastic Stack 5.0
9. 9 Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats
10. 10 Working beautifully together es kibana ls beats 6.0 7.0 5.0 5.0 5.0 5.0
11. 30+ other community Beats shipping The Beat(le)s
12. Topbeat Metricbeat 12
13. Demo: Metricbeat
14. Container monitoring 14 Metricbeat
15. Reading cgroup data from /proc/ • Doesn’t require access to the Docker API (can be a security issue) • Works for any container runtime (Docker, rkt, runC, LXD, etc.) • Automatically enhances process data with cgroup information 15
16. Querying the Docker API • Dedicated Docker module • Has access to container names and labels • It’s somehow easier to setup 16
17. Run as a container 17 App1 App2 App3 Host
18. Elasticsearch as a time series DB 18 Metricbeat
19. #velo Elasticsearch BKD trees 19 • Added for Geo-points • faster to index • faster to query • more disk-efficient • more memory efficient
20. 0 10000 20000 30000 40000 50000 60000 70000 80000 float half float scaled float (factor = 4000) scaled float (factor = 100) On Disk Usage in kb Points disk usage (kb) docs_values disk usage (kb) Float values 20 • half floats • scaled floats - great for things like percentage points
21. #velo Why Elasticsearch for time series • Horizontal scalability. Mature and battle tested cluster support. • Flexible aggregations (incl moving averages & Holt Winters) • One system for both logs and metrics • Timelion UI, Grafana • Great ecosystem: e.g. alerting tools 21
22. Filebeat 22
23. • Supports multiple file rotation strategies • “At least once” guarantees, handles backpressure • Extra powers: • Multiline • Filtering • JSON decoding 23 Filebeat
24. Back pressure handling 24 Filebeat
25. #velo 25 Why is backpressure key
26. #velo 26 batch of messages ack Synchronous sending stream of log lines read read acked registry file
27. #velo 27 batch of messages When things go wrong ack 0 (still alive) ack 50% ack 100% read acked
28. • Filebeat adapts it’s speed automatically to as much as the next stage can ingest • Simliar to the “pull” model • But: beaware when benchmarking 28 This means..
29. • Filebeat patiently waits • Log lines are not lost • It doesn’t allocate memory, it doesn’t buffer things on disk 29 When the next stage is down..
30. #velo 30 Using an intermediary queue
31. “At least once” guarantees 31 Filebeat
32. #velo No such think as “exactly once” 32
33. #velo 33 batch of messages ack batch of messages same batch of messages ack duplicates!
34. #velo Potential strategy to reduce dupes • Filebeat generates an UUID for each log line • When indexing to Elasticsearch, use the create API • Deduplication happens in Elasticsearch • But ✦ Duplicates can still happen on Filebeat crashes ✦ Performance penalty at index time 34
35. JSON decoding + Structured logging 35 Filebeat
36. Classic logging logging.debug("User '{}' (id: {}) successfully logged in. Session id: {}” .format(user["name"], user["id"], session_id)) which results in: DEBUG:root:User 'arthur' (id: 42) successfully logged in. Session id: 91e5b9d 36
37. Structured logging • Use a logging library that allows code like: log = log.bind(user='arthur', id=42, verified=False) log.msg(‘logged_in’) • Which creates log lines like: {"verified": false, "user": "arthur", "session_id": "91e5b9d", "id": 42, "event": “logged_in"} 37
38. Ask Me (Almost) Anything Tudor Golubenco / @tudor_g