Logging in a Microservice World

My talk about distributed logging and the ELK Stack
The Talk was given at the DevOpsCon Munich in November 2015

Transcript

1. Logging in a Microservice World_ The ELK Stack and Microservices DevOpsCon 2015, Munich
2. Agenda 1. Motivation for (distributed) logging 2. Really annoying things about logfiles 3. Basic ELK Architecture 4. Docker! 2
3. Motivation for (distributed) Logging 3
4. Motivation for (distributed) logging 4 The BIG Database The BIG Application Server LOG LOG DEV OP
5. Motivation for (distributed) logging suddenly... Microservices! smallest viable infrastructure becomes... 5
6. Elastic Elastic Mongo Mongo ETCD Cluster Cluster Cluster Messag e Broker Messag e Broker Service Service Service Service Motivation for (distributed) logging 6 Proxy Cluster ETCD Service Service Service Service Mongo Elastic Cassandra Postgres Message Broker DevOps
7. Logs are containing more information than ever before •  The usual errors •  State of the environment •  Behaviour of the system during normal operation 7
8. Motivation for (distributed) logging Result: Logging gets more complicated. We need to deal with it. But logging can also be much more useful: 8
9. Motivation for (distributed) logging Things we can do with Logfiles •  Debugging / Post Mortem •  Monitoring •  Analytics •  Business Decisions 9
10. Really annoying things about logfiles 10
11. 11 “The server went down because the disk was full. The logfile got really big” -Some OPs Guy
12. Annoying things about logfiles Every logfile is a unique snowflake •  Each system has it‘s own log format •  Subcomponents might have additional ones •  Stacktraces / Multiline exceptions •  That one DEV who does println 12
13. Annoying things about logfiles Human readable != Machine Readable •  JSON log output makes things so much easier to parse •  JSON log output is ridiculous to read You can always generate Strings from JSON, the other way not so much 13
14. Annoying things about logfiles Correlation is hard •  Correlate by time is hard when there is actual load on your system •  There is no clever automated way to get requests traced •  It needs to happen in your application logic 14
15. Annoying things about logfiles They contain more than you think they do •  Customer Data (credit card numbers) •  „Betriebsratsrelevant“ •  SSH Keys and AWS Keys •  Loglevels all over the place 15
16. Annoying things about logfiles •  Every Logfile is a unique, beautiful Snowflake •  They need Space •  Human readable != Machine Readable •  They contain more than you think they do •  Correlation is hard 16
17. Basic ELK Architecture 17
18. Basic ELK Architecture 18 System that logs logstash elastic kibana Log parse store visualize clean
19. Basic ELK Architecture Unifiy data: extract values from logs •  Describe Logs with a RegEx – similar pattern •  Use predefined patters for popular logs •  Use Json and be done with it 19
20. Basic ELK Architecture Ingestion Development Tips •  Use online tooling and tests •  Filter as needed •  JSON and plain Logs can coexist 20
21. Ingestion Pipeline 21 System that logs shipper logstash logstash Ingestion Pipeline redis Push through buffer parse
22. Basic ELK Architecture 22 Shipping •  Logstash brings a JVM •  Logstash forwarder is deprecated •  Beats! Filebeat, Topbeat, libbeat •  GELF (or syslog if you must)
23. Do you need more POWER? 23 logstash logstash logstash Client node Client node Data node Data node Data node Data node Master node Master node Master node
24. Basic ELK Architecture Infrastructure Demands •  Logstash likely CPU bound •  Buffers are Memory bound •  Elasticsearch CPU, IO and FS Cache bound •  Kibana doesn‘t really matter => Mostly balanced if run on one Machine 24
25. Basic ELK Architecture Dedicated Elasticsearch •  FAST IO •  1 / 40 Mem to Disk Ratio is a good start 25
26. Basic ELK Architecture Sizing •  Count your Logfiles •  Size ends up 0.5 – 2 x Logsize •  Balance by retention 26
27. Visualize Kibana! •  Easy drill-down and visualization tool •  Aggregate and view whatever is in your Elasticsearch 27
28. Visualize 28
29. Visualize Democratization of the logs. So what about security? •  Authentication & Authorisation available by plug-in •  Encrypted transport out-of-the-box 29
30. DockerDockerDockerDocker 30
31. Docker! Immutable Containers and persistent logs do not match When Infrastructure lives for minutes instead of days – central logging becomes unavoidable 31
32. Docker! 32 Safely aggregate the logs from a LOT of machines 1.  logstash forwarder / filebeat in the container 2. Shared data volume 3. Pipe stdout / stderr 4. Logappender
33. Docker! 33 Logstash forwarder / filebeat in the container •  Easiest thing to do •  Violates “one thing per container” rule •  Lots of shared configuration
34. Docker! 34 Shared data volume •  Less forwarding instances needed •  Integrates nicely with some clusters •  Backpressure instead of queueing
35. Docker! 35 Pipe stderr/stdout •  Less forwarding instances needed •  More freedom where to put the stream •  No buffers, no space
36. Docker! 36 Logging framework / appender talks to logstash directly •  Easy infastructure (as in none) •  Does not work in all ecosystems •  No buffering •  No balancing
37. Docker! 37 Most people seem to be happy with a shared volume
38. codecentric AG Merscheider Straße 1 42699 Solingen, Deutschland mobil: +49 (0) 151.16706496 fax: +49 (0) 212.23362879 www.codecentric.de blog.codecentric.de Some text here Fragen? 38 Christian Uhl, christian.uhl@codecentric .de @chrisuhlcc IT Consultant, Search Guy