Shared Nothing Logging Infrastructure

Der Betrieb einer Microservices- und/oder einer Containerinfrastruktur birgt im Logging – gelinde gesagt – Herausforderungen. Die Infrastruktur ist volatil, das Debugging einer Benutzersession über Logfiles eines einzigen Servers ist nicht mehr möglich. Eine zentrale Logging-Infrastruktur zum Einsammeln, Verarbeiten und Speichern von Logs muss her. Allerdings ist eine solche zentrale Logging-Infrastruktur nicht mit einer "Shared Nothing"-Architektur kompatibel. Konsequenterweise muss dort jedes Team seine eigene Logging-Infrastruktur betreiben. Doch dazu fehlen den meisten Teams Zeit und Wissen, weshalb der Architekturansatz hier bricht. Am Beispiel von Shopping24 zeigt dieser Talk, wie wir die Herausforderung mit Methoden der Softwareentwicklung gemeistert haben. Gemeinsam werden Schnittstellen definiert, es wird eine Default-Logging-Implementierung bereitgestellt, und die Teams können diese Implementierung nach Bedarf erweitern. Mit ein wenig Governance können wir dann sicherstellen, dass die Teams untereinander von neuen Features und Versionen profitieren.

Transcript

1. @backstreetkiwi ✨ @tboeghk Shared nothing logging infrastructure Logging for containers and the cloud
2. @backstreetkiwi ✨ @tboeghk Nikolaus Winter @backstreetkiwi #java #python #spring #ci #cd #observability
3. @backstreetkiwi ✨ @tboeghk #search #solr #ansible #kubernetes #observability #photography Torsten Bøgh Köster @tboeghk
4. @backstreetkiwi ✨ @tboeghk agenda Motivation Logging & Containers Default Logging-Stack Adaptions Pitfalls & Learnings Demo
5. @backstreetkiwi ✨ @tboeghk (1) Motivation Photo by Kobu Agency on Unsplash
6. @backstreetkiwi ✨ @tboeghk We build logging systems
7. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure self contained systems Photo by Edwin Andrade on Unsplash
8. @backstreetkiwi ✨ @tboeghk self contained systems Popular in e-commerce architectures Photo by Edwin Andrade on Unsplash
9. @backstreetkiwi ✨ @tboeghk Teams in self contained systems ... ... develop a system independently ... have individual preferences ... have administrative constraints
10. @backstreetkiwi ✨ @tboeghk Operational excellence We are building ops infrastructure for a specific target audience: developers. Our solutions need to be easy to navigate, adapt and extend Photo by Arif Riyanto on Unsplash
11. @backstreetkiwi ✨ @tboeghk Kubernetes primary target platform most services still run on "classic" systems Photo by Maximilian Weisbecker on Unsplash
12. Cattle not pets, but for clusters Photo by Stijn te Strake on Unsplash
13. @backstreetkiwi ✨ @tboeghk (2) Container and logging STDOUT as lowest common denominator Photo by Sculpture Qode on Unsplash
14. @backstreetkiwi ✨ @tboeghk Kubernetes and logging there is no silver bullet Photo by Radek Grzybowski on Unsplash
15. @backstreetkiwi ✨ @tboeghk pod observability via API is the key to logging and monitoring in Kubernetes Photo by Rishabh Varshney on Unsplash
16. @backstreetkiwi ✨ @tboeghk (3) Our logging stack Introducing the ELG stack
17. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing deployment
18. @backstreetkiwi ✨ @tboeghk Default logging stack at scale // will it scale?
19. @backstreetkiwi ✨ @tboeghk Roll-out? YAML? JSON? Helm? CRD (Operator)? Ansible?
20. @backstreetkiwi ✨ @tboeghk Logging-Operator https://github.com/banzaicloud/logging-operator
21. @backstreetkiwi ✨ @tboeghk Elastic-Operator
22. @backstreetkiwi ✨ @tboeghk " "
23. @backstreetkiwi ✨ @tboeghk (4) Classes of real- world adaptions (1) Preconfigured, selectable customizations (2) Team- and application specific extensions Photo by Nigel Tadyanehondo on Unsplash
24. @backstreetkiwi ✨ @tboeghk Log flow control Use Kubernetes annotations to customize log processing
25. @backstreetkiwi ✨ @tboeghk Log Flow Control using deployment annotations
26. @backstreetkiwi ✨ @tboeghk Filter messages Control messages to collect
27. @backstreetkiwi ✨ @tboeghk Fine grained parsing spring-boot (multi line)
28. @backstreetkiwi ✨ @tboeghk Log enrichment HTTP elements
29. @backstreetkiwi ✨ @tboeghk Log enrichment geoip lookup
30. @backstreetkiwi ✨ @tboeghk Storage classes specify log retention time // sharding
31. @backstreetkiwi ✨ @tboeghk Photo by Nigel Tadyanehondo on Unsplash Interfaces Use Kubernetes services as interface to swappable and hence highly customizable log processing implementations
32. @backstreetkiwi ✨ @tboeghk Managing extensibility it's all about the interfaces
33. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing
34. @backstreetkiwi ✨ @tboeghk Pimp my logstash branching off clickstream for machine learning
35. @backstreetkiwi ✨ @tboeghk Serving team preferences Kibana as log ui
36. @backstreetkiwi ✨ @tboeghk Centralized log storage Reducing total cost of ownership
37. @backstreetkiwi ✨ @tboeghk Photo by Nigel Tadyanehondo on Unsplash (6) Lessons learned Challenges of large volume message handling apply.
38. @backstreetkiwi ✨ @tboeghk ⚡ Data integrity
39. @backstreetkiwi ✨ @tboeghk ⚡ mixed log handling reality sucks. again.
40. @backstreetkiwi ✨ @tboeghk ⚡ log governance Store everything. Find nothing?
41. @backstreetkiwi ✨ @tboeghk Demo Examples & Helpers @ o12stack.org Photo by Federica Galli on Unsplash
42. @backstreetkiwi ✨ @tboeghk Logging is fun fun fun Photo by Courtney Hedger on Unsplash