Shared Nothing Logging Infrastructure

Der Betrieb einer Microservices- und/oder einer Containerinfrastruktur birgt beim Logging — gelinde gesagt — einige Herausforderungen. Die Infrastruktur ist volatil, das Debugging einer Benutzersession über Logfiles eines einzigen Servers ist nicht mehr möglich. Eine zentrale Logging-Infrastruktur zum Einsammeln, Verarbeiten und Speichern von Logs muss her!

Allerdings ist solch eine zentrale Logging-Infrastruktur nicht mit einer "Shared Nothing"-Architektur kompatibel. Konsequenterweise muss dort jedes Team seine eigene Logging-Infrastruktur betreiben. Ehrlicherweise fehlt es dazu den meisten Teams an Zeit und Wissen, weshalb der Architekturansatz hier dann bricht.

Am Beispiel von Shopping24 zeigt dieser Talk, wie wir die Herausforderung mit Methoden der Softwareentwicklung gemeistert haben. Gemeinsam werden Schnittstellen definiert, eine Default-Logging-Implementation bereitgestellt und die Teams können diese Implementation nach Bedarf erweitern. Mit ein wenig Governance können wir dann sicherstellen, dass die Teams untereinander von neuen Features und Versionen profitieren.

Transcript

1. @backstreetkiwi ✨ @tboeghk Shared nothing logging infrastructure Logging for containers and the cloud
2. @backstreetkiwi ✨ @tboeghk Nikolaus Winter @backstreetkiwi #search #logging #java #spring #kotlin
3. @backstreetkiwi ✨ @tboeghk #search #solr #server #ansible #kubernetes #photography Torsten Bøgh Köster @tboeghk
4. @backstreetkiwi ✨ @tboeghk shopping24 commerce network
5. @backstreetkiwi ✨ @tboeghk agenda Definitions Log message flow control Multi cluster logging Pitfalls & Learnings Demo
6. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure Photo by Edwin Andrade on Unsplash
7. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure self contained systems Photo by Edwin Andrade on Unsplash
8. @backstreetkiwi ✨ @tboeghk self contained systems Popular in e-commerce architectures Photo by Edwin Andrade on Unsplash
9. @backstreetkiwi ✨ @tboeghk Container and logging STDOUT as lowest common denominator Photo by Sculpture Qode on Unsplash
10. @backstreetkiwi ✨ @tboeghk Kubernetes container scheduler Photo by SpaceX on Unsplash
11. @backstreetkiwi ✨ @tboeghk Kubernetes and logging there is no silver bullet Photo by Radek Grzybowski on Unsplash
12. @backstreetkiwi ✨ @tboeghk Observability is the key to logging and monitoring in Kubernetes Photo by Rishabh Varshney on Unsplash
13. @backstreetkiwi ✨ @tboeghk This is how we roll Introducing the ELG stack
14. @backstreetkiwi ✨ @tboeghk Reality sucks centralized logging approach
15. @backstreetkiwi ✨ @tboeghk Flow control & Interfaces Two approaches to meet increasing logging customization needs Photo by Nigel Tadyanehondo on Unsplash
16. @backstreetkiwi ✨ @tboeghk Log flow control Use Kubernetes annotations to manipulate log processing
17. @backstreetkiwi ✨ @tboeghk Filter messages Control messages to collect
18. @backstreetkiwi ✨ @tboeghk Fine grained parsing spring-boot (multi line)
19. @backstreetkiwi ✨ @tboeghk Fine grained parsing web-access-logs
20. @backstreetkiwi ✨ @tboeghk Log enrichment HTTP elements
21. @backstreetkiwi ✨ @tboeghk Log enrichment geoip lookup
22. @backstreetkiwi ✨ @tboeghk Storage classes specify log retention time // sharding
23. Cattle not pets, but for clusters Photo by Stijn te Strake on Unsplash
24. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing deployment
25. @backstreetkiwi ✨ @tboeghk Default logging stack at scale // will it scale?
26. @backstreetkiwi ✨ @tboeghk Interfaces Use Kubernetes services as interface to swappable and hence highly customizable log processing implementations
27. @backstreetkiwi ✨ @tboeghk Managing extensibility it's all about the interfaces
28. @backstreetkiwi ✨ @tboeghk Templating log configuration Photo by Andreas Gücklhorn on Unsplash
29. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing
30. @backstreetkiwi ✨ @tboeghk Centralized log storage Reducing total cost of ownership
31. @backstreetkiwi ✨ @tboeghk Pimp my logstash branching off clickstream for machine learning
32. @backstreetkiwi ✨ @tboeghk Serving special needs Kibana as log ui
33. @backstreetkiwi ✨ @tboeghk Lessons learned Challenges of large volume message handling apply.
34. @backstreetkiwi ✨ @tboeghk ⚡ Data integrity
35. @backstreetkiwi ✨ @tboeghk ⚡ mixed log handling reality sucks. again.
36. @backstreetkiwi ✨ @tboeghk ⚡ log governance Store everything. Find nothing?
37. @backstreetkiwi ✨ @tboeghk Demo Examples & Helpers @ o12stack.org Photo by Federica Galli on Unsplash
38. @backstreetkiwi ✨ @tboeghk Logging is fun fun fun Photo by Courtney Hedger on Unsplash