Pro Yearly is on sale from $80 to $50! »

Shared Nothing Logging Infrastructure

Shared Nothing Logging Infrastructure

Der Betrieb einer Microservices- und/oder einer Containerinfrastruktur birgt beim Logging — gelinde gesagt — einige Herausforderungen. Die Infrastruktur ist volatil, das Debugging einer Benutzersession über Logfiles eines einzigen Servers ist nicht mehr möglich. Eine zentrale Logging-Infrastruktur zum Einsammeln, Verarbeiten und Speichern von Logs muss her!

Allerdings ist solch eine zentrale Logging-Infrastruktur nicht mit einer "Shared Nothing"-Architektur kompatibel. Konsequenterweise muss dort jedes Team seine eigene Logging-Infrastruktur betreiben. Ehrlicherweise fehlt es dazu den meisten Teams an Zeit und Wissen, weshalb der Architekturansatz hier dann bricht.

Am Beispiel von Shopping24 zeigt dieser Talk, wie wir die Herausforderung mit Methoden der Softwareentwicklung gemeistert haben. Gemeinsam werden Schnittstellen definiert, eine Default-Logging-Implementation bereitgestellt und die Teams können diese Implementation nach Bedarf erweitern. Mit ein wenig Governance können wir dann sicherstellen, dass die Teams untereinander von neuen Features und Versionen profitieren.

A6bb61c55fa41db28e68cd476cb54ab9?s=128

Torsten Bøgh Köster

May 09, 2019
Tweet

Transcript

  1. @backstreetkiwi ✨ @tboeghk Shared nothing logging infrastructure Logging for containers

    and the cloud
  2. @backstreetkiwi ✨ @tboeghk Nikolaus Winter @backstreetkiwi #search #logging #java #spring

    #kotlin
  3. @backstreetkiwi ✨ @tboeghk #search #solr #server #ansible #kubernetes #photography Torsten

    Bøgh Köster @tboeghk
  4. @backstreetkiwi ✨ @tboeghk shopping24 commerce network

  5. @backstreetkiwi ✨ @tboeghk agenda Definitions Log message flow control Multi

    cluster logging Pitfalls & Learnings Demo
  6. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure Photo by Edwin Andrade

    on Unsplash
  7. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure self contained systems Photo

    by Edwin Andrade on Unsplash
  8. @backstreetkiwi ✨ @tboeghk self contained systems Popular in e-commerce architectures

    Photo by Edwin Andrade on Unsplash
  9. @backstreetkiwi ✨ @tboeghk Container and logging STDOUT as lowest common

    denominator Photo by Sculpture Qode on Unsplash
  10. @backstreetkiwi ✨ @tboeghk Kubernetes container scheduler Photo by SpaceX on

    Unsplash
  11. @backstreetkiwi ✨ @tboeghk Kubernetes and logging there is no silver

    bullet Photo by Radek Grzybowski on Unsplash
  12. @backstreetkiwi ✨ @tboeghk Observability is the key to logging and

    monitoring in Kubernetes Photo by Rishabh Varshney on Unsplash
  13. @backstreetkiwi ✨ @tboeghk This is how we roll Introducing the

    ELG stack
  14. @backstreetkiwi ✨ @tboeghk Reality sucks centralized logging approach

  15. @backstreetkiwi ✨ @tboeghk Flow control & Interfaces Two approaches to

    meet increasing logging customization needs Photo by Nigel Tadyanehondo on Unsplash
  16. @backstreetkiwi ✨ @tboeghk Log flow control Use Kubernetes annotations to

    manipulate log processing
  17. @backstreetkiwi ✨ @tboeghk Filter messages Control messages to collect

  18. @backstreetkiwi ✨ @tboeghk Fine grained parsing spring-boot (multi line)

  19. @backstreetkiwi ✨ @tboeghk Fine grained parsing web-access-logs

  20. @backstreetkiwi ✨ @tboeghk Log enrichment HTTP elements

  21. @backstreetkiwi ✨ @tboeghk Log enrichment geoip lookup

  22. @backstreetkiwi ✨ @tboeghk Storage classes specify log retention time //

    sharding
  23. Cattle not pets, but for clusters Photo by Stijn te

    Strake on Unsplash
  24. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing deployment

  25. @backstreetkiwi ✨ @tboeghk Default logging stack at scale // will

    it scale?
  26. @backstreetkiwi ✨ @tboeghk Interfaces Use Kubernetes services as interface to

    swappable and hence highly customizable log processing implementations
  27. @backstreetkiwi ✨ @tboeghk Managing extensibility it's all about the interfaces

  28. @backstreetkiwi ✨ @tboeghk Templating log configuration Photo by Andreas Gücklhorn

    on Unsplash
  29. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing

  30. @backstreetkiwi ✨ @tboeghk Centralized log storage Reducing total cost of

    ownership
  31. @backstreetkiwi ✨ @tboeghk Pimp my logstash branching off clickstream for

    machine learning
  32. @backstreetkiwi ✨ @tboeghk Serving special needs Kibana as log ui

  33. @backstreetkiwi ✨ @tboeghk Lessons learned Challenges of large volume message

    handling apply.
  34. @backstreetkiwi ✨ @tboeghk ⚡ Data integrity

  35. @backstreetkiwi ✨ @tboeghk ⚡ mixed log handling reality sucks. again.

  36. @backstreetkiwi ✨ @tboeghk ⚡ log governance Store everything. Find nothing?

  37. @backstreetkiwi ✨ @tboeghk Demo Examples & Helpers @ o12stack.org Photo

    by Federica Galli on Unsplash
  38. @backstreetkiwi ✨ @tboeghk Logging is fun fun fun Photo by

    Courtney Hedger on Unsplash