Shared Nothing Logging Infrastructure

Shared Nothing Logging Infrastructure

Der Betrieb einer Microservices- und/oder einer Containerinfrastruktur birgt im Logging – gelinde gesagt – Herausforderungen. Die Infrastruktur ist volatil, das Debugging einer Benutzersession über Logfiles eines einzigen Servers ist nicht mehr möglich. Eine zentrale Logging-Infrastruktur zum Einsammeln, Verarbeiten und Speichern von Logs muss her. Allerdings ist eine solche zentrale Logging-Infrastruktur nicht mit einer "Shared Nothing"-Architektur kompatibel. Konsequenterweise muss dort jedes Team seine eigene Logging-Infrastruktur betreiben. Doch dazu fehlen den meisten Teams Zeit und Wissen, weshalb der Architekturansatz hier bricht. Am Beispiel von Shopping24 zeigt dieser Talk, wie wir die Herausforderung mit Methoden der Softwareentwicklung gemeistert haben. Gemeinsam werden Schnittstellen definiert, es wird eine Default-Logging-Implementierung bereitgestellt, und die Teams können diese Implementierung nach Bedarf erweitern. Mit ein wenig Governance können wir dann sicherstellen, dass die Teams untereinander von neuen Features und Versionen profitieren.

A6bb61c55fa41db28e68cd476cb54ab9?s=128

Torsten Bøgh Köster

November 07, 2019
Tweet

Transcript

  1. @backstreetkiwi ✨ @tboeghk Shared nothing logging infrastructure Logging for containers

    and the cloud
  2. @backstreetkiwi ✨ @tboeghk Nikolaus Winter @backstreetkiwi #java #python #spring #ci

    #cd #observability
  3. @backstreetkiwi ✨ @tboeghk #search #solr #ansible #kubernetes #observability #photography Torsten

    Bøgh Köster @tboeghk
  4. @backstreetkiwi ✨ @tboeghk agenda Motivation Logging & Containers Default Logging-Stack

    Adaptions Pitfalls & Learnings Demo
  5. @backstreetkiwi ✨ @tboeghk (1) Motivation Photo by Kobu Agency on

    Unsplash
  6. @backstreetkiwi ✨ @tboeghk We build logging systems

  7. @backstreetkiwi ✨ @tboeghk shared nothing infrastructure self contained systems Photo

    by Edwin Andrade on Unsplash
  8. @backstreetkiwi ✨ @tboeghk self contained systems Popular in e-commerce architectures

    Photo by Edwin Andrade on Unsplash
  9. @backstreetkiwi ✨ @tboeghk Teams in self contained systems ... ...

    develop a system independently ... have individual preferences ... have administrative constraints
  10. @backstreetkiwi ✨ @tboeghk Operational excellence We are building ops infrastructure

    for a specific target audience: developers. Our solutions need to be easy to navigate, adapt and extend Photo by Arif Riyanto on Unsplash
  11. @backstreetkiwi ✨ @tboeghk Kubernetes primary target platform most services still

    run on "classic" systems Photo by Maximilian Weisbecker on Unsplash
  12. Cattle not pets, but for clusters Photo by Stijn te

    Strake on Unsplash
  13. @backstreetkiwi ✨ @tboeghk (2) Container and logging STDOUT as lowest

    common denominator Photo by Sculpture Qode on Unsplash
  14. @backstreetkiwi ✨ @tboeghk Kubernetes and logging there is no silver

    bullet Photo by Radek Grzybowski on Unsplash
  15. @backstreetkiwi ✨ @tboeghk pod observability via API is the key

    to logging and monitoring in Kubernetes Photo by Rishabh Varshney on Unsplash
  16. @backstreetkiwi ✨ @tboeghk (3) Our logging stack Introducing the ELG

    stack
  17. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing deployment

  18. @backstreetkiwi ✨ @tboeghk Default logging stack at scale // will

    it scale?
  19. @backstreetkiwi ✨ @tboeghk Roll-out? YAML? JSON? Helm? CRD (Operator)? Ansible?

  20. @backstreetkiwi ✨ @tboeghk Logging-Operator https://github.com/banzaicloud/logging-operator

  21. @backstreetkiwi ✨ @tboeghk Elastic-Operator

  22. @backstreetkiwi ✨ @tboeghk " "

  23. @backstreetkiwi ✨ @tboeghk (4) Classes of real- world adaptions (1)

    Preconfigured, selectable customizations (2) Team- and application specific extensions Photo by Nigel Tadyanehondo on Unsplash
  24. @backstreetkiwi ✨ @tboeghk Log flow control Use Kubernetes annotations to

    customize log processing
  25. @backstreetkiwi ✨ @tboeghk Log Flow Control using deployment annotations

  26. @backstreetkiwi ✨ @tboeghk Filter messages Control messages to collect

  27. @backstreetkiwi ✨ @tboeghk Fine grained parsing spring-boot (multi line)

  28. @backstreetkiwi ✨ @tboeghk Log enrichment HTTP elements

  29. @backstreetkiwi ✨ @tboeghk Log enrichment geoip lookup

  30. @backstreetkiwi ✨ @tboeghk Storage classes specify log retention time //

    sharding
  31. @backstreetkiwi ✨ @tboeghk Photo by Nigel Tadyanehondo on Unsplash Interfaces

    Use Kubernetes services as interface to swappable and hence highly customizable log processing implementations
  32. @backstreetkiwi ✨ @tboeghk Managing extensibility it's all about the interfaces

  33. @backstreetkiwi ✨ @tboeghk Default logging stack Fully shared nothing

  34. @backstreetkiwi ✨ @tboeghk Pimp my logstash branching off clickstream for

    machine learning
  35. @backstreetkiwi ✨ @tboeghk Serving team preferences Kibana as log ui

  36. @backstreetkiwi ✨ @tboeghk Centralized log storage Reducing total cost of

    ownership
  37. @backstreetkiwi ✨ @tboeghk Photo by Nigel Tadyanehondo on Unsplash (6)

    Lessons learned Challenges of large volume message handling apply.
  38. @backstreetkiwi ✨ @tboeghk ⚡ Data integrity

  39. @backstreetkiwi ✨ @tboeghk ⚡ mixed log handling reality sucks. again.

  40. @backstreetkiwi ✨ @tboeghk ⚡ log governance Store everything. Find nothing?

  41. @backstreetkiwi ✨ @tboeghk Demo Examples & Helpers @ o12stack.org Photo

    by Federica Galli on Unsplash
  42. @backstreetkiwi ✨ @tboeghk Logging is fun fun fun Photo by

    Courtney Hedger on Unsplash