Centralized Logging Patterns

Centralized Logging Patterns

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
* Parse: Take the log files of your applications and extract the relevant pieces of information.
* Send: Add a log appender to send out your events directly without persisting them to a log file.
* Structure: Write your events in a structured file, which you can then centralize.
* Containerize: Keep track of short lived containers and configure their logging correctly.
* Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Ce4685da897c912aa41a815435b40a5a?s=128

Philipp Krenn

April 04, 2019
Tweet

Transcript

  1. Centralized Logging Patterns Philipp Krenn̴̴̴̴̴@xeraa ̴̴@xeraa

  2. ̴̴@xeraa

  3. ̴̴@xeraa

  4. ̴̴@xeraa

  5. ̴̴@xeraa

  6. ̴̴@xeraa

  7. ̴̴@xeraa

  8. ̴̴@xeraa

  9. None
  10. ̴̴@xeraa

  11. None
  12. Developer ̴̴@xeraa

  13. ̴̴@xeraa

  14. ̴̴@xeraa

  15. ̴̴@xeraa

  16. ̴̴@xeraa

  17. ̴̴@xeraa

  18. None
  19. Disclaimer I build highly monitored Hello World apps ̴̴@xeraa

  20. Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout ̴̴@xeraa

  21. And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP:

    Monolog ̴̴@xeraa
  22. Anti-Pattern: print System.out.println("Oops"); ̴̴@xeraa

  23. Anti-Pattern: Coupling ̴̴@xeraa

  24. Parse ̴̴ ̴̴@xeraa

  25. ̴̴@xeraa

  26. Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log

    ̴̴@xeraa
  27. None
  28. Grok https://github.com/logstash-plugins/logstash-patterns-core/blob/ master/patterns/grok-patterns ̴̴@xeraa

  29. Dev Tools Grok Debugger ̴̴@xeraa

  30. [2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] - user_experience= , session=46, loop=15

    - Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}\[%{WORD:thread}\] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?<stacktrace>(?:.|\r|\n)+))? ̴̴@xeraa
  31. Machine Learning Data Visualizer ̴̴@xeraa

  32. Logstash Key Value Filter for MDC kv { source =>

    "labels" field_split => "," trim_key => " " } ̴̴@xeraa
  33. Pro: No change Con: Regular expression, multiline, format changes ̴̴@xeraa

  34. Send ̴̴ ̴̴@xeraa

  35. ̴̴@xeraa

  36. logback.xml <appender name="logstash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender"> <destination>logstash:4560</destination> <encoder class="net.logstash.logback.encoder.LogstashEncoder"/> </appender> ̴̴@xeraa

  37. Pro: No files Con: Outages & coupling ̴̴@xeraa

  38. Structure ̴̴ ̴̴@xeraa

  39. ̴̴@xeraa

  40. Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json:

    message_key: message keys_under_root: true ̴̴@xeraa
  41. Stack(trace) Hash ̴̴@xeraa

  42. Pro: Right format Con: JSON serialization overhead ̴̴@xeraa

  43. Containerize ̴̴ ̴̴@xeraa

  44. ̴̴@xeraa

  45. Where to put Filebeat? Sidecar ̴̴@xeraa

  46. ̴̴@xeraa

  47. https://github.com/elastic/beats/tree/ master/deploy/docker ̴̴@xeraa

  48. Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors:

    - add_docker_metadata: ~ ̴̴@xeraa
  49. Metadata No Docker metadata with the other methods { "docker":

    { "container": { "image": "java-logging_java_app", "labels": { "com": { "docker": { "compose": { "container-number": "1", "project": "java-logging", "service": "java_app", "version": "1.23.2", "oneoff": "False", "config-hash": "2b38df3c73c6 1a68a37443c2006f3f3e4fc16c3c 2a1d7793f2a38841e274b607" } } }, "app": "fizzbuzz" }, "id": "9d6d5a7640a457a1e08c422cb0a08 f96ff3631fb5356f749b2ac7d8f3719687f" , "name": "java_app" } } } ̴̴@xeraa
  50. Missing the Last Line Waiting for the newline ̴̴@xeraa

  51. Hints labels: - "app=fizzbuzz" - "co.elastic.logs/multiline.pattern=^\\[" - "co.elastic.logs/multiline.negate=true" - "co.elastic.logs/multiline.match=after"

    ̴̴@xeraa
  52. Registry File filebeat.registry_file: /usr/share/filebeat/data/registry ̴̴@xeraa

  53. Ingest Pipeline output.elasticsearch: hosts: ["http://elasticsearch:9200"] index: "docker" pipelines: - pipeline:

    "parse_java" when.contains: docker.container.name: "java_app" ̴̴@xeraa
  54. Ingest Pipeline { "description" : "Parse Java log lines", "processors":

    [ { "grok": { "field": "message", "patterns": [ "^\\[%{TIMESTAMP_ISO8601:timestamp}\\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\\[%{WORD:log.method}\\]%{SPACE}- %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\\n+(?<stacktrace>(?:.|\\r|\\n)+))?" ], "ignore_failure": true } } ] } ̴̴@xeraa
  55. ASCII Art _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis

    4.0.9 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 55757 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' ̴̴@xeraa
  56. Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition:

    equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\\s+[\\-`('.|_]"] ̴̴@xeraa
  57. Who Logs the Logger Avoid loops Process without -e filebeat.yml:

    logging.to_files: true ̴̴@xeraa
  58. Pro: Hot Con: Complexity ̴̴@xeraa

  59. Orchestrate ̴̴ ̴̴@xeraa

  60. ̴̴@xeraa

  61. Where to put Filebeat? DaemonSet ̴̴@xeraa

  62. https://github.com/elastic/beats/tree/ master/deploy/kubernetes ̴̴@xeraa

  63. Metadata Either in cluster or not processors: - add_kubernetes_metadata: in_cluster:

    true - add_kubernetes_metadata: in_cluster: false host: <hostname> kube_config: ${HOME}/.kube/config ̴̴@xeraa
  64. Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": {

    "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } ̴̴@xeraa
  65. Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition:

    equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\\s+[\\-`('.|_]"] ̴̴@xeraa
  66. Customize Indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" ̴̴@xeraa

  67. Pro: Hot Con: Complexity++ ̴̴@xeraa

  68. Moar ̴̴̴ ̴ ̴̴@xeraa

  69. Index Patterns Time based (default: daily) Versioned ̴̴@xeraa

  70. Sizing Daily volume * Retention * Replication Number of shards

    ̴̴@xeraa
  71. Index Lifecycle Management ! " ̴̴@xeraa

  72. Frozen Indices https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ frozen-indices.html ̴̴@xeraa

  73. Conclusion̴̴ ̴̴@xeraa

  74. Examples https://github.com/xeraa/java-logging ̴̴@xeraa

  75. Parse Send Structure Containerize Orchestrate ̴̴@xeraa

  76. Questions?̴̴ Philipp Krenn̴̴̴̴̴@xeraa ̴̴@xeraa