Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Centralized Logging Patterns

Centralized Logging Patterns

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
* Parse: Take the log files of your applications and extract the relevant pieces of information.
* Send: Add a log appender to send out your events directly without persisting them to a log file.
* Structure: Write your events in a structured file, which you can then centralize.
* Containerize: Keep track of short lived containers and configure their logging correctly.
* Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Philipp Krenn

April 04, 2019
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. Centralized
    Logging Patterns
    Philipp Krenn̴̴̴̴̴@xeraa
    ̴̴@xeraa

    View Slide

  2. ̴̴@xeraa

    View Slide

  3. ̴̴@xeraa

    View Slide

  4. ̴̴@xeraa

    View Slide

  5. ̴̴@xeraa

    View Slide

  6. ̴̴@xeraa

    View Slide

  7. ̴̴@xeraa

    View Slide

  8. ̴̴@xeraa

    View Slide

  9. View Slide

  10. ̴̴@xeraa

    View Slide

  11. View Slide

  12. Developer
    ̴̴@xeraa

    View Slide

  13. ̴̴@xeraa

    View Slide

  14. ̴̴@xeraa

    View Slide

  15. ̴̴@xeraa

    View Slide

  16. ̴̴@xeraa

    View Slide

  17. ̴̴@xeraa

    View Slide

  18. View Slide

  19. Disclaimer
    I build highly monitored Hello World
    apps
    ̴̴@xeraa

    View Slide

  20. Example: Java
    SLF4J, Logback, MDC
    with logstash-logback-encoder
    Alternative https://github.com/vy/log4j2-logstash-layout
    ̴̴@xeraa

    View Slide

  21. And Everywhere Else
    .NET: NLog
    JavaScript: Winston
    Python: structlog
    PHP: Monolog
    ̴̴@xeraa

    View Slide

  22. Anti-Pattern: print
    System.out.println("Oops");
    ̴̴@xeraa

    View Slide

  23. Anti-Pattern: Coupling
    ̴̴@xeraa

    View Slide

  24. Parse ̴̴
    ̴̴@xeraa

    View Slide

  25. ̴̴@xeraa

    View Slide

  26. Collect Log Lines
    filebeat.inputs:
    - type: log
    paths:
    - /mnt/logs/*.log
    ̴̴@xeraa

    View Slide

  27. View Slide

  28. Grok
    https://github.com/logstash-plugins/logstash-patterns-core/blob/
    master/patterns/grok-patterns
    ̴̴@xeraa

    View Slide

  29. Dev Tools
    Grok Debugger
    ̴̴@xeraa

    View Slide

  30. [2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] -
    user_experience= , session=46, loop=15 -
    Wake me up at night
    java.lang.RuntimeException: Bad runtime...
    at net.xeraa.logging.LogMe.main(LogMe.java:30)
    ^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level}
    %{SPACE}%{USERNAME:logger}%{SPACE}\[%{WORD:thread}\]
    %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE}
    %{GREEDYDATA:themessage}(?:\n+(?(?:.|\r|\n)+))?
    ̴̴@xeraa

    View Slide

  31. Machine Learning
    Data Visualizer
    ̴̴@xeraa

    View Slide

  32. Logstash Key Value Filter for MDC
    kv {
    source => "labels"
    field_split => ","
    trim_key => " "
    }
    ̴̴@xeraa

    View Slide

  33. Pro: No change
    Con: Regular expression, multiline,
    format changes
    ̴̴@xeraa

    View Slide

  34. Send ̴̴
    ̴̴@xeraa

    View Slide

  35. ̴̴@xeraa

    View Slide

  36. logback.xml

    logstash:4560


    ̴̴@xeraa

    View Slide

  37. Pro: No files
    Con: Outages & coupling
    ̴̴@xeraa

    View Slide

  38. Structure ̴̴
    ̴̴@xeraa

    View Slide

  39. ̴̴@xeraa

    View Slide

  40. Collect JSON
    filebeat.input:
    - type: log
    paths:
    - /mnt/logs/*.json
    json:
    message_key: message
    keys_under_root: true
    ̴̴@xeraa

    View Slide

  41. Stack(trace) Hash
    ̴̴@xeraa

    View Slide

  42. Pro: Right format
    Con: JSON serialization overhead
    ̴̴@xeraa

    View Slide

  43. Containerize ̴̴
    ̴̴@xeraa

    View Slide

  44. ̴̴@xeraa

    View Slide

  45. Where to put Filebeat?
    Sidecar
    ̴̴@xeraa

    View Slide

  46. ̴̴@xeraa

    View Slide

  47. https://github.com/elastic/beats/tree/
    master/deploy/docker
    ̴̴@xeraa

    View Slide

  48. Docker Logs
    filebeat.autodiscover:
    providers:
    - type: docker
    hints.enabled: true
    processors:
    - add_docker_metadata: ~
    ̴̴@xeraa

    View Slide

  49. Metadata
    No Docker metadata with the other methods
    {
    "docker": {
    "container": {
    "image": "java-logging_java_app",
    "labels": {
    "com": {
    "docker": {
    "compose": {
    "container-number": "1",
    "project": "java-logging",
    "service": "java_app",
    "version": "1.23.2",
    "oneoff": "False",
    "config-hash": "2b38df3c73c6 1a68a37443c2006f3f3e4fc16c3c 2a1d7793f2a38841e274b607"
    }
    }
    },
    "app": "fizzbuzz"
    },
    "id": "9d6d5a7640a457a1e08c422cb0a08 f96ff3631fb5356f749b2ac7d8f3719687f" ,
    "name": "java_app"
    }
    }
    }
    ̴̴@xeraa

    View Slide

  50. Missing the Last Line
    Waiting for the newline
    ̴̴@xeraa

    View Slide

  51. Hints
    labels:
    - "app=fizzbuzz"
    - "co.elastic.logs/multiline.pattern=^\\["
    - "co.elastic.logs/multiline.negate=true"
    - "co.elastic.logs/multiline.match=after"
    ̴̴@xeraa

    View Slide

  52. Registry File
    filebeat.registry_file: /usr/share/filebeat/data/registry
    ̴̴@xeraa

    View Slide

  53. Ingest Pipeline
    output.elasticsearch:
    hosts: ["http://elasticsearch:9200"]
    index: "docker"
    pipelines:
    - pipeline: "parse_java"
    when.contains:
    docker.container.name: "java_app"
    ̴̴@xeraa

    View Slide

  54. Ingest Pipeline
    {
    "description" : "Parse Java log lines",
    "processors": [
    {
    "grok": {
    "field": "message",
    "patterns": [ "^\\[%{TIMESTAMP_ISO8601:timestamp}\\]%{SPACE}%{LOGLEVEL:log.level}
    %{SPACE}%{USERNAME:log.package}%{SPACE}\\[%{WORD:log.method}\\]%{SPACE}-
    %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed}
    (?:\\n+(?(?:.|\\r|\\n)+))?" ],
    "ignore_failure": true
    }
    }
    ]
    }
    ̴̴@xeraa

    View Slide

  55. ASCII Art
    _._
    _.-``__ ''-._
    _.-`` `. `_. ''-._ Redis 4.0.9 (00000000/0) 64 bit
    .-`` .-```. ```\/ _.,_ ''-._
    ( ' , .-` | `, ) Running in stand alone mode
    |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
    | `-._ `._ / _.-' | PID: 55757
    `-._ `-._ `-./ _.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' | http://redis.io
    `-._ `-._`-.__.-'_.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' |
    `-._ `-._`-.__.-'_.-' _.-'
    `-._ `-.__.-' _.-'
    `-._ _.-'
    `-.__.-'
    ̴̴@xeraa

    View Slide

  56. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: docker
    templates:
    - condition:
    equals:
    docker.container.image: redis
    config:
    - type: docker
    containers.ids:
    - "${data.docker.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa

    View Slide

  57. Who Logs the Logger
    Avoid loops
    Process without -e
    filebeat.yml: logging.to_files: true
    ̴̴@xeraa

    View Slide

  58. Pro: Hot
    Con: Complexity
    ̴̴@xeraa

    View Slide

  59. Orchestrate ̴̴
    ̴̴@xeraa

    View Slide

  60. ̴̴@xeraa

    View Slide

  61. Where to put Filebeat?
    DaemonSet
    ̴̴@xeraa

    View Slide

  62. https://github.com/elastic/beats/tree/
    master/deploy/kubernetes
    ̴̴@xeraa

    View Slide

  63. Metadata
    Either in cluster or not
    processors:
    - add_kubernetes_metadata:
    in_cluster: true
    - add_kubernetes_metadata:
    in_cluster: false
    host:
    kube_config: ${HOME}/.kube/config
    ̴̴@xeraa

    View Slide

  64. Metadata
    {
    "host": "172.17.0.21",
    "port": 9090,
    "kubernetes": {
    "container": {
    "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51",
    "image": "my-java:1.0.0",
    "name": "my-java"
    },
    "labels": {
    "app": "java",
    },
    "namespace": "default",
    "node": {
    "name": "minikube"
    },
    "pod": {
    "name": "java-2657348378-k1pnh"
    }
    },
    }
    ̴̴@xeraa

    View Slide

  65. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    templates:
    - condition:
    equals:
    kubernetes.namespace: redis
    config:
    - type: docker
    containers.ids:
    - "${data.kubernetes.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa

    View Slide

  66. Customize Indices
    output.elasticsearch:
    index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
    ̴̴@xeraa

    View Slide

  67. Pro: Hot
    Con: Complexity++
    ̴̴@xeraa

    View Slide

  68. Moar ̴̴̴
    ̴
    ̴̴@xeraa

    View Slide

  69. Index Patterns
    Time based (default: daily)
    Versioned
    ̴̴@xeraa

    View Slide

  70. Sizing
    Daily volume * Retention * Replication
    Number of shards
    ̴̴@xeraa

    View Slide

  71. Index Lifecycle Management
    ! "
    ̴̴@xeraa

    View Slide

  72. Frozen Indices
    https://www.elastic.co/guide/en/elasticsearch/reference/6.6/
    frozen-indices.html
    ̴̴@xeraa

    View Slide

  73. Conclusion̴̴
    ̴̴@xeraa

    View Slide

  74. Examples
    https://github.com/xeraa/java-logging
    ̴̴@xeraa

    View Slide

  75. Parse
    Send
    Structure
    Containerize
    Orchestrate
    ̴̴@xeraa

    View Slide

  76. Questions?̴̴
    Philipp Krenn̴̴̴̴̴@xeraa
    ̴̴@xeraa

    View Slide