Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Centralized Logging Patterns

Centralized Logging Patterns

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
* Parse: Take the log files of your applications and extract the relevant pieces of information.
* Send: Add a log appender to send out your events directly without persisting them to a log file.
* Structure: Write your events in a structured file, which you can then centralize.
* Containerize: Keep track of short lived containers and configure their logging correctly.
* Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Philipp Krenn

April 04, 2019
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. Centralized
    Logging Patterns
    Philipp Krenn̴̴̴̴̴@xeraa
    ̴̴@xeraa

    View full-size slide

  2. Developer
    ̴̴@xeraa

    View full-size slide

  3. Disclaimer
    I build highly monitored Hello World
    apps
    ̴̴@xeraa

    View full-size slide

  4. Example: Java
    SLF4J, Logback, MDC
    with logstash-logback-encoder
    Alternative https://github.com/vy/log4j2-logstash-layout
    ̴̴@xeraa

    View full-size slide

  5. And Everywhere Else
    .NET: NLog
    JavaScript: Winston
    Python: structlog
    PHP: Monolog
    ̴̴@xeraa

    View full-size slide

  6. Anti-Pattern: print
    System.out.println("Oops");
    ̴̴@xeraa

    View full-size slide

  7. Anti-Pattern: Coupling
    ̴̴@xeraa

    View full-size slide

  8. Parse ̴̴
    ̴̴@xeraa

    View full-size slide

  9. Collect Log Lines
    filebeat.inputs:
    - type: log
    paths:
    - /mnt/logs/*.log
    ̴̴@xeraa

    View full-size slide

  10. Grok
    https://github.com/logstash-plugins/logstash-patterns-core/blob/
    master/patterns/grok-patterns
    ̴̴@xeraa

    View full-size slide

  11. Dev Tools
    Grok Debugger
    ̴̴@xeraa

    View full-size slide

  12. [2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] -
    user_experience= , session=46, loop=15 -
    Wake me up at night
    java.lang.RuntimeException: Bad runtime...
    at net.xeraa.logging.LogMe.main(LogMe.java:30)
    ^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level}
    %{SPACE}%{USERNAME:logger}%{SPACE}\[%{WORD:thread}\]
    %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE}
    %{GREEDYDATA:themessage}(?:\n+(?(?:.|\r|\n)+))?
    ̴̴@xeraa

    View full-size slide

  13. Machine Learning
    Data Visualizer
    ̴̴@xeraa

    View full-size slide

  14. Logstash Key Value Filter for MDC
    kv {
    source => "labels"
    field_split => ","
    trim_key => " "
    }
    ̴̴@xeraa

    View full-size slide

  15. Pro: No change
    Con: Regular expression, multiline,
    format changes
    ̴̴@xeraa

    View full-size slide

  16. Send ̴̴
    ̴̴@xeraa

    View full-size slide

  17. logback.xml

    logstash:4560


    ̴̴@xeraa

    View full-size slide

  18. Pro: No files
    Con: Outages & coupling
    ̴̴@xeraa

    View full-size slide

  19. Structure ̴̴
    ̴̴@xeraa

    View full-size slide

  20. Collect JSON
    filebeat.input:
    - type: log
    paths:
    - /mnt/logs/*.json
    json:
    message_key: message
    keys_under_root: true
    ̴̴@xeraa

    View full-size slide

  21. Stack(trace) Hash
    ̴̴@xeraa

    View full-size slide

  22. Pro: Right format
    Con: JSON serialization overhead
    ̴̴@xeraa

    View full-size slide

  23. Containerize ̴̴
    ̴̴@xeraa

    View full-size slide

  24. Where to put Filebeat?
    Sidecar
    ̴̴@xeraa

    View full-size slide

  25. https://github.com/elastic/beats/tree/
    master/deploy/docker
    ̴̴@xeraa

    View full-size slide

  26. Docker Logs
    filebeat.autodiscover:
    providers:
    - type: docker
    hints.enabled: true
    processors:
    - add_docker_metadata: ~
    ̴̴@xeraa

    View full-size slide

  27. Metadata
    No Docker metadata with the other methods
    {
    "docker": {
    "container": {
    "image": "java-logging_java_app",
    "labels": {
    "com": {
    "docker": {
    "compose": {
    "container-number": "1",
    "project": "java-logging",
    "service": "java_app",
    "version": "1.23.2",
    "oneoff": "False",
    "config-hash": "2b38df3c73c6 1a68a37443c2006f3f3e4fc16c3c 2a1d7793f2a38841e274b607"
    }
    }
    },
    "app": "fizzbuzz"
    },
    "id": "9d6d5a7640a457a1e08c422cb0a08 f96ff3631fb5356f749b2ac7d8f3719687f" ,
    "name": "java_app"
    }
    }
    }
    ̴̴@xeraa

    View full-size slide

  28. Missing the Last Line
    Waiting for the newline
    ̴̴@xeraa

    View full-size slide

  29. Hints
    labels:
    - "app=fizzbuzz"
    - "co.elastic.logs/multiline.pattern=^\\["
    - "co.elastic.logs/multiline.negate=true"
    - "co.elastic.logs/multiline.match=after"
    ̴̴@xeraa

    View full-size slide

  30. Registry File
    filebeat.registry_file: /usr/share/filebeat/data/registry
    ̴̴@xeraa

    View full-size slide

  31. Ingest Pipeline
    output.elasticsearch:
    hosts: ["http://elasticsearch:9200"]
    index: "docker"
    pipelines:
    - pipeline: "parse_java"
    when.contains:
    docker.container.name: "java_app"
    ̴̴@xeraa

    View full-size slide

  32. Ingest Pipeline
    {
    "description" : "Parse Java log lines",
    "processors": [
    {
    "grok": {
    "field": "message",
    "patterns": [ "^\\[%{TIMESTAMP_ISO8601:timestamp}\\]%{SPACE}%{LOGLEVEL:log.level}
    %{SPACE}%{USERNAME:log.package}%{SPACE}\\[%{WORD:log.method}\\]%{SPACE}-
    %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed}
    (?:\\n+(?(?:.|\\r|\\n)+))?" ],
    "ignore_failure": true
    }
    }
    ]
    }
    ̴̴@xeraa

    View full-size slide

  33. ASCII Art
    _._
    _.-``__ ''-._
    _.-`` `. `_. ''-._ Redis 4.0.9 (00000000/0) 64 bit
    .-`` .-```. ```\/ _.,_ ''-._
    ( ' , .-` | `, ) Running in stand alone mode
    |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
    | `-._ `._ / _.-' | PID: 55757
    `-._ `-._ `-./ _.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' | http://redis.io
    `-._ `-._`-.__.-'_.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' |
    `-._ `-._`-.__.-'_.-' _.-'
    `-._ `-.__.-' _.-'
    `-._ _.-'
    `-.__.-'
    ̴̴@xeraa

    View full-size slide

  34. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: docker
    templates:
    - condition:
    equals:
    docker.container.image: redis
    config:
    - type: docker
    containers.ids:
    - "${data.docker.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa

    View full-size slide

  35. Who Logs the Logger
    Avoid loops
    Process without -e
    filebeat.yml: logging.to_files: true
    ̴̴@xeraa

    View full-size slide

  36. Pro: Hot
    Con: Complexity
    ̴̴@xeraa

    View full-size slide

  37. Orchestrate ̴̴
    ̴̴@xeraa

    View full-size slide

  38. Where to put Filebeat?
    DaemonSet
    ̴̴@xeraa

    View full-size slide

  39. https://github.com/elastic/beats/tree/
    master/deploy/kubernetes
    ̴̴@xeraa

    View full-size slide

  40. Metadata
    Either in cluster or not
    processors:
    - add_kubernetes_metadata:
    in_cluster: true
    - add_kubernetes_metadata:
    in_cluster: false
    host:
    kube_config: ${HOME}/.kube/config
    ̴̴@xeraa

    View full-size slide

  41. Metadata
    {
    "host": "172.17.0.21",
    "port": 9090,
    "kubernetes": {
    "container": {
    "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51",
    "image": "my-java:1.0.0",
    "name": "my-java"
    },
    "labels": {
    "app": "java",
    },
    "namespace": "default",
    "node": {
    "name": "minikube"
    },
    "pod": {
    "name": "java-2657348378-k1pnh"
    }
    },
    }
    ̴̴@xeraa

    View full-size slide

  42. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    templates:
    - condition:
    equals:
    kubernetes.namespace: redis
    config:
    - type: docker
    containers.ids:
    - "${data.kubernetes.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa

    View full-size slide

  43. Customize Indices
    output.elasticsearch:
    index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
    ̴̴@xeraa

    View full-size slide

  44. Pro: Hot
    Con: Complexity++
    ̴̴@xeraa

    View full-size slide

  45. Moar ̴̴̴
    ̴
    ̴̴@xeraa

    View full-size slide

  46. Index Patterns
    Time based (default: daily)
    Versioned
    ̴̴@xeraa

    View full-size slide

  47. Sizing
    Daily volume * Retention * Replication
    Number of shards
    ̴̴@xeraa

    View full-size slide

  48. Index Lifecycle Management
    ! "
    ̴̴@xeraa

    View full-size slide

  49. Frozen Indices
    https://www.elastic.co/guide/en/elasticsearch/reference/6.6/
    frozen-indices.html
    ̴̴@xeraa

    View full-size slide

  50. Conclusion̴̴
    ̴̴@xeraa

    View full-size slide

  51. Examples
    https://github.com/xeraa/java-logging
    ̴̴@xeraa

    View full-size slide

  52. Parse
    Send
    Structure
    Containerize
    Orchestrate
    ̴̴@xeraa

    View full-size slide

  53. Questions?̴̴
    Philipp Krenn̴̴̴̴̴@xeraa
    ̴̴@xeraa

    View full-size slide