Centralized Logging Patterns

Most organizations feel the need to centralize their logs — once you have more than a couple of servers or containers, SSH and tail will not serve you well any more. However, the common question or struggle is how to achieve that.

This talk presents multiple approaches and patterns with their advantages and disadvantages, so you can pick the one that fits your organization best:
* Parse: Take the log files of your applications and extract the relevant pieces of information.
* Send: Add a log appender to send out your events directly without persisting them to a log file.
* Structure: Write your events in a structured file, which you can then centralize.
* Containerize: Keep track of short lived containers and configure their logging correctly.
* Orchestrate: Stay on top of your logs even when services are short lived and dynamically allocated on Kubernetes.

Each pattern has its own demo with the open source Elastic Stack (previously called ELK Stack), so you can easily try out the different approaches in your environment. Though the general patterns are applicable with any centralized logging system.

Transcript

1. Centralized
   Logging Patterns
   Philipp Krenn̴̴̴̴̴@xeraa
   ̴̴@xeraa
   

   View Slide

2. ̴̴@xeraa
   

   View Slide

3. ̴̴@xeraa
   

   View Slide

4. ̴̴@xeraa
   

   View Slide

5. ̴̴@xeraa
   

   View Slide

6. ̴̴@xeraa
   

   View Slide

7. ̴̴@xeraa
   

   View Slide

8. ̴̴@xeraa
   

   View Slide

9. View Slide

10. ̴̴@xeraa
    

    View Slide

11. View Slide

12. Developer
    ̴̴@xeraa
    

    View Slide

13. ̴̴@xeraa
    

    View Slide

14. ̴̴@xeraa
    

    View Slide

15. ̴̴@xeraa
    

    View Slide

16. ̴̴@xeraa
    

    View Slide

17. ̴̴@xeraa
    

    View Slide

18. View Slide

19. Disclaimer
    I build highly monitored Hello World
    apps
    ̴̴@xeraa
    

    View Slide

20. Example: Java
    SLF4J, Logback, MDC
    with logstash-logback-encoder
    Alternative https://github.com/vy/log4j2-logstash-layout
    ̴̴@xeraa
    

    View Slide

21. And Everywhere Else
    .NET: NLog
    JavaScript: Winston
    Python: structlog
    PHP: Monolog
    ̴̴@xeraa
    

    View Slide

22. Anti-Pattern: print
    System.out.println("Oops");
    ̴̴@xeraa
    

    View Slide

23. Anti-Pattern: Coupling
    ̴̴@xeraa
    

    View Slide

24. Parse ̴̴
    ̴̴@xeraa
    

    View Slide

25. ̴̴@xeraa
    

    View Slide

26. Collect Log Lines
    filebeat.inputs:
    - type: log
    paths:
    - /mnt/logs/*.log
    ̴̴@xeraa
    

    View Slide

27. View Slide

28. Grok
    https://github.com/logstash-plugins/logstash-patterns-core/blob/
    master/patterns/grok-patterns
    ̴̴@xeraa
    

    View Slide

29. Dev Tools
    Grok Debugger
    ̴̴@xeraa
    

    View Slide

30. [2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] -
    user_experience= , session=46, loop=15 -
    Wake me up at night
    java.lang.RuntimeException: Bad runtime...
    at net.xeraa.logging.LogMe.main(LogMe.java:30)
    ^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level}
    %{SPACE}%{USERNAME:logger}%{SPACE}\[%{WORD:thread}\]
    %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE}
    %{GREEDYDATA:themessage}(?:\n+(?(?:.|\r|\n)+))?
    ̴̴@xeraa
    

    View Slide

31. Machine Learning
    Data Visualizer
    ̴̴@xeraa
    

    View Slide

32. Logstash Key Value Filter for MDC
    kv {
    source => "labels"
    field_split => ","
    trim_key => " "
    }
    ̴̴@xeraa
    

    View Slide

33. Pro: No change
    Con: Regular expression, multiline,
    format changes
    ̴̴@xeraa
    

    View Slide

34. Send ̴̴
    ̴̴@xeraa
    

    View Slide

35. ̴̴@xeraa
    

    View Slide

36. logback.xml
    
    logstash:4560
    
    
    ̴̴@xeraa
    

    View Slide

37. Pro: No files
    Con: Outages & coupling
    ̴̴@xeraa
    

    View Slide

38. Structure ̴̴
    ̴̴@xeraa
    

    View Slide

39. ̴̴@xeraa
    

    View Slide

40. Collect JSON
    filebeat.input:
    - type: log
    paths:
    - /mnt/logs/*.json
    json:
    message_key: message
    keys_under_root: true
    ̴̴@xeraa
    

    View Slide

41. Stack(trace) Hash
    ̴̴@xeraa
    

    View Slide

42. Pro: Right format
    Con: JSON serialization overhead
    ̴̴@xeraa
    

    View Slide

43. Containerize ̴̴
    ̴̴@xeraa
    

    View Slide

44. ̴̴@xeraa
    

    View Slide

45. Where to put Filebeat?
    Sidecar
    ̴̴@xeraa
    

    View Slide

46. ̴̴@xeraa
    

    View Slide

47. https://github.com/elastic/beats/tree/
    master/deploy/docker
    ̴̴@xeraa
    

    View Slide

48. Docker Logs
    filebeat.autodiscover:
    providers:
    - type: docker
    hints.enabled: true
    processors:
    - add_docker_metadata: ~
    ̴̴@xeraa
    

    View Slide

49. Metadata
    No Docker metadata with the other methods
    {
    "docker": {
    "container": {
    "image": "java-logging_java_app",
    "labels": {
    "com": {
    "docker": {
    "compose": {
    "container-number": "1",
    "project": "java-logging",
    "service": "java_app",
    "version": "1.23.2",
    "oneoff": "False",
    "config-hash": "2b38df3c73c6 1a68a37443c2006f3f3e4fc16c3c 2a1d7793f2a38841e274b607"
    }
    }
    },
    "app": "fizzbuzz"
    },
    "id": "9d6d5a7640a457a1e08c422cb0a08 f96ff3631fb5356f749b2ac7d8f3719687f" ,
    "name": "java_app"
    }
    }
    }
    ̴̴@xeraa
    

    View Slide

50. Missing the Last Line
    Waiting for the newline
    ̴̴@xeraa
    

    View Slide

51. Hints
    labels:
    - "app=fizzbuzz"
    - "co.elastic.logs/multiline.pattern=^\\["
    - "co.elastic.logs/multiline.negate=true"
    - "co.elastic.logs/multiline.match=after"
    ̴̴@xeraa
    

    View Slide

52. Registry File
    filebeat.registry_file: /usr/share/filebeat/data/registry
    ̴̴@xeraa
    

    View Slide

53. Ingest Pipeline
    output.elasticsearch:
    hosts: ["http://elasticsearch:9200"]
    index: "docker"
    pipelines:
    - pipeline: "parse_java"
    when.contains:
    docker.container.name: "java_app"
    ̴̴@xeraa
    

    View Slide

54. Ingest Pipeline
    {
    "description" : "Parse Java log lines",
    "processors": [
    {
    "grok": {
    "field": "message",
    "patterns": [ "^\\[%{TIMESTAMP_ISO8601:timestamp}\\]%{SPACE}%{LOGLEVEL:log.level}
    %{SPACE}%{USERNAME:log.package}%{SPACE}\\[%{WORD:log.method}\\]%{SPACE}-
    %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed}
    (?:\\n+(?(?:.|\\r|\\n)+))?" ],
    "ignore_failure": true
    }
    }
    ]
    }
    ̴̴@xeraa
    

    View Slide

55. ASCII Art
    _._
    _.-``__ ''-._
    _.-`` `. `_. ''-._ Redis 4.0.9 (00000000/0) 64 bit
    .-`` .-```. ```\/ _.,_ ''-._
    ( ' , .-` | `, ) Running in stand alone mode
    |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
    | `-._ `._ / _.-' | PID: 55757
    `-._ `-._ `-./ _.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' | http://redis.io
    `-._ `-._`-.__.-'_.-' _.-'
    |`-._`-._ `-.__.-' _.-'_.-'|
    | `-._`-._ _.-'_.-' |
    `-._ `-._`-.__.-'_.-' _.-'
    `-._ `-.__.-' _.-'
    `-._ _.-'
    `-.__.-'
    ̴̴@xeraa
    

    View Slide

56. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: docker
    templates:
    - condition:
    equals:
    docker.container.image: redis
    config:
    - type: docker
    containers.ids:
    - "${data.docker.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa
    

    View Slide

57. Who Logs the Logger
    Avoid loops
    Process without -e
    filebeat.yml: logging.to_files: true
    ̴̴@xeraa
    

    View Slide

58. Pro: Hot
    Con: Complexity
    ̴̴@xeraa
    

    View Slide

59. Orchestrate ̴̴
    ̴̴@xeraa
    

    View Slide

60. ̴̴@xeraa
    

    View Slide

61. Where to put Filebeat?
    DaemonSet
    ̴̴@xeraa
    

    View Slide

62. https://github.com/elastic/beats/tree/
    master/deploy/kubernetes
    ̴̴@xeraa
    

    View Slide

63. Metadata
    Either in cluster or not
    processors:
    - add_kubernetes_metadata:
    in_cluster: true
    - add_kubernetes_metadata:
    in_cluster: false
    host:
    kube_config: ${HOME}/.kube/config
    ̴̴@xeraa
    

    View Slide

64. Metadata
    {
    "host": "172.17.0.21",
    "port": 9090,
    "kubernetes": {
    "container": {
    "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51",
    "image": "my-java:1.0.0",
    "name": "my-java"
    },
    "labels": {
    "app": "java",
    },
    "namespace": "default",
    "node": {
    "name": "minikube"
    },
    "pod": {
    "name": "java-2657348378-k1pnh"
    }
    },
    }
    ̴̴@xeraa
    

    View Slide

65. Configuration Templates
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    templates:
    - condition:
    equals:
    kubernetes.namespace: redis
    config:
    - type: docker
    containers.ids:
    - "${data.kubernetes.container.id}"
    exclude_lines: ["^\\s+[\\-`('.|_]"]
    ̴̴@xeraa
    

    View Slide

66. Customize Indices
    output.elasticsearch:
    index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
    ̴̴@xeraa
    

    View Slide

67. Pro: Hot
    Con: Complexity++
    ̴̴@xeraa
    

    View Slide

68. Moar ̴̴̴
    ̴
    ̴̴@xeraa
    

    View Slide

69. Index Patterns
    Time based (default: daily)
    Versioned
    ̴̴@xeraa
    

    View Slide

70. Sizing
    Daily volume * Retention * Replication
    Number of shards
    ̴̴@xeraa
    

    View Slide

71. Index Lifecycle Management
    ! "
    ̴̴@xeraa
    

    View Slide

72. Frozen Indices
    https://www.elastic.co/guide/en/elasticsearch/reference/6.6/
    frozen-indices.html
    ̴̴@xeraa
    

    View Slide

73. Conclusion̴̴
    ̴̴@xeraa
    

    View Slide

74. Examples
    https://github.com/xeraa/java-logging
    ̴̴@xeraa
    

    View Slide

75. Parse
    Send
    Structure
    Containerize
    Orchestrate
    ̴̴@xeraa
    

    View Slide

76. Questions?̴̴
    Philipp Krenn̴̴̴̴̴@xeraa
    ̴̴@xeraa
    

    View Slide