Tweet
Tweet

Slide 1

Slide 1 text

Centralized Logging Patterns Philipp Krenn̴̴̴̴̴@xeraa ̴̴@xeraa

Slide 2

Slide 2 text

̴̴@xeraa

Slide 3

Slide 3 text

̴̴@xeraa

Slide 4

Slide 4 text

̴̴@xeraa

Slide 5

Slide 5 text

̴̴@xeraa

Slide 6

Slide 6 text

̴̴@xeraa

Slide 7

Slide 7 text

̴̴@xeraa

Slide 8

Slide 8 text

̴̴@xeraa

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

̴̴@xeraa

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Developer ̴̴@xeraa

Slide 13

Slide 13 text

̴̴@xeraa

Slide 14

Slide 14 text

̴̴@xeraa

Slide 15

Slide 15 text

̴̴@xeraa

Slide 16

Slide 16 text

̴̴@xeraa

Slide 17

Slide 17 text

̴̴@xeraa

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Disclaimer I build highly monitored Hello World apps ̴̴@xeraa

Slide 20

Slide 20 text

Example: Java SLF4J, Logback, MDC with logstash-logback-encoder Alternative https://github.com/vy/log4j2-logstash-layout ̴̴@xeraa

Slide 21

Slide 21 text

And Everywhere Else .NET: NLog JavaScript: Winston Python: structlog PHP: Monolog ̴̴@xeraa

Slide 22

Slide 22 text

Anti-Pattern: print System.out.println("Oops"); ̴̴@xeraa

Slide 23

Slide 23 text

Anti-Pattern: Coupling ̴̴@xeraa

Slide 24

Slide 24 text

Parse ̴̴ ̴̴@xeraa

Slide 25

Slide 25 text

̴̴@xeraa

Slide 26

Slide 26 text

Collect Log Lines filebeat.inputs: - type: log paths: - /mnt/logs/*.log ̴̴@xeraa

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

Grok https://github.com/logstash-plugins/logstash-patterns-core/blob/ master/patterns/grok-patterns ̴̴@xeraa

Slide 29

Slide 29 text

Dev Tools Grok Debugger ̴̴@xeraa

Slide 30

Slide 30 text

[2018-09-28 10:30:38.516] ERROR net.xeraa.logging.LogMe [main] - user_experience= , session=46, loop=15 - Wake me up at night java.lang.RuntimeException: Bad runtime... at net.xeraa.logging.LogMe.main(LogMe.java:30) ^\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}%{LOGLEVEL:level} %{SPACE}%{USERNAME:logger}%{SPACE}\[%{WORD:thread}\] %{SPACE}-%{SPACE}%{GREEDYDATA:mdc}%{SPACE}-%{SPACE} %{GREEDYDATA:themessage}(?:\n+(?(?:.|\r|\n)+))? ̴̴@xeraa

Slide 31

Slide 31 text

Machine Learning Data Visualizer ̴̴@xeraa

Slide 32

Slide 32 text

Logstash Key Value Filter for MDC kv { source => "labels" field_split => "," trim_key => " " } ̴̴@xeraa

Slide 33

Slide 33 text

Pro: No change Con: Regular expression, multiline, format changes ̴̴@xeraa

Slide 34

Slide 34 text

Send ̴̴ ̴̴@xeraa

Slide 35

Slide 35 text

̴̴@xeraa

Slide 36

Slide 36 text

logback.xml logstash:4560 ̴̴@xeraa

Slide 37

Slide 37 text

Pro: No files Con: Outages & coupling ̴̴@xeraa

Slide 38

Slide 38 text

Structure ̴̴ ̴̴@xeraa

Slide 39

Slide 39 text

̴̴@xeraa

Slide 40

Slide 40 text

Collect JSON filebeat.input: - type: log paths: - /mnt/logs/*.json json: message_key: message keys_under_root: true ̴̴@xeraa

Slide 41

Slide 41 text

Stack(trace) Hash ̴̴@xeraa

Slide 42

Slide 42 text

Pro: Right format Con: JSON serialization overhead ̴̴@xeraa

Slide 43

Slide 43 text

Containerize ̴̴ ̴̴@xeraa

Slide 44

Slide 44 text

̴̴@xeraa

Slide 45

Slide 45 text

Where to put Filebeat? Sidecar ̴̴@xeraa

Slide 46

Slide 46 text

̴̴@xeraa

Slide 47

Slide 47 text

https://github.com/elastic/beats/tree/ master/deploy/docker ̴̴@xeraa

Slide 48

Slide 48 text

Docker Logs filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ ̴̴@xeraa

Slide 49

Slide 49 text

Metadata No Docker metadata with the other methods { "docker": { "container": { "image": "java-logging_java_app", "labels": { "com": { "docker": { "compose": { "container-number": "1", "project": "java-logging", "service": "java_app", "version": "1.23.2", "oneoff": "False", "config-hash": "2b38df3c73c6 1a68a37443c2006f3f3e4fc16c3c 2a1d7793f2a38841e274b607" } } }, "app": "fizzbuzz" }, "id": "9d6d5a7640a457a1e08c422cb0a08 f96ff3631fb5356f749b2ac7d8f3719687f" , "name": "java_app" } } } ̴̴@xeraa

Slide 50

Slide 50 text

Missing the Last Line Waiting for the newline ̴̴@xeraa

Slide 51

Slide 51 text

Hints labels: - "app=fizzbuzz" - "co.elastic.logs/multiline.pattern=^\\[" - "co.elastic.logs/multiline.negate=true" - "co.elastic.logs/multiline.match=after" ̴̴@xeraa

Slide 52

Slide 52 text

Registry File filebeat.registry_file: /usr/share/filebeat/data/registry ̴̴@xeraa

Slide 53

Slide 53 text

Ingest Pipeline output.elasticsearch: hosts: ["http://elasticsearch:9200"] index: "docker" pipelines: - pipeline: "parse_java" when.contains: docker.container.name: "java_app" ̴̴@xeraa

Slide 54

Slide 54 text

Ingest Pipeline { "description" : "Parse Java log lines", "processors": [ { "grok": { "field": "message", "patterns": [ "^\\[%{TIMESTAMP_ISO8601:timestamp}\\]%{SPACE}%{LOGLEVEL:log.level} %{SPACE}%{USERNAME:log.package}%{SPACE}\\[%{WORD:log.method}\\]%{SPACE}- %{SPACE}%{GREEDYDATA:labels}%{SPACE}-%{SPACE}%{GREEDYDATA:message_parsed} (?:\\n+(?(?:.|\\r|\\n)+))?" ], "ignore_failure": true } } ] } ̴̴@xeraa

Slide 55

Slide 55 text

ASCII Art _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 4.0.9 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 55757 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' ̴̴@xeraa

Slide 56

Slide 56 text

Configuration Templates filebeat.autodiscover: providers: - type: docker templates: - condition: equals: docker.container.image: redis config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ["^\\s+[\\-`('.|_]"] ̴̴@xeraa

Slide 57

Slide 57 text

Who Logs the Logger Avoid loops Process without -e filebeat.yml: logging.to_files: true ̴̴@xeraa

Slide 58

Slide 58 text

Pro: Hot Con: Complexity ̴̴@xeraa

Slide 59

Slide 59 text

Orchestrate ̴̴ ̴̴@xeraa

Slide 60

Slide 60 text

̴̴@xeraa

Slide 61

Slide 61 text

Where to put Filebeat? DaemonSet ̴̴@xeraa

Slide 62

Slide 62 text

https://github.com/elastic/beats/tree/ master/deploy/kubernetes ̴̴@xeraa

Slide 63

Slide 63 text

Metadata Either in cluster or not processors: - add_kubernetes_metadata: in_cluster: true - add_kubernetes_metadata: in_cluster: false host: kube_config: ${HOME}/.kube/config ̴̴@xeraa

Slide 64

Slide 64 text

Metadata { "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51", "image": "my-java:1.0.0", "name": "my-java" }, "labels": { "app": "java", }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "java-2657348378-k1pnh" } }, } ̴̴@xeraa

Slide 65

Slide 65 text

Configuration Templates filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: redis config: - type: docker containers.ids: - "${data.kubernetes.container.id}" exclude_lines: ["^\\s+[\\-`('.|_]"] ̴̴@xeraa

Slide 66

Slide 66 text

Customize Indices output.elasticsearch: index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}" ̴̴@xeraa

Slide 67

Slide 67 text

Pro: Hot Con: Complexity++ ̴̴@xeraa

Slide 68

Slide 68 text

Moar ̴̴̴ ̴ ̴̴@xeraa

Slide 69

Slide 69 text

Index Patterns Time based (default: daily) Versioned ̴̴@xeraa

Slide 70

Slide 70 text

Sizing Daily volume * Retention * Replication Number of shards ̴̴@xeraa

Slide 71

Slide 71 text

Index Lifecycle Management ! " ̴̴@xeraa

Slide 72

Slide 72 text

Frozen Indices https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ frozen-indices.html ̴̴@xeraa

Slide 73

Slide 73 text

Conclusion̴̴ ̴̴@xeraa

Slide 74

Slide 74 text

Examples https://github.com/xeraa/java-logging ̴̴@xeraa

Slide 75

Slide 75 text

Parse Send Structure Containerize Orchestrate ̴̴@xeraa

Slide 76

Slide 76 text

Questions?̴̴ Philipp Krenn̴̴̴̴̴@xeraa ̴̴@xeraa