Getting Started with Logging in Kubernetes

A good practice when deploying applications in Kubernetes is to set proper instrumentation to gather insights and solve general monitoring needs. Logging is a fundamental piece of the instrumentation cycle and is continually evolving to solve pains associated with unstructured formats, performance and monitoring.

In this presentation you will learn the concepts involved in log processing for containerized applications. You will also be introduced to these hot new features in Logging: metering the logging pipeline with Prometheus, performance improvements, scalability and the ability to customize the log processor behavior through declarative resource annotations.

Transcript

1. KubeCon Europe!, Copenhagen 2018 Eduardo Silva <eduardo@treasure-data.com> @edsiper Getting Started with Logging in Kubernetes
2. /about • Open Source Software Engineer at Treasure Data • Maintainer of Fluent Bit (a Fluentd ecosystem project) #Logging #Kubernetes #Blockchain #Linux #Scalability /about
3. Applications & Logging
4. /about Logging / File System Log File Logging Writing messages to a log file Application
5. /about Standard I/O Streams STDIN STDOUT STDERR STDOUT STDERR Application
6. /about Standard I/O Streams Logging STDIN STDOUT STDERR Application
7. Logging & Docker Containers
8. /about Logging & Docker Containers STDOUT Application Hey KubeCon! Application writes a message to the standard output interface
9. /about Logging & Docker Containers Message is encapsulated in a JSON Map By Docker Engine Hey KubeCon!
10. /about Logging & Docker Containers JSON Map string is appended to the Container Log File /var/lib/docker/containers/df0ac7aee...48a3d8df/df0ac7aee...48a3d8df-json.log
11. /about Logging & Docker Containers Operational perspective • Discover Containers log files • Read each log entry = read each JSON Map • Read content of Log key message • Append metadata (container name/id)
12. Demo #1: Docker Log Streams
13. Applications in Kubernetes (Quick Overview)
14. Application /about Application in Kubernetes
15. Application Container /about Application in Kubernetes
16. Application Container 1 Application Container N Pod /about Application in Kubernetes
17. Pod Pod Node /about Application in Kubernetes Application Container 1 Container N Application Application Container 1 Container N Application
18. Node 1 Node 2 Node 3 Node N Master /about Application in Kubernetes
19. Logging Context in Kubernetes
20. /about Logging & Kubernetes In addition to the container context we have: • Pod name and Pod ID • Namespace • Node (Host) • Labels • Annotations
21. /about Logging & Kubernetes Log Processor • Container name and Container ID • Pod name and Pod ID • Namespace • Node (Host) • Labels • Annotations Context
22. /about Logging & Kubernetes Log Processor • Container name and Container ID • Pod name and Pod ID • Namespace • Node (Host) • Labels • Annotations Context File System / Journald
23. Kubernetes API Server /about Logging & Kubernetes Log Processor • Container name and Container ID • Pod name and Pod ID • Namespace • Node (Host) • Labels • Annotations Context File System / Journald
24. Kubernetes API Server /about Logging & Kubernetes Log Processor Correlate Context File System / Journald
25. Kubernetes API Server /about Logging & Kubernetes File System / Journald Logs Source Log Processor Storage
26. Log Processing in Kubernetes
27. /about Logging & Kubernetes Node Master /var/log/containers/* Fluent Bit Pod Pod 1 Pod N Run Log Processor as a DaemonSet
28. /about Logging & Kubernetes Node Master /var/log/containers/* Fluent Bit Pod Pod 1 Pod N Read Logs from the Filesystem (or Journald) Read Logs
29. /about Logging & Kubernetes Node Master /var/log/containers/* Fluent Bit Pod Pod 1 Pod N Lookup for Metadata Associated with each Pod Read Logs Lookup Metadata
30. Demo #2: Simple Pod and Log Visualization
31. Fluent Ecosystem
32. /about Ecosystem
33. /about Fluent Ecosystem
34. /about What’s new ? • Annotated Pods • Metrics: JSON & Prometheus • Throttle Filter • Enterprise Connectors
35. /about Fluent Bit v0.13 Annotated Pods • fluentbit.io/parser suggest a parser • fluentbit.io/exclude suggest to exclude logs
36. /about Fluent Bit v0.13 Annotated Pods apiVersion: v1 kind: Pod metadata: name: apache-logs labels: app: apache-logs annotations: fluentbit.io/parser: apache spec: containers: - name: apache image: edsiper/apache_logs Suggest a Parser
37. /about Fluent Bit v0.13 Built-in Metrics (JSON & Prometheus)
38. Demo #3: Fluent Bit v0.13
39. /about Stats • 74 Releases • 32 Contributors in 3 years • > 50k Docker Pulls (downloads) per day! • > 3M downloads in the last year Fluent Bit Status
40. /about Stats Fluent Bit Adoption (in the last 11 months)
41. /about Roadmap / Join us! We are just starting v0.14 development! • Features ◦ Load Balancing for output plugins ◦ Nested Multiline in Docker JSON logs • We need help with ◦ Documentation ◦ Unit tests, CI/CD ◦ Multiarch Docker images
42. /about Getting Started with Logging in Kubernetes Thanks! Eduardo Silva <eduardo@treasure-data.com> @edsiper fluent/fluent-bit fluentbit.io