Overcoming Logging Challenges for Cloud Native Applications at Scale

Logging is not that easy as it looks and when dealing with a Cloud Native environment where applications runs at scale and likely applications generate their own logging format, do log analysis becomes a challenge.

The following presentation was given at Open Source Summit Europe 2017 (Prague) and it have a strong focus on Kubernetes and Docker specifically.

Transcript

1. Open Source Summit Europe, Prague 2017 Eduardo Silva <eduardo@treasure-data.com> @edsiper Overcoming Logging Challenges for Cloud Native Apps
2. /about • SW Engineer at Treasure Data • Maintainer of Fluent Bit (a Fluentd ecosystem project) #C #Kubernetes #CloudNative #Linux #Performance #Scalability
3. Logging
4. Logging • Why ? ◦ Monitoring ◦ Troubleshooting
5. Logging • Standalone Applications ◦ Writes to a log file ◦ Writes to local logging service ◦ Writes to a remote logging service
6. Logging • Operational Perspective (reality) ◦ Multiple Applications ◦ Multiple Hosts ◦ Logging == Complexity
7. Distributed Systems
8. Distributed Systems • Considerations ◦ “Application” might be composed by multiple services ◦ Services might run in multiple Nodes/Hosts ◦ Applications & Services might scale (likely on demand)
9. Distributed Systems • From a Logging perspective ◦ Log Sources might be many ◦ Applications and Services have their own “Logging way” ◦ Logs requires identity: origin + metadata
10. Distributed Systems • What is Cloud Native ? ◦ “It’s about Scale and Resilience...” • Cloud Native Logging ? ◦ New challenges
11. Scale Applications Requires Scale Logging
12. Cloud Native Logging • Requirements ◦ Consume logs in many ways (sources) ◦ Structure log records (more than human readable) ◦ Enrich logs (identity) ◦ Deliver logs to a central or multiple places for Analysis
13. Cloud Native Logging • A Web Server log looks simple 195.168.207.127 - [16/Oct/2017 17:43:26 +0000] GET /documentation/0.12/installation/requirements.html HTTP/1.1 200 32062 195.168.207.127 - [16/Oct/2017 17:43:44 +0000] GET /documentation/0.12/configuration/schema.html HTTP/1.1 200 34046 186.15.28.177 - [16/Oct/2017 17:45:12 +0000] GET /stats/hub.fluent-bit.stats HTTP/1.1 200 2921 186.15.28.177 - [16/Oct/2017 17:45:13 +0000] GET /stats/hub.fluentd.stats HTTP/1.1 200 3406 195.168.207.127 - [16/Oct/2017 17:46:50 +0000] GET /documentation/0.12/configuration/file.html HTTP/1.1 200 36318 195.168.207.127 - [16/Oct/2017 17:47:21 +0000] GET /documentation/0.12/configuration/variables.html HTTP/1.1 200 32090 186.15.28.177 - [16/Oct/2017 17:49:54 +0000] GET /stats/hub.fluent-bit.stats HTTP/1.1 200 2921 186.15.28.177 - [16/Oct/2017 17:49:54 +0000] GET /stats/hub.fluentd.stats HTTP/1.1 200 3406 180.76.15.15 - [16/Oct/2017 17:54:16 +0000] GET /documentation/0.12/about/fluentd_and_fluentbit.html HTTP/1.1 200 32750
14. Cloud Native Logging • Analysis for Applications at Scale (simple?)
15. Cloud Native Logging • How to approach Data Analysis at Scale ? ◦ Different Sources / Applications ◦ Different log formats ◦ Different nodes/hosts
16. Logging Pipeline
17. Cloud Native / Logging Pipeline
18. Cloud Native / Logging Pipeline: Input
19. Cloud Native / Logging Pipeline: Parser
20. Cloud Native / Logging Pipeline: Filter
21. Cloud Native / Logging Pipeline: Buffer
22. Cloud Native / Logging Pipeline: Routing
23. Cloud Native / Logging Pipeline: Output
24. UnStructured Data
25. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206
26. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 IP Address
27. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Date and Time (timestamp)
28. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 HTTP Method
29. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 URI
30. 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File Protocol
31. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Status
32. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Bytes
33. Cloud Native Logging • Unstructured Log / Raw Text HTTP Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 IP Address Date and Time (timestamp) HTTP Method URI Protocol Status Bytes
34. Cloud Native Logging • Structure ?
35. Cloud Native Logging • Structured Logs ◦ Easy to query specific data ◦ Empower Filtering ◦ Analytics
36. Cloud Native Logging • Logging Pipeline • Aware of Distributed Applications • Strategies for Applications at Scale • Unstructured to Structured Logs • Centralize logs in Backends
37. Cloud Native Logging
38. Cloud Native Logging • CNCF Project • More than 700 plugins! • Reliable and Secure • Built-in Reliability • Written in Ruby + C
39. Fluentd Users
40. More than a Project… Logging Ecosystem
41. Cloud Native Logging • Feedback: Fluentd is a Cloud Native Project, but ◦ “Sometimes we need something lightweight” ◦ “Performance is good, but can we make it better ?” ◦ “I wish features A, B, C…”
42. Fluentd Ecosystem A new project was created
43. Cloud Native Logging fluent/fluent-bit
44. Cloud Native Logging
45. Cloud Native Logging • Fluent Bit Highlights ◦ Written in C language ◦ Pluggable Architecture (more than 35 plugins) ◦ Async I/O, Event Driven, SSL/TLS, Monitoring...
46. Cloud Native Logging • Some Plugins ◦ Input: Tail, Systemd, Syslog, Tcp, Metrics (cpu, mem, disk) ◦ Filters: Kubernetes, Parser, Record Modifier.. ◦ Output: Elasticsearch, InfluxDB, Kafka REST, Nats, HTTP
47. Cloud Native Logging • Fluent Bit Primary Focus
48. Kubernetes & Logging 101
49. Kubernetes & Logging 101 Application
50. Kubernetes & Logging 101 Application Container
51. Kubernetes & Logging 101 Application Container 1 Application Container N POD
52. Kubernetes & Logging 101 Application Container 1 Application Container N POD Application Container 1 Application Container N POD Node
53. Kubernetes & Logging 101 Node 1 Node 2 Node 3 Node N Master
54. Kubernetes & Logging 101 Node Master /var/log/containers/*
55. Kubernetes & Logging 101 Node Master /var/log/containers/* Read Logs
56. Kubernetes & Logging 101 Node Master /var/log/containers/* Lookup Metadata Read Logs
57. Kubernetes & Logging 101 Nodes Master /var/log/containers/* /var/log/containers/*
58. DEMO
59. • Kubernetes Cluster (Raspberry Pi) ◦ node0 : master / API Server ◦ node1 : worker node ◦ node2 : worker node ◦ node3 : worker node • PODS generate random data • Fluent Bit as DaemonSet • Data aggregated into Elasticsearch • Logs visualization with Kibana
60. Cloud Native Logging • Fluent Bit & Monitoring (v0.13 dev)
61. Cloud Native Logging Thanks! Eduardo Silva <eduardo@treasure-data.com> @edsiper fluent/fluent-bit fluentbit.io