Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Overcoming Logging Challenges for Cloud Native Applications at Scale

Overcoming Logging Challenges for Cloud Native Applications at Scale

Logging is not that easy as it looks and when dealing with a Cloud Native environment where applications runs at scale and likely applications generate their own logging format, do log analysis becomes a challenge.

The following presentation was given at Open Source Summit Europe 2017 (Prague) and it have a strong focus on Kubernetes and Docker specifically.

Eduardo Silva

October 23, 2017
Tweet

More Decks by Eduardo Silva

Other Decks in Technology

Transcript

  1. /about • SW Engineer at Treasure Data • Maintainer of

    Fluent Bit (a Fluentd ecosystem project) #C #Kubernetes #CloudNative #Linux #Performance #Scalability
  2. Logging • Standalone Applications ◦ Writes to a log file

    ◦ Writes to local logging service ◦ Writes to a remote logging service
  3. Distributed Systems • Considerations ◦ “Application” might be composed by

    multiple services ◦ Services might run in multiple Nodes/Hosts ◦ Applications & Services might scale (likely on demand)
  4. Distributed Systems • From a Logging perspective ◦ Log Sources

    might be many ◦ Applications and Services have their own “Logging way” ◦ Logs requires identity: origin + metadata
  5. Distributed Systems • What is Cloud Native ? ◦ “It’s

    about Scale and Resilience...” • Cloud Native Logging ? ◦ New challenges
  6. Cloud Native Logging • Requirements ◦ Consume logs in many

    ways (sources) ◦ Structure log records (more than human readable) ◦ Enrich logs (identity) ◦ Deliver logs to a central or multiple places for Analysis
  7. Cloud Native Logging • A Web Server log looks simple

    195.168.207.127 - [16/Oct/2017 17:43:26 +0000] GET /documentation/0.12/installation/requirements.html HTTP/1.1 200 32062 195.168.207.127 - [16/Oct/2017 17:43:44 +0000] GET /documentation/0.12/configuration/schema.html HTTP/1.1 200 34046 186.15.28.177 - [16/Oct/2017 17:45:12 +0000] GET /stats/hub.fluent-bit.stats HTTP/1.1 200 2921 186.15.28.177 - [16/Oct/2017 17:45:13 +0000] GET /stats/hub.fluentd.stats HTTP/1.1 200 3406 195.168.207.127 - [16/Oct/2017 17:46:50 +0000] GET /documentation/0.12/configuration/file.html HTTP/1.1 200 36318 195.168.207.127 - [16/Oct/2017 17:47:21 +0000] GET /documentation/0.12/configuration/variables.html HTTP/1.1 200 32090 186.15.28.177 - [16/Oct/2017 17:49:54 +0000] GET /stats/hub.fluent-bit.stats HTTP/1.1 200 2921 186.15.28.177 - [16/Oct/2017 17:49:54 +0000] GET /stats/hub.fluentd.stats HTTP/1.1 200 3406 180.76.15.15 - [16/Oct/2017 17:54:16 +0000] GET /documentation/0.12/about/fluentd_and_fluentbit.html HTTP/1.1 200 32750
  8. Cloud Native Logging • How to approach Data Analysis at

    Scale ? ◦ Different Sources / Applications ◦ Different log formats ◦ Different nodes/hosts
  9. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206
  10. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 IP Address
  11. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Date and Time (timestamp)
  12. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 HTTP Method
  13. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 URI
  14. 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Cloud

    Native Logging • Unstructured Log / Raw Text HTTP Server Log File Protocol
  15. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Status
  16. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 Bytes
  17. Cloud Native Logging • Unstructured Log / Raw Text HTTP

    Server Log File 195.168.207.127 - [16/Oct/2017 17:43:26] GET /docs/ HTTP/1.1 200 3206 IP Address Date and Time (timestamp) HTTP Method URI Protocol Status Bytes
  18. Cloud Native Logging • Structured Logs ◦ Easy to query

    specific data ◦ Empower Filtering ◦ Analytics
  19. Cloud Native Logging • Logging Pipeline • Aware of Distributed

    Applications • Strategies for Applications at Scale • Unstructured to Structured Logs • Centralize logs in Backends
  20. Cloud Native Logging • CNCF Project • More than 700

    plugins! • Reliable and Secure • Built-in Reliability • Written in Ruby + C
  21. Cloud Native Logging • Feedback: Fluentd is a Cloud Native

    Project, but ◦ “Sometimes we need something lightweight” ◦ “Performance is good, but can we make it better ?” ◦ “I wish features A, B, C…”
  22. Cloud Native Logging • Fluent Bit Highlights ◦ Written in

    C language ◦ Pluggable Architecture (more than 35 plugins) ◦ Async I/O, Event Driven, SSL/TLS, Monitoring...
  23. Cloud Native Logging • Some Plugins ◦ Input: Tail, Systemd,

    Syslog, Tcp, Metrics (cpu, mem, disk) ◦ Filters: Kubernetes, Parser, Record Modifier.. ◦ Output: Elasticsearch, InfluxDB, Kafka REST, Nats, HTTP
  24. Kubernetes & Logging 101 Application Container 1 Application Container N

    POD Application Container 1 Application Container N POD Node
  25. • Kubernetes Cluster (Raspberry Pi) ◦ node0 : master /

    API Server ◦ node1 : worker node ◦ node2 : worker node ◦ node3 : worker node • PODS generate random data • Fluent Bit as DaemonSet • Data aggregated into Elasticsearch • Logs visualization with Kibana