Log Aggregation - Using Fluentd + Elasticsearch + Kibana

Ruby Tuesday #32 @Taiwan

Transcript

1. Log Aggregation Using Fluentd + Elasticsearch + Kibana❤️ hsatac@KKTIX, 2014 1
2. Ash Wu hSATAC hsatac@KKTIX, 2014 2
3. hsatac@KKTIX, 2014 3
4. Issues to Solve • Log files are scattered across servers. • grep & zgrep through all the mess. • Servers come and go(autoscaling) and we lose log data. • Keep logs of last 6 months. => Querying & Archiving hsatac@KKTIX, 2014 4
5. Fluentd Comes to Rescue hsatac@KKTIX, 2014 5
6. Fluentd • Data collector • Open sourced • Written in Ruby (Easy to write plugins) • Combined C language and Ruby for performance • Process 13,000 events/second/core • Simple, flexible and reliable. hsatac@KKTIX, 2014 6
7. Before hsatac@KKTIX, 2014 7
8. After hsatac@KKTIX, 2014 8
9. No Fixed 2-tier Setup • Where's the server? Where's the clients? • A fluentd intance is a node. • Each node and be a server, or a client, or both at the same time. • Define the data flow and servers topology as you like. hsatac@KKTIX, 2014 9
10. Data Flow Input & Output hsatac@KKTIX, 2014 10
11. Input • in_forward (listen on udp & tcp) • in_http • in_tail (tail a regular file) • in_exec • in_syslog hsatac@KKTIX, 2014 11
12. Output • out_file • out_foward (Send to other fluentd instance) • out_exec • out_copy • out_stdout • out_s3 hsatac@KKTIX, 2014 12
13. Collector Example hsatac@KKTIX, 2014 13
14. Buffered Output e.g. out_file, out_forward Events will be flushed when: • Events chunk size exceeds buffer_chunk_limit • Time limie flush_interval hsatac@KKTIX, 2014 14
15. hsatac@KKTIX, 2014 15
16. Time Sliced Plugin • Flush regularly (daily, hourly...) according to time_slice_format. • e.g. %Y%m%d for daily chunk • Just like logrotate hsatac@KKTIX, 2014 16
17. Station Example hsatac@KKTIX, 2014 17
18. Overview hsatac@KKTIX, 2014 18
19. Overview hsatac@KKTIX, 2014 19
20. Or... hsatac@KKTIX, 2014 20
21. Or... hsatac@KKTIX, 2014 21
22. Or... hsatac@KKTIX, 2014 22
23. Elasticsearch & Kibana • Elasticsearch • Distributed restful search and analytics (schema free) • Kibana • Realtime query frontend • Data visualizer • No code required hsatac@KKTIX, 2014 23
24. Kibana Basics • Query & filters • Panels (charts) hsatac@KKTIX, 2014 24
25. hsatac@KKTIX, 2014 25
26. hsatac@KKTIX, 2014 26
27. hsatac@KKTIX, 2014 27
28. hsatac@KKTIX, 2014 28
29. hsatac@KKTIX, 2014 29
30. hsatac@KKTIX, 2014 30
31. hsatac@KKTIX, 2014 31
32. hsatac@KKTIX, 2014 32
33. hsatac@KKTIX, 2014 33
34. Rails & Fluentd Integration hsatac@KKTIX, 2014 34
35. Attempt 1 Official Document hsatac@KKTIX, 2014 35
36. Official Document http://www.fluentd.org/datasources/rails • Rails side: • lograge gem (supress log) • act-fluent-logger-rails send log to fluentd hsatac@KKTIX, 2014 36
37. Official Document http://www.fluentd.org/datasources/rails • Fluentd side: • fluent-plugin-parser plugin to parse JSON. • fluent-plugin-elasticsearch plugin send parsed data to elasticsearch. hsatac@KKTIX, 2014 37
38. FAIL hsatac@KKTIX, 2014 38
39. Official Document http://www.fluentd.org/datasources/rails • lograge is good for access log, but what about other logs? • act-fluent-logger-rails crashes puma • What's all the fuss about JSON encode / decode thing? hsatac@KKTIX, 2014 39
40. Attempt 2 hsatac@KKTIX, 2014 40
41. Attempt 2 • act-fluent-logger-rails only to keep all the logs. • Replace puma with unicorn to avoid threading issue. hsatac@KKTIX, 2014 41
42. FAIL hsatac@KKTIX, 2014 42
43. Attempt 2 • All logs are kept, but the format is not search friendly. 2013-01-18T15:04:50+09:00 foo { "messages":"Started GET \"/\" for 127.0.0.1 at 2013-01-18 15:04:49 +0900\n Processing by TopController#index as HTML\n Completed 200 OK in 635ms (Views: 479.3ms | ActiveRecord: 39.6ms)"], "level":"INFO" } • Unicorn return weird 404 with act-fluent-logger-rails gem under massive requests. hsatac@KKTIX, 2014 43
44. Logging should never interfere with the functionality of your application. — Jack Ma hsatac@KKTIX, 2014 44
45. I didn't say that. — Jack Ma hsatac@KKTIX, 2014 45
46. Attempt 3 New Strategy hsatac@KKTIX, 2014 46
47. Attempt 3 • Rails side: • Remove act-fluent-logger-rails, unstable. • Keep lograge gem, using Logstash formatter. • logstash-logger gem converts other logs into logstash format. • Output logs to production.log. hsatac@KKTIX, 2014 47
48. Attempt 3 • Fluentd side: • Tail production.log with format json • Forward to station fluentd. • Done. hsatac@KKTIX, 2014 48
49. Thanks Q & A hsatac@KKTIX, 2014 49