Monitoring with syslog-ng, Riemann and Kibana

Presented at OSDC 2014 in Berlin.

Transcript

1. Monitoring with syslog-ng, Riemann and Kibana @algernoone @balabit
2. syslog-ng
3. syslog-ng • Open source event processor and swiss army knife
4. syslog-ng • Open source event processor and swiss army knife • Developed since 1998, LGPL + GPL
5. syslog-ng • Open source event processor and swiss army knife • Developed since 1998, LGPL + GPL – (Commercial offering since 2007)
6. syslog-ng • Open source event processor and swiss army knife • Developed since 1998, LGPL + GPL – (Commercial offering since 2007) • Collects, parses, filters, transforms, transfers events
7. syslog-ng • Open source event processor and swiss army knife • Developed since 1998, LGPL + GPL – (Commercial offering since 2007) • Collects, parses, filters, transforms, transfers events • Wide variety of plugins
8. syslog-ng • Open source event processor and swiss army knife • Developed since 1998, LGPL + GPL – (Commercial offering since 2007) • Collects, parses, filters, transforms, transfers events • Wide variety of plugins • A sizable, helpful and very inclusive community
9. Riemann
10. Riemann • Riemann monitors distributed systems
11. Riemann • Riemann monitors distributed systems • Event aggregator with a powerful stream processing language
12. Riemann • Riemann monitors distributed systems • Event aggregator with a powerful stream processing language • Provides a low-latency, transient shared state
13. Kibana
14. Kibana • Visualize logs and time-stamped data
15. Kibana • Visualize logs and time-stamped data • Powerful search syntax
16. Kibana • Visualize logs and time-stamped data • Powerful search syntax • Flexible, powerful, yet intuitive interface
17. Kibana
18. Monitoring
19. Monitoring What
20. Monitoring • System state What
21. Monitoring • System state • Application state What
22. Monitoring • System state • Application state • Exceptions What
23. Monitoring • System state • Application state • Exceptions • Activity What
24. Monitoring Tools
25. Monitoring • Nagios • Collectd • Munin Tools
26. Monitoring • Nagios • Collectd • Munin • Riemann Tools
27. Logging
28. Logging
29. Logging • Persisting application state
30. Logging • Persisting application state • Format is usually application specific
31. Logging • Persisting application state • Format is usually application specific • Structured vs unstructured
32. Logging • Persisting application state • Format is usually application specific • Structured vs unstructured • Great source for monitoring too!
33. Monitoring + Logging
34. Monitoring + Logging • What do we already have?
35. Monitoring + Logging • What do we already have? • What can we add?
36. Monitoring + Logging • What do we already have? • What can we add? • How will we benefit from all of this?
37. Q & A
38. Thanks! • http://www.syslog-ng.org/ • https://github.com/balabit/syslog-ng-incubator • https://talien.blogs.balabit.com/ • https://algernon.blogs.balabit.com/