Get the best out of Graylog2 & Icinga 2

Integrate Graylog2 and Icinga2. OSMC 2014

Transcript

1. Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2
2. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA BEFORE WE START …
3. Agenda
4. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA AGENDA • Introduction • Tools • Log History • Logs & Monitoring • Demo • „The Future“ • Resources • Q&A
5. Introduction
6. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA WHO‘S WHO Bernd Ahlers @berndahlers German, 34, Graylog2 Developer Graylog2 Team since 2014 Developer @ TORCH GmbH Michael Friedrich @dnsmichi Austrian, 31, Icinga Developer Icinga Team since May 2009 Application Developer @ NETWAYS
7. Tools: Graylog2
8. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Started as open source project by Lennart Koopmann in 2010 – Developed entirely in his free time – Free & open source log management tool • TORCH GmbH founded as company behind Graylog2 in late 2012 – after seeing massive growth and worldwide distribution in large scale setups • Team of 8 engineers working full-time on it
9. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Big rewrite of Graylog2 started in 2012 • Finished with releasing a final v0.20.0 in February 2014 • Addresses what we learnt from our first customers and all users • Unified REST API communication – easy extending and integrating with other products, tools and scripts • New web interface focusing on powerful analytics • Current stable version: 0.91.3
10. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2
11. Tools: Icinga 2
12. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 • Monitoring core engine – Checks, alerts, notifications – Backend interfaces for frontend visualization • Scalable for high performance & real-time monitoring – check_interval = 1s • Dynamic configuration format • Cluster & remote clients, SSL x509 & IPv4/6
13. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 • Modular feature set & connectors – DB IDO, Livestatus, Perfdata, Graphite, Gelf • Supports Monitoring Plugins API • Rewritten from scratch – Stable version: 2.2.0 (17.11.2014)
14. Log History
15. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Logs everywhere • How to collect them? – Splunk (4500$+ for 1GB/day) – Syslog-ng + Custom scripts • Purpose of your collection? – Regex for log parsing – Filters – Alerts? Notifications? Correlation? – Reporting • #devops Stack – Graylog2, Logstash (ELK) + $monitoring + $metrics + $cfgmgmt
16. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Problems with remote syslog checks – Failure: where‘s the context? – Pattern matching – Seek files (state history, rate calculation) – Configuration inside Icinga/Plugin • Collect them – Central log cluster (failover) – Correlate events from other servers – Defined streams and alert triggers – Defined input types (e.g. GELF) – Query alert API from Icinga
17. Logs & Monitoring
18. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG & MONITORING • Monitor your logs – Call check plugin or receive passive events – Generate alerts based on thresholds (configuration) – Notifications based on alerts – Visualize the current state & history for SLA reporting – Trigger event handlers (e.g. iptables on flood) • Popular plugins – check_logfiles – check_splunk • Collector APIs & Hooks – Graylog2 alert API & alert callback plugin – Logstash Nagios output
19. Logs & Monitoring: Strategy
20. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA STRATEGY • Out-of-the-box support or external addons? • Add hook to streams for passive event sending? • Query a defined API for alerts? • Visualize alerts, and where? (we want dashboards!) • Re-usable & customizable URL for notifications • Combine Log Events & Monitoring notifications and handlers
21. Logs & Monitoring: Push
22. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA PUSH: GRAYLOG2 ALARM CALLBACK • Requirements – Icinga API (Command Pipe) – Graylog2 Plugin Alarm Callback http://www.graylog2.org/resources/documentation/general/streams http://www.graylog2.org/resources/documentation/general/plugins • Ideas – Exec Callback+NSCA http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/ – (Ab)Use the notification plugin http://everythingshouldbevirtual.com/graylog2-streams-via-email – Custom Rake Plugin http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html • Solution – There is no simple & secure unified Core API (yet) – Use local Icinga2 client & poll check plugin instead
23. Logs & Monitoring: Poll
24. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK • Requirements – Graylog2 REST API – Icinga Check Plugin • Ideas – Wrapper for Python API calls? https://github.com/qmetric/graylog2-api-tools – Compile check_graylog2_stream? https://github.com/emind-systems/check_graylog2_stream • Solution – New Icinga Plugin by Graylog2 https://github.com/Graylog2/check-graylog2-stream
25. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK
26. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK # ./check-graylog2-stream usage: -condition="<ID>": Condition ID, set only to check a single alert (optional) -password="<password>": API password (mandatory) -stream="<ID>": Stream ID (mandatory) -url="http://localhost:12900": URL to Graylog2 api (optional) -user="<username>": API username (mandatory)
27. Combining Graylog2 & Icinga 2
28. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA COMBINING GRAYLOG2 & ICINGA 2 • Events triggered by Icinga 2 – Check results – State changes – Notifications • Sent to Graylog2 using `GelfWriter` feature # icinga2 feature enable gelf && service icinga2 restart • Visualize in Graylog2 – Filter based on type (e.g. state != OK) – Alert streams based on counts, etc
29. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA NOTIFICATIONS • „Default Monitoring Alerts are awful“ http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/ – You want to see what‘s wrong. No additional click on your mobile. • Icinga 2 triggers a notification – Fetch additional information from Graylog2 API – Include ‚notes_url‘ with stream id in notification • Requirements – Custom notification script – Stream ids as custom attributes – Icinga2 v2.2 Apply For Rules
30. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA MONITOR THE MONITORING CORE • Check Plugin – Query Graylog2 Alert Stream API for Icinga 2 alerts – Use Stream ID for notifications & notes_url • See what‘s happening in Icinga 2 – Restrict views based on user roles – Debug plugin & check problems – Combine cluster mal-function log – Filter events – Additional dashboard
31. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA GRAYLOG2: GELFWRITER VISUALIZED
32. Demo
33. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA DEMO • Graylog2 0.91.x • Icinga 2 2.2.0 • check-graylog2-stream Plugin • Configuration – Graylog2 icinga2 stream & alert – Icinga2 check plugin & host/service/notification apply rules
34. „The FUTURE“
35. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA „THE FUTURE“ • Build your own stack • Combine existing interfaces into one – Graylog2 streams in Icinga Web 2 (ask Tom!) – Icinga 2 Events in Graylog2 (more? We want more!) • Correlate your monitoring events with events & logs of any kind • Think about – Simple and secure event receiver – Auto-Discover checkable objects from log alerts – Alert stream rules for monitoring
36. RESOURCES
37. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA • Code https://github.com/graylog2 https://github.com/icinga/icinga2 • Vagrant Box icinga2x-graylog2 New @ https://github.com/icinga/icinga-vagrant/ • Documentation http://www.graylog2.org/resources/documentation http://docs.icinga.org/icinga2/latest
38. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA Q&A Web www.{graylog2,icinga}.org Releases github.com/{graylog2,Icinga} IRC #graylog2 #icinga on FreeNode Support support.{graylog2,icinga}.org Twitter twitter.com/{graylog2,icinga} …….. Everywhere! ? Questions & Answers