Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Get the best out of Graylog2 & Icinga 2

A428c190cba8985990a410b0450b680f?s=47 Bernd Ahlers
November 20, 2014

Get the best out of Graylog2 & Icinga 2

Integrate Graylog2 and Icinga2. OSMC 2014

A428c190cba8985990a410b0450b680f?s=128

Bernd Ahlers

November 20, 2014
Tweet

More Decks by Bernd Ahlers

Other Decks in Programming

Transcript

  1. Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best

    out of Graylog2 & Icinga 2
  2. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA BEFORE WE START …

  3. Agenda

  4. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA AGENDA • Introduction •

    Tools • Log History • Logs & Monitoring • Demo • „The Future“ • Resources • Q&A
  5. Introduction

  6. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA WHO‘S WHO Bernd Ahlers

    @berndahlers German, 34, Graylog2 Developer Graylog2 Team since 2014 Developer @ TORCH GmbH Michael Friedrich @dnsmichi Austrian, 31, Icinga Developer Icinga Team since May 2009 Application Developer @ NETWAYS
  7. Tools: Graylog2

  8. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Started

    as open source project by Lennart Koopmann in 2010 – Developed entirely in his free time – Free & open source log management tool • TORCH GmbH founded as company behind Graylog2 in late 2012 – after seeing massive growth and worldwide distribution in large scale setups • Team of 8 engineers working full-time on it
  9. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Big

    rewrite of Graylog2 started in 2012 • Finished with releasing a final v0.20.0 in February 2014 • Addresses what we learnt from our first customers and all users • Unified REST API communication – easy extending and integrating with other products, tools and scripts • New web interface focusing on powerful analytics • Current stable version: 0.91.3
  10. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2

  11. Tools: Icinga 2

  12. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •

    Monitoring core engine – Checks, alerts, notifications – Backend interfaces for frontend visualization • Scalable for high performance & real-time monitoring – check_interval = 1s • Dynamic configuration format • Cluster & remote clients, SSL x509 & IPv4/6
  13. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •

    Modular feature set & connectors – DB IDO, Livestatus, Perfdata, Graphite, Gelf • Supports Monitoring Plugins API • Rewritten from scratch – Stable version: 2.2.0 (17.11.2014)
  14. Log History

  15. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Logs

    everywhere • How to collect them? – Splunk (4500$+ for 1GB/day) – Syslog-ng + Custom scripts • Purpose of your collection? – Regex for log parsing – Filters – Alerts? Notifications? Correlation? – Reporting • #devops Stack – Graylog2, Logstash (ELK) + $monitoring + $metrics + $cfgmgmt
  16. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Problems

    with remote syslog checks – Failure: where‘s the context? – Pattern matching – Seek files (state history, rate calculation) – Configuration inside Icinga/Plugin • Collect them – Central log cluster (failover) – Correlate events from other servers – Defined streams and alert triggers – Defined input types (e.g. GELF) – Query alert API from Icinga
  17. Logs & Monitoring

  18. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG & MONITORING •

    Monitor your logs – Call check plugin or receive passive events – Generate alerts based on thresholds (configuration) – Notifications based on alerts – Visualize the current state & history for SLA reporting – Trigger event handlers (e.g. iptables on flood) • Popular plugins – check_logfiles – check_splunk • Collector APIs & Hooks – Graylog2 alert API & alert callback plugin – Logstash Nagios output
  19. Logs & Monitoring: Strategy

  20. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA STRATEGY • Out-of-the-box support

    or external addons? • Add hook to streams for passive event sending? • Query a defined API for alerts? • Visualize alerts, and where? (we want dashboards!) • Re-usable & customizable URL for notifications • Combine Log Events & Monitoring notifications and handlers
  21. Logs & Monitoring: Push

  22. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA PUSH: GRAYLOG2 ALARM CALLBACK

    • Requirements – Icinga API (Command Pipe) – Graylog2 Plugin Alarm Callback http://www.graylog2.org/resources/documentation/general/streams http://www.graylog2.org/resources/documentation/general/plugins • Ideas – Exec Callback+NSCA http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/ – (Ab)Use the notification plugin http://everythingshouldbevirtual.com/graylog2-streams-via-email – Custom Rake Plugin http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html • Solution – There is no simple & secure unified Core API (yet) – Use local Icinga2 client & poll check plugin instead
  23. Logs & Monitoring: Poll

  24. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK •

    Requirements – Graylog2 REST API – Icinga Check Plugin • Ideas – Wrapper for Python API calls? https://github.com/qmetric/graylog2-api-tools – Compile check_graylog2_stream? https://github.com/emind-systems/check_graylog2_stream • Solution – New Icinga Plugin by Graylog2 https://github.com/Graylog2/check-graylog2-stream
  25. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK

  26. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK #

    ./check-graylog2-stream usage: -condition="<ID>": Condition ID, set only to check a single alert (optional) -password="<password>": API password (mandatory) -stream="<ID>": Stream ID (mandatory) -url="http://localhost:12900": URL to Graylog2 api (optional) -user="<username>": API username (mandatory)
  27. Combining Graylog2 & Icinga 2

  28. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA COMBINING GRAYLOG2 & ICINGA

    2 • Events triggered by Icinga 2 – Check results – State changes – Notifications • Sent to Graylog2 using `GelfWriter` feature # icinga2 feature enable gelf && service icinga2 restart • Visualize in Graylog2 – Filter based on type (e.g. state != OK) – Alert streams based on counts, etc
  29. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA NOTIFICATIONS • „Default Monitoring

    Alerts are awful“ http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/ – You want to see what‘s wrong. No additional click on your mobile. • Icinga 2 triggers a notification – Fetch additional information from Graylog2 API – Include ‚notes_url‘ with stream id in notification • Requirements – Custom notification script – Stream ids as custom attributes – Icinga2 v2.2 Apply For Rules
  30. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA MONITOR THE MONITORING CORE

    • Check Plugin – Query Graylog2 Alert Stream API for Icinga 2 alerts – Use Stream ID for notifications & notes_url • See what‘s happening in Icinga 2 – Restrict views based on user roles – Debug plugin & check problems – Combine cluster mal-function log – Filter events – Additional dashboard
  31. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA GRAYLOG2: GELFWRITER VISUALIZED

  32. Demo

  33. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA DEMO • Graylog2 0.91.x

    • Icinga 2 2.2.0 • check-graylog2-stream Plugin • Configuration – Graylog2 icinga2 stream & alert – Icinga2 check plugin & host/service/notification apply rules
  34. „The FUTURE“

  35. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA „THE FUTURE“ • Build

    your own stack • Combine existing interfaces into one – Graylog2 streams in Icinga Web 2 (ask Tom!) – Icinga 2 Events in Graylog2 (more? We want more!) • Correlate your monitoring events with events & logs of any kind • Think about – Simple and secure event receiver – Auto-Discover checkable objects from log alerts – Alert stream rules for monitoring
  36. RESOURCES

  37. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA • Code https://github.com/graylog2 https://github.com/icinga/icinga2

    • Vagrant Box icinga2x-graylog2 New @ https://github.com/icinga/icinga-vagrant/ • Documentation http://www.graylog2.org/resources/documentation http://docs.icinga.org/icinga2/latest
  38. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA Q&A Web www.{graylog2,icinga}.org Releases

    github.com/{graylog2,Icinga} IRC #graylog2 #icinga on FreeNode Support support.{graylog2,icinga}.org Twitter twitter.com/{graylog2,icinga} …….. Everywhere! ? Questions & Answers