Pro Yearly is on sale from $80 to $50! »

Get the best out of Graylog2 & Icinga 2

A428c190cba8985990a410b0450b680f?s=47 Bernd Ahlers
November 20, 2014

Get the best out of Graylog2 & Icinga 2

Integrate Graylog2 and Icinga2. OSMC 2014

A428c190cba8985990a410b0450b680f?s=128

Bernd Ahlers

November 20, 2014
Tweet

Transcript

  1. Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best

    out of Graylog2 & Icinga 2
  2. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA BEFORE WE START …

  3. Agenda

  4. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA AGENDA • Introduction •

    Tools • Log History • Logs & Monitoring • Demo • „The Future“ • Resources • Q&A
  5. Introduction

  6. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA WHO‘S WHO Bernd Ahlers

    @berndahlers German, 34, Graylog2 Developer Graylog2 Team since 2014 Developer @ TORCH GmbH Michael Friedrich @dnsmichi Austrian, 31, Icinga Developer Icinga Team since May 2009 Application Developer @ NETWAYS
  7. Tools: Graylog2

  8. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Started

    as open source project by Lennart Koopmann in 2010 – Developed entirely in his free time – Free & open source log management tool • TORCH GmbH founded as company behind Graylog2 in late 2012 – after seeing massive growth and worldwide distribution in large scale setups • Team of 8 engineers working full-time on it
  9. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 • Big

    rewrite of Graylog2 started in 2012 • Finished with releasing a final v0.20.0 in February 2014 • Addresses what we learnt from our first customers and all users • Unified REST API communication – easy extending and integrating with other products, tools and scripts • New web interface focusing on powerful analytics • Current stable version: 0.91.3
  10. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2

  11. Tools: Icinga 2

  12. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •

    Monitoring core engine – Checks, alerts, notifications – Backend interfaces for frontend visualization • Scalable for high performance & real-time monitoring – check_interval = 1s • Dynamic configuration format • Cluster & remote clients, SSL x509 & IPv4/6
  13. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •

    Modular feature set & connectors – DB IDO, Livestatus, Perfdata, Graphite, Gelf • Supports Monitoring Plugins API • Rewritten from scratch – Stable version: 2.2.0 (17.11.2014)
  14. Log History

  15. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Logs

    everywhere • How to collect them? – Splunk (4500$+ for 1GB/day) – Syslog-ng + Custom scripts • Purpose of your collection? – Regex for log parsing – Filters – Alerts? Notifications? Correlation? – Reporting • #devops Stack – Graylog2, Logstash (ELK) + $monitoring + $metrics + $cfgmgmt
  16. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY • Problems

    with remote syslog checks – Failure: where‘s the context? – Pattern matching – Seek files (state history, rate calculation) – Configuration inside Icinga/Plugin • Collect them – Central log cluster (failover) – Correlate events from other servers – Defined streams and alert triggers – Defined input types (e.g. GELF) – Query alert API from Icinga
  17. Logs & Monitoring

  18. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG & MONITORING •

    Monitor your logs – Call check plugin or receive passive events – Generate alerts based on thresholds (configuration) – Notifications based on alerts – Visualize the current state & history for SLA reporting – Trigger event handlers (e.g. iptables on flood) • Popular plugins – check_logfiles – check_splunk • Collector APIs & Hooks – Graylog2 alert API & alert callback plugin – Logstash Nagios output
  19. Logs & Monitoring: Strategy

  20. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA STRATEGY • Out-of-the-box support

    or external addons? • Add hook to streams for passive event sending? • Query a defined API for alerts? • Visualize alerts, and where? (we want dashboards!) • Re-usable & customizable URL for notifications • Combine Log Events & Monitoring notifications and handlers
  21. Logs & Monitoring: Push

  22. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA PUSH: GRAYLOG2 ALARM CALLBACK

    • Requirements – Icinga API (Command Pipe) – Graylog2 Plugin Alarm Callback http://www.graylog2.org/resources/documentation/general/streams http://www.graylog2.org/resources/documentation/general/plugins • Ideas – Exec Callback+NSCA http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/ – (Ab)Use the notification plugin http://everythingshouldbevirtual.com/graylog2-streams-via-email – Custom Rake Plugin http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html • Solution – There is no simple & secure unified Core API (yet) – Use local Icinga2 client & poll check plugin instead
  23. Logs & Monitoring: Poll

  24. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK •

    Requirements – Graylog2 REST API – Icinga Check Plugin • Ideas – Wrapper for Python API calls? https://github.com/qmetric/graylog2-api-tools – Compile check_graylog2_stream? https://github.com/emind-systems/check_graylog2_stream • Solution – New Icinga Plugin by Graylog2 https://github.com/Graylog2/check-graylog2-stream
  25. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK

  26. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK #

    ./check-graylog2-stream usage: -condition="<ID>": Condition ID, set only to check a single alert (optional) -password="<password>": API password (mandatory) -stream="<ID>": Stream ID (mandatory) -url="http://localhost:12900": URL to Graylog2 api (optional) -user="<username>": API username (mandatory)
  27. Combining Graylog2 & Icinga 2

  28. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA COMBINING GRAYLOG2 & ICINGA

    2 • Events triggered by Icinga 2 – Check results – State changes – Notifications • Sent to Graylog2 using `GelfWriter` feature # icinga2 feature enable gelf && service icinga2 restart • Visualize in Graylog2 – Filter based on type (e.g. state != OK) – Alert streams based on counts, etc
  29. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA NOTIFICATIONS • „Default Monitoring

    Alerts are awful“ http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/ – You want to see what‘s wrong. No additional click on your mobile. • Icinga 2 triggers a notification – Fetch additional information from Graylog2 API – Include ‚notes_url‘ with stream id in notification • Requirements – Custom notification script – Stream ids as custom attributes – Icinga2 v2.2 Apply For Rules
  30. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA MONITOR THE MONITORING CORE

    • Check Plugin – Query Graylog2 Alert Stream API for Icinga 2 alerts – Use Stream ID for notifications & notes_url • See what‘s happening in Icinga 2 – Restrict views based on user roles – Debug plugin & check problems – Combine cluster mal-function log – Filter events – Additional dashboard
  31. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA GRAYLOG2: GELFWRITER VISUALIZED

  32. Demo

  33. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA DEMO • Graylog2 0.91.x

    • Icinga 2 2.2.0 • check-graylog2-stream Plugin • Configuration – Graylog2 icinga2 stream & alert – Icinga2 check plugin & host/service/notification apply rules
  34. „The FUTURE“

  35. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA „THE FUTURE“ • Build

    your own stack • Combine existing interfaces into one – Graylog2 streams in Icinga Web 2 (ask Tom!) – Icinga 2 Events in Graylog2 (more? We want more!) • Correlate your monitoring events with events & logs of any kind • Think about – Simple and secure event receiver – Auto-Discover checkable objects from log alerts – Alert stream rules for monitoring
  36. RESOURCES

  37. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA • Code https://github.com/graylog2 https://github.com/icinga/icinga2

    • Vagrant Box icinga2x-graylog2 New @ https://github.com/icinga/icinga-vagrant/ • Documentation http://www.graylog2.org/resources/documentation http://docs.icinga.org/icinga2/latest
  38. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA Q&A Web www.{graylog2,icinga}.org Releases

    github.com/{graylog2,Icinga} IRC #graylog2 #icinga on FreeNode Support support.{graylog2,icinga}.org Twitter twitter.com/{graylog2,icinga} …….. Everywhere! ? Questions & Answers