Get the best out of Graylog2 & Icinga 2

Bernd Ahlers
Michael Friedrich
Log Monitoring Simplified
Get the best out of Graylog2 & Icinga 2

View Slide

WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
#OSMC #GRAYLOG2 #ICINGA
BEFORE WE START …

View Slide

Agenda

View Slide

AGENDA
• Introduction
• Tools
• Log History
• Logs & Monitoring
• Demo
• „The Future“
• Resources
• Q&A

View Slide

Introduction

View Slide

WHO‘S WHO
Bernd Ahlers
@berndahlers
German, 34, Graylog2 Developer
Graylog2 Team since 2014
Developer @ TORCH GmbH
Michael Friedrich
@dnsmichi
Austrian, 31, Icinga Developer
Icinga Team since May 2009
Application Developer @
NETWAYS

View Slide

Tools: Graylog2

View Slide

TOOLS: GRAYLOG2
• Started as open source project by Lennart Koopmann in 2010
– Developed entirely in his free time
– Free & open source log management tool
• TORCH GmbH founded as company behind Graylog2 in late
2012
– after seeing massive growth and worldwide distribution in
large scale setups
• Team of 8 engineers working full-time on it

View Slide

TOOLS: GRAYLOG2
• Big rewrite of Graylog2 started in 2012
• Finished with releasing a final v0.20.0 in February 2014
• Addresses what we learnt from our first customers and all
users
• Unified REST API communication
– easy extending and integrating with other products, tools
and scripts
• New web interface focusing on powerful analytics
• Current stable version: 0.91.3

View Slide

TOOLS: GRAYLOG2

View Slide

Tools: Icinga 2

View Slide

TOOLS: ICINGA 2
• Monitoring core engine
– Checks, alerts, notifications
– Backend interfaces for frontend visualization
• Scalable for high performance & real-time monitoring
– check_interval = 1s
• Dynamic configuration format
• Cluster & remote clients, SSL x509 & IPv4/6

View Slide

TOOLS: ICINGA 2
• Modular feature set & connectors
– DB IDO, Livestatus, Perfdata, Graphite, Gelf
• Supports Monitoring Plugins API
• Rewritten from scratch
– Stable version: 2.2.0 (17.11.2014)

View Slide

Log History

View Slide

LOG HISTORY
• Logs everywhere
• How to collect them?
– Splunk (4500$+ for 1GB/day)
– Syslog-ng + Custom scripts
• Purpose of your collection?
– Regex for log parsing
– Filters
– Alerts? Notifications? Correlation?
– Reporting
• #devops Stack
– Graylog2, Logstash (ELK) + $monitoring + $metrics +
$cfgmgmt

View Slide

LOG HISTORY
• Problems with remote syslog checks
– Failure: where‘s the context?
– Pattern matching
– Seek files (state history, rate calculation)
– Configuration inside Icinga/Plugin
• Collect them
– Central log cluster (failover)
– Correlate events from other servers
– Defined streams and alert triggers
– Defined input types (e.g. GELF)
– Query alert API from Icinga

View Slide

Logs & Monitoring

View Slide

LOG & MONITORING
• Monitor your logs
– Call check plugin or receive passive events
– Generate alerts based on thresholds (configuration)
– Notifications based on alerts
– Visualize the current state & history for SLA reporting
– Trigger event handlers (e.g. iptables on flood)
• Popular plugins
– check_logfiles
– check_splunk
• Collector APIs & Hooks
– Graylog2 alert API & alert callback plugin
– Logstash Nagios output

View Slide

Logs & Monitoring: Strategy

View Slide

STRATEGY
• Out-of-the-box support or external addons?
• Add hook to streams for passive event sending?
• Query a defined API for alerts?
• Visualize alerts, and where? (we want dashboards!)
• Re-usable & customizable URL for notifications
• Combine Log Events & Monitoring notifications and handlers

View Slide

Logs & Monitoring: Push

View Slide

PUSH: GRAYLOG2 ALARM CALLBACK
• Requirements
– Icinga API (Command Pipe)
– Graylog2 Plugin Alarm Callback
http://www.graylog2.org/resources/documentation/general/streams
http://www.graylog2.org/resources/documentation/general/plugins
• Ideas
– Exec Callback+NSCA
http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/
– (Ab)Use the notification plugin
http://everythingshouldbevirtual.com/graylog2-streams-via-email
– Custom Rake Plugin
http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html
• Solution
– There is no simple & secure unified Core API (yet)
– Use local Icinga2 client & poll check plugin instead

View Slide

Logs & Monitoring: Poll

View Slide

POLL: ICINGA CHECK
• Requirements
– Graylog2 REST API
– Icinga Check Plugin
• Ideas
– Wrapper for Python API calls?
https://github.com/qmetric/graylog2-api-tools
– Compile check_graylog2_stream?
https://github.com/emind-systems/check_graylog2_stream
• Solution
– New Icinga Plugin by Graylog2
https://github.com/Graylog2/check-graylog2-stream

View Slide

POLL: ICINGA CHECK

View Slide

POLL: ICINGA CHECK
# ./check-graylog2-stream
usage:
-condition="": Condition ID, set only to check a single alert (optional)
-password="": API password (mandatory)
-stream="": Stream ID (mandatory)
-url="http://localhost:12900": URL to Graylog2 api (optional)
-user="": API username (mandatory)

View Slide

Combining Graylog2 & Icinga 2

View Slide

COMBINING GRAYLOG2 & ICINGA 2
• Events triggered by Icinga 2
– Check results
– State changes
– Notifications
• Sent to Graylog2 using `GelfWriter` feature
# icinga2 feature enable gelf && service icinga2 restart
• Visualize in Graylog2
– Filter based on type (e.g. state != OK)
– Alert streams based on counts, etc

View Slide

NOTIFICATIONS
• „Default Monitoring Alerts are awful“
http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/
– You want to see what‘s wrong. No additional click on your
mobile.
• Icinga 2 triggers a notification
– Fetch additional information from Graylog2 API
– Include ‚notes_url‘ with stream id in notification
• Requirements
– Custom notification script
– Stream ids as custom attributes
– Icinga2 v2.2 Apply For Rules

View Slide

MONITOR THE MONITORING CORE
• Check Plugin
– Query Graylog2 Alert Stream API for Icinga 2 alerts
– Use Stream ID for notifications & notes_url
• See what‘s happening in Icinga 2
– Restrict views based on user roles
– Debug plugin & check problems
– Combine cluster mal-function log
– Filter events
– Additional dashboard

View Slide

GRAYLOG2: GELFWRITER VISUALIZED

View Slide

Demo

View Slide

DEMO
• Graylog2 0.91.x
• Icinga 2 2.2.0
• check-graylog2-stream Plugin
• Configuration
– Graylog2 icinga2 stream & alert
– Icinga2 check plugin & host/service/notification apply rules

View Slide

„The FUTURE“

View Slide

„THE FUTURE“
• Build your own stack
• Combine existing interfaces into one
– Graylog2 streams in Icinga Web 2 (ask Tom!)
– Icinga 2 Events in Graylog2 (more? We want more!)
• Correlate your monitoring events with events & logs of any
kind
• Think about
– Simple and secure event receiver
– Auto-Discover checkable objects from log alerts
– Alert stream rules for monitoring

View Slide

RESOURCES

View Slide

• Code
https://github.com/graylog2
https://github.com/icinga/icinga2
• Vagrant Box icinga2x-graylog2
New @ https://github.com/icinga/icinga-vagrant/
• Documentation
http://www.graylog2.org/resources/documentation
http://docs.icinga.org/icinga2/latest

View Slide

Q&A
Web www.{graylog2,icinga}.org
Releases github.com/{graylog2,Icinga}
IRC #graylog2 #icinga on FreeNode
Support support.{graylog2,icinga}.org
Twitter twitter.com/{graylog2,icinga}
…….. Everywhere!
?
Questions & Answers

View Slide

Get the best out of Graylog2 & Icinga 2

Get the best out of Graylog2 & Icinga 2

Bernd Ahlers

More Decks by Bernd Ahlers

Other Decks in Programming

Featured

Transcript