$30 off During Our Annual Pro Sale. View Details »

Get the best out of Graylog2 & Icinga 2

Bernd Ahlers
November 20, 2014

Get the best out of Graylog2 & Icinga 2

Integrate Graylog2 and Icinga2. OSMC 2014

Bernd Ahlers

November 20, 2014
Tweet

More Decks by Bernd Ahlers

Other Decks in Programming

Transcript

  1. Bernd Ahlers
    Michael Friedrich
    Log Monitoring Simplified
    Get the best out of Graylog2 & Icinga 2

    View Slide

  2. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    BEFORE WE START …

    View Slide

  3. Agenda

    View Slide

  4. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    AGENDA
    • Introduction
    • Tools
    • Log History
    • Logs & Monitoring
    • Demo
    • „The Future“
    • Resources
    • Q&A

    View Slide

  5. Introduction

    View Slide

  6. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    WHO‘S WHO
    Bernd Ahlers
    @berndahlers
    German, 34, Graylog2 Developer
    Graylog2 Team since 2014
    Developer @ TORCH GmbH
    Michael Friedrich
    @dnsmichi
    Austrian, 31, Icinga Developer
    Icinga Team since May 2009
    Application Developer @
    NETWAYS

    View Slide

  7. Tools: Graylog2

    View Slide

  8. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    TOOLS: GRAYLOG2
    • Started as open source project by Lennart Koopmann in 2010
    – Developed entirely in his free time
    – Free & open source log management tool
    • TORCH GmbH founded as company behind Graylog2 in late
    2012
    – after seeing massive growth and worldwide distribution in
    large scale setups
    • Team of 8 engineers working full-time on it

    View Slide

  9. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    TOOLS: GRAYLOG2
    • Big rewrite of Graylog2 started in 2012
    • Finished with releasing a final v0.20.0 in February 2014
    • Addresses what we learnt from our first customers and all
    users
    • Unified REST API communication
    – easy extending and integrating with other products, tools
    and scripts
    • New web interface focusing on powerful analytics
    • Current stable version: 0.91.3

    View Slide

  10. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    TOOLS: GRAYLOG2

    View Slide

  11. Tools: Icinga 2

    View Slide

  12. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    TOOLS: ICINGA 2
    • Monitoring core engine
    – Checks, alerts, notifications
    – Backend interfaces for frontend visualization
    • Scalable for high performance & real-time monitoring
    – check_interval = 1s
    • Dynamic configuration format
    • Cluster & remote clients, SSL x509 & IPv4/6

    View Slide

  13. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    TOOLS: ICINGA 2
    • Modular feature set & connectors
    – DB IDO, Livestatus, Perfdata, Graphite, Gelf
    • Supports Monitoring Plugins API
    • Rewritten from scratch
    – Stable version: 2.2.0 (17.11.2014)

    View Slide

  14. Log History

    View Slide

  15. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    LOG HISTORY
    • Logs everywhere
    • How to collect them?
    – Splunk (4500$+ for 1GB/day)
    – Syslog-ng + Custom scripts
    • Purpose of your collection?
    – Regex for log parsing
    – Filters
    – Alerts? Notifications? Correlation?
    – Reporting
    • #devops Stack
    – Graylog2, Logstash (ELK) + $monitoring + $metrics +
    $cfgmgmt

    View Slide

  16. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    LOG HISTORY
    • Problems with remote syslog checks
    – Failure: where‘s the context?
    – Pattern matching
    – Seek files (state history, rate calculation)
    – Configuration inside Icinga/Plugin
    • Collect them
    – Central log cluster (failover)
    – Correlate events from other servers
    – Defined streams and alert triggers
    – Defined input types (e.g. GELF)
    – Query alert API from Icinga

    View Slide

  17. Logs & Monitoring

    View Slide

  18. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    LOG & MONITORING
    • Monitor your logs
    – Call check plugin or receive passive events
    – Generate alerts based on thresholds (configuration)
    – Notifications based on alerts
    – Visualize the current state & history for SLA reporting
    – Trigger event handlers (e.g. iptables on flood)
    • Popular plugins
    – check_logfiles
    – check_splunk
    • Collector APIs & Hooks
    – Graylog2 alert API & alert callback plugin
    – Logstash Nagios output

    View Slide

  19. Logs & Monitoring: Strategy

    View Slide

  20. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    STRATEGY
    • Out-of-the-box support or external addons?
    • Add hook to streams for passive event sending?
    • Query a defined API for alerts?
    • Visualize alerts, and where? (we want dashboards!)
    • Re-usable & customizable URL for notifications
    • Combine Log Events & Monitoring notifications and handlers

    View Slide

  21. Logs & Monitoring: Push

    View Slide

  22. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    PUSH: GRAYLOG2 ALARM CALLBACK
    • Requirements
    – Icinga API (Command Pipe)
    – Graylog2 Plugin Alarm Callback
    http://www.graylog2.org/resources/documentation/general/streams
    http://www.graylog2.org/resources/documentation/general/plugins
    • Ideas
    – Exec Callback+NSCA
    http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/
    – (Ab)Use the notification plugin
    http://everythingshouldbevirtual.com/graylog2-streams-via-email
    – Custom Rake Plugin
    http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html
    • Solution
    – There is no simple & secure unified Core API (yet)
    – Use local Icinga2 client & poll check plugin instead

    View Slide

  23. Logs & Monitoring: Poll

    View Slide

  24. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    POLL: ICINGA CHECK
    • Requirements
    – Graylog2 REST API
    – Icinga Check Plugin
    • Ideas
    – Wrapper for Python API calls?
    https://github.com/qmetric/graylog2-api-tools
    – Compile check_graylog2_stream?
    https://github.com/emind-systems/check_graylog2_stream
    • Solution
    – New Icinga Plugin by Graylog2
    https://github.com/Graylog2/check-graylog2-stream

    View Slide

  25. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    POLL: ICINGA CHECK

    View Slide

  26. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    POLL: ICINGA CHECK
    # ./check-graylog2-stream
    usage:
    -condition="": Condition ID, set only to check a single alert (optional)
    -password="": API password (mandatory)
    -stream="": Stream ID (mandatory)
    -url="http://localhost:12900": URL to Graylog2 api (optional)
    -user="": API username (mandatory)

    View Slide

  27. Combining Graylog2 & Icinga 2

    View Slide

  28. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    COMBINING GRAYLOG2 & ICINGA 2
    • Events triggered by Icinga 2
    – Check results
    – State changes
    – Notifications
    • Sent to Graylog2 using `GelfWriter` feature
    # icinga2 feature enable gelf && service icinga2 restart
    • Visualize in Graylog2
    – Filter based on type (e.g. state != OK)
    – Alert streams based on counts, etc

    View Slide

  29. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    NOTIFICATIONS
    • „Default Monitoring Alerts are awful“
    http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/
    – You want to see what‘s wrong. No additional click on your
    mobile.
    • Icinga 2 triggers a notification
    – Fetch additional information from Graylog2 API
    – Include ‚notes_url‘ with stream id in notification
    • Requirements
    – Custom notification script
    – Stream ids as custom attributes
    – Icinga2 v2.2 Apply For Rules

    View Slide

  30. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    MONITOR THE MONITORING CORE
    • Check Plugin
    – Query Graylog2 Alert Stream API for Icinga 2 alerts
    – Use Stream ID for notifications & notes_url
    • See what‘s happening in Icinga 2
    – Restrict views based on user roles
    – Debug plugin & check problems
    – Combine cluster mal-function log
    – Filter events
    – Additional dashboard

    View Slide

  31. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    GRAYLOG2: GELFWRITER VISUALIZED

    View Slide

  32. Demo

    View Slide

  33. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    DEMO
    • Graylog2 0.91.x
    • Icinga 2 2.2.0
    • check-graylog2-stream Plugin
    • Configuration
    – Graylog2 icinga2 stream & alert
    – Icinga2 check plugin & host/service/notification apply rules

    View Slide

  34. „The FUTURE“

    View Slide

  35. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    „THE FUTURE“
    • Build your own stack
    • Combine existing interfaces into one
    – Graylog2 streams in Icinga Web 2 (ask Tom!)
    – Icinga 2 Events in Graylog2 (more? We want more!)
    • Correlate your monitoring events with events & logs of any
    kind
    • Think about
    – Simple and secure event receiver
    – Auto-Discover checkable objects from log alerts
    – Alert stream rules for monitoring

    View Slide

  36. RESOURCES

    View Slide

  37. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    • Code
    https://github.com/graylog2
    https://github.com/icinga/icinga2
    • Vagrant Box icinga2x-graylog2
    New @ https://github.com/icinga/icinga-vagrant/
    • Documentation
    http://www.graylog2.org/resources/documentation
    http://docs.icinga.org/icinga2/latest

    View Slide

  38. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG
    #OSMC #GRAYLOG2 #ICINGA
    Q&A
    Web www.{graylog2,icinga}.org
    Releases github.com/{graylog2,Icinga}
    IRC #graylog2 #icinga on FreeNode
    Support support.{graylog2,icinga}.org
    Twitter twitter.com/{graylog2,icinga}
    …….. Everywhere!
    ?
    Questions & Answers

    View Slide