Introduction to Sensu A monitoring framework that aims to be simple, malleable, and scalable.

Who am I? Joe Miller Systems Engineer @ Pantheon (getpantheon.com) @miller_joe / https://github.com/joemiller

Why A New Framework? ● Existing systems ... ● ... not built with config management tools in mind ● ... not built with "cloud" in mind. Server lifecycles can be very short. ● ... don't scale well So ... ● Sonian, Inc. wrote Sensu to replace their struggling Nagios infrastructure ● released as open-source in 2011 (MIT license)

Sensu Core Features ● Simple, small codebase (<1200 LOC) ● CM friendly (Chef, Puppet, etc) ● Simple plugin interface, write them in any language ● Re-use Nagios plugins! There are thousands. ● Available as Ruby Gem or .. ● .. a simple "Omnibus"-style installer (.deb, .rpm)

Sensu Buzzword Sheet ● Ruby ( + EventMachine) ● RabbitMQ ● Redis ● JSON

Sensu Components ● sensu-client ● sensu-server ● sensu-api ● sensu-dashboard

How it works - the TL;DR ● sensu-server publishes check request ... ● ... sensu-client executes check and send results to sensu-server ... ● ... sensu-server executes handlers using the output of check requests. ● all communication over RabbitMQ

How it works - in pictures

Install ● Recommended: Use the "Omnibus"-style installers ○ Installs Sensu with its own embedded Ruby (1.9) and gems to /opt/sensu. Well-tested package. $ yum install sensu $ apt-get install sensu ● Alternative: install via gem $ gem install sensu

Configure ● Works best with config management tools ● Chef, Puppet, and Salt and examples available on Github to get you started ● All configuration is JSON ● Distribute same configs to clients and server ● Configs loaded from: ○ /etc/sensu/config.json ○ /etc/sensu/conf.d/*.json

Client config ● unique on each node. /etc/sensu/conf.d/client.json { "client": { "name": "web01.dom.tld", "address": "10.0.0.1", "subscriptions": [ "base", "nginx" ] } }

Checks ● Familiar API - (Nagios): ○ Exit status indicates result: ■ 0 = ok ■ 1 = warning ■ 2 = critical ■ 3 = uknown ○ Message on STDOUT: CheckPort OK: Port 7199 is LISTENING

Checks ● Re-use Nagios plugins. There's over a decade worth of pre-existing code here. ● Or, pick and choose from the "sensu-community- plugins" repo on github ● Or, write your own. ● If you want to use Ruby, install "sensu-plugin" gem and subclass Sensu::Plugin::Check::CLI

Checks - JSON config { "checks": { "load_average": { "handler": "pagerduty", "command": "/etc/sensu/plugins/check-load.rb --warn 2,2,2 --crit 4,4,4", "occurrences": 2, "interval": 60, "subscribers": [ "base" ] } } }

Custom Attributes ● Arbitrary attributes can be set on "client" config. ● Uses: ○ parameter substitution in checks ○ also available to handlers { "client": { "name": "db01.dom.tld", "address": "10.0.0.1", "subscriptions": [ "base", "mysql_servers" ], "mysql_user": "user", "mysql_pass": "pass" } }

Custom Attributes { "checks": { "mysql_alive_check": { "handler": "default", "command": "/etc/sensu/plugins/mysql-alive.rb --user :::mysql_user::: --mysql_pass :::mysql_pass:::" "interval": 30, "subscribers": [ "mysql_servers" ] }

Available Handlers ● https://github.com/sensu/sensu-community-plugins ○ Pagerduty (most popular) ○ IRC ○ GELF (graylog2) ○ Twitter ○ Campfire ○ email ○ Graphite

Writing Your Own Handlers ● Write in any language, but Ruby is easiest. ● Sensu::Handler class from "sensu-plugin" gem contains a lot of helpful boilerplate. ● Handlers receive JSON data on STDIN, Example: https://gist.github.com/1723079

Handlers - JSON config { "handlers": { "pagerduty": { "type": "pipe", "command": "/etc/sensu/handlers/pantheon_pagerduty.rb", "api_key": "11111aaaaa22222233333", "handle_warnings": false } } }

Handler Sets ● Send check results to multiple handlers easily { "handlers": { "default": { "description": "send alerts to a lot of places", "type": "set", "handlers": [ "pagerduty", "gelf", "irc" ] } } }

sensu-dashboard ● Screenshots !!!

sensu-dashboard ● More screenshots !!!

sensu-api ● RESTful API ● Integrate Sensu with other infrastructure tools ○ Remove a server from Sensu: curl -X DELETE http:///client/

Other Cool Stuff ● Multi-master Sensu (HA) ● Standalone Checks ● Sensu-client Socket ● Aggregated Checks (coming soon): ○ "Alert if 20% of my webservers fail" ● sensu-admin (coming soon): ○ Replaces sensu-dashboard ○ downtime scheduling, expiration on silences, audit log, users accounts & permissions.

Useful Links ● https://github.com/sensu ● https://github.com/sensu/sensu-community-plugins ● IRC: irc.freenode.net #sensu

Get Help (wiki) https://github.com/sensu/sensu/wiki (IRC) irc.freenode.net #sensu