Introduction to Sensu

Slides from the Sensu SF Meetup @ Pantheon offices (9/26/2012)

Transcript

1. Introduction to Sensu A monitoring framework that aims to be simple, malleable, and scalable.
2. Who am I? Joe Miller Systems Engineer @ Pantheon (getpantheon.com) @miller_joe / https://github.com/joemiller
3. Why A New Framework? • Existing systems ... • ... not built with config management tools in mind • ... not built with "cloud" in mind. Server lifecycles can be very short. • ... don't scale well So ... • Sonian, Inc. wrote Sensu to replace their struggling Nagios infrastructure • released as open-source in 2011 (MIT license)
4. Sensu Core Features • Simple, small codebase (<1200 LOC) • CM friendly (Chef, Puppet, etc) • Simple plugin interface, write them in any language • Re-use Nagios plugins! There are thousands. • Available as Ruby Gem or .. • .. a simple "Omnibus"-style installer (.deb, .rpm)
5. Sensu Buzzword Sheet • Ruby ( + EventMachine) • RabbitMQ • Redis • JSON
6. Sensu Components • sensu-client • sensu-server • sensu-api • sensu-dashboard
7. How it works - the TL;DR • sensu-server publishes check request ... • ... sensu-client executes check and send results to sensu-server ... • ... sensu-server executes handlers using the output of check requests. • all communication over RabbitMQ
8. How it works - in pictures
9. Install • Recommended: Use the "Omnibus"-style installers ◦ Installs Sensu with its own embedded Ruby (1.9) and gems to /opt/sensu. Well-tested package. $ yum install sensu $ apt-get install sensu • Alternative: install via gem $ gem install sensu
10. Configure • Works best with config management tools • Chef, Puppet, and Salt and examples available on Github to get you started • All configuration is JSON • Distribute same configs to clients and server • Configs loaded from: ◦ /etc/sensu/config.json ◦ /etc/sensu/conf.d/*.json
11. Client config • unique on each node. /etc/sensu/conf.d/client.json { "client": { "name": "web01.dom.tld", "address": "10.0.0.1", "subscriptions": [ "base", "nginx" ] } }
12. Checks • Familiar API - (Nagios): ◦ Exit status indicates result: ▪ 0 = ok ▪ 1 = warning ▪ 2 = critical ▪ 3 = uknown ◦ Message on STDOUT: CheckPort OK: Port 7199 is LISTENING
13. Checks • Re-use Nagios plugins. There's over a decade worth of pre-existing code here. • Or, pick and choose from the "sensu-community- plugins" repo on github • Or, write your own. • If you want to use Ruby, install "sensu-plugin" gem and subclass Sensu::Plugin::Check::CLI
14. Checks - JSON config { "checks": { "load_average": { "handler": "pagerduty", "command": "/etc/sensu/plugins/check-load.rb --warn 2,2,2 --crit 4,4,4", "occurrences": 2, "interval": 60, "subscribers": [ "base" ] } } }
15. Custom Attributes • Arbitrary attributes can be set on "client" config. • Uses: ◦ parameter substitution in checks ◦ also available to handlers { "client": { "name": "db01.dom.tld", "address": "10.0.0.1", "subscriptions": [ "base", "mysql_servers" ], "mysql_user": "user", "mysql_pass": "pass" } }
16. Custom Attributes { "checks": { "mysql_alive_check": { "handler": "default", "command": "/etc/sensu/plugins/mysql-alive.rb --user :::mysql_user::: --mysql_pass :::mysql_pass:::" "interval": 30, "subscribers": [ "mysql_servers" ] }
17. Available Handlers • https://github.com/sensu/sensu-community-plugins ◦ Pagerduty (most popular) ◦ IRC ◦ GELF (graylog2) ◦ Twitter ◦ Campfire ◦ email ◦ Graphite
18. Writing Your Own Handlers • Write in any language, but Ruby is easiest. • Sensu::Handler class from "sensu-plugin" gem contains a lot of helpful boilerplate. • Handlers receive JSON data on STDIN, Example: https://gist.github.com/1723079
19. Handlers - JSON config { "handlers": { "pagerduty": { "type": "pipe", "command": "/etc/sensu/handlers/pantheon_pagerduty.rb", "api_key": "11111aaaaa22222233333", "handle_warnings": false } } }
20. Handler Sets • Send check results to multiple handlers easily { "handlers": { "default": { "description": "send alerts to a lot of places", "type": "set", "handlers": [ "pagerduty", "gelf", "irc" ] } } }
21. sensu-dashboard • Screenshots !!!
22. sensu-dashboard • More screenshots !!!
23. sensu-api • RESTful API • Integrate Sensu with other infrastructure tools ◦ Remove a server from Sensu: curl -X DELETE http://<sensu-api>/client/<node>
24. Other Cool Stuff • Multi-master Sensu (HA) • Standalone Checks • Sensu-client Socket • Aggregated Checks (coming soon): ◦ "Alert if 20% of my webservers fail" • sensu-admin (coming soon): ◦ Replaces sensu-dashboard ◦ downtime scheduling, expiration on silences, audit log, users accounts & permissions.
25. Useful Links • https://github.com/sensu • https://github.com/sensu/sensu-community-plugins • IRC: irc.freenode.net #sensu
26. Get Help (wiki) https://github.com/sensu/sensu/wiki (IRC) irc.freenode.net #sensu