FOCUS ● The Sensu origin story - where it came from ● What makes Sensu different ● IaC development workflows ● Testing & monitoring in an IaC workflow ● Chef & Sensu in practice
THE SENSU ORIGIN STORY ● I started at Sonian ~ 2010 ○ Chef 0.7 with AWS #YOLO infrastructure ○ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite
CHECK EXECUTION METHODS ● Pub/Sub (central orchestration) ○ e.g. execute http check on all API nodes ● Standalone ○ Define checks while provisioning node(s) ○ Scheduled by the local Sensu client
Writing IaC tests “A procedure intended to establish the quality, performance, or reliability of something, especially before it is taken into widespread use” - Google.
TESTING TOOLS ● Serverspec ○ RSpec tests for your servers ○ serverspec.org ● Bats ○ Bash Automated Testing System ○ Bash script with special syntax for defining test cases
SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
BATS #!/usr/bin/env bats @test "httpd should be running" { run service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
RUNNING TESTS AS SENSU CHECKS ● Use the Sensu Serverspec check plugin ○ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' ● Run Bats scripts
SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu ● Recipes to install & manage services ● LWRPs for configuring handlers, checks, etc. ● Intended to be used by wrapper cookbook ○ e.g. recipe['monitor::haproxy']
EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe' command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end
USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check = data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end