Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure as Code & Monitoring

portertech
September 16, 2015

Infrastructure as Code & Monitoring

Automacon 2015 talk: Infrastructure as Code & monitoring, sharing the same development workflow.

portertech

September 16, 2015
Tweet

More Decks by portertech

Other Decks in Programming

Transcript

  1. FOCUS • The Sensu origin story - where it came

    from • What makes Sensu different • IaC development workflows • Testing & monitoring in an IaC workflow • Chef & Sensu in practice
  2. THE SENSU ORIGIN STORY • I started at Sonian ~

    2010 ◦ Chef 0.7 with AWS #YOLO infrastructure ◦ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite
  3. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  4. CHECK EXECUTION METHODS • Pub/Sub (central orchestration) ◦ e.g. execute

    http check on all API nodes • Standalone ◦ Define checks while provisioning node(s) ◦ Scheduled by the local Sensu client
  5. LOCAL CLIENT SOCKET echo '{ \ "name": "mysql_backup", \ "output":

    "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  6. Let’s talk about IaC workflows “The sequence of processes through

    which a piece of work passes from initiation to completion” - Google.
  7. Writing IaC tests “A procedure intended to establish the quality,

    performance, or reliability of something, especially before it is taken into widespread use” - Google.
  8. TESTING TOOLS • Serverspec ◦ RSpec tests for your servers

    ◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
  9. SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'

    do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  10. BATS #!/usr/bin/env bats @test "httpd should be running" { run

    service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
  11. RUNNING TESTS • Test Kitchen ◦ kitchen.ci • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  12. RUNNING TESTS AS SENSU CHECKS • Use the Sensu Serverspec

    check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
  13. SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu • Recipes to install & manage

    services • LWRPs for configuring handlers, checks, etc. • Intended to be used by wrapper cookbook ◦ e.g. recipe['monitor::haproxy']
  14. EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe'

    command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end
  15. USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check =

    data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end
  16. SENSU SERVERSPEC CHECKS # recipe['monitor::serverspec'] sensu_gem 'sensu-plugins-serverspec' sensu_check 'serverspec' do

    command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb ' standalone true interval 30 end # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures