Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure as Code & Monitoring

98f9dfc2e5e1318ac78b8c716582cd30?s=47 portertech
September 16, 2015

Infrastructure as Code & Monitoring

Automacon 2015 talk: Infrastructure as Code & monitoring, sharing the same development workflow.

98f9dfc2e5e1318ac78b8c716582cd30?s=128

portertech

September 16, 2015
Tweet

Transcript

  1. Infrastructure as Code & Monitoring Sharing the same development workflow.

  2. Sean Porter @PorterTech

  3. FOCUS • The Sensu origin story - where it came

    from • What makes Sensu different • IaC development workflows • Testing & monitoring in an IaC workflow • Chef & Sensu in practice
  4. None
  5. THE SENSU ORIGIN STORY • I started at Sonian ~

    2010 ◦ Chef 0.7 with AWS #YOLO infrastructure ◦ Several traditional monitoring tools: Nagios, Ganglia, Collectd, & Graphite
  6. ~ 50 alerts a day Mostly noise.

  7. THE SENSU ORIGIN STORY Triggered @lusis to write: “Why Monitoring

    Sucks”
  8. Wanted MOAR!

  9. Build it. Automatic (de)registration Config Management friendly Secure connectivity REST

    API Elastic scalability UNIX
  10. July 11th, 2011

  11. What makes Sensu different? What makes it better for CM

    driven environments?
  12. None
  13. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  14. CHECK EXECUTION METHODS • Pub/Sub (central orchestration) ◦ e.g. execute

    http check on all API nodes • Standalone ◦ Define checks while provisioning node(s) ◦ Scheduled by the local Sensu client
  15. LOCAL CLIENT SOCKET echo '{ \ "name": "mysql_backup", \ "output":

    "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  16. None
  17. Let’s talk about IaC workflows “The sequence of processes through

    which a piece of work passes from initiation to completion” - Google.
  18. BASIC IaC WORKFLOW “It’s all software.”

  19. BASIC IaC WORKFLOW

  20. BASIC IaC WORKFLOW “It’s all software.”

  21. BASIC IaC WORKFLOW Use tests. Still need to review tests

    & code quality.
  22. BASIC IaC WORKFLOW TEST ≈ MONITOR

  23. BASIC IaC WORKFLOW Shorten the feedback loop.

  24. Writing IaC tests “A procedure intended to establish the quality,

    performance, or reliability of something, especially before it is taken into widespread use” - Google.
  25. TESTING TOOLS • Serverspec ◦ RSpec tests for your servers

    ◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
  26. SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'

    do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  27. BATS #!/usr/bin/env bats @test "httpd should be running" { run

    service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
  28. RUNNING TESTS • Test Kitchen ◦ kitchen.ci • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  29. RUNNING TESTS AS SENSU CHECKS • Use the Sensu Serverspec

    check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
  30. Chef & Sensu In practice.

  31. SENSU CHEF COOKBOOK supermarket.chef.io/cookbooks/sensu • Recipes to install & manage

    services • LWRPs for configuring handlers, checks, etc. • Intended to be used by wrapper cookbook ◦ e.g. recipe['monitor::haproxy']
  32. EXAMPLE CHEF RESOURCES sensu_handler 'default' do # recipe['monitor::_handlers'] type 'pipe'

    command 'pagerduty.rb' end sensu_check 'redis_process' do # recipe['monitor::redis'] command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1' standalone true interval 30 end
  33. USING DATA BAGS # recipe['monitor::pubsub'] data_bag('sensu_checks').each do |data_bag_item| check =

    data_bag_item('sensu_checks', data_bag_item) sensu_check check['id'] do check.each do |key, value| send(key.to_sym, value) if respond_to?(key.to_sym) end end end
  34. SENSU SERVERSPEC CHECKS # recipe['monitor::serverspec'] sensu_gem 'sensu-plugins-serverspec' sensu_check 'serverspec' do

    command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb ' standalone true interval 30 end # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures
  35. BASIC IaC WORKFLOW TEST ≈ MONITOR

  36. Questions? Sean Porter (@PorterTech) sensuapp.org