Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure as Code & Monitoring

portertech
September 16, 2015

Infrastructure as Code & Monitoring

Automacon 2015 talk: Infrastructure as Code & monitoring, sharing the same development workflow.

portertech

September 16, 2015
Tweet

More Decks by portertech

Other Decks in Programming

Transcript

  1. Infrastructure as Code &
    Monitoring
    Sharing the same development workflow.

    View Slide

  2. Sean Porter
    @PorterTech

    View Slide

  3. FOCUS
    ● The Sensu origin story - where it came from
    ● What makes Sensu different
    ● IaC development workflows
    ● Testing & monitoring in an IaC workflow
    ● Chef & Sensu in practice

    View Slide

  4. View Slide

  5. THE SENSU ORIGIN STORY
    ● I started at Sonian ~ 2010
    ○ Chef 0.7 with AWS #YOLO infrastructure
    ○ Several traditional monitoring tools:
    Nagios, Ganglia, Collectd, & Graphite

    View Slide

  6. ~ 50 alerts a day
    Mostly noise.

    View Slide

  7. THE SENSU ORIGIN STORY
    Triggered @lusis to write: “Why Monitoring Sucks”

    View Slide

  8. Wanted MOAR!

    View Slide

  9. Build it.
    Automatic (de)registration
    Config Management friendly
    Secure connectivity
    REST API
    Elastic scalability
    UNIX

    View Slide

  10. July 11th, 2011

    View Slide

  11. What makes Sensu different?
    What makes it better for CM driven environments?

    View Slide

  12. View Slide

  13. JSON CONFIGURATION
    {
    "checks": {
    "mysql_replication": {
    "command": "check-mysql-replication.rb",
    "subscribers": ["mysql"],
    "interval": 30,
    "playbook": "http://wiki.example.com/mysql-replication-playbook"
    }
    }
    }

    View Slide

  14. CHECK EXECUTION METHODS
    ● Pub/Sub (central orchestration)
    ○ e.g. execute http check on all API nodes
    ● Standalone
    ○ Define checks while provisioning node(s)
    ○ Scheduled by the local Sensu client

    View Slide

  15. LOCAL CLIENT SOCKET
    echo '{ \
    "name": "mysql_backup", \
    "output": "could not connect to mysql", \
    "status": 2, \
    "ttl": 90000 }' | nc localhost 3030

    View Slide

  16. View Slide

  17. Let’s talk about IaC workflows
    “The sequence of processes through which a piece of
    work passes from initiation to completion” - Google.

    View Slide

  18. BASIC IaC WORKFLOW
    “It’s all software.”

    View Slide

  19. BASIC IaC WORKFLOW

    View Slide

  20. BASIC IaC WORKFLOW
    “It’s all software.”

    View Slide

  21. BASIC IaC WORKFLOW
    Use tests. Still need to review tests & code quality.

    View Slide

  22. BASIC IaC WORKFLOW
    TEST ≈ MONITOR

    View Slide

  23. BASIC IaC WORKFLOW
    Shorten the feedback loop.

    View Slide

  24. Writing IaC tests
    “A procedure intended to establish the quality, performance,
    or reliability of something, especially before it is taken into
    widespread use” - Google.

    View Slide

  25. TESTING TOOLS
    ● Serverspec
    ○ RSpec tests for your servers
    ○ serverspec.org
    ● Bats
    ○ Bash Automated Testing System
    ○ Bash script with special syntax for defining
    test cases

    View Slide

  26. SERVERSPEC
    require 'spec_helper'
    describe service('httpd'), :if => os[:family] == 'redhat' do
    it { should be_enabled }
    it { should be_running }
    end
    describe port(80) do
    it { should be_listening }
    end

    View Slide

  27. BATS
    #!/usr/bin/env bats
    @test "httpd should be running" {
    run service httpd status
    [ "$status" -eq 0 ]
    }
    @test "httpd should be listening for connections" {
    [ "$(netstat -plant | grep httpd)" ]
    }

    View Slide

  28. RUNNING TESTS
    ● Test Kitchen
    ○ kitchen.ci
    ● Vagrant plugins
    ○ github.com/jvoorhis/vagrant-serverspec
    ● Serverspec SSH
    ● … choose your own adventure!

    View Slide

  29. RUNNING TESTS AS SENSU CHECKS
    ● Use the Sensu Serverspec check plugin
    ○ gem install sensu-plugins-serverspec
    check-serverspec.rb \
    -d /etc/sensu/serverspec -t '*_spec.rb'
    ● Run Bats scripts

    View Slide

  30. Chef & Sensu
    In practice.

    View Slide

  31. SENSU CHEF COOKBOOK
    supermarket.chef.io/cookbooks/sensu
    ● Recipes to install & manage services
    ● LWRPs for configuring handlers, checks, etc.
    ● Intended to be used by wrapper cookbook
    ○ e.g. recipe['monitor::haproxy']

    View Slide

  32. EXAMPLE CHEF RESOURCES
    sensu_handler 'default' do # recipe['monitor::_handlers']
    type 'pipe'
    command 'pagerduty.rb'
    end
    sensu_check 'redis_process' do # recipe['monitor::redis']
    command 'check-procs.rb -p redis-server -w 2 -c 3 -C 1'
    standalone true
    interval 30
    end

    View Slide

  33. USING DATA BAGS
    # recipe['monitor::pubsub']
    data_bag('sensu_checks').each do |data_bag_item|
    check = data_bag_item('sensu_checks', data_bag_item)
    sensu_check check['id'] do
    check.each do |key, value|
    send(key.to_sym, value) if respond_to?(key.to_sym)
    end
    end
    end

    View Slide

  34. SENSU SERVERSPEC CHECKS
    # recipe['monitor::serverspec']
    sensu_gem 'sensu-plugins-serverspec'
    sensu_check 'serverspec' do
    command 'check-serverspec.rb -d /etc/sensu/serverspec -t \*_spec.rb '
    standalone true
    interval 30
    end
    # e.g. CheckServerspec CRITICAL: 12 examples, 2 failures

    View Slide

  35. BASIC IaC WORKFLOW
    TEST ≈ MONITOR

    View Slide

  36. Questions?
    Sean Porter (@PorterTech)
    sensuapp.org

    View Slide