Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet & Sensu - Infrastructure as Code & Monit...

portertech
October 08, 2015

Puppet & Sensu - Infrastructure as Code & Monitoring

First draft of my PuppetConf 2015 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=KAAhwzTuhYU

portertech

October 08, 2015
Tweet

More Decks by portertech

Other Decks in Programming

Transcript

  1. +

  2. FOCUS • What is Infrastructure as Code? • IaC development

    workflows • Testing & monitoring in an IaC workflow • What is Sensu? • Puppet & Sensu in practice
  3. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the

    business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
  4. Let’s talk about IaC workflows “The sequence of processes through

    which a piece of work passes from initiation to completion” - Google.
  5. Writing IaC tests “A procedure intended to establish the quality,

    performance, or reliability of something, especially before it is taken into widespread use” - Google.
  6. TESTING TOOLS • Serverspec ◦ RSpec tests for your servers

    ◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
  7. SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'

    do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  8. BATS #!/usr/bin/env bats @test "httpd should be running" { run

    service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
  9. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  10. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern

    architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for CM workflows (Puppet, etc.)
  11. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, etc.)

    ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
  12. SERVICE CHECKS • Simple to write & understand ◦ STDOUT

    & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
  13. INPUTS & OUTPUTS echo '{ \ "name": "mysql_backup", \ "output":

    "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  14. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org

    • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
  15. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  16. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and

    configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! ◦ jlambert121, jamtur01, rodjek, and more!
  17. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', server => true, api => true }
  18. SENSU CLIENT node 'api-42.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
  19. Let’s configure a check Run an HTTP endpoint check on

    ALL API machines. This check is configured on the Sensu server.
  20. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u

    https://127.0.0.1/health', interval => 20, subscribers => 'api', timeout => 60, handlers => [ 'pagerduty', 'slack' ] }
  21. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu

    Serverspec check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
  22. SENSU CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb -d

    /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => [ 'pagerduty', 'slack' ] }