Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet & Sensu - Infrastructure as Code & Monitoring

98f9dfc2e5e1318ac78b8c716582cd30?s=47 portertech
October 08, 2015

Puppet & Sensu - Infrastructure as Code & Monitoring

First draft of my PuppetConf 2015 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=KAAhwzTuhYU

98f9dfc2e5e1318ac78b8c716582cd30?s=128

portertech

October 08, 2015
Tweet

Transcript

  1. Infrastructure as Code & Monitoring Puppet & Sensu Sharing the

    same development workflow.
  2. Sean Porter @PorterTech

  3. +

  4. FOCUS • What is Infrastructure as Code? • IaC development

    workflows • Testing & monitoring in an IaC workflow • What is Sensu? • Puppet & Sensu in practice
  5. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the

    business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
  6. Let’s talk about IaC workflows “The sequence of processes through

    which a piece of work passes from initiation to completion” - Google.
  7. BASIC IaC WORKFLOW “It’s all software.”

  8. BASIC IaC WORKFLOW

  9. BASIC IaC WORKFLOW “It’s all software.”

  10. BASIC IaC WORKFLOW Use tests. Still need to review tests

    & code quality.
  11. BASIC IaC WORKFLOW TEST ≈ MONITOR

  12. BASIC IaC WORKFLOW Shorten the feedback loop.

  13. Writing IaC tests “A procedure intended to establish the quality,

    performance, or reliability of something, especially before it is taken into widespread use” - Google.
  14. TESTING TOOLS • Serverspec ◦ RSpec tests for your servers

    ◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
  15. SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat'

    do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  16. BATS #!/usr/bin/env bats @test "httpd should be running" { run

    service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
  17. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  18. What is Sensu? Monitoring for today’s infrastructure.

  19. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern

    architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for CM workflows (Puppet, etc.)
  20. July 11th, 2011

  21. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, etc.)

    ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
  22. None
  23. SERVICE CHECKS • Simple to write & understand ◦ STDOUT

    & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
  24. INPUTS & OUTPUTS echo '{ \ "name": "mysql_backup", \ "output":

    "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  25. INPUTS & OUTPUTS

  26. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org

    • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
  27. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  28. None
  29. Puppet & Sensu In practice.

  30. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and

    configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! ◦ jlambert121, jamtur01, rodjek, and more!
  31. Let’s configure a Sensu server Sensu servers publish check requests

    and process check results and events.
  32. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', server => true, api => true }
  33. Let’s configure a Sensu client On an HTTP API machine.

  34. SENSU CLIENT node 'api-42.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
  35. Let’s configure a check Run an HTTP endpoint check on

    ALL API machines. This check is configured on the Sensu server.
  36. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u

    https://127.0.0.1/health', interval => 20, subscribers => 'api', timeout => 60, handlers => [ 'pagerduty', 'slack' ] }
  37. TEST ≈ MONITOR TEST ≈ MONITOR

  38. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu

    Serverspec check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
  39. SENSU CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb -d

    /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => [ 'pagerduty', 'slack' ] }
  40. None
  41. None
  42. None
  43. sensuapp.org Sean Porter - @PorterTech Ask me about Sensu Enterprise!