Puppet & Sensu - Infrastructure as Code & Monitoring

First draft of my PuppetConf 2015 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=KAAhwzTuhYU

1. Infrastructure as Code & Monitoring Puppet & Sensu Sharing the same development workflow.
2. Sean Porter @PorterTech
3. +
4. FOCUS • What is Infrastructure as Code? • IaC development workflows • Testing & monitoring in an IaC workflow • What is Sensu? • Puppet & Sensu in practice
5. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
6. Let’s talk about IaC workflows “The sequence of processes through which a piece of work passes from initiation to completion” - Google.
7. BASIC IaC WORKFLOW “It’s all software.”
8. BASIC IaC WORKFLOW
9. BASIC IaC WORKFLOW “It’s all software.”
10. BASIC IaC WORKFLOW Use tests. Still need to review tests & code quality.
11. BASIC IaC WORKFLOW TEST ≈ MONITOR
12. BASIC IaC WORKFLOW Shorten the feedback loop.
13. Writing IaC tests “A procedure intended to establish the quality, performance, or reliability of something, especially before it is taken into widespread use” - Google.
14. TESTING TOOLS • Serverspec ◦ RSpec tests for your servers ◦ serverspec.org • Bats ◦ Bash Automated Testing System ◦ Bash script with special syntax for defining test cases
15. SERVERSPEC require 'spec_helper' describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
16. BATS #!/usr/bin/env bats @test "httpd should be running" { run service httpd status [ "$status" -eq 0 ] } @test "httpd should be listening for connections" { [ "$(netstat -plant | grep httpd)" ] }
17. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
18. What is Sensu? Monitoring for today’s infrastructure.
19. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for CM workflows (Puppet, etc.)
20. July 11th, 2011
21. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, etc.) ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
23. SERVICE CHECKS • Simple to write & understand ◦ STDOUT & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
24. INPUTS & OUTPUTS echo '{ \ "name": "mysql_backup", \ "output": "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
25. INPUTS & OUTPUTS
26. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
27. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers": ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
29. Puppet & Sensu In practice.
30. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! ◦ jlambert121, jamtur01, rodjek, and more!
31. Let’s configure a Sensu server Sensu servers publish check requests and process check results and events.
32. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', server => true, api => true }
33. Let’s configure a Sensu client On an HTTP API machine.
34. SENSU CLIENT node 'api-42.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
35. Let’s configure a check Run an HTTP endpoint check on ALL API machines. This check is configured on the Sensu server.
36. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, subscribers => 'api', timeout => 60, handlers => [ 'pagerduty', 'slack' ] }
37. TEST ≈ MONITOR TEST ≈ MONITOR
38. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu Serverspec check plugin ◦ gem install sensu-plugins-serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb' • Run Bats scripts
39. SENSU CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => [ 'pagerduty', 'slack' ] }
43. sensuapp.org Sean Porter - @PorterTech Ask me about Sensu Enterprise!