Watching the Puppet Show

My PuppetConf 2016 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=AbRA-bqqOrg

1. Watching the Puppet Show Puppet & Sensu Delivering reliable services.
2. Sean Porter @PorterTech
3. +
4. FOCUS • The challenges • DevOps & Infrastructure as Code • Common pitfalls & failure cases • Sensu • Puppet & Sensu in practice
5. Let’s talk about software It is eating the world.
6. “Software is eating the world” - Marc Andreessen (2011)
7. SOFTWARE IS EATING THE WORLD! • Society has an insatiable hunger for software ◦ It is becoming part of every facet of our lives • Companies deliver value with software • We need to deliver more software, better software, faster, & reliably - Easy right?
8. Let’s talk about infrastructure A plethora of technologies.
12. NETFLIX 2013
13. “Set it all on fire, child” - Overlord Manatee
14. Let’s talk about DevOps What is DevOps?
15. WHAT IS DEVOPS? “DevOps is continuously looking for new ways to break down silos, eliminate inefficiencies, and remove the risks that prevent the rapid and reliable delivery of software based services” - Damon Edwards, DevOps Cafe
16. WHAT IS DEVOPS? • Continuous improvement - there is no end • Requires a culture that supports learning ◦ Measurement - move the needles ◦ Retrospectives (or blameless post-mortems) • All about delivering better software, faster
17. Let’s talk about IaC What is Infrastructure as Code?
18. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
19. WHAT IS INFRASTRUCTURE AS CODE? • It’s not just about reconstruction & repeatability • IaC techniques scale effectively to manage large numbers of hosts and services • Apply & revert* changes quickly - move faster! • All about delivering software, faster, & reliably
20. BASIC IaC WORKFLOW It’s all software.
21. BASIC IaC WORKFLOW
22. No safeties.
23. INFRASTRUCTURE AS CODE • Break things at scale! • Some changes cannot easily be undone • System state & service health • Coordinating with application deployments • “Erosion” - Entropy
24. BASIC IaC WORKFLOW
25. BASIC IaC WORKFLOW
26. BASIC IaC WORKFLOW Improve the feedback loop. Provides continuous testing.
27. Let’s talk about Sensu What is Sensu?
28. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for IaC workflows
29. WHAT IS SENSU? • A global community ◦ 300+ contributors • Scalable, monitor tens of thousands of systems • Commercially backed ◦ Enterprise version (RBAC etc.) ◦ Support, training, & professional services
30. WHAT PLATFORMS CAN SENSU MONITOR? • Fantastic multi-platform support! • Linux (Debian, RHEL) • Windows • OS X • FreeBSD • Solaris (10, 11) • AIX
31. July 11th, 2011
32. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, Docker, etc.) ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
34. SERVICE CHECKS • Simple to write & understand ◦ STDOUT & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
35. SENSU CLIENT SOCKET INPUT echo '{ \ "name": "mysql_backup", \ "output": "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
36. THE SENSU PIPELINE
37. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
38. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers": ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
39. Puppet & Sensu In practice.
40. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! (101+) ◦ jlambert121, jamtur01, rodjek, and more!
41. Let’s configure a Sensu server Sensu servers publish check requests and process check results and events.
42. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', redis_host => 'redis.foo.com', redis_password => 'p4s5w0rd', server => true, api => true }
43. Let’s configure a Sensu client On an HTTP API host.
44. SENSU CLIENT node 'api-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
45. Let’s configure a Sensu handler On the Sensu server.
46. SENSU HANDLER CONFIG sensu::handler { 'slack': command => 'handler-slack.rb', timeout => 30, config => { 'webhook_url' => 'https://...', 'channel' => 'alerts', 'username' => 'sensu' } } sensu::plugin { 'sensu-plugins-slack': type => 'package', pkg_provider => sensu_gem }
47. Let’s configure a check Run an HTTP endpoint check on ALL API machines. This check is configured on the Sensu server.
48. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, subscribers => ['api'], aggregate => 'api_health', timeout => 60, handlers => ['slack'] }
49. SENSU CHECK DEPENDENCIES Install the check plugin on hosts expected to run it: sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
50. Let’s configure a standalone check Run an HTTP endpoint check on the local API machine. This check is configured on the API machine.
51. SENSU STANDALONE CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, standalone => true, aggregate => 'api_health', timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
52. SENSU IN OTHER PUPPET MODULES Create a new class to be included: e.g. apache/manifests/monitoring/sensu.pp class apache::monitoring::sensu { sensu::check { 'apache-running': command => 'check-procs.rb -p /usr/sbin/httpd -w 100 -c 200 -C 1', handlers => ['slack'] } }
53. SENSU IN OTHER PUPPET MODULES Add client subscriptions and custom attributes: class apache::monitoring::sensu { sensu::subscription { 'apache': 'custom' => { 'ntp_server' => $ntp::servers[0], 'health_endpoint' => '/healthz' } } }
54. THE SENSU PIPELINE
55. PUPPET & SENSU
56. Let’s take it to the next level Puppet module testing & Sensu.
57. SERVERSPEC RSpec tests for your servers: describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
58. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
59. TEST ≈ MONITOR TEST ≈ MONITOR
60. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu Serverspec check plugin ◦ sensu-install -p serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb'
61. SENSU SERVERSPEC CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-serverspec': type => 'package', pkg_provider => sensu_gem }
63. SUMMARY • More software & infrastructure • DevOps & IaC help us deliver software - faster! ◦ No safeties! • Monitoring MUST be part of the workflow • Puppet & Sensu have a mutualistic relationship
64. sensuapp.org Sean Porter - @PorterTech Questions?