Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Watching the Puppet Show

Watching the Puppet Show

My PuppetConf 2016 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=AbRA-bqqOrg

98f9dfc2e5e1318ac78b8c716582cd30?s=128

portertech

October 20, 2016
Tweet

Transcript

  1. Watching the Puppet Show Puppet & Sensu Delivering reliable services.

  2. Sean Porter @PorterTech

  3. +

  4. FOCUS • The challenges • DevOps & Infrastructure as Code

    • Common pitfalls & failure cases • Sensu • Puppet & Sensu in practice
  5. Let’s talk about software It is eating the world.

  6. “Software is eating the world” - Marc Andreessen (2011)

  7. SOFTWARE IS EATING THE WORLD! • Society has an insatiable

    hunger for software ◦ It is becoming part of every facet of our lives • Companies deliver value with software • We need to deliver more software, better software, faster, & reliably - Easy right?
  8. Let’s talk about infrastructure A plethora of technologies.

  9. None
  10. None
  11. None
  12. NETFLIX 2013

  13. “Set it all on fire, child” - Overlord Manatee

  14. Let’s talk about DevOps What is DevOps?

  15. WHAT IS DEVOPS? “DevOps is continuously looking for new ways

    to break down silos, eliminate inefficiencies, and remove the risks that prevent the rapid and reliable delivery of software based services” - Damon Edwards, DevOps Cafe
  16. WHAT IS DEVOPS? • Continuous improvement - there is no

    end • Requires a culture that supports learning ◦ Measurement - move the needles ◦ Retrospectives (or blameless post-mortems) • All about delivering better software, faster
  17. Let’s talk about IaC What is Infrastructure as Code?

  18. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the

    business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
  19. WHAT IS INFRASTRUCTURE AS CODE? • It’s not just about

    reconstruction & repeatability • IaC techniques scale effectively to manage large numbers of hosts and services • Apply & revert* changes quickly - move faster! • All about delivering software, faster, & reliably
  20. BASIC IaC WORKFLOW It’s all software.

  21. BASIC IaC WORKFLOW

  22. No safeties.

  23. INFRASTRUCTURE AS CODE • Break things at scale! • Some

    changes cannot easily be undone • System state & service health • Coordinating with application deployments • “Erosion” - Entropy
  24. BASIC IaC WORKFLOW

  25. BASIC IaC WORKFLOW

  26. BASIC IaC WORKFLOW Improve the feedback loop. Provides continuous testing.

  27. Let’s talk about Sensu What is Sensu?

  28. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern

    architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for IaC workflows
  29. WHAT IS SENSU? • A global community ◦ 300+ contributors

    • Scalable, monitor tens of thousands of systems • Commercially backed ◦ Enterprise version (RBAC etc.) ◦ Support, training, & professional services
  30. WHAT PLATFORMS CAN SENSU MONITOR? • Fantastic multi-platform support! •

    Linux (Debian, RHEL) • Windows • OS X • FreeBSD • Solaris (10, 11) • AIX
  31. July 11th, 2011

  32. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, Docker,

    etc.) ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
  33. None
  34. SERVICE CHECKS • Simple to write & understand ◦ STDOUT

    & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
  35. SENSU CLIENT SOCKET INPUT echo '{ \ "name": "mysql_backup", \

    "output": "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  36. THE SENSU PIPELINE

  37. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org

    • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
  38. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  39. Puppet & Sensu In practice.

  40. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and

    configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! (101+) ◦ jlambert121, jamtur01, rodjek, and more!
  41. Let’s configure a Sensu server Sensu servers publish check requests

    and process check results and events.
  42. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', redis_host => 'redis.foo.com', redis_password => 'p4s5w0rd', server => true, api => true }
  43. Let’s configure a Sensu client On an HTTP API host.

  44. SENSU CLIENT node 'api-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
  45. Let’s configure a Sensu handler On the Sensu server.

  46. SENSU HANDLER CONFIG sensu::handler { 'slack': command => 'handler-slack.rb', timeout

    => 30, config => { 'webhook_url' => 'https://...', 'channel' => 'alerts', 'username' => 'sensu' } } sensu::plugin { 'sensu-plugins-slack': type => 'package', pkg_provider => sensu_gem }
  47. Let’s configure a check Run an HTTP endpoint check on

    ALL API machines. This check is configured on the Sensu server.
  48. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u

    https://127.0.0.1/health', interval => 20, subscribers => ['api'], aggregate => 'api_health', timeout => 60, handlers => ['slack'] }
  49. SENSU CHECK DEPENDENCIES Install the check plugin on hosts expected

    to run it: sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  50. Let’s configure a standalone check Run an HTTP endpoint check

    on the local API machine. This check is configured on the API machine.
  51. SENSU STANDALONE CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb

    -u https://127.0.0.1/health', interval => 20, standalone => true, aggregate => 'api_health', timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  52. SENSU IN OTHER PUPPET MODULES Create a new class to

    be included: e.g. apache/manifests/monitoring/sensu.pp class apache::monitoring::sensu { sensu::check { 'apache-running': command => 'check-procs.rb -p /usr/sbin/httpd -w 100 -c 200 -C 1', handlers => ['slack'] } }
  53. SENSU IN OTHER PUPPET MODULES Add client subscriptions and custom

    attributes: class apache::monitoring::sensu { sensu::subscription { 'apache': 'custom' => { 'ntp_server' => $ntp::servers[0], 'health_endpoint' => '/healthz' } } }
  54. THE SENSU PIPELINE

  55. PUPPET & SENSU

  56. Let’s take it to the next level Puppet module testing

    & Sensu.
  57. SERVERSPEC RSpec tests for your servers: describe service('httpd'), :if =>

    os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  58. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  59. TEST ≈ MONITOR TEST ≈ MONITOR

  60. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu

    Serverspec check plugin ◦ sensu-install -p serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb'
  61. SENSU SERVERSPEC CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb

    -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-serverspec': type => 'package', pkg_provider => sensu_gem }
  62. None
  63. SUMMARY • More software & infrastructure • DevOps & IaC

    help us deliver software - faster! ◦ No safeties! • Monitoring MUST be part of the workflow • Puppet & Sensu have a mutualistic relationship
  64. sensuapp.org Sean Porter - @PorterTech Questions?