Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Watching the Puppet Show

Watching the Puppet Show

My PuppetConf 2016 slide deck.

The presentation video recording can be found at: https://www.youtube.com/watch?v=AbRA-bqqOrg

portertech

October 20, 2016
Tweet

More Decks by portertech

Other Decks in Programming

Transcript

  1. +

  2. FOCUS • The challenges • DevOps & Infrastructure as Code

    • Common pitfalls & failure cases • Sensu • Puppet & Sensu in practice
  3. SOFTWARE IS EATING THE WORLD! • Society has an insatiable

    hunger for software ◦ It is becoming part of every facet of our lives • Companies deliver value with software • We need to deliver more software, better software, faster, & reliably - Easy right?
  4. WHAT IS DEVOPS? “DevOps is continuously looking for new ways

    to break down silos, eliminate inefficiencies, and remove the risks that prevent the rapid and reliable delivery of software based services” - Damon Edwards, DevOps Cafe
  5. WHAT IS DEVOPS? • Continuous improvement - there is no

    end • Requires a culture that supports learning ◦ Measurement - move the needles ◦ Retrospectives (or blameless post-mortems) • All about delivering better software, faster
  6. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the

    business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
  7. WHAT IS INFRASTRUCTURE AS CODE? • It’s not just about

    reconstruction & repeatability • IaC techniques scale effectively to manage large numbers of hosts and services • Apply & revert* changes quickly - move faster! • All about delivering software, faster, & reliably
  8. INFRASTRUCTURE AS CODE • Break things at scale! • Some

    changes cannot easily be undone • System state & service health • Coordinating with application deployments • “Erosion” - Entropy
  9. WHAT IS SENSU? • It’s a monitoring tool ◦ Modern

    architecture ◦ Uses service checks with a simple plugin spec ◦ Defined inputs/outputs & very composable ◦ Designed for IaC workflows
  10. WHAT IS SENSU? • A global community ◦ 300+ contributors

    • Scalable, monitor tens of thousands of systems • Commercially backed ◦ Enterprise version (RBAC etc.) ◦ Support, training, & professional services
  11. WHAT PLATFORMS CAN SENSU MONITOR? • Fantastic multi-platform support! •

    Linux (Debian, RHEL) • Windows • OS X • FreeBSD • Solaris (10, 11) • AIX
  12. MODERN ARCHITECTURE • Designed for: ◦ Dynamic infrastructure (EC2, Docker,

    etc.) ◦ Public networks ◦ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
  13. SERVICE CHECKS • Simple to write & understand ◦ STDOUT

    & exit status code • Provide context in multiple forms ◦ Human readable messages ◦ Formatted metrics (PerfData, Graphite, etc.) • Placed top to bottom - service dependency chain
  14. SENSU CLIENT SOCKET INPUT echo '{ \ "name": "mysql_backup", \

    "output": "could not connect to mysql", \ "status": 2, \ "ttl": 90000 }' | nc localhost 3030
  15. PLUGINS & EXTENSIONS • github.com/sensu-plugins (checks, handlers, etc.) • monitoring-plugins.org

    • Many extensions to add protocols etc. ◦ StatsD ◦ InfluxDB ◦ System Profile (metric collection)
  16. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers":

    ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  17. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu • A module to install and

    configure Sensu • Well documented & tested (score ~ 5.0) • Types e.g. sensu_check_config • Awesome contributors! (101+) ◦ jlambert121, jamtur01, rodjek, and more!
  18. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', redis_host => 'redis.foo.com', redis_password => 'p4s5w0rd', server => true, api => true }
  19. SENSU CLIENT node 'api-01.foo.com' { class { 'sensu': rabbitmq_host =>

    'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
  20. SENSU HANDLER CONFIG sensu::handler { 'slack': command => 'handler-slack.rb', timeout

    => 30, config => { 'webhook_url' => 'https://...', 'channel' => 'alerts', 'username' => 'sensu' } } sensu::plugin { 'sensu-plugins-slack': type => 'package', pkg_provider => sensu_gem }
  21. Let’s configure a check Run an HTTP endpoint check on

    ALL API machines. This check is configured on the Sensu server.
  22. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u

    https://127.0.0.1/health', interval => 20, subscribers => ['api'], aggregate => 'api_health', timeout => 60, handlers => ['slack'] }
  23. SENSU CHECK DEPENDENCIES Install the check plugin on hosts expected

    to run it: sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  24. Let’s configure a standalone check Run an HTTP endpoint check

    on the local API machine. This check is configured on the API machine.
  25. SENSU STANDALONE CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb

    -u https://127.0.0.1/health', interval => 20, standalone => true, aggregate => 'api_health', timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  26. SENSU IN OTHER PUPPET MODULES Create a new class to

    be included: e.g. apache/manifests/monitoring/sensu.pp class apache::monitoring::sensu { sensu::check { 'apache-running': command => 'check-procs.rb -p /usr/sbin/httpd -w 100 -c 200 -C 1', handlers => ['slack'] } }
  27. SENSU IN OTHER PUPPET MODULES Add client subscriptions and custom

    attributes: class apache::monitoring::sensu { sensu::subscription { 'apache': 'custom' => { 'ntp_server' => $ntp::servers[0], 'health_endpoint' => '/healthz' } } }
  28. SERVERSPEC RSpec tests for your servers: describe service('httpd'), :if =>

    os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  29. RUNNING TESTS • Test Kitchen ◦ github.com/neillturner/kitchen-puppet • Vagrant plugins

    ◦ github.com/jvoorhis/vagrant-serverspec • Serverspec SSH • … choose your own adventure!
  30. PUPPET MODULE TESTS AS SENSU CHECKS • Use the Sensu

    Serverspec check plugin ◦ sensu-install -p serverspec check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb'
  31. SENSU SERVERSPEC CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb

    -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-serverspec': type => 'package', pkg_provider => sensu_gem }
  32. SUMMARY • More software & infrastructure • DevOps & IaC

    help us deliver software - faster! ◦ No safeties! • Monitoring MUST be part of the workflow • Puppet & Sensu have a mutualistic relationship