Advanced Vagrant Usage with Puppet

Advanced Vagrant Usage with Puppet

This talk was given at PuppetConf 2012. It covers how I use Vagrant with Puppet and how I automate a lot of what I do with Puppet.

2828f28fb012308a7786eee83b8293c5?s=128

Mitchell Hashimoto

September 27, 2012
Tweet

Transcript

  1. Advanced Vagrant Usage with Puppet

  2. I’m Mitchell Hashimoto Also known as @mitchellh

  3. I made Vagrant. Hopefully you use it. I think you’ll

    like it. http://vagrantup.com
  4. I’m an automation freak. This talk will show this to

    be true.
  5. Vagrant Usage (Ops Focused)

  6. Benefits we want from Vagrant...

  7. “The Cloud” but on your machine.

  8. Self service. Instant provisioning. Cost efficient. Elastic. Pay per use.

    Paul Strong’s Cloud
  9. - Manifest development, both simple and not so simple Benefits

    We Want
  10. - Manifest development, both simple and not so simple -

    Repeatability Benefits We Want
  11. - Manifest development, both simple and not so simple -

    Repeatability - Fast feedback Benefits We Want
  12. - Manifest development, both simple and not so simple -

    Repeatability - Fast feedback - Confidence Benefits We Want
  13. Confession: I’ve been doing Puppet full time for awhile now.

  14. Current state of Vagrant + Puppet...

  15. Basic manifest development and testing.

  16. ... Yep.

  17. We can do better. We can do much better.

  18. We can do better with what is available right now.

  19. Teaser: We will do magic with what is coming in

    the future. <3 <3 <3
  20. My state of Vagrant + Puppet...

  21. Fully automated Puppet Master setup.

  22. Testing exported resources, hiera, and nodes.

  23. Common deploy process across Vagrant and EC2.

  24. Repeatable workflow of dev to staging to prod.

  25. Golden master box creation for development.

  26. Time to share what I’ve learned.

  27. Advanced Vagrant Usage

  28. Advanced Vagrant Usage

  29. Advanced Automation for Puppet Work

  30. Fully Automated Puppet Master Setup

  31. I asked: “How do people bring up or recover a

    Puppet master?”
  32. “Most people roll their Puppet Master by hand.” - Anonymous

    PuppetLabs Employee
  33. None
  34. Puppet Master is crucial to testing realistic scenarios.

  35. Automated Puppet Master - Pushing broken Puppet crashes the server.

  36. Automated Puppet Master - Pushing broken Puppet crashes the server.

    - Local development against a Puppet Master has benefits.
  37. Automated Puppet Master - Pushing broken Puppet crashes the server.

    - Local development against a Puppet Master has benefits. - Automation all the way down.
  38. Multi-level bootstrap.

  39. 1. Bash script to minimally install Puppet master and agent.

  40. 2. puppet apply to minimally setup Puppet Master infrastructure

  41. 3. puppet agent to completely setup and harden the master.

  42. Result: Production-quality Puppet Master whenever you need it.

  43. Vagrant::Config.run do |config| # ... config.vm.hostname = "puppet" config.vm.provision :shell,

    :path => "bootstrap.sh" end
  44. Testing Exported Resources, Hiera, and nodes.

  45. I asked: “How do people test more than the most

    basic Puppet module?”
  46. “I suspect the answer is that they just don't test

    their modules adequately.” - Anonymous PuppetLabs Employee
  47. None
  48. Solution: Automated Puppet Master + Multi-VM

  49. Automated Puppet Master is production ready: PuppetDB, Hiera, etc.

  50. Multi-VM enables Vagrant to manage a cluster of machines that

    can communicate. http://vagrantup.com/v1/docs/multivm.html
  51. Testing Exported Resources

  52. Create two nodes. Export one. Collect other. Ruby/Shell script. Testing

    Exported Resources
  53. node 'test_exporter' { @@nginx::site { "test": content => "\n", tag

    => "origin", } } node 'test_collector' { include role::origin }
  54. Vagrant::Config.run do |config| config.vm.define :export do |n| n.vm.hostname = "test_exporter"

    n.vm.provision :puppet_server, :options => "--verbose --debug" end config.vm.define :collect do |n| n.vm.hostname = "test_collecter" n.vm.provision :puppet_server, :options => "--verbose --debug" end end
  55. #!/bin/bash test -f /etc/nginx/sites-available/test

  56. Testing Hiera

  57. Create full Hiera hierarchy. Launch node. Test hierarchy.

  58. --- test_region: "us-east-1" --- test_role: "hiera" --- test_name: "test-hiera”

  59. Vagrant::Config.run do |config| config.vm.hostname = "test_hiera" config.vm.provision :puppet_server, :options =>

    "--verbose --debug" end end
  60. node 'test_hiera' { $region = hiera("test_region") $role = hiera("test_role") $name

    = hiera("test_name") file { "/tmp/results": content => "$region $role $name", mode => "0644", } }
  61. Testing Nodes

  62. Create node. Provision. Test behavior.

  63. Vagrant::Config.run do |config| config.vm.define :master do |master| master.vm.hostname = "puppet"

    master.vm.provision :shell, :path => "bootstrap.sh" end config.vm.define :node do |node| node.vm.hostname = "postgresql" node.vm.provision :puppet_server, :options => "--verbose --debug" end end
  64. Pain points: Node destroy/up requires cert clean on master plus

    a PuppetDB deactivate.
  65. Common Deploy Process Across Vagrant and EC2

  66. Getting your Puppet code to your masters. Solved?

  67. Goal: Make it the same for Vagrant, production, and anything

    in between.
  68. My solution: Bash script to git pull, rsync, and restart

    the master.
  69. fab deploy:vagrant fab deploy:production fab deploy:dev-mitchellh http://fabfile.org

  70. @task def deploy(environment): # ... run("sudo /opt/puppet-updater/update")

  71. - Git pull - Find env-* branches for environments. -

    RSync - Restart Puppet Master http://bit.ly/Qyg3RW Updater Script
  72. Note: I don’t use Puppet environments for dev because I

    like to keep production master just for production.
  73. (Plus, the automated Puppet Master setup is just so easy!)

  74. Repeatable Workflow From Dev to Staging to Production

  75. Dev is in VirtualBox. Staging is in EC2. Production is

    in EC2.
  76. Goal: Same workflow.

  77. rake launch:vagrant,postgresql rake destroy:production,riak-001 rake provision:staging,haproxy-002

  78. Wrapper around vagrant and AWS library.

  79. Hides some cruft: destroy will cert clean and deactivate from

    PuppetDB, for example.
  80. Golden Master Box Creation for Development

  81. vagrant up a complete dev environment can be slow.

  82. Take advantage of Puppet’s idempotence and vagrant package

  83. Two-pass Puppet run for development.

  84. Pass 1 (pre-package): Installation and configuration.

  85. Pass 2 (vagrant up): Service starting and maybe configuration.

  86. vagrant package takes current Vagrant VM and produces a distributable

    box.
  87. Build discipline around updating the base box.

  88. Example: Work on any box you want, update to latest

    base box prior to committing.
  89. Bonus points: Put this in a CI.

  90. Automate All the things

  91. THANKS! @mitchellh