Advanced Vagrant Usage with Puppet

This talk was given at PuppetConf 2012. It covers how I use Vagrant with Puppet and how I automate a lot of what I do with Puppet.

Transcript

1. Advanced Vagrant Usage with Puppet
2. I’m Mitchell Hashimoto Also known as @mitchellh
3. I made Vagrant. Hopefully you use it. I think you’ll like it. http://vagrantup.com
4. I’m an automation freak. This talk will show this to be true.
5. Vagrant Usage (Ops Focused)
6. Benefits we want from Vagrant...
7. “The Cloud” but on your machine.
8. Self service. Instant provisioning. Cost efficient. Elastic. Pay per use. Paul Strong’s Cloud
9. - Manifest development, both simple and not so simple Benefits We Want
10. - Manifest development, both simple and not so simple - Repeatability Benefits We Want
11. - Manifest development, both simple and not so simple - Repeatability - Fast feedback Benefits We Want
12. - Manifest development, both simple and not so simple - Repeatability - Fast feedback - Confidence Benefits We Want
13. Confession: I’ve been doing Puppet full time for awhile now.
14. Current state of Vagrant + Puppet...
15. Basic manifest development and testing.
16. ... Yep.
17. We can do better. We can do much better.
18. We can do better with what is available right now.
19. Teaser: We will do magic with what is coming in the future. <3 <3 <3
20. My state of Vagrant + Puppet...
21. Fully automated Puppet Master setup.
22. Testing exported resources, hiera, and nodes.
23. Common deploy process across Vagrant and EC2.
24. Repeatable workflow of dev to staging to prod.
25. Golden master box creation for development.
26. Time to share what I’ve learned.
27. Advanced Vagrant Usage
28. Advanced Vagrant Usage
29. Advanced Automation for Puppet Work
30. Fully Automated Puppet Master Setup
31. I asked: “How do people bring up or recover a Puppet master?”
32. “Most people roll their Puppet Master by hand.” - Anonymous PuppetLabs Employee
34. Puppet Master is crucial to testing realistic scenarios.
35. Automated Puppet Master - Pushing broken Puppet crashes the server.
36. Automated Puppet Master - Pushing broken Puppet crashes the server. - Local development against a Puppet Master has benefits.
37. Automated Puppet Master - Pushing broken Puppet crashes the server. - Local development against a Puppet Master has benefits. - Automation all the way down.
38. Multi-level bootstrap.
39. 1. Bash script to minimally install Puppet master and agent.
40. 2. puppet apply to minimally setup Puppet Master infrastructure
41. 3. puppet agent to completely setup and harden the master.
42. Result: Production-quality Puppet Master whenever you need it.
43. Vagrant::Config.run do |config| # ... config.vm.hostname = "puppet" config.vm.provision :shell, :path => "bootstrap.sh" end
44. Testing Exported Resources, Hiera, and nodes.
45. I asked: “How do people test more than the most basic Puppet module?”
46. “I suspect the answer is that they just don't test their modules adequately.” - Anonymous PuppetLabs Employee
48. Solution: Automated Puppet Master + Multi-VM
49. Automated Puppet Master is production ready: PuppetDB, Hiera, etc.
50. Multi-VM enables Vagrant to manage a cluster of machines that can communicate. http://vagrantup.com/v1/docs/multivm.html
51. Testing Exported Resources
52. Create two nodes. Export one. Collect other. Ruby/Shell script. Testing Exported Resources
53. node 'test_exporter' { @@nginx::site { "test": content => "\n", tag => "origin", } } node 'test_collector' { include role::origin }
54. Vagrant::Config.run do |config| config.vm.define :export do |n| n.vm.hostname = "test_exporter" n.vm.provision :puppet_server, :options => "--verbose --debug" end config.vm.define :collect do |n| n.vm.hostname = "test_collecter" n.vm.provision :puppet_server, :options => "--verbose --debug" end end
55. #!/bin/bash test -f /etc/nginx/sites-available/test
56. Testing Hiera
57. Create full Hiera hierarchy. Launch node. Test hierarchy.
58. --- test_region: "us-east-1" --- test_role: "hiera" --- test_name: "test-hiera”
59. Vagrant::Config.run do |config| config.vm.hostname = "test_hiera" config.vm.provision :puppet_server, :options => "--verbose --debug" end end
60. node 'test_hiera' { $region = hiera("test_region") $role = hiera("test_role") $name = hiera("test_name") file { "/tmp/results": content => "$region $role $name", mode => "0644", } }
61. Testing Nodes
62. Create node. Provision. Test behavior.
63. Vagrant::Config.run do |config| config.vm.define :master do |master| master.vm.hostname = "puppet" master.vm.provision :shell, :path => "bootstrap.sh" end config.vm.define :node do |node| node.vm.hostname = "postgresql" node.vm.provision :puppet_server, :options => "--verbose --debug" end end
64. Pain points: Node destroy/up requires cert clean on master plus a PuppetDB deactivate.
65. Common Deploy Process Across Vagrant and EC2
66. Getting your Puppet code to your masters. Solved?
67. Goal: Make it the same for Vagrant, production, and anything in between.
68. My solution: Bash script to git pull, rsync, and restart the master.
69. fab deploy:vagrant fab deploy:production fab deploy:dev-mitchellh http://fabfile.org
70. @task def deploy(environment): # ... run("sudo /opt/puppet-updater/update")
71. - Git pull - Find env-* branches for environments. - RSync - Restart Puppet Master http://bit.ly/Qyg3RW Updater Script
72. Note: I don’t use Puppet environments for dev because I like to keep production master just for production.
73. (Plus, the automated Puppet Master setup is just so easy!)
74. Repeatable Workflow From Dev to Staging to Production
75. Dev is in VirtualBox. Staging is in EC2. Production is in EC2.
76. Goal: Same workflow.
77. rake launch:vagrant,postgresql rake destroy:production,riak-001 rake provision:staging,haproxy-002
78. Wrapper around vagrant and AWS library.
79. Hides some cruft: destroy will cert clean and deactivate from PuppetDB, for example.
80. Golden Master Box Creation for Development
81. vagrant up a complete dev environment can be slow.
82. Take advantage of Puppet’s idempotence and vagrant package
83. Two-pass Puppet run for development.
84. Pass 1 (pre-package): Installation and configuration.
85. Pass 2 (vagrant up): Service starting and maybe configuration.
86. vagrant package takes current Vagrant VM and produces a distributable box.
87. Build discipline around updating the base box.
88. Example: Work on any box you want, update to latest base box prior to committing.
89. Bonus points: Put this in a CI.
90. Automate All the things
91. THANKS! @mitchellh