Advanced Vagrant Usage with Puppet

Slide 1

Slide 1 text

Slide 2

Slide 2 text

I’m Mitchell Hashimoto Also known as @mitchellh

Slide 3

Slide 3 text

I made Vagrant. Hopefully you use it. I think you’ll like it. http://vagrantup.com

Slide 4

Slide 4 text

I’m an automation freak. This talk will show this to be true.

Slide 5

Slide 5 text

Vagrant Usage (Ops Focused)

Slide 6

Slide 6 text

Benefits we want from Vagrant...

Slide 7

Slide 7 text

“The Cloud” but on your machine.

Slide 8

Slide 8 text

Self service. Instant provisioning. Cost efficient. Elastic. Pay per use. Paul Strong’s Cloud

Slide 9

Slide 9 text

- Manifest development, both simple and not so simple Benefits We Want

Slide 10

Slide 10 text

- Manifest development, both simple and not so simple - Repeatability Benefits We Want

Slide 11

Slide 11 text

- Manifest development, both simple and not so simple - Repeatability - Fast feedback Benefits We Want

Slide 12

Slide 12 text

- Manifest development, both simple and not so simple - Repeatability - Fast feedback - Confidence Benefits We Want

Slide 13

Slide 13 text

Confession: I’ve been doing Puppet full time for awhile now.

Slide 14

Slide 14 text

Current state of Vagrant + Puppet...

Slide 15

Slide 15 text

Basic manifest development and testing.

Slide 16

Slide 16 text

... Yep.

Slide 17

Slide 17 text

We can do better. We can do much better.

Slide 18

Slide 18 text

We can do better with what is available right now.

Slide 19

Slide 19 text

Teaser: We will do magic with what is coming in the future. <3 <3 <3

Slide 20

Slide 20 text

My state of Vagrant + Puppet...

Slide 21

Slide 21 text

Fully automated Puppet Master setup.

Slide 22

Slide 22 text

Testing exported resources, hiera, and nodes.

Slide 23

Slide 23 text

Common deploy process across Vagrant and EC2.

Slide 24

Slide 24 text

Repeatable workflow of dev to staging to prod.

Slide 25

Slide 25 text

Golden master box creation for development.

Slide 26

Slide 26 text

Time to share what I’ve learned.

Slide 27

Slide 27 text

Advanced Vagrant Usage

Slide 28

Slide 28 text

Advanced Vagrant Usage

Slide 29

Slide 29 text

Advanced Automation for Puppet Work

Slide 30

Slide 30 text

Fully Automated Puppet Master Setup

Slide 31

Slide 31 text

I asked: “How do people bring up or recover a Puppet master?”

Slide 32

Slide 32 text

“Most people roll their Puppet Master by hand.” - Anonymous PuppetLabs Employee

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Puppet Master is crucial to testing realistic scenarios.

Slide 35

Slide 35 text

Automated Puppet Master - Pushing broken Puppet crashes the server.

Slide 36

Slide 36 text

Automated Puppet Master - Pushing broken Puppet crashes the server. - Local development against a Puppet Master has benefits.

Slide 37

Slide 37 text

Automated Puppet Master - Pushing broken Puppet crashes the server. - Local development against a Puppet Master has benefits. - Automation all the way down.

Slide 38

Slide 38 text

Multi-level bootstrap.

Slide 39

Slide 39 text

1. Bash script to minimally install Puppet master and agent.

Slide 40

Slide 40 text

2. puppet apply to minimally setup Puppet Master infrastructure

Slide 41

Slide 41 text

3. puppet agent to completely setup and harden the master.

Slide 42

Slide 42 text

Result: Production-quality Puppet Master whenever you need it.

Slide 43

Slide 43 text

Vagrant::Config.run do |config| # ... config.vm.hostname = "puppet" config.vm.provision :shell, :path => "bootstrap.sh" end

Slide 44

Slide 44 text

Testing Exported Resources, Hiera, and nodes.

Slide 45

Slide 45 text

I asked: “How do people test more than the most basic Puppet module?”

Slide 46

Slide 46 text

“I suspect the answer is that they just don't test their modules adequately.” - Anonymous PuppetLabs Employee

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Solution: Automated Puppet Master + Multi-VM

Slide 49

Slide 49 text

Automated Puppet Master is production ready: PuppetDB, Hiera, etc.

Slide 50

Slide 50 text

Multi-VM enables Vagrant to manage a cluster of machines that can communicate. http://vagrantup.com/v1/docs/multivm.html

Slide 51

Slide 51 text

Testing Exported Resources

Slide 52

Slide 52 text

Create two nodes. Export one. Collect other. Ruby/Shell script. Testing Exported Resources

Slide 53

Slide 53 text

node 'test_exporter' { @@nginx::site { "test": content => "\n", tag => "origin", } } node 'test_collector' { include role::origin }

Slide 54

Slide 54 text

Vagrant::Config.run do |config| config.vm.define :export do |n| n.vm.hostname = "test_exporter" n.vm.provision :puppet_server, :options => "--verbose --debug" end config.vm.define :collect do |n| n.vm.hostname = "test_collecter" n.vm.provision :puppet_server, :options => "--verbose --debug" end end

Slide 55

Slide 55 text

#!/bin/bash test -f /etc/nginx/sites-available/test

Slide 56

Slide 56 text

Testing Hiera

Slide 57

Slide 57 text

Create full Hiera hierarchy. Launch node. Test hierarchy.

Slide 58

Slide 58 text

--- test_region: "us-east-1" --- test_role: "hiera" --- test_name: "test-hiera”

Slide 59

Slide 59 text

Vagrant::Config.run do |config| config.vm.hostname = "test_hiera" config.vm.provision :puppet_server, :options => "--verbose --debug" end end

Slide 60

Slide 60 text

node 'test_hiera' { $region = hiera("test_region") $role = hiera("test_role") $name = hiera("test_name") file { "/tmp/results": content => "$region $role $name", mode => "0644", } }

Slide 61

Slide 61 text

Testing Nodes

Slide 62

Slide 62 text

Create node. Provision. Test behavior.

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Pain points: Node destroy/up requires cert clean on master plus a PuppetDB deactivate.

Slide 65

Slide 65 text

Common Deploy Process Across Vagrant and EC2

Slide 66

Slide 66 text

Getting your Puppet code to your masters. Solved?

Slide 67

Slide 67 text

Goal: Make it the same for Vagrant, production, and anything in between.

Slide 68

Slide 68 text

My solution: Bash script to git pull, rsync, and restart the master.

Slide 69

Slide 69 text

fab deploy:vagrant fab deploy:production fab deploy:dev-mitchellh http://fabﬁle.org

Slide 70

Slide 70 text

@task def deploy(environment): # ... run("sudo /opt/puppet-updater/update")

Slide 71

Slide 71 text

- Git pull - Find env-* branches for environments. - RSync - Restart Puppet Master http://bit.ly/Qyg3RW Updater Script

Slide 72

Slide 72 text

Note: I don’t use Puppet environments for dev because I like to keep production master just for production.

Slide 73

Slide 73 text

(Plus, the automated Puppet Master setup is just so easy!)

Slide 74

Slide 74 text

Repeatable Workﬂow From Dev to Staging to Production

Slide 75

Slide 75 text

Dev is in VirtualBox. Staging is in EC2. Production is in EC2.

Slide 76

Slide 76 text

Goal: Same workflow.

Slide 77

Slide 77 text

rake launch:vagrant,postgresql rake destroy:production,riak-001 rake provision:staging,haproxy-002

Slide 78

Slide 78 text

Wrapper around vagrant and AWS library.

Slide 79

Slide 79 text

Hides some cruft: destroy will cert clean and deactivate from PuppetDB, for example.

Slide 80

Slide 80 text

Golden Master Box Creation for Development

Slide 81

Slide 81 text

vagrant up a complete dev environment can be slow.

Slide 82

Slide 82 text

Take advantage of Puppet’s idempotence and vagrant package

Slide 83

Slide 83 text

Two-pass Puppet run for development.

Slide 84

Slide 84 text

Pass 1 (pre-package): Installation and configuration.

Slide 85

Slide 85 text

Pass 2 (vagrant up): Service starting and maybe configuration.

Slide 86

Slide 86 text

vagrant package takes current Vagrant VM and produces a distributable box.

Slide 87

Slide 87 text

Build discipline around updating the base box.

Slide 88

Slide 88 text

Example: Work on any box you want, update to latest base box prior to committing.