Slide 1
Slide 1 text
Virtualize Your Stack With
Vagrant and Puppet
Jacob Mather
Software Engineer at Mashery
Slide 2
Slide 2 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Who am I?
What I do:
• Software Engineer at Mashery
• Co-organizer of the
San Francisco PHP User Group
• Community Evangelist
Where I can be found:
• Blog: http://jmather.com
• Twitter: @thejmather
Slide 3
Slide 3 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Why I am giving this talk
Slide 4
Slide 4 text
Let’s talk about
Vagrant
Slide 5
Slide 5 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
What is Vagrant?
A tool that allows you to create and configure lightweight,
reproducible, and portable development environments.
Slide 6
Slide 6 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant is built...
• in Ruby
• to be simple
Slide 7
Slide 7 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant terms
• Box, or base box, is the ‘initial image’ of a virtual machine.
• Host, or your computer, the machine running the VMs
• Guest, or vm, referring to an individual VM instance
• Provider, an adapter to a VMS (like VirtualBox or VMWare)
• Provisioner, provides post-boot configuration of guests
Slide 8
Slide 8 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
How to use Vagrant
• Install Vagrant - http://vagrantup.com
• Check out your repository
• Run: vagrant up
Slide 9
Slide 9 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrant Commands
• vagrant up - Turn guest on
• vagrant halt - Turn guest off
• vagrant status - Show guest status
• vagrant destroy - Delete guest
• vagrant suspend - Suspend guest
• vagrant resume - Resume guest
• vagrant ssh - SSH into guest
Slide 10
Slide 10 text
Let’s talk about
code
Slide 11
Slide 11 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "base-box-name"
config.vm.box_url = ”http://some/url.box"
end
Slide 12
Slide 12 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (add file system)
Vagrant.configure("2") do |config|
config.vm.box = "base-box-name"
config.vm.box_url = ”http://some/url.box"
config.vm.synced_folder "/data", "/vagrant_data"
end
Slide 13
Slide 13 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (add port forwarding)
Vagrant.configure("2") do |config|
config.vm.box = "base-box-name"
config.vm.box_url = ”http://some/url.box"
config.vm.synced_folder "/data", "/vagrant_data"
config.vm.network :forwarded_port, guest:80, host: 8080
end
Slide 14
Slide 14 text
Let’s talk about
clouds
Slide 15
Slide 15 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Replicating production matters
If you are writing code that will be deployed to a multiple-host
environment, why aren’t you writing your code in a multiple-
host environment?
Slide 16
Slide 16 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile
Vagrant.configure("2") do |config|
config.vm.hostname = “box”
config.vm.box = "base-box-name"
config.vm.box_url = ”http://some/url.box"
end
Slide 17
Slide 17 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (Multi-Guest ready)
Vagrant.configure("2") do |config|
config.vm.define “box” do |node|
node.vm.hostname = “box”
node.vm.box = "base-box-name"
node.vm.box_url = ”http://some/url.box"
end
end
Slide 18
Slide 18 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (Multi-Guest)
Vagrant.configure("2") do |config|
config.vm.define “box” do |node|
node.vm.hostname = “box”
node.vm.box = "base-box-name"
node.vm.box_url = ”http://some/url.box"
end
config.vm.define “other-box” do |node|
node.vm.hostname = “other-box”
node.vm.box = "base-box-name"
node.vm.box_url = ”http://some/url.box"
end
end
Slide 19
Slide 19 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (Multi-Guest, useful, part 1)
nodes = {
:'web-server' => {
:hostname => 'server.example.com',
:ipaddress => '192.168.56.60',
},
:'db-server' => {
:hostname => 'db.example.com',
:ipaddress => '192.168.56.61',
}
}
Slide 20
Slide 20 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (Multi-Guest, useful, part 2)
nodes = { … node configuration … }
Vagrant.configure("2") do |config|
nodes.each_pair do |name,options|
config.vm.define name do |node|
node.vm.box = "base-box-name"
node.vm.box_url = "http://some/url.box"
node.vm.hostname = options[:hostname]
node.vm.network :private_network, ip: options[:ipaddress]
end
end
end
Slide 21
Slide 21 text
Let’s talk about
Providers
Slide 22
Slide 22 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Providers
Providers
Vagrant
Slide 23
Slide 23 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Providers
VirtualBox
VMWare (WS or Fusion)
Parallels Desktop
Providers
Vagrant
Slide 24
Slide 24 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Providers
VirtualBox
VMWare (WS or Fusion)
Parallels Desktop
Amazon AWS
Digital Ocean
Providers
Vagrant
Slide 25
Slide 25 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Providers (desktop)
• VirtualBox is FREE! (sort of)
• Some people have trouble with crashes
• Some people have trouble with suspend/resume
• Built in FS bridging is pretty slow
• VMWare (Workstation or Fusion!) is PAID!
• You have to buy both VMWare ($49) and provider ($79)
• Supposed to be more stable and faster
• Parallels Desktop is FREE!
• Open source projects FOR THE WIN!
Slide 26
Slide 26 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Providers (cloud)
• Amazon AWS
• Digital Ocean
• RackSpace Cloud
Slide 27
Slide 27 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Provider Limits
Spinning up guests is cool, but what do I do with a box once
it’s up? How do I control it? How do I make it useful?
Computing hardware without software is nearly useless.
Slide 28
Slide 28 text
Let’s talk about
Provisioners
Slide 29
Slide 29 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Provisioners
Provisioners
Vagrant
Slide 30
Slide 30 text
Shell Scripts
Puppet
Chef
Ansible
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Provisioners
Provisioners
Vagrant
Slide 31
Slide 31 text
Let’s talk about
Puppet
Slide 32
Slide 32 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
What is Puppet?
Puppet is IT automation software that helps system administrators
manage infrastructure throughout its lifecycle, from provisioning
and configuration to patch management and compliance.
Slide 33
Slide 33 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
What?
Slide 34
Slide 34 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
What is Puppet in simpler words?
Puppet gives you an object oriented way to manage the
configuration of your servers, either on your laptop, in the cloud, or
in your favorite data center.
Slide 35
Slide 35 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet is built...
• to be simple
• with it’s own declarative language
Slide 36
Slide 36 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet terms
• Manifests - This is “application” code
• Modules - This is “library” code
• Templates - Exactly as it sounds
• Facter - Environment data
• Hiera - Configuration data
Slide 37
Slide 37 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
How to use Puppet?
Select a base box for Vagrant from Puppet Labs
http://puppet-vagrant-boxes.puppetlabs.com/
I tend to use CentOS 6.4 for VirtualBox
Slide 38
Slide 38 text
Let’s talk about
code
Slide 39
Slide 39 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Vagrantfile (add Puppet)
Vagrant.configure("2") do |config|
config.vm.box = "base-box-name"
config.vm.box_url = "http://some/url.box"
config.vm.provision :puppet do |puppet|
puppet.manifests_path = "puppet/manifests/"
puppet.manifest_file = "guest.pp"
puppet.module_path = "puppet/modules/"
end
end
Slide 40
Slide 40 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Puppet Manifest (guest.pp)
class guest {
notify { “Look! My puppet code is working!”: }
}
Slide 41
Slide 41 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Puppet Manifest (add Apache)
class guest {
notify { “Look! My puppet code is working!”: }
package { “httpd”:
ensure => present,
}
service { “httpd”:
require => Package[“httpd”],
ensure => running,
enabled => true,
}
}
Slide 42
Slide 42 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Basic Puppet Manifest (refactored Apache)
# manifests/guest.pp
class guest {
notify { “Code working!”: }
require server::httpd
}
# modules/server/httpd.pp
class server::httpd {
package { “httpd”:
ensure => present,
}
service { “httpd”:
require => Package[“httpd”],
ensure => running,
enabled => true,
}
}
Slide 43
Slide 43 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (controlling flow)
class guest {
notify { “Say me first”: }
notify { “Say me second”:
require => Notify[“Say me first”],
}
notify { “Say me third”:
require => Notify[“Say me second”],
}
}
Slide 44
Slide 44 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (adding parameters)
# modules/server/writetestfile.pp
class server::writetestfile ($content) {
file { “/tmp/test”:
content => $content,
}
}
# modules/server/writetestfiledefault.pp
class server::writetestfiledefault ($content = “default”) {
file { “/tmp/test”:
content => $content
}
}
Slide 45
Slide 45 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (passing parameters)
# manifests/guest.pp -- using defaults
class guest {
include server::writetestfiledefault
}
# manifests/guest.pp -- passing custom value
class guest {
class { “server::writetestfiledefault”:
content => “test content”,
}
}
Slide 46
Slide 46 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (including classes)
class guest {
# the class below must be loaded BEFORE guest
require server::writetestfiledefault
# the class below must be loaded WITH guest
include server::writetestfiledefault
# the class below must be loaded AFTER guest
class { “server::writetestfiledefault”:
content => “test content”,
}
}
Slide 47
Slide 47 text
Let’s talk about
how to do things conditionally
Slide 48
Slide 48 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
External Input in Puppet
Remember there are three primary sources of where data that
branches your logic will come from:
• User Space Variables
• $things = “you set yourself”
• Facts
• Specific details about your current environment, such as
the name of the host, ip address, operating system, etc...
• Hiera
• Configuration you pass in to enable building for multiple
configurations from the same shared codebase
Slide 49
Slide 49 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (variables)
class guest {
# local variable
$host = “mybox”
if ($host == “mybox”) { notify { “Box!”: } } # Box!
if ($host == “mybox”) { notify { “${host}!”: } } # mybox!
}
Slide 50
Slide 50 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (facts)
class guest {
# fact - environment configuration
if ($::hostname == “box”) { notify { “Box!”: } } # output: Box!
if ($hostname == “box”) { notify { “Box!”: } } # output: Box!
# assign $hostname locally...
$hostname = “some string”
if ($::hostname == “box”) { notify { “Box!”: } } # output: Box!
if ($hostname == “box”) { notify { “Box!”: } } # no output
}
Slide 51
Slide 51 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (facts, debugging)
class guest {
# what facts are there? Let’s find out!
# this should be one line, but it is just too long to fit
$template = "<%= scope.to_hash.reject { |k,v| !( k.is_a?
(String) && v.is_a?(String) ) }.to_yaml %>"
# create a file...
file { "/tmp/facts.yaml":
content => inline_template($template),
}
}
Slide 52
Slide 52 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (if conditionals)
class guest {
if (“a” == “b”) {
notify { “Inconceivable!”: }
} elsif (“hello” =~ “help”) {
notify { “Regex!”: }
} else {
notify { “Echo something...”: }
}
}
Slide 53
Slide 53 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (case conditionals)
class guest {
# $::operatingsystem is a fact, provided directly by Facter
case $::operatingsystem {
centos, redhat: { $apache = "httpd" }
debian, ubuntu: { $apache = "apache2" }
default: { fail("Unrecognized operating system") }
}
case $::operatingsystem {
/linux/: { fail(“We don’t support anything with Linux in it”) }
}
}
Slide 54
Slide 54 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Puppet Manifest (selector conditionals)
class guest {
# $::operatingsystem is a fact, provided directly by Facter
$package = $::operatingsystem ? {
centos => “httpd”,
redhat => “httpd”,
/(i?)(ubuntu|debian)/ => “apache2”,
default => undef
}
}
Slide 55
Slide 55 text
Let’s talk about
real multiple guest configuration
Slide 56
Slide 56 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Configure our guest nodes...
nodes = {
:'web-server' => {
:hostname => 'server.example.com',
:ipaddress => '192.168.56.60',
},
:'db-server' => {
:hostname => 'db.example.com',
:ipaddress => '192.168.56.61',
}
}
Slide 57
Slide 57 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Tell Vagrant about them
nodes = { … node configuration … }
Vagrant.configure("2") do |config|
nodes.each_pair do |name,options|
config.vm.define name do |node|
node.vm.box = "base-box-name"
node.vm.box_url = "http://some/url.box"
node.vm.hostname = options[:hostname]
node.vm.network :private_network, ip: options[:ipaddress]
end
end
end
Slide 58
Slide 58 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Tell Vagrant about Puppet
nodes = { … node configuration … }
Vagrant.configure("2") do |config|
nodes.each_pair do |name,options|
config.vm.define name do |node|
node.vm... # omit general config
node.vm.provision :puppet do |puppet|
puppet.manifests_path = "puppet/manifests/"
puppet.manifest_file = "guests.pp"
puppet.module_path = "puppet/modules/"
end
end
end
end
Slide 59
Slide 59 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Tell Puppet about the guest nodes
class guests {
if ($::hostname == “web-server”) {
# only run for web server
}
if ($::hostname == “db-server”) {
# only run for db server
}
}
Slide 60
Slide 60 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Tell Puppet about the guest nodes (redux)
class guests {
if ($::hostname =~ /^web-/) {
# only run for web servers
}
if ($::hostname =~ /^db-/) {
# only run for db servers
}
}
Slide 61
Slide 61 text
Let’s talk about
how everything ties together
Slide 62
Slide 62 text
Shell Scripts
Puppet
Chef
Ansible
VirtualBox
VMWare (WS or Fusion)
Parallels Desktop
Amazon AWS
Digital Ocean
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Vagrant Architecture
Providers
Provisioners
Vagrant
Slide 63
Slide 63 text
Let’s talk about
how to convince your boss
Slide 64
Slide 64 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Development environments come in all
shapes and sizes
• Some develop in production
– Yes, really.
• Some develop on a non production server
– Not a copy of production, not by a long shot
• Some develop locally on a “non production server”
– Home grown solutions that “work” until they don’t
• Some develop on a development server
– Actually tries to be like production
– Not always like production
– May have additional tuning to help with developmenty things
• Some develop in a true development environment
– Locally or remote, coming soon to a computer near you
Slide 65
Slide 65 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Why developing in production is really bad
Assuming, of course, that you are testing your code before you
make it live.
Production should always move from one stable version of code to
another stable version of code. When you edit code in production,
you inherently introduce instability, and can no longer be assured
that your code is running as tested.
Slide 66
Slide 66 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Why developing on a non production setup
is also bad
Debugging environmental issues is slow and painful.
Developers should be solving problems with software, not
debugging configuration issues.
When you build your code in an environment that isn’t related to
production in any tangible way, you will often discover just how
different your development environment is from production.
Slide 67
Slide 67 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Where your development environment lives
Remote
• Easy for operations to set up
• Easy to enforce a rigid structure
• Limits developer’s choice of tool
chain selection
• Can get expensive and
complicated to maintain
• Developers must be online to do
any development
Local
• Can require beefier hardware
• Can take extra work to set up
• Developers have the freedom to
work how they work best
• Developers can assist with
infrastructural initiatives
Slide 68
Slide 68 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Bad development environments will sap the resources out of
your team. In addition to the time spent developing and testing
new code and fixing bugs, you will spend time:
When development environments attack
Debugging broken environments or settings
Debugging broken behavior between dev and prod
Debugging broken behavior between two dev systems
Working in broken ways because “it works”
Slide 69
Slide 69 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Failure to specifically build an environment for
development will directly impact developer
productivity and product quality.
Slide 70
Slide 70 text
Let’s talk about
what you should remember
Slide 71
Slide 71 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Takeaways
• Bad (or unstable) environments (be it dev, qa, staging, or
prod) hurt developers.
• Puppet lets you automate all of the configuration of your
systems (dev, qa, staging, and prod), ensuring
consistency between them.
• Vagrant is easy to use, and will let you use your
production puppet configs to build development
environments.
• You will manage server configuration, whether you
decide to or not. If you choose not to do it formally it will
come back to bite you when you forget something later.
Slide 72
Slide 72 text
Let’s talk about
where to go from here
Slide 73
Slide 73 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
If you’re still not sold...
h"p://PuPHPet.com/
/
PuPHPet/provides/a/
simple/config/tool/for/
those/who/don’t/want/
to/=nker/with/puppet/
and/need/a/LAMP/stack/
up/and/running/NOW./
V2 was released this morning!!!
Slide 74
Slide 74 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
And also look at...
• Packer
• http://packer.io
• Take your Vagrant and Puppet usage to the next level by
using Packer to pre-provision images for easy first spins
• Build prebuilt systems for multiple VMS and cloud
platforms
• Docker
• http://docker.io
• A different idea of how to build multiple machine VM
environments
Slide 75
Slide 75 text
Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
Thank you!
More information about this talk can be found at:
http://jmather.com/talks/2013-10-12