Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtualize your stack with Vagrant and Puppet - Columbus Code Camp 2013 Edition

Jacob Mather
October 12, 2013

Virtualize your stack with Vagrant and Puppet - Columbus Code Camp 2013 Edition

These are the slides from my Virtualizing your stack with Vagrant and Puppet talk, as done at Columbus Code Camp.

For more information on this talk, go to: http://jmather.com/talks/2013-10-12

Jacob Mather

October 12, 2013
Tweet

More Decks by Jacob Mather

Other Decks in Programming

Transcript

  1. Virtualize Your Stack With
    Vagrant and Puppet
    Jacob Mather
    Software Engineer at Mashery

    View full-size slide

  2. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Who am I?
    What I do:
    • Software Engineer at Mashery
    • Co-organizer of the
    San Francisco PHP User Group
    • Community Evangelist
    Where I can be found:
    • Blog: http://jmather.com
    • Twitter: @thejmather

    View full-size slide

  3. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Why I am giving this talk

    View full-size slide

  4. Let’s talk about
    Vagrant

    View full-size slide

  5. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    What is Vagrant?
    A tool that allows you to create and configure lightweight,
    reproducible, and portable development environments.

    View full-size slide

  6. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant is built...
    • in Ruby
    • to be simple

    View full-size slide

  7. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant terms
    • Box, or base box, is the ‘initial image’ of a virtual machine.
    • Host, or your computer, the machine running the VMs
    • Guest, or vm, referring to an individual VM instance
    • Provider, an adapter to a VMS (like VirtualBox or VMWare)
    • Provisioner, provides post-boot configuration of guests

    View full-size slide

  8. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    How to use Vagrant
    • Install Vagrant - http://vagrantup.com
    • Check out your repository
    • Run: vagrant up

    View full-size slide

  9. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrant Commands
    • vagrant up - Turn guest on
    • vagrant halt - Turn guest off
    • vagrant status - Show guest status
    • vagrant destroy - Delete guest
    • vagrant suspend - Suspend guest
    • vagrant resume - Resume guest
    • vagrant ssh - SSH into guest

    View full-size slide

  10. Let’s talk about
    code

    View full-size slide

  11. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile
    Vagrant.configure("2") do |config|
    config.vm.box = "base-box-name"
    config.vm.box_url = ”http://some/url.box"
    end

    View full-size slide

  12. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (add file system)
    Vagrant.configure("2") do |config|
    config.vm.box = "base-box-name"
    config.vm.box_url = ”http://some/url.box"
    config.vm.synced_folder "/data", "/vagrant_data"
    end

    View full-size slide

  13. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (add port forwarding)
    Vagrant.configure("2") do |config|
    config.vm.box = "base-box-name"
    config.vm.box_url = ”http://some/url.box"
    config.vm.synced_folder "/data", "/vagrant_data"
    config.vm.network :forwarded_port, guest:80, host: 8080
    end

    View full-size slide

  14. Let’s talk about
    clouds

    View full-size slide

  15. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Replicating production matters
    If you are writing code that will be deployed to a multiple-host
    environment, why aren’t you writing your code in a multiple-
    host environment?

    View full-size slide

  16. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile
    Vagrant.configure("2") do |config|
    config.vm.hostname = “box”
    config.vm.box = "base-box-name"
    config.vm.box_url = ”http://some/url.box"
    end

    View full-size slide

  17. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (Multi-Guest ready)
    Vagrant.configure("2") do |config|
    config.vm.define “box” do |node|
    node.vm.hostname = “box”
    node.vm.box = "base-box-name"
    node.vm.box_url = ”http://some/url.box"
    end
    end

    View full-size slide

  18. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (Multi-Guest)
    Vagrant.configure("2") do |config|
    config.vm.define “box” do |node|
    node.vm.hostname = “box”
    node.vm.box = "base-box-name"
    node.vm.box_url = ”http://some/url.box"
    end
    config.vm.define “other-box” do |node|
    node.vm.hostname = “other-box”
    node.vm.box = "base-box-name"
    node.vm.box_url = ”http://some/url.box"
    end
    end

    View full-size slide

  19. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (Multi-Guest, useful, part 1)
    nodes = {
    :'web-server' => {
    :hostname => 'server.example.com',
    :ipaddress => '192.168.56.60',
    },
    :'db-server' => {
    :hostname => 'db.example.com',
    :ipaddress => '192.168.56.61',
    }
    }

    View full-size slide

  20. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (Multi-Guest, useful, part 2)
    nodes = { … node configuration … }
    Vagrant.configure("2") do |config|
    nodes.each_pair do |name,options|
    config.vm.define name do |node|
    node.vm.box = "base-box-name"
    node.vm.box_url = "http://some/url.box"
    node.vm.hostname = options[:hostname]
    node.vm.network :private_network, ip: options[:ipaddress]
    end
    end
    end

    View full-size slide

  21. Let’s talk about
    Providers

    View full-size slide

  22. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Providers
    Providers
    Vagrant

    View full-size slide

  23. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Providers
    VirtualBox
    VMWare (WS or Fusion)
    Parallels Desktop
    Providers
    Vagrant

    View full-size slide

  24. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Providers
    VirtualBox
    VMWare (WS or Fusion)
    Parallels Desktop
    Amazon AWS
    Digital Ocean
    Providers
    Vagrant

    View full-size slide

  25. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Providers (desktop)
    • VirtualBox is FREE! (sort of)
    • Some people have trouble with crashes
    • Some people have trouble with suspend/resume
    • Built in FS bridging is pretty slow
    • VMWare (Workstation or Fusion!) is PAID!
    • You have to buy both VMWare ($49) and provider ($79)
    • Supposed to be more stable and faster
    • Parallels Desktop is FREE!
    • Open source projects FOR THE WIN!

    View full-size slide

  26. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Providers (cloud)
    • Amazon AWS
    • Digital Ocean
    • RackSpace Cloud

    View full-size slide

  27. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Provider Limits
    Spinning up guests is cool, but what do I do with a box once
    it’s up? How do I control it? How do I make it useful?
    Computing hardware without software is nearly useless.

    View full-size slide

  28. Let’s talk about
    Provisioners

    View full-size slide

  29. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Provisioners
    Provisioners
    Vagrant

    View full-size slide

  30. Shell Scripts
    Puppet
    Chef
    Ansible
    Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Provisioners
    Provisioners
    Vagrant

    View full-size slide

  31. Let’s talk about
    Puppet

    View full-size slide

  32. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    What is Puppet?
    Puppet is IT automation software that helps system administrators
    manage infrastructure throughout its lifecycle, from provisioning
    and configuration to patch management and compliance.

    View full-size slide

  33. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    What?

    View full-size slide

  34. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    What is Puppet in simpler words?
    Puppet gives you an object oriented way to manage the
    configuration of your servers, either on your laptop, in the cloud, or
    in your favorite data center.

    View full-size slide

  35. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet is built...
    • to be simple
    • with it’s own declarative language

    View full-size slide

  36. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet terms
    • Manifests - This is “application” code
    • Modules - This is “library” code
    • Templates - Exactly as it sounds
    • Facter - Environment data
    • Hiera - Configuration data

    View full-size slide

  37. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    How to use Puppet?
    Select a base box for Vagrant from Puppet Labs
    http://puppet-vagrant-boxes.puppetlabs.com/
    I tend to use CentOS 6.4 for VirtualBox

    View full-size slide

  38. Let’s talk about
    code

    View full-size slide

  39. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Vagrantfile (add Puppet)
    Vagrant.configure("2") do |config|
    config.vm.box = "base-box-name"
    config.vm.box_url = "http://some/url.box"
    config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests/"
    puppet.manifest_file = "guest.pp"
    puppet.module_path = "puppet/modules/"
    end
    end

    View full-size slide

  40. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Puppet Manifest (guest.pp)
    class guest {
    notify { “Look! My puppet code is working!”: }
    }

    View full-size slide

  41. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Puppet Manifest (add Apache)
    class guest {
    notify { “Look! My puppet code is working!”: }
    package { “httpd”:
    ensure => present,
    }
    service { “httpd”:
    require => Package[“httpd”],
    ensure => running,
    enabled => true,
    }
    }

    View full-size slide

  42. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Basic Puppet Manifest (refactored Apache)
    # manifests/guest.pp
    class guest {
    notify { “Code working!”: }
    require server::httpd
    }
    # modules/server/httpd.pp
    class server::httpd {
    package { “httpd”:
    ensure => present,
    }
    service { “httpd”:
    require => Package[“httpd”],
    ensure => running,
    enabled => true,
    }
    }

    View full-size slide

  43. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (controlling flow)
    class guest {
    notify { “Say me first”: }
    notify { “Say me second”:
    require => Notify[“Say me first”],
    }
    notify { “Say me third”:
    require => Notify[“Say me second”],
    }
    }

    View full-size slide

  44. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (adding parameters)
    # modules/server/writetestfile.pp
    class server::writetestfile ($content) {
    file { “/tmp/test”:
    content => $content,
    }
    }
    # modules/server/writetestfiledefault.pp
    class server::writetestfiledefault ($content = “default”) {
    file { “/tmp/test”:
    content => $content
    }
    }

    View full-size slide

  45. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (passing parameters)
    # manifests/guest.pp -- using defaults
    class guest {
    include server::writetestfiledefault
    }
    # manifests/guest.pp -- passing custom value
    class guest {
    class { “server::writetestfiledefault”:
    content => “test content”,
    }
    }

    View full-size slide

  46. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (including classes)
    class guest {
    # the class below must be loaded BEFORE guest
    require server::writetestfiledefault
    # the class below must be loaded WITH guest
    include server::writetestfiledefault
    # the class below must be loaded AFTER guest
    class { “server::writetestfiledefault”:
    content => “test content”,
    }
    }

    View full-size slide

  47. Let’s talk about
    how to do things conditionally

    View full-size slide

  48. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    External Input in Puppet
    Remember there are three primary sources of where data that
    branches your logic will come from:
    • User Space Variables
    • $things = “you set yourself”
    • Facts
    • Specific details about your current environment, such as
    the name of the host, ip address, operating system, etc...
    • Hiera
    • Configuration you pass in to enable building for multiple
    configurations from the same shared codebase

    View full-size slide

  49. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (variables)
    class guest {
    # local variable
    $host = “mybox”
    if ($host == “mybox”) { notify { “Box!”: } } # Box!
    if ($host == “mybox”) { notify { “${host}!”: } } # mybox!
    }

    View full-size slide

  50. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (facts)
    class guest {
    # fact - environment configuration
    if ($::hostname == “box”) { notify { “Box!”: } } # output: Box!
    if ($hostname == “box”) { notify { “Box!”: } } # output: Box!
    # assign $hostname locally...
    $hostname = “some string”
    if ($::hostname == “box”) { notify { “Box!”: } } # output: Box!
    if ($hostname == “box”) { notify { “Box!”: } } # no output
    }

    View full-size slide

  51. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (facts, debugging)
    class guest {
    # what facts are there? Let’s find out!
    # this should be one line, but it is just too long to fit
    $template = "<%= scope.to_hash.reject { |k,v| !( k.is_a?
    (String) && v.is_a?(String) ) }.to_yaml %>"
    # create a file...
    file { "/tmp/facts.yaml":
    content => inline_template($template),
    }
    }

    View full-size slide

  52. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (if conditionals)
    class guest {
    if (“a” == “b”) {
    notify { “Inconceivable!”: }
    } elsif (“hello” =~ “help”) {
    notify { “Regex!”: }
    } else {
    notify { “Echo something...”: }
    }
    }

    View full-size slide

  53. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (case conditionals)
    class guest {
    # $::operatingsystem is a fact, provided directly by Facter
    case $::operatingsystem {
    centos, redhat: { $apache = "httpd" }
    debian, ubuntu: { $apache = "apache2" }
    default: { fail("Unrecognized operating system") }
    }
    case $::operatingsystem {
    /linux/: { fail(“We don’t support anything with Linux in it”) }
    }
    }

    View full-size slide

  54. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Puppet Manifest (selector conditionals)
    class guest {
    # $::operatingsystem is a fact, provided directly by Facter
    $package = $::operatingsystem ? {
    centos => “httpd”,
    redhat => “httpd”,
    /(i?)(ubuntu|debian)/ => “apache2”,
    default => undef
    }
    }

    View full-size slide

  55. Let’s talk about
    real multiple guest configuration

    View full-size slide

  56. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Configure our guest nodes...
    nodes = {
    :'web-server' => {
    :hostname => 'server.example.com',
    :ipaddress => '192.168.56.60',
    },
    :'db-server' => {
    :hostname => 'db.example.com',
    :ipaddress => '192.168.56.61',
    }
    }

    View full-size slide

  57. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Tell Vagrant about them
    nodes = { … node configuration … }
    Vagrant.configure("2") do |config|
    nodes.each_pair do |name,options|
    config.vm.define name do |node|
    node.vm.box = "base-box-name"
    node.vm.box_url = "http://some/url.box"
    node.vm.hostname = options[:hostname]
    node.vm.network :private_network, ip: options[:ipaddress]
    end
    end
    end

    View full-size slide

  58. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Tell Vagrant about Puppet
    nodes = { … node configuration … }
    Vagrant.configure("2") do |config|
    nodes.each_pair do |name,options|
    config.vm.define name do |node|
    node.vm... # omit general config
    node.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests/"
    puppet.manifest_file = "guests.pp"
    puppet.module_path = "puppet/modules/"
    end
    end
    end
    end

    View full-size slide

  59. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Tell Puppet about the guest nodes
    class guests {
    if ($::hostname == “web-server”) {
    # only run for web server
    }
    if ($::hostname == “db-server”) {
    # only run for db server
    }
    }

    View full-size slide

  60. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Tell Puppet about the guest nodes (redux)
    class guests {
    if ($::hostname =~ /^web-/) {
    # only run for web servers
    }
    if ($::hostname =~ /^db-/) {
    # only run for db servers
    }
    }

    View full-size slide

  61. Let’s talk about
    how everything ties together

    View full-size slide

  62. Shell Scripts
    Puppet
    Chef
    Ansible
    VirtualBox
    VMWare (WS or Fusion)
    Parallels Desktop
    Amazon AWS
    Digital Ocean
    Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Vagrant Architecture
    Providers
    Provisioners
    Vagrant

    View full-size slide

  63. Let’s talk about
    how to convince your boss

    View full-size slide

  64. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Development environments come in all
    shapes and sizes
    • Some develop in production
    – Yes, really.
    • Some develop on a non production server
    – Not a copy of production, not by a long shot
    • Some develop locally on a “non production server”
    – Home grown solutions that “work” until they don’t
    • Some develop on a development server
    – Actually tries to be like production
    – Not always like production
    – May have additional tuning to help with developmenty things
    • Some develop in a true development environment
    – Locally or remote, coming soon to a computer near you

    View full-size slide

  65. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Why developing in production is really bad
    Assuming, of course, that you are testing your code before you
    make it live.
    Production should always move from one stable version of code to
    another stable version of code. When you edit code in production,
    you inherently introduce instability, and can no longer be assured
    that your code is running as tested.

    View full-size slide

  66. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Why developing on a non production setup
    is also bad
    Debugging environmental issues is slow and painful.
    Developers should be solving problems with software, not
    debugging configuration issues.
    When you build your code in an environment that isn’t related to
    production in any tangible way, you will often discover just how
    different your development environment is from production.

    View full-size slide

  67. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Where your development environment lives
    Remote
    • Easy for operations to set up
    • Easy to enforce a rigid structure
    • Limits developer’s choice of tool
    chain selection
    • Can get expensive and
    complicated to maintain
    • Developers must be online to do
    any development
    Local
    • Can require beefier hardware
    • Can take extra work to set up
    • Developers have the freedom to
    work how they work best
    • Developers can assist with
    infrastructural initiatives

    View full-size slide

  68. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Bad development environments will sap the resources out of
    your team. In addition to the time spent developing and testing
    new code and fixing bugs, you will spend time:
    When development environments attack
    Debugging broken environments or settings
    Debugging broken behavior between dev and prod
    Debugging broken behavior between two dev systems
    Working in broken ways because “it works”

    View full-size slide

  69. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Failure to specifically build an environment for
    development will directly impact developer
    productivity and product quality.

    View full-size slide

  70. Let’s talk about
    what you should remember

    View full-size slide

  71. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Takeaways
    • Bad (or unstable) environments (be it dev, qa, staging, or
    prod) hurt developers.
    • Puppet lets you automate all of the configuration of your
    systems (dev, qa, staging, and prod), ensuring
    consistency between them.
    • Vagrant is easy to use, and will let you use your
    production puppet configs to build development
    environments.
    • You will manage server configuration, whether you
    decide to or not. If you choose not to do it formally it will
    come back to bite you when you forget something later.

    View full-size slide

  72. Let’s talk about
    where to go from here

    View full-size slide

  73. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    If you’re still not sold...
    h"p://PuPHPet.com/
    /
    PuPHPet/provides/a/
    simple/config/tool/for/
    those/who/don’t/want/
    to/=nker/with/puppet/
    and/need/a/LAMP/stack/
    up/and/running/NOW./
    V2 was released this morning!!!

    View full-size slide

  74. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    And also look at...
    • Packer
    • http://packer.io
    • Take your Vagrant and Puppet usage to the next level by
    using Packer to pre-provision images for easy first spins
    • Build prebuilt systems for multiple VMS and cloud
    platforms
    • Docker
    • http://docker.io
    • A different idea of how to build multiple machine VM
    environments

    View full-size slide

  75. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013
    Thank you!
    More information about this talk can be found at:
    http://jmather.com/talks/2013-10-12

    View full-size slide