Develop and Test Configuration Management Scripts with Vagrant

Configuration management and systems automation has really taken off the past few years, and the tooling around this has been rapidly improving as well. However, many engineers working with these tools still write entire scripts, spin up a new server, test them on that server, and then work through any issues they may face. Incremental development is rare or unheard of and basic testing of such scripts is cumbersome.

In this talk I will present using local virtual machines along with Vagrant as a way of enabling incremental development for server provisioning, and how it can be used to provide more stability and predictability during development. Additionally, I will talk about how organizations are using Vagrant as a way to also do basic integration testing of server provisioning scripts.

By enabling incremental development and providing a method for testing these scripts, the entire feedback loop during development is shortened which leads to higher stability and confidence of server provisioning.

Vagrant is an open source tool that has been around for about 2 years now. It provides a command line tool to create and manage virtual machines and is used by companies and organizations such as Rackspace, LivingSocial, EventBrite, Yammer, Nokia, and many more.

Transcript

1. Develop and Test
   Config Management Scripts
   with Vagrant
   

   View Slide

2. Mitchell Hashimoto
   @mitchellh
   

   View Slide

3. View Slide

4. jobs.kiip.me
   

   View Slide

5. View Slide

6. A Look at
   Web Ops Today
   Let’s take a look at what
   the daily life in web ops
   looks like today...
   

   View Slide

7. “Automate the installation
   and configuration
   of Apache.”
   So your boss comes to you and says this...
   what do you do next?
   NOTE: We’re going to use Chef for
   examples because thats what I do, but it
   shouldn’t get in the way of any
   understanding.
   

   View Slide

8. cookbooks ! mkdir -p apache2/recipes
   cookbooks ! vim apache2/recipes/default.rb
   First you’ll probably go into
   some “cookbooks” directory
   and make an apache2
   cookbook and a default
   recipe.
   

   View Slide

9. 1 package "apache2" do
   2 action :install
   3 end
   4
   5 service "apache2" do
   6 action :enable
   7 end
   8
   9 template "apache2.conf" do
   10 path "#{node[:apache][:dir]}/” +
   “apache2.conf"
   11 mode 0644
   12 notifies :restart,
   resources(:service => "apache2")
   13 end
   You might type something like this into
   your recipe, using your best guess as to
   what it takes to install and configure
   Apache. Or perhaps you have a server
   somewhere you’re trying this out on as you
   go via SSH. Either way, you’re kind of
   guessing here.
   

   View Slide

10. cookbooks ! ./start_ec2_instance_for_test.sh
    So now you have a semi-working
    cookbook and you want to test it. Let’s
    be generous and assume you have some
    process for “quickly” bringing up some
    remote server (in this case EC2) for
    test.
    You run that script.
    

    View Slide

11. Coffee/Tea +
    Sword Fight.
    Now you wait.
    

    View Slide

12. cookbooks ! ./start_ec2_instance_for_test.sh
    Instance ready for testing!
    ec2-1-2-3-4.compute-1.amazonaws.com
    Eventually your EC2 instance is ready
    and you get the address to SSH to.
    

    View Slide

13. cookbooks ! ssh ubuntu@...
    Welcome to Ubuntu 12.04 LTS!
    ubuntu$
    You SSH into it.
    

    View Slide

14. cookbooks ! ssh ubuntu@...
    Welcome to Ubuntu 12.04 LTS!
    ubuntu$ vim config.json
    You have to configure Chef now to run
    the cookbook you just made.
    Note that we’re making more
    assumptions here that your cookbooks
    were somehow rsyncced or found their
    way onto this server.
    

    View Slide

15. 1 {
    2 "run_list": ["recipe[apache2]"]
    3 }
    Chef configuration to run your single
    recipe that you’re developing.
    

    View Slide

16. cookbooks ! ssh ubuntu@...
    Welcome to Ubuntu 12.04 LTS!
    ubuntu$ sudo chef-solo -j config.json
    Next you run Chef..
    

    View Slide

17. cookbooks ! ssh ubuntu@...
    Welcome to Ubuntu 12.04 LTS!
    ubuntu$ sudo chef-solo -j config.json
    [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 ***
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]]
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2]
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1)
    [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update)
    [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully
    [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3)
    [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9
    [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7)
    [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds
    [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers
    [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
    And you get output that looks like this.
    Normal chef output.
    Looks like things worked! Alright. We’re
    getting somewhere.
    So the next step is that you probably
    need to be able to configure vhosts on
    the instance, so we want to setup Chef
    to be able to configure vhosts.
    

    View Slide

18. cookbooks ! mkdir -p apache2/definitions
    cookbooks ! vim apache2/definitions/site.rb
    So you start working on an
    “
    apache2_site” definition, which will be
    a resource in Chef to create vhosts.
    

    View Slide

19. 1 define :apache_site, :enable => true do
    2 ...
    3
    4 if params[:enable]
    5 execute "a2ensite #{params[:name]}" do
    6 command "/usr/sbin/a2ensite” +
    “#{params[:name]}"
    8 notifies :restart,
    resources(:service => "apache2")
    9 end
    10 else
    11 ...
    16 end
    17 end
    Here is what it might look like. Don’t
    worry about understanding this if you
    don’t know Chef, just know that you’re
    WRITING CODE.
    

    View Slide

20. cookbooks ! ./start_ec2_instance_for_test.sh
    And nowwwwww we want to test it
    so we need to bring up a new instance.
    Note: Yes, maybe you can re-use your old
    instance, because this time we’re just
    ADDING things to Chef, but what if you
    were CHANGING previous resources?
    Chef can’t undo, so you’d need a new
    instance. Wahhhhh!
    

    View Slide

21. Repeat.
    Again.
    And again.
    Anddddd you follow this process and
    repeat it over and over until you get to
    the state you want and feel its ready
    for stage/prod.
    

    View Slide

22. PRODUCTIVITY
    FAILURE.
    This is a huge failure.
    This process has a ton of problems
    associated with it...
    

    View Slide

23. Extreme case?
    Maybe.
    Maybe not.
    Is this an extreme case? For some
    people yes, but for some people no. I
    assumed a lot of things:
    * Automated server startup + Chef
    setup.
    * Chef somehow configured with
    cookbooks already on server.
    * Chef-solo, not chef-client.
    Therefore, kind of a toss-up. It can
    be extreme, but I think for a lot of
    people this rings true.
    

    View Slide

24. Problems:
    • Slow feedback loop
    • Encourages avoiding
    incremental development
    • No real automation!
    

    View Slide

25. A Look at
    Web Dev Today
    So we saw the ops side of things, let’s
    take a look at the dev side of things.
    Realizing Velocity is an “Ops”
    conference, but we’re also responsible
    for providing sane development
    environments as well. So let’s see what
    devs typically go through nowadays...
    

    View Slide

26. “You’ve been hired! How
    about you get the site up
    and running your shiny new
    MacBook Air?” A developer gets hired, and the boss
    says THIS! What is the next step?
    

    View Slide

27. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    The general first step is to clone out the
    source, so lets do that...
    

    View Slide

28. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    sites ! cd facebook_killer
    

    View Slide

29. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    sites ! cd facebook_killer
    facebook_killer ! ...
    Next, SOMETHING happens. We’ll get
    back to this, but SOMETHING HAPPENS.
    

    View Slide

30. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    sites ! cd facebook_killer
    facebook_killer ! ...
    facebook_killer ! ./start.sh
    Server listening on http://localhost:3000/...
    Then you run some script that starts a
    server and you’re OFF AND RUNNING!
    Ready to work!
    

    View Slide

31. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    sites ! cd facebook_killer
    facebook_killer ! ...
    A small sacrifice is made
    to Flying Spaghetti Monster
    Well, let’s take a look at this “...” step...
    Typically this step requires a small
    sacrifice to some deity. What goes on
    here?
    

    View Slide

32. sites ! git clone git://.../facebook_killer.git
    Cloning into “facebook_killer”...
    sites ! cd facebook_killer
    facebook_killer ! ...
    Let’s take a closer look...
    

    View Slide

33. Scenario 1:
    You have a setup script.
    First potential scenario is that you
    have a setup script, which is a script
    you run which sets up your environment
    for development.
    

    View Slide

34. facebook_killer ! ./setup.sh
    Installing some server...
    Installing some software...
    Configuring it all...
    Successful sacrifice made to FSM.
    It MIGHT look like this.
    

    View Slide

35. Scenario 1:
    You have a setup script.
    UNLIKELY
    But this is an unlikely scenario. Its
    possible, but unlikely. Even if it does
    exist, there are some problems.
    

    View Slide

36. • Your Mac is not what is
    running in production.
    Your Mac/Windows machine is not
    what is running in production. Your
    linux machine may be pretty similar, but
    ignoring that.
    Therefore, you’re trying to install
    server software on your own machine,
    in an environment it was not meant to
    run. You’re asking for trouble.
    

    View Slide

37. • Unlikely that all the software you need
    will work the same on your machine
    Sure, Apache, MySQL, etc. work and
    compile fine on Windows and Mac. But
    then you start getting some weird
    dependencies... or you write your own
    libraries.
    Its unlikely you want to spend any
    serious amount of time working on
    getting compilation working in every
    environment.
    

    View Slide

38. • Unlikely it is configured the same,
    which leads to pain later
    Example: MySQL has HUNDREDS of
    configuration parameters. You may
    configure them similarly so that it
    appears to work on your machine, but
    undoubtedly at some point in the future
    these slight configuration changes will
    cause HUGE WEIRD bugs.
    

    View Slide

39. • Requires special maintenance
    And perhaps most importantly, this
    setup script requires individual
    maintenance.
    That means that it’ll likely just decay
    and end up not working for periods of a
    time without large amounts of
    developer discipline.
    Things that require EXTRA WORK,
    typically don’t work.
    

    View Slide

40. • What if people want to work in
    Windows? Linux?
    So if you have other operating systems
    out there for dev, you’re going to cater
    these setup scripts to work in all
    environments? Yeah, right.
    

    View Slide

41. Scenario 1:
    You have a setup script.
    UNLIKELY
    

    View Slide

42. Scenario 2:
    The Über-README.
    Next scenario: You have an UBER
    README. This is a README that doesn’t
    DO anything except tell the developer
    how to set up his/her environment.
    

    View Slide

43. View Slide

44. UNLIKELY
    Scenario 2:
    The Über-README.
    

    View Slide

45. • Extremely prone to
    user error.
    Actual execution of the setup is left to
    the user, who will likely do things wrong
    or differently.
    

    View Slide

46. • Heavy Maintenance
    Very careful steps must be maintained
    for every operating system for every
    piece of infrastructure.
    Like I said before, extra work doesn’t
    usually work.
    

    View Slide

47. • Time consuming
    The time for a new hire to get a site up
    and running goes from hours to days.
    I’ve heard of larger companies doing this
    (> 250 employees) where the time for
    an engineer to get a site up and running
    actually took 5 days. Whoa.
    

    View Slide

48. UNLIKELY
    Scenario 2:
    The Über-README.
    

    View Slide

49. Scenario 3:
    Good luck. Have fun.
    Scenario 3: GOOD LUCK!
    

    View Slide

50. facebook_killer ! ruby main.rb
    Failed to connect to MySQL at localhost:3306
    

    View Slide

51. Install MySQL.
    

    View Slide

52. facebook_killer ! ruby main.rb
    Failed to connect to Redis at localhost:6379!
    

    View Slide

53. Install Redis.
    

    View Slide

54. facebook_killer ! ruby main.rb
    Missing ImageMagick extensions!
    

    View Slide

55. Install ImageMagick.
    (Starting to hate yourself)
    

    View Slide

56. facebook_killer ! ruby main.rb
    Site running on localhost:3000.
    

    View Slide

57. View Slide

58. View Slide

59. View Slide

60. Scenario 3:
    Good luck. Have fun.
    MOST
    LIKELY!
    

    View Slide

61. PRODUCTIVITY
    FAILURE.
    

    View Slide

62. Development Setup
    is also an Operations Problem
    Dev setup is also an ops problem,
    because we’re responsible for making
    sure they’re environments are working
    properly.
    Ops people work very hard to have
    stable systems up and running in
    production, why can’t that transfer
    over to development?
    

    View Slide

63. Problems:
    • Not repeatable
    • Not verifiably correct
    • Not isolated
    • Difficult to understand
    • SLOW, SLOW, SLOW!
    

    View Slide

64. Virtual Machines
    So is there some technology out there
    that can help solve some of this ops/dev
    pain that exists today?
    VIRTUAL MACHINES!
    This has been known for some time, this
    isn’t revolutionary.
    

    View Slide

65. • Isolated
    • Repeatable
    • Model complex relationships
    For Ops: VMs allow Ops to work in isolated
    environments, just how we’d like to test
    ops things initially.
    VMs are scriptable, ephemeral. We can
    use our programming skills to make a
    repeatable working environment.
    VMs are full mini-machines that are
    cheap, so we can just make lots of them
    to model complex relationships (such as
    distributed systems).
    

    View Slide

66. • Automated
    • Local
    • Host-OS agnostic
    For Dev: For dev, VMs are automated, which has
    numerous benefits: repeatable, easy,
    verifiable, testable.
    The VMs are or can be local, so
    developers can work on their own
    machine.
    Most VM providers provide software
    that works on most major platforms, so
    developers are free to work on the
    platform they want.
    

    View Slide

67. • Faster on-boarding
    • Anyone can do it
    • Less bugs
    For Biz: And there are also BUSINESS benefits.
    On-boarding times are slashed. When
    you hire a new employee, they can have
    a new environment up and running in a
    matter of minutes rather than hours or
    days.
    Anyone can do it, so even designers can
    bring up development environments if
    they want to.
    And there are less bugs, because the
    development environments are all the
    same so you get rid of the “works on my
    machine” bugs.
    

    View Slide

68. The Idea:
    Build local VMs to match production and
    develop there. So the idea to solve our problems is this.
    

    View Slide

69. We have many options
    when it comes down to
    local virtualization...
    So do we just pick one?
    LXC
    

    View Slide

70. Vagrant
    A standard workflow for
    working with VMs.
    http://vagrantup.com
    

    View Slide

71. Open source tool for building and
    distributing development
    environments.
    https://github.com/mitchellh/vagrant
    

    View Slide

72. Stable v1.0.3
    Development since Jan 2010
    Used by hundreds of companies
    

    View Slide

73. basho
    

    View Slide

74. Feature Breakdown
    • Simple management interface (CLI)
    • VM Creation
    • Provisioning
    • Scripted config of VM properties
    • Scripted config of networks
    • Scripted config of shared file system
    • Single command setup
    

    View Slide

75. VirtualBox Only,
    for now.
    Vagrant is currently VirtualBox only.
    When Vagrant started, VirtualBox is
    the only freely available virtualization
    tech that works on every major
    platform that also has a
    comprehensive API. It was the best
    choice to spur adoption.
    That being said, this will change in the
    

    View Slide

76. Setting up Vagrant
    http://vagrantup.com
    

    View Slide

77. Download and install VirtualBox:
    https://www.virtualbox.org/wiki/Downloads
    Download and install Vagrant:
    https://vagrantup.com
    

    View Slide

78. ~ ! vagrant
    Usage: vagrant [-v] [-h] command []
    -v, --version Print the version and exit.
    -h, --help Print this help.
    Available subcommands:
    box
    destroy
    gem
    halt
    init
    package
    provision
    reload
    resume
    ssh
    ssh-config
    status
    suspend
    up
    For help on any individual command run `vagrant COMMAND -h`
    

    View Slide

79. How does it work?
    http://vagrantup.com
    

    View Slide

80. How does it work?
    • Vagrantfile
    • vagrant command line interface
    These are the two components that
    Vagrant needs to work. We’ll go over
    each in detail a bit later, the main point
    now is that there are simply two things
    that make things work.
    

    View Slide

81. Vagrantfile
    • Per-project configuration
    • Configure VMs
    • Configure networking, provisioning, shared file
    system
    • Ruby DSL, no Ruby knowledge needed
    

    View Slide

82. vagrant
    • Single command to control lifecycle of your virtual
    machines.
    • git-style. “vagrant ”
    

    View Slide

83. vagrant up
    The only command that matters for most people.
    

    View Slide

84. vagrant up
    • Single setup command
    • Creates VM
    • Configures properties of VM
    • Configures networking and shared file system on
    both host and guest
    • Provisions the guest
    

    View Slide

85. “Welcome to our company!
    Get the website up and
    running on your shiny new
    laptop!”
    

    View Slide

86. vagrant up
    To get that site up and running as a developer.
    

    View Slide

87. “Hey can you fix some bugs in
    the Apache2 cookbook?”
    

    View Slide

88. vagrant up
    To develop and test configuration management scripts
    

    View Slide

89. http://vagrantup.com
    Vagrant Concepts
    Now we’re going to go over the concepts
    of Vagrant. This section will be light on
    sticky notes because the slides are
    mostly self-explanatory.
    

    View Slide

90. Base Boxes
    http://vagrantup.com
    

    View Slide

91. Base Boxes
    • Base images for creating VMs
    • Minimal software set installed
    - SSH
    - Chef/Puppet (optional)
    • Includes VirtualBox Guest Additions
    

    View Slide

92. Box Management
    • Built-in to Vagrant
    • vagrant box
    

    View Slide

93. ~ ! vagrant box
    Usage: vagrant box []
    Available subcommands:
    add
    list
    remove
    repackage
    For help on any individual command run `vagrant box COMMAND -h`
    

    View Slide

94. Adding a Box
    $ vagrant box add
    • Name: Local name referenced in config
    • URL: File path or HTTP URL.
    

    View Slide

95. Finding Boxes
    • “Official” Boxes:
    - lucid32, lucid64, precise32, precise64
    - http://files.vagrantup.com/NAME.box
    • 3rd Party:
    - http://vagrantbox.es
    

    View Slide

96. ~ ! vagrant box add lucid32 \
    http://files.vagrantup.com/lucid32.box
    

    View Slide

97. ~ ! vagrant box list
    lucid32
    

    View Slide

98. A First Vagrant Project
    http://vagrantup.com
    

    View Slide

99. • Vagrant is project-oriented
    • VMs exist in the context of a “project”
    • Project denoted by existence of a Vagrantfile
    Projects?
    

    View Slide

100. ~ ! vagrant init lucid32
     A `Vagrantfile` has been placed in this directory. You are now
     ready to `vagrant up` your first virtual environment! Please
     read the comments in the Vagrantfile as well as documentation on
     `vagrantup.com` for more information on using Vagrant.
     

     View Slide

101. • Configures Vagrant for your project
     • Ruby DSL
     • No Ruby knowledge needed
     Vagrantfile?
     

     View Slide

102. vagrant up
     

     View Slide

103. ~ ! vagrant up
     [default] Importing base box 'lucid32'...
     [default] Matching MAC address for NAT networking...
     [default] Clearing any previously set forwarded ports...
     [default] Forwarding ports...
     [default] -- 22 => 2222 (adapter 1)
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

104. vagrant ssh
     

     View Slide

105. ~ ! vagrant ssh
     Ubuntu 10.04.3 LTS
     Welcome to Ubuntu!
     vagrant@lucid32:~$ echo Hello Velocity!
     Hello Velocity!
     vagrant@lucid32:~$
     

     View Slide

106. Fully functional VM!
     

     View Slide

107. In... 30 seconds!
     (Ignoring download times)
     

     View Slide

108. • Base box used as a skeleton, but private copy is
     created for each `up`
     • Therefore, files in your VM don’t affect past/future
     VMs.
     A Note on “up”
     

     View Slide

109. • vagrant up
     • vagrant status
     • vagrant reload
     • vagrant suspend
     • vagrant halt
     • vagrant destroy
     VM Lifecycle
     

     View Slide

110. Working in a VM
     http://vagrantup.com
     

     View Slide

111. File Synchronization
     http://vagrantup.com
     

     View Slide

112. • VirtualBox shared folders
     • NFS (Depends on host*)
     • Use your preferred editor on the host
     • No need to sync files manually!
     File Synchronization
     

     View Slide

113. • By default “/vagrant” is your project directory.
     Shared Folders
     

     View Slide

114. vagrant@lucid32:~$ ls /vagrant
     Vagrantfile
     vagrant@lucid32:~$
     

     View Slide

115. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”)
     Shared Folders
     

     View Slide

116. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”)
     Logical name
     Shared Folders
     

     View Slide

117. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”)
     Logical name
     Guest path
     Shared Folders
     

     View Slide

118. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”)
     Logical name
     Guest path
     Host path
     Shared Folders
     

     View Slide

119. vagrant reload
     

     View Slide

120. VirtualBox Shared Folders: 5m 14s
     Host File System: 10s
     Native VM File System: 13s
     NFS Shared Folders: 22s
     NFS Shared Folders (warm cache): 14s
     NFS
     

     View Slide

121. NFS
     • Windows does not support NFS.
     • Most linux distros do, but obscure ones may not.
     

     View Slide

122. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”,
     :nfs => true)
     Shared Folders + NFS
     

     View Slide

123. config.vm.share_folder(
     “static”,
     “/srv/static”,
     “./static”,
     :nfs => true)
     Shared Folders + NFS
     

     View Slide

124. vagrant reload
     

     View Slide

125. Provisioning
     http://vagrantup.com
     

     View Slide

126. • Scripted installation of packages
     • Scripted configuration management
     • Scripted starting of services
     Provisioning
     I’m probably preaching to the choir here
     

     View Slide

127. • Shell
     • Puppet (via apply or server)
     • Chef (solo and server)
     • CFEngine coming soon!
     Provisioning
     

     View Slide

128. • Setup development using same scripts as
     production.
     • Test scripts prior to putting into production
     • Without it, just moving the annoying setup of dev
     environment into a VM...
     Why Provision?
     

     View Slide

129. • First class support in Vagrantfile
     • Run with `vagrant up`,
     `vagrant reload`, or `vagrant provision`
     Provisioning
     

     View Slide

130. Chef Example
     1 config.vm.provision :chef_solo do |c|
     2 c.cookbooks_path = “cookbooks”
     3 c.add_recipe “apache2”
     4 end
     

     View Slide

131. vagrant reload
     

     View Slide

132. ~ ! vagrant reload
     [default] Attempting graceful shutdown of VM...
     ...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks
     [default] Running provisioner: Vagrant::Provisioners::ChefSolo...
     [default] Generating chef JSON and uploading...
     [default] Running chef-solo...
     stdin: is not a tty
     [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 ***
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]]
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2]
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1)
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update)
     [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully
     [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3)
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7)
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
     

     View Slide

133. “Welcome to our company!
     Get the website up and
     running on your shiny new
     laptop!”
     

     View Slide

134. vagrant up
     To get that site up and running as a developer.
     

     View Slide

135. vagrant provision
     

     View Slide

136. • Runs the provisioner again.
     • Allows iterative provisioning script development!
     vagrant provision
     

     View Slide

137. Networking
     http://vagrantup.com
     

     View Slide

138. We have a code for some service in
     the VM... how do we access it?
     The Problem
     

     View Slide

139. vagrant@lucid32:~$ curl http://localhost/
     This workflow would never work.
     

     View Slide

140. vagrant@lucid32:~$ curl http://localhost/
     This workflow would never work.
     

     View Slide

141. Network into the VM
     

     View Slide

142. • Port forwarding
     • Host-only Networking
     • Bridged Networking
     Networking
     

     View Slide

143. Port Forwarding
     

     View Slide

144. Expose some port on the guest via a
     port on the host.
     

     View Slide

145. Outside World
     Host Machine
     VM
     :22 :1234
     :80 :8080
     

     View Slide

146. Port Forwarding
     1 config.vm.forward_port 22, 1234
     2 config.vm.forward_port 80, 8080
     

     View Slide

147. ~ ! vagrant reload
     [default] Attempting graceful shutdown of VM...
     [default] Clearing any previously set forwarded ports...
     [default] Forwarding ports...
     [default] -- 22 => 1234 (adapter 1)
     [default] -- 80 => 8080 (adapter 1)
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

148. ~ ! vagrant reload
     [default] Attempting graceful shutdown of VM...
     [default] Clearing any previously set forwarded ports...
     [default] Forwarding ports...
     [default] -- 22 => 1234 (adapter 1)
     [default] -- 80 => 8080 (adapter 1)
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

149. vagrant@lucid32$ cd /Vagrant
     vagrant@lucid32$ sudo python -m SimpleHTTPServer 80
     Serving HTTP on 0.0.0.0 port 80 ...
     

     View Slide

150. 700x492 viewport
     

     View Slide

151. Common Gotcha:
     Make sure your listening socket binds to
     “0.0.0.0”
     

     View Slide

152. Pros/Cons
     • Pro: Simple
     • Con: Port exposed globally
     • Con: Port-by-port
     • Con: Host port >= 1024 (*)
     

     View Slide

153. Host-only Networking
     

     View Slide

154. Private network on your machine that
     can include your host and other VMs.
     

     View Slide

155. Outside World
     Host Machine
     VM
     10.0.0.5
     VM
     10.0.0.6
     VM
     10.0.0.7
     Virtual Router (VirtualBox)
     10.0.0.1
     

     View Slide

156. Host-only Networking
     1 config.vm.network :hostonly, "10.40.40.40"
     

     View Slide

157. ~ ! vagrant reload
     ...
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Preparing network interfaces based on configuration...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Configuring and enabling network interfaces...
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

158. ~ ! vagrant reload
     ...
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Preparing network interfaces based on configuration...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Configuring and enabling network interfaces...
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

159. vagrant@lucid32$ cd /Vagrant
     vagrant@lucid32$ sudo python -m SimpleHTTPServer 80
     Serving HTTP on 0.0.0.0 port 80 ...
     

     View Slide

160. 700x492 viewport
     

     View Slide

161. 700x492 viewport
     

     View Slide

162. Common Gotcha:
     If the static IP you assign collides with
     your internet subnet, it won’t work.
     

     View Slide

163. Pros/Cons
     • Pro: Local-only, secure
     • Pro: VM can address the host
     • Pro: Multiple VMs can communicate
     • Con: Local-only
     

     View Slide

164. Bridged Networking
     

     View Slide

165. Bridge VM onto a networking device on
     your host machine.
     

     View Slide

166. Outside World
     Host Machine
     10.0.1.14
     Host Machine
     10.0.1.23
     Host Machine
     10.0.1.35
     LAN - Router
     VM
     10.0.1.10
     VM
     10.0.1.16
     

     View Slide

167. Bridged Networking
     1 config.vm.network :bridged
     

     View Slide

168. ~ ! vagrant reload
     ...
     [default] Available bridged network interfaces:
     1) en0: Wi-Fi (AirPort)
     2) p2p0
     What interface should the network bridge to? 1
     [default] Preparing network interfaces based on configuration...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Configuring and enabling network interfaces...
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

169. ~ ! vagrant reload
     ...
     [default] Available bridged network interfaces:
     1) en0: Wi-Fi (AirPort)
     2) p2p0
     What interface should the network bridge to? 1
     [default] Preparing network interfaces based on configuration...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Configuring and enabling network interfaces...
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

170. vagrant@lucid32$ ifconfig
     eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b
     inet addr:10.1.10.54 Bcast:10.1.10.255 Mask:255.255.255.0
     inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link
     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
     RX packets:64 errors:0 dropped:0 overruns:0 frame:0
     TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
     

     View Slide

171. vagrant@lucid32$ ifconfig
     eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b
     inet addr:10.1.10.54 Bcast:10.1.10.255 Mask:255.255.255.0
     inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link
     UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
     RX packets:64 errors:0 dropped:0 overruns:0 frame:0
     TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
     

     View Slide

172. vagrant@lucid32$ cd /Vagrant
     vagrant@lucid32$ sudo python -m SimpleHTTPServer 80
     Serving HTTP on 0.0.0.0 port 80 ...
     

     View Slide

173. 700x492 viewport
     

     View Slide

174. 700x492 viewport
     

     View Slide

175. Common Gotcha:
     Bridging may not work on some router
     configs. Typically hotels.
     

     View Slide

176. Pros/Cons
     • Pro: Other machines on LAN can
     access the VM.
     • Con: Must select a network device
     • Con: Doesn’t always work depending
     on router config
     

     View Slide

177. • Mix multiple together!
     • Flexibility you need to work how you
     want
     Networking
     

     View Slide

178. Multi-VM
     http://vagrantup.com
     

     View Slide

179. Testing a SOA or distributed system
     properly.
     The Problem
     

     View Slide

180. • Configure multiple VMs in a single
     Vagrantfile
     • Network them together
     Multi-VM
     

     View Slide

181. Multi-VM Config
     1 config.vm.box = "lucid64"
     2
     3 config.vm.define :web do |web|
     4 web.vm.network :hostonly, "10.1.40.5"
     5 end
     6
     7 config.vm.define :db do |db|
     8 db.vm.network :hostonly, "10.1.40.6"
     9 end
     

     View Slide

182. • vagrant up
     • vagrant status
     • vagrant reload
     • vagrant suspend
     • vagrant halt
     • vagrant destroy
     VM Lifecycle
     

     View Slide

183. • vagrant up
     • vagrant status
     • vagrant reload
     • vagrant suspend
     • vagrant halt
     • vagrant destroy
     VM Lifecycle + Multi-VM
     

     View Slide

184. ~ ! vagrant up web
     [web] Importing base box 'lucid64'...
     [web] Matching MAC address for NAT networking...
     [web] Clearing any previously set forwarded ports...
     [web] Forwarding ports...
     [web] -- 22 => 2222 (adapter 1)
     [web] Creating shared folders metadata...
     [web] Clearing any previously set network interfaces...
     [web] Preparing network interfaces based on configuration...
     [web] Booting VM...
     [web] Waiting for VM to boot. This can take a few minutes.
     [web] VM booted and ready for use!
     [web] Configuring and enabling network interfaces...
     [web] Mounting shared folders...
     [web] -- v-root: /vagrant
     

     View Slide

185. ~ ! vagrant ssh web
     Ubuntu 10.04.3 LTS
     Welcome to Ubuntu!
     vagrant@lucid32:~$
     

     View Slide

186. • Web, DB, queue, etc. on separate
     machines.
     • Allow developers to work on single
     service in SOA
     • Distributed system testing (fault
     tolerance, network partition, etc.)
     Use Cases
     

     View Slide

187. Packaging
     http://vagrantup.com
     

     View Slide

188. Provisioning takes a long time!
     The Problem
     

     View Slide

189. vagrant package
     

     View Slide

190. • Exports currently created Vagrant environment as a
     box.
     vagrant package
     

     View Slide

191. ~ ! vagrant package
     [default] Attempting graceful shutdown of VM...
     [default] Clearing any previously set forwarded ports...
     [default] Creating temporary directory for export...
     [default] Exporting VM...
     [default] Compressing package to: /Users/mitchellh/Dropbox/
     Documents/presentations/2012_velocity/package.box
     

     View Slide

192. ~ ! vagrant box add my_site package.box
     [vagrant] Downloading with Vagrant::Downloaders::File...
     [vagrant] Copying box to temporary location...
     [vagrant] Extracting box...
     [vagrant] Verifying box...
     [vagrant] Cleaning up downloaded box...
     

     View Slide

193. Packaged Box
     1 config.vm.box = “my_site”
     

     View Slide

194. ~ ! vagrant up
     [default] Importing base box 'my_site'...
     [default] Matching MAC address for NAT networking...
     [default] Clearing any previously set forwarded ports...
     [default] Forwarding ports...
     [default] -- 22 => 2222 (adapter 1)
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

195. Best Practices
     • Package up installed software
     • Use provisioners for configuration and
     managing services
     

     View Slide

196. Modern WebOps
     with Vagrant
     http://vagrantup.com
     

     View Slide

197. 1. Manually work in Vagrant
     2. Automate a piece
     3. Test in Vagrant, repeat.
     4. Push to stage, then prod
     

     View Slide

198. “Automate the installation
     and configuration
     of Apache.”
     

     View Slide

199. 1. Manually work in
     Vagrant
     

     View Slide

200. ~ ! vagrant up
     [default] Importing base box 'lucid32'...
     [default] Matching MAC address for NAT networking...
     [default] Clearing any previously set forwarded ports...
     [default] Forwarding ports...
     [default] -- 22 => 2222 (adapter 1)
     [default] Creating shared folders metadata...
     [default] Clearing any previously set network interfaces...
     [default] Booting VM...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     

     View Slide

201. ~ ! vagrant ssh
     Ubuntu 10.04.3 LTS
     Welcome to Ubuntu!
     vagrant@lucid32:~$ sudo apt-get install apache2
     ...
     vagrant@lucid32:~$ vi /etc/apache2/apache2.conf
     ...
     

     View Slide

202. Get to the target system manually using
     Vagrant
     

     View Slide

203. 2. Automate a piece
     

     View Slide

204. 1 package "apache2" do
     2 action :install
     3 end
     4
     5 service "apache2" do
     6 action :enable
     7 end
     8
     9 template "apache2.conf" do
     10 path "#{node[:apache][:dir]}/” +
     “apache2.conf"
     11 mode 0644
     12 notifies :restart,
     resources(:service => "apache2")
     13 end
     

     View Slide

205. 3. Test in Vagrant
     

     View Slide

206. ~ ! vagrant reload
     [default] Attempting graceful shutdown of VM...
     ...
     [default] Waiting for VM to boot. This can take a few minutes.
     [default] VM booted and ready for use!
     [default] Mounting shared folders...
     [default] -- v-root: /vagrant
     [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks
     [default] Running provisioner: Vagrant::Provisioners::ChefSolo...
     [default] Generating chef JSON and uploading...
     [default] Running chef-solo...
     stdin: is not a tty
     [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 ***
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]]
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2]
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1)
     [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update)
     [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully
     [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3)
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7)
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers
     [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
     

     View Slide

207. Piece-by-piece get to target system
     previously reached in Vagrant
     

     View Slide

208. Repeat.
     Destroy VM if needed.
     

     View Slide

209. 4. Push to stage/prod
     

     View Slide

210. ~ ! knife cookbook upload apache2
     

     View Slide

211. Uses (Ops)
     • Developing Chef/Puppet/etc. scripts
     • Testing Chef/Puppet/etc. scripts
     • Using production-quality ops scripts to
     setup dev environments.
     • QA
     

     View Slide

212. Vagrant:
     Where is it going?
     http://vagrantup.com
     

     View Slide

213. workflow
     Predictable, usable, sensible.
     

     View Slide

214. A standard workflow for working
     with development
     environments.
     

     View Slide

215. so what’s next?
     

     View Slide

216. 1. Any virt software
     (VMWare, EC2, KVM)
     

     View Slide

217. ~ ! vagrant up --provider=vmware
     ...
     ~ ! vagrant up --provider=ec2
     ...
     

     View Slide

218. Same workflow, but using the right
     environment for the right job.
     

     View Slide

219. 2. Vagrant Builder
     

     View Slide

220. Work in identical environments in both
     development and production
     

     View Slide

221. Vagrant Builder
     ISO
     Bootstrap
     Scripts
     VirtualBox
     VMWare
     AMI
     ...
     

     View Slide

222. 3. Any Guest OS
     

     View Slide

223. Windows, Mac OS,
     MyNextCustomOS? Sure.
     

     View Slide

224. • Dev/Prod Parity
     • Frictionless development
     • Flexibility
     Goals
     

     View Slide

225. http://vagrantup.com
     Thank you!
     Questions?
     @mitchellh
     

     View Slide