Develop and Test Configuration Management Scripts with Vagrant

Develop and Test Configuration Management Scripts with Vagrant

Configuration management and systems automation has really taken off the past few years, and the tooling around this has been rapidly improving as well. However, many engineers working with these tools still write entire scripts, spin up a new server, test them on that server, and then work through any issues they may face. Incremental development is rare or unheard of and basic testing of such scripts is cumbersome.

In this talk I will present using local virtual machines along with Vagrant as a way of enabling incremental development for server provisioning, and how it can be used to provide more stability and predictability during development. Additionally, I will talk about how organizations are using Vagrant as a way to also do basic integration testing of server provisioning scripts.

By enabling incremental development and providing a method for testing these scripts, the entire feedback loop during development is shortened which leads to higher stability and confidence of server provisioning.

Vagrant is an open source tool that has been around for about 2 years now. It provides a command line tool to create and manage virtual machines and is used by companies and organizations such as Rackspace, LivingSocial, EventBrite, Yammer, Nokia, and many more.

2828f28fb012308a7786eee83b8293c5?s=128

Mitchell Hashimoto

June 26, 2012
Tweet

Transcript

  1. Develop and Test Config Management Scripts with Vagrant

  2. Mitchell Hashimoto @mitchellh

  3. None
  4. jobs.kiip.me

  5. None
  6. A Look at Web Ops Today Let’s take a look

    at what the daily life in web ops looks like today...
  7. “Automate the installation and configuration of Apache.” So your boss

    comes to you and says this... what do you do next? NOTE: We’re going to use Chef for examples because thats what I do, but it shouldn’t get in the way of any understanding.
  8. cookbooks ! mkdir -p apache2/recipes cookbooks ! vim apache2/recipes/default.rb First

    you’ll probably go into some “cookbooks” directory and make an apache2 cookbook and a default recipe.
  9. 1 package "apache2" do 2 action :install 3 end 4

    5 service "apache2" do 6 action :enable 7 end 8 9 template "apache2.conf" do 10 path "#{node[:apache][:dir]}/” + “apache2.conf" 11 mode 0644 12 notifies :restart, resources(:service => "apache2") 13 end You might type something like this into your recipe, using your best guess as to what it takes to install and configure Apache. Or perhaps you have a server somewhere you’re trying this out on as you go via SSH. Either way, you’re kind of guessing here.
  10. cookbooks ! ./start_ec2_instance_for_test.sh So now you have a semi-working cookbook

    and you want to test it. Let’s be generous and assume you have some process for “quickly” bringing up some remote server (in this case EC2) for test. You run that script.
  11. Coffee/Tea + Sword Fight. Now you wait.

  12. cookbooks ! ./start_ec2_instance_for_test.sh Instance ready for testing! ec2-1-2-3-4.compute-1.amazonaws.com Eventually your

    EC2 instance is ready and you get the address to SSH to.
  13. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$

    You SSH into it.
  14. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$

    vim config.json You have to configure Chef now to run the cookbook you just made. Note that we’re making more assumptions here that your cookbooks were somehow rsyncced or found their way onto this server.
  15. 1 { 2 "run_list": ["recipe[apache2]"] 3 } Chef configuration to

    run your single recipe that you’re developing.
  16. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$

    sudo chef-solo -j config.json Next you run Chef..
  17. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$

    sudo chef-solo -j config.json [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete And you get output that looks like this. Normal chef output. Looks like things worked! Alright. We’re getting somewhere. So the next step is that you probably need to be able to configure vhosts on the instance, so we want to setup Chef to be able to configure vhosts.
  18. cookbooks ! mkdir -p apache2/definitions cookbooks ! vim apache2/definitions/site.rb So

    you start working on an “ apache2_site” definition, which will be a resource in Chef to create vhosts.
  19. 1 define :apache_site, :enable => true do 2 ... 3

    4 if params[:enable] 5 execute "a2ensite #{params[:name]}" do 6 command "/usr/sbin/a2ensite” + “#{params[:name]}" 8 notifies :restart, resources(:service => "apache2") 9 end 10 else 11 ... 16 end 17 end Here is what it might look like. Don’t worry about understanding this if you don’t know Chef, just know that you’re WRITING CODE.
  20. cookbooks ! ./start_ec2_instance_for_test.sh And nowwwwww we want to test it

    so we need to bring up a new instance. Note: Yes, maybe you can re-use your old instance, because this time we’re just ADDING things to Chef, but what if you were CHANGING previous resources? Chef can’t undo, so you’d need a new instance. Wahhhhh!
  21. Repeat. Again. And again. Anddddd you follow this process and

    repeat it over and over until you get to the state you want and feel its ready for stage/prod.
  22. PRODUCTIVITY FAILURE. This is a huge failure. This process has

    a ton of problems associated with it...
  23. Extreme case? Maybe. Maybe not. Is this an extreme case?

    For some people yes, but for some people no. I assumed a lot of things: * Automated server startup + Chef setup. * Chef somehow configured with cookbooks already on server. * Chef-solo, not chef-client. Therefore, kind of a toss-up. It can be extreme, but I think for a lot of people this rings true.
  24. Problems: • Slow feedback loop • Encourages avoiding incremental development

    • No real automation!
  25. A Look at Web Dev Today So we saw the

    ops side of things, let’s take a look at the dev side of things. Realizing Velocity is an “Ops” conference, but we’re also responsible for providing sane development environments as well. So let’s see what devs typically go through nowadays...
  26. “You’ve been hired! How about you get the site up

    and running your shiny new MacBook Air?” A developer gets hired, and the boss says THIS! What is the next step?
  27. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... The general

    first step is to clone out the source, so lets do that...
  28. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites !

    cd facebook_killer
  29. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites !

    cd facebook_killer facebook_killer ! ... Next, SOMETHING happens. We’ll get back to this, but SOMETHING HAPPENS.
  30. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites !

    cd facebook_killer facebook_killer ! ... facebook_killer ! ./start.sh Server listening on http://localhost:3000/... Then you run some script that starts a server and you’re OFF AND RUNNING! Ready to work!
  31. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites !

    cd facebook_killer facebook_killer ! ... A small sacrifice is made to Flying Spaghetti Monster Well, let’s take a look at this “...” step... Typically this step requires a small sacrifice to some deity. What goes on here?
  32. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites !

    cd facebook_killer facebook_killer ! ... Let’s take a closer look...
  33. Scenario 1: You have a setup script. First potential scenario

    is that you have a setup script, which is a script you run which sets up your environment for development.
  34. facebook_killer ! ./setup.sh Installing some server... Installing some software... Configuring

    it all... Successful sacrifice made to FSM. It MIGHT look like this.
  35. Scenario 1: You have a setup script. UNLIKELY But this

    is an unlikely scenario. Its possible, but unlikely. Even if it does exist, there are some problems.
  36. • Your Mac is not what is running in production.

    Your Mac/Windows machine is not what is running in production. Your linux machine may be pretty similar, but ignoring that. Therefore, you’re trying to install server software on your own machine, in an environment it was not meant to run. You’re asking for trouble.
  37. • Unlikely that all the software you need will work

    the same on your machine Sure, Apache, MySQL, etc. work and compile fine on Windows and Mac. But then you start getting some weird dependencies... or you write your own libraries. Its unlikely you want to spend any serious amount of time working on getting compilation working in every environment.
  38. • Unlikely it is configured the same, which leads to

    pain later Example: MySQL has HUNDREDS of configuration parameters. You may configure them similarly so that it appears to work on your machine, but undoubtedly at some point in the future these slight configuration changes will cause HUGE WEIRD bugs.
  39. • Requires special maintenance And perhaps most importantly, this setup

    script requires individual maintenance. That means that it’ll likely just decay and end up not working for periods of a time without large amounts of developer discipline. Things that require EXTRA WORK, typically don’t work.
  40. • What if people want to work in Windows? Linux?

    So if you have other operating systems out there for dev, you’re going to cater these setup scripts to work in all environments? Yeah, right.
  41. Scenario 1: You have a setup script. UNLIKELY

  42. Scenario 2: The Über-README. Next scenario: You have an UBER

    README. This is a README that doesn’t DO anything except tell the developer how to set up his/her environment.
  43. None
  44. UNLIKELY Scenario 2: The Über-README.

  45. • Extremely prone to user error. Actual execution of the

    setup is left to the user, who will likely do things wrong or differently.
  46. • Heavy Maintenance Very careful steps must be maintained for

    every operating system for every piece of infrastructure. Like I said before, extra work doesn’t usually work.
  47. • Time consuming The time for a new hire to

    get a site up and running goes from hours to days. I’ve heard of larger companies doing this (> 250 employees) where the time for an engineer to get a site up and running actually took 5 days. Whoa.
  48. UNLIKELY Scenario 2: The Über-README.

  49. Scenario 3: Good luck. Have fun. Scenario 3: GOOD LUCK!

  50. facebook_killer ! ruby main.rb Failed to connect to MySQL at

    localhost:3306
  51. Install MySQL.

  52. facebook_killer ! ruby main.rb Failed to connect to Redis at

    localhost:6379!
  53. Install Redis.

  54. facebook_killer ! ruby main.rb Missing ImageMagick extensions!

  55. Install ImageMagick. (Starting to hate yourself)

  56. facebook_killer ! ruby main.rb Site running on localhost:3000.

  57. None
  58. None
  59. None
  60. Scenario 3: Good luck. Have fun. MOST LIKELY!

  61. PRODUCTIVITY FAILURE.

  62. Development Setup is also an Operations Problem Dev setup is

    also an ops problem, because we’re responsible for making sure they’re environments are working properly. Ops people work very hard to have stable systems up and running in production, why can’t that transfer over to development?
  63. Problems: • Not repeatable • Not verifiably correct • Not

    isolated • Difficult to understand • SLOW, SLOW, SLOW!
  64. Virtual Machines So is there some technology out there that

    can help solve some of this ops/dev pain that exists today? VIRTUAL MACHINES! This has been known for some time, this isn’t revolutionary.
  65. • Isolated • Repeatable • Model complex relationships For Ops:

    VMs allow Ops to work in isolated environments, just how we’d like to test ops things initially. VMs are scriptable, ephemeral. We can use our programming skills to make a repeatable working environment. VMs are full mini-machines that are cheap, so we can just make lots of them to model complex relationships (such as distributed systems).
  66. • Automated • Local • Host-OS agnostic For Dev: For

    dev, VMs are automated, which has numerous benefits: repeatable, easy, verifiable, testable. The VMs are or can be local, so developers can work on their own machine. Most VM providers provide software that works on most major platforms, so developers are free to work on the platform they want.
  67. • Faster on-boarding • Anyone can do it • Less

    bugs For Biz: And there are also BUSINESS benefits. On-boarding times are slashed. When you hire a new employee, they can have a new environment up and running in a matter of minutes rather than hours or days. Anyone can do it, so even designers can bring up development environments if they want to. And there are less bugs, because the development environments are all the same so you get rid of the “works on my machine” bugs.
  68. The Idea: Build local VMs to match production and develop

    there. So the idea to solve our problems is this.
  69. We have many options when it comes down to local

    virtualization... So do we just pick one? LXC
  70. Vagrant A standard workflow for working with VMs. http://vagrantup.com

  71. Open source tool for building and distributing development environments. https://github.com/mitchellh/vagrant

  72. Stable v1.0.3 Development since Jan 2010 Used by hundreds of

    companies
  73. basho

  74. Feature Breakdown • Simple management interface (CLI) • VM Creation

    • Provisioning • Scripted config of VM properties • Scripted config of networks • Scripted config of shared file system • Single command setup
  75. VirtualBox Only, for now. Vagrant is currently VirtualBox only. When

    Vagrant started, VirtualBox is the only freely available virtualization tech that works on every major platform that also has a comprehensive API. It was the best choice to spur adoption. That being said, this will change in the
  76. Setting up Vagrant http://vagrantup.com

  77. Download and install VirtualBox: https://www.virtualbox.org/wiki/Downloads Download and install Vagrant: https://vagrantup.com

  78. ~ ! vagrant Usage: vagrant [-v] [-h] command [<args>] -v,

    --version Print the version and exit. -h, --help Print this help. Available subcommands: box destroy gem halt init package provision reload resume ssh ssh-config status suspend up For help on any individual command run `vagrant COMMAND -h`
  79. How does it work? http://vagrantup.com

  80. How does it work? • Vagrantfile • vagrant command line

    interface These are the two components that Vagrant needs to work. We’ll go over each in detail a bit later, the main point now is that there are simply two things that make things work.
  81. Vagrantfile • Per-project configuration • Configure VMs • Configure networking,

    provisioning, shared file system • Ruby DSL, no Ruby knowledge needed
  82. vagrant • Single command to control lifecycle of your virtual

    machines. • git-style. “vagrant <subcommand>”
  83. vagrant up The only command that matters for most people.

  84. vagrant up • Single setup command • Creates VM •

    Configures properties of VM • Configures networking and shared file system on both host and guest • Provisions the guest
  85. “Welcome to our company! Get the website up and running

    on your shiny new laptop!”
  86. vagrant up To get that site up and running as

    a developer.
  87. “Hey can you fix some bugs in the Apache2 cookbook?”

  88. vagrant up To develop and test configuration management scripts

  89. http://vagrantup.com Vagrant Concepts Now we’re going to go over the

    concepts of Vagrant. This section will be light on sticky notes because the slides are mostly self-explanatory.
  90. Base Boxes http://vagrantup.com

  91. Base Boxes • Base images for creating VMs • Minimal

    software set installed - SSH - Chef/Puppet (optional) • Includes VirtualBox Guest Additions
  92. Box Management • Built-in to Vagrant • vagrant box

  93. ~ ! vagrant box Usage: vagrant box <command> [<args>] Available

    subcommands: add list remove repackage For help on any individual command run `vagrant box COMMAND -h`
  94. Adding a Box $ vagrant box add <name> <url> •

    Name: Local name referenced in config • URL: File path or HTTP URL.
  95. Finding Boxes • “Official” Boxes: - lucid32, lucid64, precise32, precise64

    - http://files.vagrantup.com/NAME.box • 3rd Party: - http://vagrantbox.es
  96. ~ ! vagrant box add lucid32 \ http://files.vagrantup.com/lucid32.box

  97. ~ ! vagrant box list lucid32

  98. A First Vagrant Project http://vagrantup.com

  99. • Vagrant is project-oriented • VMs exist in the context

    of a “project” • Project denoted by existence of a Vagrantfile Projects?
  100. ~ ! vagrant init lucid32 A `Vagrantfile` has been placed

    in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `vagrantup.com` for more information on using Vagrant.
  101. • Configures Vagrant for your project • Ruby DSL •

    No Ruby knowledge needed Vagrantfile?
  102. vagrant up

  103. ~ ! vagrant up [default] Importing base box 'lucid32'... [default]

    Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  104. vagrant ssh

  105. ~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu!

    vagrant@lucid32:~$ echo Hello Velocity! Hello Velocity! vagrant@lucid32:~$
  106. Fully functional VM!

  107. In... 30 seconds! (Ignoring download times)

  108. • Base box used as a skeleton, but private copy

    is created for each `up` • Therefore, files in your VM don’t affect past/future VMs. A Note on “up”
  109. • vagrant up • vagrant status • vagrant reload •

    vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle
  110. Working in a VM http://vagrantup.com

  111. File Synchronization http://vagrantup.com

  112. • VirtualBox shared folders • NFS (Depends on host*) •

    Use your preferred editor on the host • No need to sync files manually! File Synchronization
  113. • By default “/vagrant” is your project directory. Shared Folders

  114. vagrant@lucid32:~$ ls /vagrant Vagrantfile vagrant@lucid32:~$

  115. config.vm.share_folder( “static”, “/srv/static”, “./static”) Shared Folders

  116. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Shared Folders

  117. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Shared Folders

  118. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Host path

    Shared Folders
  119. vagrant reload

  120. VirtualBox Shared Folders: 5m 14s Host File System: 10s Native

    VM File System: 13s NFS Shared Folders: 22s NFS Shared Folders (warm cache): 14s NFS
  121. NFS • Windows does not support NFS. • Most linux

    distros do, but obscure ones may not.
  122. config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders +

    NFS
  123. config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders +

    NFS
  124. vagrant reload

  125. Provisioning http://vagrantup.com

  126. • Scripted installation of packages • Scripted configuration management •

    Scripted starting of services Provisioning I’m probably preaching to the choir here
  127. • Shell • Puppet (via apply or server) • Chef

    (solo and server) • CFEngine coming soon! Provisioning
  128. • Setup development using same scripts as production. • Test

    scripts prior to putting into production • Without it, just moving the annoying setup of dev environment into a VM... Why Provision?
  129. • First class support in Vagrantfile • Run with `vagrant

    up`, `vagrant reload`, or `vagrant provision` Provisioning
  130. Chef Example 1 config.vm.provision :chef_solo do |c| 2 c.cookbooks_path =

    “cookbooks” 3 c.add_recipe “apache2” 4 end
  131. vagrant reload

  132. ~ ! vagrant reload [default] Attempting graceful shutdown of VM...

    ... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... stdin: is not a tty [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
  133. “Welcome to our company! Get the website up and running

    on your shiny new laptop!”
  134. vagrant up To get that site up and running as

    a developer.
  135. vagrant provision

  136. • Runs the provisioner again. • Allows iterative provisioning script

    development! vagrant provision
  137. Networking http://vagrantup.com

  138. We have a code for some service in the VM...

    how do we access it? The Problem
  139. vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.

  140. vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.

  141. Network into the VM

  142. • Port forwarding • Host-only Networking • Bridged Networking Networking

  143. Port Forwarding

  144. Expose some port on the guest via a port on

    the host.
  145. Outside World Host Machine VM :22 :1234 :80 :8080

  146. Port Forwarding 1 config.vm.forward_port 22, 1234 2 config.vm.forward_port 80, 8080

  147. ~ ! vagrant reload [default] Attempting graceful shutdown of VM...

    [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 1234 (adapter 1) [default] -- 80 => 8080 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  148. ~ ! vagrant reload [default] Attempting graceful shutdown of VM...

    [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 1234 (adapter 1) [default] -- 80 => 8080 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  149. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving

    HTTP on 0.0.0.0 port 80 ...
  150. 700x492 viewport

  151. Common Gotcha: Make sure your listening socket binds to “0.0.0.0”

  152. Pros/Cons • Pro: Simple • Con: Port exposed globally •

    Con: Port-by-port • Con: Host port >= 1024 (*)
  153. Host-only Networking

  154. Private network on your machine that can include your host

    and other VMs.
  155. Outside World Host Machine VM 10.0.0.5 VM 10.0.0.6 VM 10.0.0.7

    Virtual Router (VirtualBox) 10.0.0.1
  156. Host-only Networking 1 config.vm.network :hostonly, "10.40.40.40"

  157. ~ ! vagrant reload ... [default] Creating shared folders metadata...

    [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
  158. ~ ! vagrant reload ... [default] Creating shared folders metadata...

    [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
  159. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving

    HTTP on 0.0.0.0 port 80 ...
  160. 700x492 viewport

  161. 700x492 viewport

  162. Common Gotcha: If the static IP you assign collides with

    your internet subnet, it won’t work.
  163. Pros/Cons • Pro: Local-only, secure • Pro: VM can address

    the host • Pro: Multiple VMs can communicate • Con: Local-only
  164. Bridged Networking

  165. Bridge VM onto a networking device on your host machine.

  166. Outside World Host Machine 10.0.1.14 Host Machine 10.0.1.23 Host Machine

    10.0.1.35 LAN - Router VM 10.0.1.10 VM 10.0.1.16
  167. Bridged Networking 1 config.vm.network :bridged

  168. ~ ! vagrant reload ... [default] Available bridged network interfaces:

    1) en0: Wi-Fi (AirPort) 2) p2p0 What interface should the network bridge to? 1 [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
  169. ~ ! vagrant reload ... [default] Available bridged network interfaces:

    1) en0: Wi-Fi (AirPort) 2) p2p0 What interface should the network bridge to? 1 [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
  170. vagrant@lucid32$ ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b inet addr:10.1.10.54 Bcast:10.1.10.255

    Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
  171. vagrant@lucid32$ ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b inet addr:10.1.10.54 Bcast:10.1.10.255

    Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
  172. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving

    HTTP on 0.0.0.0 port 80 ...
  173. 700x492 viewport

  174. 700x492 viewport

  175. Common Gotcha: Bridging may not work on some router configs.

    Typically hotels.
  176. Pros/Cons • Pro: Other machines on LAN can access the

    VM. • Con: Must select a network device • Con: Doesn’t always work depending on router config
  177. • Mix multiple together! • Flexibility you need to work

    how you want Networking
  178. Multi-VM http://vagrantup.com

  179. Testing a SOA or distributed system properly. The Problem

  180. • Configure multiple VMs in a single Vagrantfile • Network

    them together Multi-VM
  181. Multi-VM Config 1 config.vm.box = "lucid64" 2 3 config.vm.define :web

    do |web| 4 web.vm.network :hostonly, "10.1.40.5" 5 end 6 7 config.vm.define :db do |db| 8 db.vm.network :hostonly, "10.1.40.6" 9 end
  182. • vagrant up • vagrant status • vagrant reload •

    vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle
  183. • vagrant up <target> • vagrant status <target> • vagrant

    reload <target> • vagrant suspend <target> • vagrant halt <target> • vagrant destroy <target> VM Lifecycle + Multi-VM
  184. ~ ! vagrant up web [web] Importing base box 'lucid64'...

    [web] Matching MAC address for NAT networking... [web] Clearing any previously set forwarded ports... [web] Forwarding ports... [web] -- 22 => 2222 (adapter 1) [web] Creating shared folders metadata... [web] Clearing any previously set network interfaces... [web] Preparing network interfaces based on configuration... [web] Booting VM... [web] Waiting for VM to boot. This can take a few minutes. [web] VM booted and ready for use! [web] Configuring and enabling network interfaces... [web] Mounting shared folders... [web] -- v-root: /vagrant
  185. ~ ! vagrant ssh web Ubuntu 10.04.3 LTS Welcome to

    Ubuntu! vagrant@lucid32:~$
  186. • Web, DB, queue, etc. on separate machines. • Allow

    developers to work on single service in SOA • Distributed system testing (fault tolerance, network partition, etc.) Use Cases
  187. Packaging http://vagrantup.com

  188. Provisioning takes a long time! The Problem

  189. vagrant package

  190. • Exports currently created Vagrant environment as a box. vagrant

    package
  191. ~ ! vagrant package [default] Attempting graceful shutdown of VM...

    [default] Clearing any previously set forwarded ports... [default] Creating temporary directory for export... [default] Exporting VM... [default] Compressing package to: /Users/mitchellh/Dropbox/ Documents/presentations/2012_velocity/package.box
  192. ~ ! vagrant box add my_site package.box [vagrant] Downloading with

    Vagrant::Downloaders::File... [vagrant] Copying box to temporary location... [vagrant] Extracting box... [vagrant] Verifying box... [vagrant] Cleaning up downloaded box...
  193. Packaged Box 1 config.vm.box = “my_site”

  194. ~ ! vagrant up [default] Importing base box 'my_site'... [default]

    Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  195. Best Practices • Package up installed software • Use provisioners

    for configuration and managing services
  196. Modern WebOps with Vagrant http://vagrantup.com

  197. 1. Manually work in Vagrant 2. Automate a piece 3.

    Test in Vagrant, repeat. 4. Push to stage, then prod
  198. “Automate the installation and configuration of Apache.”

  199. 1. Manually work in Vagrant

  200. ~ ! vagrant up [default] Importing base box 'lucid32'... [default]

    Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  201. ~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu!

    vagrant@lucid32:~$ sudo apt-get install apache2 ... vagrant@lucid32:~$ vi /etc/apache2/apache2.conf ...
  202. Get to the target system manually using Vagrant

  203. 2. Automate a piece

  204. 1 package "apache2" do 2 action :install 3 end 4

    5 service "apache2" do 6 action :enable 7 end 8 9 template "apache2.conf" do 10 path "#{node[:apache][:dir]}/” + “apache2.conf" 11 mode 0644 12 notifies :restart, resources(:service => "apache2") 13 end
  205. 3. Test in Vagrant

  206. ~ ! vagrant reload [default] Attempting graceful shutdown of VM...

    ... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... stdin: is not a tty [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
  207. Piece-by-piece get to target system previously reached in Vagrant

  208. Repeat. Destroy VM if needed.

  209. 4. Push to stage/prod

  210. ~ ! knife cookbook upload apache2

  211. Uses (Ops) • Developing Chef/Puppet/etc. scripts • Testing Chef/Puppet/etc. scripts

    • Using production-quality ops scripts to setup dev environments. • QA
  212. Vagrant: Where is it going? http://vagrantup.com

  213. workflow Predictable, usable, sensible.

  214. A standard workflow for working with development environments.

  215. so what’s next?

  216. 1. Any virt software (VMWare, EC2, KVM)

  217. ~ ! vagrant up --provider=vmware ... ~ ! vagrant up

    --provider=ec2 ...
  218. Same workflow, but using the right environment for the right

    job.
  219. 2. Vagrant Builder

  220. Work in identical environments in both development and production

  221. Vagrant Builder ISO Bootstrap Scripts VirtualBox VMWare AMI ...

  222. 3. Any Guest OS

  223. Windows, Mac OS, MyNextCustomOS? Sure.

  224. • Dev/Prod Parity • Frictionless development • Flexibility Goals

  225. http://vagrantup.com Thank you! Questions? @mitchellh