Develop and Test Configuration Management Scripts with Vagrant

Configuration management and systems automation has really taken off the past few years, and the tooling around this has been rapidly improving as well. However, many engineers working with these tools still write entire scripts, spin up a new server, test them on that server, and then work through any issues they may face. Incremental development is rare or unheard of and basic testing of such scripts is cumbersome.

In this talk I will present using local virtual machines along with Vagrant as a way of enabling incremental development for server provisioning, and how it can be used to provide more stability and predictability during development. Additionally, I will talk about how organizations are using Vagrant as a way to also do basic integration testing of server provisioning scripts.

By enabling incremental development and providing a method for testing these scripts, the entire feedback loop during development is shortened which leads to higher stability and confidence of server provisioning.

Vagrant is an open source tool that has been around for about 2 years now. It provides a command line tool to create and manage virtual machines and is used by companies and organizations such as Rackspace, LivingSocial, EventBrite, Yammer, Nokia, and many more.

Transcript

1. Develop and Test Config Management Scripts with Vagrant
2. Mitchell Hashimoto @mitchellh
4. jobs.kiip.me
6. A Look at Web Ops Today Let’s take a look at what the daily life in web ops looks like today...
7. “Automate the installation and configuration of Apache.” So your boss comes to you and says this... what do you do next? NOTE: We’re going to use Chef for examples because thats what I do, but it shouldn’t get in the way of any understanding.
8. cookbooks ! mkdir -p apache2/recipes cookbooks ! vim apache2/recipes/default.rb First you’ll probably go into some “cookbooks” directory and make an apache2 cookbook and a default recipe.
9. 1 package "apache2" do 2 action :install 3 end 4 5 service "apache2" do 6 action :enable 7 end 8 9 template "apache2.conf" do 10 path "#{node[:apache][:dir]}/” + “apache2.conf" 11 mode 0644 12 notifies :restart, resources(:service => "apache2") 13 end You might type something like this into your recipe, using your best guess as to what it takes to install and configure Apache. Or perhaps you have a server somewhere you’re trying this out on as you go via SSH. Either way, you’re kind of guessing here.
10. cookbooks ! ./start_ec2_instance_for_test.sh So now you have a semi-working cookbook and you want to test it. Let’s be generous and assume you have some process for “quickly” bringing up some remote server (in this case EC2) for test. You run that script.
11. Coffee/Tea + Sword Fight. Now you wait.
12. cookbooks ! ./start_ec2_instance_for_test.sh Instance ready for testing! ec2-1-2-3-4.compute-1.amazonaws.com Eventually your EC2 instance is ready and you get the address to SSH to.
13. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ You SSH into it.
14. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ vim config.json You have to configure Chef now to run the cookbook you just made. Note that we’re making more assumptions here that your cookbooks were somehow rsyncced or found their way onto this server.
15. 1 { 2 "run_list": ["recipe[apache2]"] 3 } Chef configuration to run your single recipe that you’re developing.
16. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ sudo chef-solo -j config.json Next you run Chef..
17. cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ sudo chef-solo -j config.json [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete And you get output that looks like this. Normal chef output. Looks like things worked! Alright. We’re getting somewhere. So the next step is that you probably need to be able to configure vhosts on the instance, so we want to setup Chef to be able to configure vhosts.
18. cookbooks ! mkdir -p apache2/definitions cookbooks ! vim apache2/definitions/site.rb So you start working on an “ apache2_site” definition, which will be a resource in Chef to create vhosts.
19. 1 define :apache_site, :enable => true do 2 ... 3 4 if params[:enable] 5 execute "a2ensite #{params[:name]}" do 6 command "/usr/sbin/a2ensite” + “#{params[:name]}" 8 notifies :restart, resources(:service => "apache2") 9 end 10 else 11 ... 16 end 17 end Here is what it might look like. Don’t worry about understanding this if you don’t know Chef, just know that you’re WRITING CODE.
20. cookbooks ! ./start_ec2_instance_for_test.sh And nowwwwww we want to test it so we need to bring up a new instance. Note: Yes, maybe you can re-use your old instance, because this time we’re just ADDING things to Chef, but what if you were CHANGING previous resources? Chef can’t undo, so you’d need a new instance. Wahhhhh!
21. Repeat. Again. And again. Anddddd you follow this process and repeat it over and over until you get to the state you want and feel its ready for stage/prod.
22. PRODUCTIVITY FAILURE. This is a huge failure. This process has a ton of problems associated with it...
23. Extreme case? Maybe. Maybe not. Is this an extreme case? For some people yes, but for some people no. I assumed a lot of things: * Automated server startup + Chef setup. * Chef somehow configured with cookbooks already on server. * Chef-solo, not chef-client. Therefore, kind of a toss-up. It can be extreme, but I think for a lot of people this rings true.
24. Problems: • Slow feedback loop • Encourages avoiding incremental development • No real automation!
25. A Look at Web Dev Today So we saw the ops side of things, let’s take a look at the dev side of things. Realizing Velocity is an “Ops” conference, but we’re also responsible for providing sane development environments as well. So let’s see what devs typically go through nowadays...
26. “You’ve been hired! How about you get the site up and running your shiny new MacBook Air?” A developer gets hired, and the boss says THIS! What is the next step?
27. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... The general first step is to clone out the source, so lets do that...
28. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer
29. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... Next, SOMETHING happens. We’ll get back to this, but SOMETHING HAPPENS.
30. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... facebook_killer ! ./start.sh Server listening on http://localhost:3000/... Then you run some script that starts a server and you’re OFF AND RUNNING! Ready to work!
31. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... A small sacrifice is made to Flying Spaghetti Monster Well, let’s take a look at this “...” step... Typically this step requires a small sacrifice to some deity. What goes on here?
32. sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... Let’s take a closer look...
33. Scenario 1: You have a setup script. First potential scenario is that you have a setup script, which is a script you run which sets up your environment for development.
34. facebook_killer ! ./setup.sh Installing some server... Installing some software... Configuring it all... Successful sacrifice made to FSM. It MIGHT look like this.
35. Scenario 1: You have a setup script. UNLIKELY But this is an unlikely scenario. Its possible, but unlikely. Even if it does exist, there are some problems.
36. • Your Mac is not what is running in production. Your Mac/Windows machine is not what is running in production. Your linux machine may be pretty similar, but ignoring that. Therefore, you’re trying to install server software on your own machine, in an environment it was not meant to run. You’re asking for trouble.
37. • Unlikely that all the software you need will work the same on your machine Sure, Apache, MySQL, etc. work and compile fine on Windows and Mac. But then you start getting some weird dependencies... or you write your own libraries. Its unlikely you want to spend any serious amount of time working on getting compilation working in every environment.
38. • Unlikely it is configured the same, which leads to pain later Example: MySQL has HUNDREDS of configuration parameters. You may configure them similarly so that it appears to work on your machine, but undoubtedly at some point in the future these slight configuration changes will cause HUGE WEIRD bugs.
39. • Requires special maintenance And perhaps most importantly, this setup script requires individual maintenance. That means that it’ll likely just decay and end up not working for periods of a time without large amounts of developer discipline. Things that require EXTRA WORK, typically don’t work.
40. • What if people want to work in Windows? Linux? So if you have other operating systems out there for dev, you’re going to cater these setup scripts to work in all environments? Yeah, right.
41. Scenario 1: You have a setup script. UNLIKELY
42. Scenario 2: The Über-README. Next scenario: You have an UBER README. This is a README that doesn’t DO anything except tell the developer how to set up his/her environment.
44. UNLIKELY Scenario 2: The Über-README.
45. • Extremely prone to user error. Actual execution of the setup is left to the user, who will likely do things wrong or differently.
46. • Heavy Maintenance Very careful steps must be maintained for every operating system for every piece of infrastructure. Like I said before, extra work doesn’t usually work.
47. • Time consuming The time for a new hire to get a site up and running goes from hours to days. I’ve heard of larger companies doing this (> 250 employees) where the time for an engineer to get a site up and running actually took 5 days. Whoa.
48. UNLIKELY Scenario 2: The Über-README.
49. Scenario 3: Good luck. Have fun. Scenario 3: GOOD LUCK!
50. facebook_killer ! ruby main.rb Failed to connect to MySQL at localhost:3306
51. Install MySQL.
52. facebook_killer ! ruby main.rb Failed to connect to Redis at localhost:6379!
53. Install Redis.
54. facebook_killer ! ruby main.rb Missing ImageMagick extensions!
55. Install ImageMagick. (Starting to hate yourself)
56. facebook_killer ! ruby main.rb Site running on localhost:3000.
60. Scenario 3: Good luck. Have fun. MOST LIKELY!
61. PRODUCTIVITY FAILURE.
62. Development Setup is also an Operations Problem Dev setup is also an ops problem, because we’re responsible for making sure they’re environments are working properly. Ops people work very hard to have stable systems up and running in production, why can’t that transfer over to development?
63. Problems: • Not repeatable • Not verifiably correct • Not isolated • Difficult to understand • SLOW, SLOW, SLOW!
64. Virtual Machines So is there some technology out there that can help solve some of this ops/dev pain that exists today? VIRTUAL MACHINES! This has been known for some time, this isn’t revolutionary.
65. • Isolated • Repeatable • Model complex relationships For Ops: VMs allow Ops to work in isolated environments, just how we’d like to test ops things initially. VMs are scriptable, ephemeral. We can use our programming skills to make a repeatable working environment. VMs are full mini-machines that are cheap, so we can just make lots of them to model complex relationships (such as distributed systems).
66. • Automated • Local • Host-OS agnostic For Dev: For dev, VMs are automated, which has numerous benefits: repeatable, easy, verifiable, testable. The VMs are or can be local, so developers can work on their own machine. Most VM providers provide software that works on most major platforms, so developers are free to work on the platform they want.
67. • Faster on-boarding • Anyone can do it • Less bugs For Biz: And there are also BUSINESS benefits. On-boarding times are slashed. When you hire a new employee, they can have a new environment up and running in a matter of minutes rather than hours or days. Anyone can do it, so even designers can bring up development environments if they want to. And there are less bugs, because the development environments are all the same so you get rid of the “works on my machine” bugs.
68. The Idea: Build local VMs to match production and develop there. So the idea to solve our problems is this.
69. We have many options when it comes down to local virtualization... So do we just pick one? LXC
70. Vagrant A standard workflow for working with VMs. http://vagrantup.com
71. Open source tool for building and distributing development environments. https://github.com/mitchellh/vagrant
72. Stable v1.0.3 Development since Jan 2010 Used by hundreds of companies
73. basho
74. Feature Breakdown • Simple management interface (CLI) • VM Creation • Provisioning • Scripted config of VM properties • Scripted config of networks • Scripted config of shared file system • Single command setup
75. VirtualBox Only, for now. Vagrant is currently VirtualBox only. When Vagrant started, VirtualBox is the only freely available virtualization tech that works on every major platform that also has a comprehensive API. It was the best choice to spur adoption. That being said, this will change in the
76. Setting up Vagrant http://vagrantup.com
77. Download and install VirtualBox: https://www.virtualbox.org/wiki/Downloads Download and install Vagrant: https://vagrantup.com
78. ~ ! vagrant Usage: vagrant [-v] [-h] command [<args>] -v, --version Print the version and exit. -h, --help Print this help. Available subcommands: box destroy gem halt init package provision reload resume ssh ssh-config status suspend up For help on any individual command run `vagrant COMMAND -h`
79. How does it work? http://vagrantup.com
80. How does it work? • Vagrantfile • vagrant command line interface These are the two components that Vagrant needs to work. We’ll go over each in detail a bit later, the main point now is that there are simply two things that make things work.
81. Vagrantfile • Per-project configuration • Configure VMs • Configure networking, provisioning, shared file system • Ruby DSL, no Ruby knowledge needed
82. vagrant • Single command to control lifecycle of your virtual machines. • git-style. “vagrant <subcommand>”
83. vagrant up The only command that matters for most people.
84. vagrant up • Single setup command • Creates VM • Configures properties of VM • Configures networking and shared file system on both host and guest • Provisions the guest
85. “Welcome to our company! Get the website up and running on your shiny new laptop!”
86. vagrant up To get that site up and running as a developer.
87. “Hey can you fix some bugs in the Apache2 cookbook?”
88. vagrant up To develop and test configuration management scripts
89. http://vagrantup.com Vagrant Concepts Now we’re going to go over the concepts of Vagrant. This section will be light on sticky notes because the slides are mostly self-explanatory.
90. Base Boxes http://vagrantup.com
91. Base Boxes • Base images for creating VMs • Minimal software set installed - SSH - Chef/Puppet (optional) • Includes VirtualBox Guest Additions
92. Box Management • Built-in to Vagrant • vagrant box
93. ~ ! vagrant box Usage: vagrant box <command> [<args>] Available subcommands: add list remove repackage For help on any individual command run `vagrant box COMMAND -h`
94. Adding a Box $ vagrant box add <name> <url> • Name: Local name referenced in config • URL: File path or HTTP URL.
95. Finding Boxes • “Official” Boxes: - lucid32, lucid64, precise32, precise64 - http://files.vagrantup.com/NAME.box • 3rd Party: - http://vagrantbox.es
96. ~ ! vagrant box add lucid32 \ http://files.vagrantup.com/lucid32.box
97. ~ ! vagrant box list lucid32
98. A First Vagrant Project http://vagrantup.com
99. • Vagrant is project-oriented • VMs exist in the context of a “project” • Project denoted by existence of a Vagrantfile Projects?
100. ~ ! vagrant init lucid32 A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `vagrantup.com` for more information on using Vagrant.
101. • Configures Vagrant for your project • Ruby DSL • No Ruby knowledge needed Vagrantfile?
102. vagrant up
103. ~ ! vagrant up [default] Importing base box 'lucid32'... [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
104. vagrant ssh
105. ~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$ echo Hello Velocity! Hello Velocity! vagrant@lucid32:~$
106. Fully functional VM!
107. In... 30 seconds! (Ignoring download times)
108. • Base box used as a skeleton, but private copy is created for each `up` • Therefore, files in your VM don’t affect past/future VMs. A Note on “up”
109. • vagrant up • vagrant status • vagrant reload • vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle
110. Working in a VM http://vagrantup.com
111. File Synchronization http://vagrantup.com
112. • VirtualBox shared folders • NFS (Depends on host*) • Use your preferred editor on the host • No need to sync files manually! File Synchronization
113. • By default “/vagrant” is your project directory. Shared Folders
114. vagrant@lucid32:~$ ls /vagrant Vagrantfile vagrant@lucid32:~$
115. config.vm.share_folder( “static”, “/srv/static”, “./static”) Shared Folders
116. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Shared Folders
117. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Shared Folders
118. config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Host path Shared Folders
119. vagrant reload
120. VirtualBox Shared Folders: 5m 14s Host File System: 10s Native VM File System: 13s NFS Shared Folders: 22s NFS Shared Folders (warm cache): 14s NFS
121. NFS • Windows does not support NFS. • Most linux distros do, but obscure ones may not.
122. config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders + NFS
123. config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders + NFS
124. vagrant reload
125. Provisioning http://vagrantup.com
126. • Scripted installation of packages • Scripted configuration management • Scripted starting of services Provisioning I’m probably preaching to the choir here
127. • Shell • Puppet (via apply or server) • Chef (solo and server) • CFEngine coming soon! Provisioning
128. • Setup development using same scripts as production. • Test scripts prior to putting into production • Without it, just moving the annoying setup of dev environment into a VM... Why Provision?
129. • First class support in Vagrantfile • Run with `vagrant up`, `vagrant reload`, or `vagrant provision` Provisioning
130. Chef Example 1 config.vm.provision :chef_solo do |c| 2 c.cookbooks_path = “cookbooks” 3 c.add_recipe “apache2” 4 end
131. vagrant reload
132. ~ ! vagrant reload [default] Attempting graceful shutdown of VM... ... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... stdin: is not a tty [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
133. “Welcome to our company! Get the website up and running on your shiny new laptop!”
134. vagrant up To get that site up and running as a developer.
135. vagrant provision
136. • Runs the provisioner again. • Allows iterative provisioning script development! vagrant provision
137. Networking http://vagrantup.com
138. We have a code for some service in the VM... how do we access it? The Problem
139. vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.
140. vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.
141. Network into the VM
142. • Port forwarding • Host-only Networking • Bridged Networking Networking
143. Port Forwarding
144. Expose some port on the guest via a port on the host.
145. Outside World Host Machine VM :22 :1234 :80 :8080
146. Port Forwarding 1 config.vm.forward_port 22, 1234 2 config.vm.forward_port 80, 8080
147. ~ ! vagrant reload [default] Attempting graceful shutdown of VM... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 1234 (adapter 1) [default] -- 80 => 8080 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
148. ~ ! vagrant reload [default] Attempting graceful shutdown of VM... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 1234 (adapter 1) [default] -- 80 => 8080 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
149. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...
150. 700x492 viewport
151. Common Gotcha: Make sure your listening socket binds to “0.0.0.0”
152. Pros/Cons • Pro: Simple • Con: Port exposed globally • Con: Port-by-port • Con: Host port >= 1024 (*)
153. Host-only Networking
154. Private network on your machine that can include your host and other VMs.
155. Outside World Host Machine VM 10.0.0.5 VM 10.0.0.6 VM 10.0.0.7 Virtual Router (VirtualBox) 10.0.0.1
156. Host-only Networking 1 config.vm.network :hostonly, "10.40.40.40"
157. ~ ! vagrant reload ... [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
158. ~ ! vagrant reload ... [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
159. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...
160. 700x492 viewport
161. 700x492 viewport
162. Common Gotcha: If the static IP you assign collides with your internet subnet, it won’t work.
163. Pros/Cons • Pro: Local-only, secure • Pro: VM can address the host • Pro: Multiple VMs can communicate • Con: Local-only
164. Bridged Networking
165. Bridge VM onto a networking device on your host machine.
166. Outside World Host Machine 10.0.1.14 Host Machine 10.0.1.23 Host Machine 10.0.1.35 LAN - Router VM 10.0.1.10 VM 10.0.1.16
167. Bridged Networking 1 config.vm.network :bridged
168. ~ ! vagrant reload ... [default] Available bridged network interfaces: 1) en0: Wi-Fi (AirPort) 2) p2p0 What interface should the network bridge to? 1 [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
169. ~ ! vagrant reload ... [default] Available bridged network interfaces: 1) en0: Wi-Fi (AirPort) 2) p2p0 What interface should the network bridge to? 1 [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant
170. vagrant@lucid32$ ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b inet addr:10.1.10.54 Bcast:10.1.10.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
171. vagrant@lucid32$ ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b inet addr:10.1.10.54 Bcast:10.1.10.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)
172. vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...
173. 700x492 viewport
174. 700x492 viewport
175. Common Gotcha: Bridging may not work on some router configs. Typically hotels.
176. Pros/Cons • Pro: Other machines on LAN can access the VM. • Con: Must select a network device • Con: Doesn’t always work depending on router config
177. • Mix multiple together! • Flexibility you need to work how you want Networking
178. Multi-VM http://vagrantup.com
179. Testing a SOA or distributed system properly. The Problem
180. • Configure multiple VMs in a single Vagrantfile • Network them together Multi-VM
181. Multi-VM Config 1 config.vm.box = "lucid64" 2 3 config.vm.define :web do |web| 4 web.vm.network :hostonly, "10.1.40.5" 5 end 6 7 config.vm.define :db do |db| 8 db.vm.network :hostonly, "10.1.40.6" 9 end
182. • vagrant up • vagrant status • vagrant reload • vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle
183. • vagrant up <target> • vagrant status <target> • vagrant reload <target> • vagrant suspend <target> • vagrant halt <target> • vagrant destroy <target> VM Lifecycle + Multi-VM
184. ~ ! vagrant up web [web] Importing base box 'lucid64'... [web] Matching MAC address for NAT networking... [web] Clearing any previously set forwarded ports... [web] Forwarding ports... [web] -- 22 => 2222 (adapter 1) [web] Creating shared folders metadata... [web] Clearing any previously set network interfaces... [web] Preparing network interfaces based on configuration... [web] Booting VM... [web] Waiting for VM to boot. This can take a few minutes. [web] VM booted and ready for use! [web] Configuring and enabling network interfaces... [web] Mounting shared folders... [web] -- v-root: /vagrant
185. ~ ! vagrant ssh web Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$
186. • Web, DB, queue, etc. on separate machines. • Allow developers to work on single service in SOA • Distributed system testing (fault tolerance, network partition, etc.) Use Cases
187. Packaging http://vagrantup.com
188. Provisioning takes a long time! The Problem
189. vagrant package
190. • Exports currently created Vagrant environment as a box. vagrant package
191. ~ ! vagrant package [default] Attempting graceful shutdown of VM... [default] Clearing any previously set forwarded ports... [default] Creating temporary directory for export... [default] Exporting VM... [default] Compressing package to: /Users/mitchellh/Dropbox/ Documents/presentations/2012_velocity/package.box
192. ~ ! vagrant box add my_site package.box [vagrant] Downloading with Vagrant::Downloaders::File... [vagrant] Copying box to temporary location... [vagrant] Extracting box... [vagrant] Verifying box... [vagrant] Cleaning up downloaded box...
193. Packaged Box 1 config.vm.box = “my_site”
194. ~ ! vagrant up [default] Importing base box 'my_site'... [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
195. Best Practices • Package up installed software • Use provisioners for configuration and managing services
196. Modern WebOps with Vagrant http://vagrantup.com
197. 1. Manually work in Vagrant 2. Automate a piece 3. Test in Vagrant, repeat. 4. Push to stage, then prod
198. “Automate the installation and configuration of Apache.”
199. 1. Manually work in Vagrant
200. ~ ! vagrant up [default] Importing base box 'lucid32'... [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
201. ~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$ sudo apt-get install apache2 ... vagrant@lucid32:~$ vi /etc/apache2/apache2.conf ...
202. Get to the target system manually using Vagrant
203. 2. Automate a piece
204. 1 package "apache2" do 2 action :install 3 end 4 5 service "apache2" do 6 action :enable 7 end 8 9 template "apache2.conf" do 10 path "#{node[:apache][:dir]}/” + “apache2.conf" 11 mode 0644 12 notifies :restart, resources(:service => "apache2") 13 end
205. 3. Test in Vagrant
206. ~ ! vagrant reload [default] Attempting graceful shutdown of VM... ... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... stdin: is not a tty [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete
207. Piece-by-piece get to target system previously reached in Vagrant
208. Repeat. Destroy VM if needed.
209. 4. Push to stage/prod
210. ~ ! knife cookbook upload apache2
211. Uses (Ops) • Developing Chef/Puppet/etc. scripts • Testing Chef/Puppet/etc. scripts • Using production-quality ops scripts to setup dev environments. • QA
212. Vagrant: Where is it going? http://vagrantup.com
213. workflow Predictable, usable, sensible.
214. A standard workflow for working with development environments.
215. so what’s next?
216. 1. Any virt software (VMWare, EC2, KVM)
217. ~ ! vagrant up --provider=vmware ... ~ ! vagrant up --provider=ec2 ...
218. Same workflow, but using the right environment for the right job.
219. 2. Vagrant Builder
220. Work in identical environments in both development and production
221. Vagrant Builder ISO Bootstrap Scripts VirtualBox VMWare AMI ...
222. 3. Any Guest OS
223. Windows, Mac OS, MyNextCustomOS? Sure.
224. • Dev/Prod Parity • Frictionless development • Flexibility Goals
225. http://vagrantup.com Thank you! Questions? @mitchellh