Develop and Test Configuration Management Scripts with Vagrant

Slide 1

Slide 1 text

Develop and Test Config Management Scripts with Vagrant

Slide 2

Slide 2 text

Mitchell Hashimoto @mitchellh

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

jobs.kiip.me

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

A Look at Web Ops Today Let’s take a look at what the daily life in web ops looks like today...

Slide 7

Slide 7 text

“Automate the installation and configuration of Apache.” So your boss comes to you and says this... what do you do next? NOTE: We’re going to use Chef for examples because thats what I do, but it shouldn’t get in the way of any understanding.

Slide 8

Slide 8 text

cookbooks ! mkdir -p apache2/recipes cookbooks ! vim apache2/recipes/default.rb First you’ll probably go into some “cookbooks” directory and make an apache2 cookbook and a default recipe.

Slide 9

Slide 9 text

1 package "apache2" do 2 action :install 3 end 4 5 service "apache2" do 6 action :enable 7 end 8 9 template "apache2.conf" do 10 path "#{node[:apache][:dir]}/” + “apache2.conf" 11 mode 0644 12 notifies :restart, resources(:service => "apache2") 13 end You might type something like this into your recipe, using your best guess as to what it takes to install and configure Apache. Or perhaps you have a server somewhere you’re trying this out on as you go via SSH. Either way, you’re kind of guessing here.

Slide 10

Slide 10 text

cookbooks ! ./start_ec2_instance_for_test.sh So now you have a semi-working cookbook and you want to test it. Let’s be generous and assume you have some process for “quickly” bringing up some remote server (in this case EC2) for test. You run that script.

Slide 11

Slide 11 text

Coffee/Tea + Sword Fight. Now you wait.

Slide 12

Slide 12 text

cookbooks ! ./start_ec2_instance_for_test.sh Instance ready for testing! ec2-1-2-3-4.compute-1.amazonaws.com Eventually your EC2 instance is ready and you get the address to SSH to.

Slide 13

Slide 13 text

cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ You SSH into it.

Slide 14

Slide 14 text

cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ vim config.json You have to configure Chef now to run the cookbook you just made. Note that we’re making more assumptions here that your cookbooks were somehow rsyncced or found their way onto this server.

Slide 15

Slide 15 text

1 { 2 "run_list": ["recipe[apache2]"] 3 } Chef configuration to run your single recipe that you’re developing.

Slide 16

Slide 16 text

cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ sudo chef-solo -j config.json Next you run Chef..

Slide 17

Slide 17 text

cookbooks ! ssh ubuntu@... Welcome to Ubuntu 12.04 LTS! ubuntu$ sudo chef-solo -j config.json [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete And you get output that looks like this. Normal chef output. Looks like things worked! Alright. We’re getting somewhere. So the next step is that you probably need to be able to configure vhosts on the instance, so we want to setup Chef to be able to configure vhosts.

Slide 18

Slide 18 text

cookbooks ! mkdir -p apache2/definitions cookbooks ! vim apache2/definitions/site.rb So you start working on an “ apache2_site” definition, which will be a resource in Chef to create vhosts.

Slide 19

Slide 19 text

1 define :apache_site, :enable => true do 2 ... 3 4 if params[:enable] 5 execute "a2ensite #{params[:name]}" do 6 command "/usr/sbin/a2ensite” + “#{params[:name]}" 8 notifies :restart, resources(:service => "apache2") 9 end 10 else 11 ... 16 end 17 end Here is what it might look like. Don’t worry about understanding this if you don’t know Chef, just know that you’re WRITING CODE.

Slide 20

Slide 20 text

cookbooks ! ./start_ec2_instance_for_test.sh And nowwwwww we want to test it so we need to bring up a new instance. Note: Yes, maybe you can re-use your old instance, because this time we’re just ADDING things to Chef, but what if you were CHANGING previous resources? Chef can’t undo, so you’d need a new instance. Wahhhhh!

Slide 21

Slide 21 text

Repeat. Again. And again. Anddddd you follow this process and repeat it over and over until you get to the state you want and feel its ready for stage/prod.

Slide 22

Slide 22 text

PRODUCTIVITY FAILURE. This is a huge failure. This process has a ton of problems associated with it...

Slide 23

Slide 23 text

Extreme case? Maybe. Maybe not. Is this an extreme case? For some people yes, but for some people no. I assumed a lot of things: * Automated server startup + Chef setup. * Chef somehow configured with cookbooks already on server. * Chef-solo, not chef-client. Therefore, kind of a toss-up. It can be extreme, but I think for a lot of people this rings true.

Slide 24

Slide 24 text

Problems: • Slow feedback loop • Encourages avoiding incremental development • No real automation!

Slide 25

Slide 25 text

A Look at Web Dev Today So we saw the ops side of things, let’s take a look at the dev side of things. Realizing Velocity is an “Ops” conference, but we’re also responsible for providing sane development environments as well. So let’s see what devs typically go through nowadays...

Slide 26

Slide 26 text

“You’ve been hired! How about you get the site up and running your shiny new MacBook Air?” A developer gets hired, and the boss says THIS! What is the next step?

Slide 27

Slide 27 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... The general first step is to clone out the source, so lets do that...

Slide 28

Slide 28 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer

Slide 29

Slide 29 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... Next, SOMETHING happens. We’ll get back to this, but SOMETHING HAPPENS.

Slide 30

Slide 30 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... facebook_killer ! ./start.sh Server listening on http://localhost:3000/... Then you run some script that starts a server and you’re OFF AND RUNNING! Ready to work!

Slide 31

Slide 31 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... A small sacrifice is made to Flying Spaghetti Monster Well, let’s take a look at this “...” step... Typically this step requires a small sacrifice to some deity. What goes on here?

Slide 32

Slide 32 text

sites ! git clone git://.../facebook_killer.git Cloning into “facebook_killer”... sites ! cd facebook_killer facebook_killer ! ... Let’s take a closer look...

Slide 33

Slide 33 text

Scenario 1: You have a setup script. First potential scenario is that you have a setup script, which is a script you run which sets up your environment for development.

Slide 34

Slide 34 text

facebook_killer ! ./setup.sh Installing some server... Installing some software... Configuring it all... Successful sacrifice made to FSM. It MIGHT look like this.

Slide 35

Slide 35 text

Scenario 1: You have a setup script. UNLIKELY But this is an unlikely scenario. Its possible, but unlikely. Even if it does exist, there are some problems.

Slide 36

Slide 36 text

• Your Mac is not what is running in production. Your Mac/Windows machine is not what is running in production. Your linux machine may be pretty similar, but ignoring that. Therefore, you’re trying to install server software on your own machine, in an environment it was not meant to run. You’re asking for trouble.

Slide 37

Slide 37 text

• Unlikely that all the software you need will work the same on your machine Sure, Apache, MySQL, etc. work and compile fine on Windows and Mac. But then you start getting some weird dependencies... or you write your own libraries. Its unlikely you want to spend any serious amount of time working on getting compilation working in every environment.

Slide 38

Slide 38 text

• Unlikely it is configured the same, which leads to pain later Example: MySQL has HUNDREDS of configuration parameters. You may configure them similarly so that it appears to work on your machine, but undoubtedly at some point in the future these slight configuration changes will cause HUGE WEIRD bugs.

Slide 39

Slide 39 text

• Requires special maintenance And perhaps most importantly, this setup script requires individual maintenance. That means that it’ll likely just decay and end up not working for periods of a time without large amounts of developer discipline. Things that require EXTRA WORK, typically don’t work.

Slide 40

Slide 40 text

• What if people want to work in Windows? Linux? So if you have other operating systems out there for dev, you’re going to cater these setup scripts to work in all environments? Yeah, right.

Slide 41

Slide 41 text

Scenario 1: You have a setup script. UNLIKELY

Slide 42

Slide 42 text

Scenario 2: The Über-README. Next scenario: You have an UBER README. This is a README that doesn’t DO anything except tell the developer how to set up his/her environment.

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

UNLIKELY Scenario 2: The Über-README.

Slide 45

Slide 45 text

• Extremely prone to user error. Actual execution of the setup is left to the user, who will likely do things wrong or differently.

Slide 46

Slide 46 text

• Heavy Maintenance Very careful steps must be maintained for every operating system for every piece of infrastructure. Like I said before, extra work doesn’t usually work.

Slide 47

Slide 47 text

• Time consuming The time for a new hire to get a site up and running goes from hours to days. I’ve heard of larger companies doing this (> 250 employees) where the time for an engineer to get a site up and running actually took 5 days. Whoa.

Slide 48

Slide 48 text

UNLIKELY Scenario 2: The Über-README.

Slide 49

Slide 49 text

Scenario 3: Good luck. Have fun. Scenario 3: GOOD LUCK!

Slide 50

Slide 50 text

facebook_killer ! ruby main.rb Failed to connect to MySQL at localhost:3306

Slide 51

Slide 51 text

Install MySQL.

Slide 52

Slide 52 text

facebook_killer ! ruby main.rb Failed to connect to Redis at localhost:6379!

Slide 53

Slide 53 text

Install Redis.

Slide 54

Slide 54 text

facebook_killer ! ruby main.rb Missing ImageMagick extensions!

Slide 55

Slide 55 text

Install ImageMagick. (Starting to hate yourself)

Slide 56

Slide 56 text

facebook_killer ! ruby main.rb Site running on localhost:3000.

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

Scenario 3: Good luck. Have fun. MOST LIKELY!

Slide 61

Slide 61 text

PRODUCTIVITY FAILURE.

Slide 62

Slide 62 text

Development Setup is also an Operations Problem Dev setup is also an ops problem, because we’re responsible for making sure they’re environments are working properly. Ops people work very hard to have stable systems up and running in production, why can’t that transfer over to development?

Slide 63

Slide 63 text

Problems: • Not repeatable • Not verifiably correct • Not isolated • Difficult to understand • SLOW, SLOW, SLOW!

Slide 64

Slide 64 text

Virtual Machines So is there some technology out there that can help solve some of this ops/dev pain that exists today? VIRTUAL MACHINES! This has been known for some time, this isn’t revolutionary.

Slide 65

Slide 65 text

• Isolated • Repeatable • Model complex relationships For Ops: VMs allow Ops to work in isolated environments, just how we’d like to test ops things initially. VMs are scriptable, ephemeral. We can use our programming skills to make a repeatable working environment. VMs are full mini-machines that are cheap, so we can just make lots of them to model complex relationships (such as distributed systems).

Slide 66

Slide 66 text

• Automated • Local • Host-OS agnostic For Dev: For dev, VMs are automated, which has numerous benefits: repeatable, easy, verifiable, testable. The VMs are or can be local, so developers can work on their own machine. Most VM providers provide software that works on most major platforms, so developers are free to work on the platform they want.

Slide 67

Slide 67 text

• Faster on-boarding • Anyone can do it • Less bugs For Biz: And there are also BUSINESS benefits. On-boarding times are slashed. When you hire a new employee, they can have a new environment up and running in a matter of minutes rather than hours or days. Anyone can do it, so even designers can bring up development environments if they want to. And there are less bugs, because the development environments are all the same so you get rid of the “works on my machine” bugs.

Slide 68

Slide 68 text

The Idea: Build local VMs to match production and develop there. So the idea to solve our problems is this.

Slide 69

Slide 69 text

We have many options when it comes down to local virtualization... So do we just pick one? LXC

Slide 70

Slide 70 text

Vagrant A standard workflow for working with VMs. http://vagrantup.com

Slide 71

Slide 71 text

Open source tool for building and distributing development environments. https://github.com/mitchellh/vagrant

Slide 72

Slide 72 text

Stable v1.0.3 Development since Jan 2010 Used by hundreds of companies

Slide 73

Slide 73 text

basho

Slide 74

Slide 74 text

Feature Breakdown • Simple management interface (CLI) • VM Creation • Provisioning • Scripted config of VM properties • Scripted config of networks • Scripted config of shared file system • Single command setup

Slide 75

Slide 75 text

VirtualBox Only, for now. Vagrant is currently VirtualBox only. When Vagrant started, VirtualBox is the only freely available virtualization tech that works on every major platform that also has a comprehensive API. It was the best choice to spur adoption. That being said, this will change in the

Slide 76

Slide 76 text

Setting up Vagrant http://vagrantup.com

Slide 77

Slide 77 text

Download and install VirtualBox: https://www.virtualbox.org/wiki/Downloads Download and install Vagrant: https://vagrantup.com

Slide 78

Slide 78 text

~ ! vagrant Usage: vagrant [-v] [-h] command [] -v, --version Print the version and exit. -h, --help Print this help. Available subcommands: box destroy gem halt init package provision reload resume ssh ssh-config status suspend up For help on any individual command run `vagrant COMMAND -h`

Slide 79

Slide 79 text

How does it work? http://vagrantup.com

Slide 80

Slide 80 text

How does it work? • Vagrantfile • vagrant command line interface These are the two components that Vagrant needs to work. We’ll go over each in detail a bit later, the main point now is that there are simply two things that make things work.

Slide 81

Slide 81 text

Vagrantfile • Per-project configuration • Configure VMs • Configure networking, provisioning, shared file system • Ruby DSL, no Ruby knowledge needed

Slide 82

Slide 82 text

vagrant • Single command to control lifecycle of your virtual machines. • git-style. “vagrant ”

Slide 83

Slide 83 text

vagrant up The only command that matters for most people.

Slide 84

Slide 84 text

vagrant up • Single setup command • Creates VM • Configures properties of VM • Configures networking and shared file system on both host and guest • Provisions the guest

Slide 85

Slide 85 text

“Welcome to our company! Get the website up and running on your shiny new laptop!”

Slide 86

Slide 86 text

vagrant up To get that site up and running as a developer.

Slide 87

Slide 87 text

“Hey can you fix some bugs in the Apache2 cookbook?”

Slide 88

Slide 88 text

vagrant up To develop and test configuration management scripts

Slide 89

Slide 89 text

http://vagrantup.com Vagrant Concepts Now we’re going to go over the concepts of Vagrant. This section will be light on sticky notes because the slides are mostly self-explanatory.

Slide 90

Slide 90 text

Base Boxes http://vagrantup.com

Slide 91

Slide 91 text

Base Boxes • Base images for creating VMs • Minimal software set installed - SSH - Chef/Puppet (optional) • Includes VirtualBox Guest Additions

Slide 92

Slide 92 text

Box Management • Built-in to Vagrant • vagrant box

Slide 93

Slide 93 text

~ ! vagrant box Usage: vagrant box [] Available subcommands: add list remove repackage For help on any individual command run `vagrant box COMMAND -h`

Slide 94

Slide 94 text

Adding a Box $ vagrant box add • Name: Local name referenced in config • URL: File path or HTTP URL.

Slide 95

Slide 95 text

Finding Boxes • “Official” Boxes: - lucid32, lucid64, precise32, precise64 - http://files.vagrantup.com/NAME.box • 3rd Party: - http://vagrantbox.es

Slide 96

Slide 96 text

~ ! vagrant box add lucid32 \ http://files.vagrantup.com/lucid32.box

Slide 97

Slide 97 text

~ ! vagrant box list lucid32

Slide 98

Slide 98 text

A First Vagrant Project http://vagrantup.com

Slide 99

Slide 99 text

• Vagrant is project-oriented • VMs exist in the context of a “project” • Project denoted by existence of a Vagrantfile Projects?

Slide 100

Slide 100 text

~ ! vagrant init lucid32 A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `vagrantup.com` for more information on using Vagrant.

Slide 101

Slide 101 text

• Configures Vagrant for your project • Ruby DSL • No Ruby knowledge needed Vagrantfile?

Slide 102

Slide 102 text

vagrant up

Slide 103

Slide 103 text

~ ! vagrant up [default] Importing base box 'lucid32'... [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant

Slide 104

Slide 104 text

vagrant ssh

Slide 105

Slide 105 text

~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$ echo Hello Velocity! Hello Velocity! vagrant@lucid32:~$

Slide 106

Slide 106 text

Fully functional VM!

Slide 107

Slide 107 text

In... 30 seconds! (Ignoring download times)

Slide 108

Slide 108 text

• Base box used as a skeleton, but private copy is created for each `up` • Therefore, files in your VM don’t affect past/future VMs. A Note on “up”

Slide 109

Slide 109 text

• vagrant up • vagrant status • vagrant reload • vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle

Slide 110

Slide 110 text

Working in a VM http://vagrantup.com

Slide 111

Slide 111 text

File Synchronization http://vagrantup.com

Slide 112

Slide 112 text

• VirtualBox shared folders • NFS (Depends on host*) • Use your preferred editor on the host • No need to sync files manually! File Synchronization

Slide 113

Slide 113 text

• By default “/vagrant” is your project directory. Shared Folders

Slide 114

Slide 114 text

vagrant@lucid32:~$ ls /vagrant Vagrantfile vagrant@lucid32:~$

Slide 115

Slide 115 text

config.vm.share_folder( “static”, “/srv/static”, “./static”) Shared Folders

Slide 116

Slide 116 text

config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Shared Folders

Slide 117

Slide 117 text

config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Shared Folders

Slide 118

Slide 118 text

config.vm.share_folder( “static”, “/srv/static”, “./static”) Logical name Guest path Host path Shared Folders

Slide 119

Slide 119 text

vagrant reload

Slide 120

Slide 120 text

VirtualBox Shared Folders: 5m 14s Host File System: 10s Native VM File System: 13s NFS Shared Folders: 22s NFS Shared Folders (warm cache): 14s NFS

Slide 121

Slide 121 text

NFS • Windows does not support NFS. • Most linux distros do, but obscure ones may not.

Slide 122

Slide 122 text

config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders + NFS

Slide 123

Slide 123 text

config.vm.share_folder( “static”, “/srv/static”, “./static”, :nfs => true) Shared Folders + NFS

Slide 124

Slide 124 text

vagrant reload

Slide 125

Slide 125 text

Provisioning http://vagrantup.com

Slide 126

Slide 126 text

• Scripted installation of packages • Scripted configuration management • Scripted starting of services Provisioning I’m probably preaching to the choir here

Slide 127

Slide 127 text

• Shell • Puppet (via apply or server) • Chef (solo and server) • CFEngine coming soon! Provisioning

Slide 128

Slide 128 text

• Setup development using same scripts as production. • Test scripts prior to putting into production • Without it, just moving the annoying setup of dev environment into a VM... Why Provision?

Slide 129

Slide 129 text

• First class support in Vagrantfile • Run with `vagrant up`, `vagrant reload`, or `vagrant provision` Provisioning

Slide 130

Slide 130 text

Chef Example 1 config.vm.provision :chef_solo do |c| 2 c.cookbooks_path = “cookbooks” 3 c.add_recipe “apache2” 4 end

Slide 131

Slide 131 text

vagrant reload

Slide 132

Slide 132 text

~ ! vagrant reload [default] Attempting graceful shutdown of VM... ... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... stdin: is not a tty [Sun, 24 Jun 2012 21:33:14 -0700] INFO: *** Chef 0.10.2 *** [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Setting the run_list to ["recipe[apache2]"] from JSON [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List is [recipe[apache2]] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Run List expands to [apache2] [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Starting Chef Run for lucid64 [Sun, 24 Jun 2012 21:33:15 -0700] INFO: Processing execute[apt-get update] action run (apache2::default line 1) [Sun, 24 Jun 2012 21:33:15 -0700] INFO: execute[apt-get update] sh(apt-get update) [Sun, 24 Jun 2012 21:35:29 -0700] INFO: execute[apt-get update] ran successfully [Sun, 24 Jun 2012 21:35:29 -0700] INFO: Processing package[apache2] action install (apache2::default line 3) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: package[apache2] installed version 2.2.14-5ubuntu8.9 [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Processing service[apache2] action enable (apache2::default line 7) [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Chef Run complete in 175.23797 seconds [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Running report handlers [Sun, 24 Jun 2012 21:36:10 -0700] INFO: Report handlers complete

Slide 133

Slide 133 text

“Welcome to our company! Get the website up and running on your shiny new laptop!”

Slide 134

Slide 134 text

vagrant up To get that site up and running as a developer.

Slide 135

Slide 135 text

vagrant provision

Slide 136

Slide 136 text

• Runs the provisioner again. • Allows iterative provisioning script development! vagrant provision

Slide 137

Slide 137 text

Networking http://vagrantup.com

Slide 138

Slide 138 text

We have a code for some service in the VM... how do we access it? The Problem

Slide 139

Slide 139 text

vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.

Slide 140

Slide 140 text

vagrant@lucid32:~$ curl http://localhost/ This workflow would never work.

Slide 141

Slide 141 text

Network into the VM

Slide 142

Slide 142 text

• Port forwarding • Host-only Networking • Bridged Networking Networking

Slide 143

Slide 143 text

Port Forwarding

Slide 144

Slide 144 text

Expose some port on the guest via a port on the host.

Slide 145

Slide 145 text

Outside World Host Machine VM :22 :1234 :80 :8080

Slide 146

Slide 146 text

Port Forwarding 1 config.vm.forward_port 22, 1234 2 config.vm.forward_port 80, 8080

Slide 147

Slide 147 text

~ ! vagrant reload [default] Attempting graceful shutdown of VM... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 1234 (adapter 1) [default] -- 80 => 8080 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant

Slide 148

Slide 148 text

Slide 149

Slide 149 text

vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...

Slide 150

Slide 150 text

700x492 viewport

Slide 151

Slide 151 text

Common Gotcha: Make sure your listening socket binds to “0.0.0.0”

Slide 152

Slide 152 text

Pros/Cons • Pro: Simple • Con: Port exposed globally • Con: Port-by-port • Con: Host port >= 1024 (*)

Slide 153

Slide 153 text

Host-only Networking

Slide 154

Slide 154 text

Private network on your machine that can include your host and other VMs.

Slide 155

Slide 155 text

Outside World Host Machine VM 10.0.0.5 VM 10.0.0.6 VM 10.0.0.7 Virtual Router (VirtualBox) 10.0.0.1

Slide 156

Slide 156 text

Host-only Networking 1 config.vm.network :hostonly, "10.40.40.40"

Slide 157

Slide 157 text

~ ! vagrant reload ... [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant

Slide 158

Slide 158 text

Slide 159

Slide 159 text

vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...

Slide 160

Slide 160 text

700x492 viewport

Slide 161

Slide 161 text

700x492 viewport

Slide 162

Slide 162 text

Common Gotcha: If the static IP you assign collides with your internet subnet, it won’t work.

Slide 163

Slide 163 text

Pros/Cons • Pro: Local-only, secure • Pro: VM can address the host • Pro: Multiple VMs can communicate • Con: Local-only

Slide 164

Slide 164 text

Bridged Networking

Slide 165

Slide 165 text

Bridge VM onto a networking device on your host machine.

Slide 166

Slide 166 text

Outside World Host Machine 10.0.1.14 Host Machine 10.0.1.23 Host Machine 10.0.1.35 LAN - Router VM 10.0.1.10 VM 10.0.1.16

Slide 167

Slide 167 text

Bridged Networking 1 config.vm.network :bridged

Slide 168

Slide 168 text

~ ! vagrant reload ... [default] Available bridged network interfaces: 1) en0: Wi-Fi (AirPort) 2) p2p0 What interface should the network bridge to? 1 [default] Preparing network interfaces based on configuration... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... [default] Mounting shared folders... [default] -- v-root: /vagrant

Slide 169

Slide 169 text

Slide 170

Slide 170 text

vagrant@lucid32$ ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:ad:7f:2b inet addr:10.1.10.54 Bcast:10.1.10.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fead:7f2b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8914 (8.9 KB) TX bytes:1152 (1.1 KB)

Slide 171

Slide 171 text

Slide 172

Slide 172 text

vagrant@lucid32$ cd /Vagrant vagrant@lucid32$ sudo python -m SimpleHTTPServer 80 Serving HTTP on 0.0.0.0 port 80 ...

Slide 173

Slide 173 text

700x492 viewport

Slide 174

Slide 174 text

700x492 viewport

Slide 175

Slide 175 text

Common Gotcha: Bridging may not work on some router configs. Typically hotels.

Slide 176

Slide 176 text

Pros/Cons • Pro: Other machines on LAN can access the VM. • Con: Must select a network device • Con: Doesn’t always work depending on router config

Slide 177

Slide 177 text

• Mix multiple together! • Flexibility you need to work how you want Networking

Slide 178

Slide 178 text

Multi-VM http://vagrantup.com

Slide 179

Slide 179 text

Testing a SOA or distributed system properly. The Problem

Slide 180

Slide 180 text

• Configure multiple VMs in a single Vagrantfile • Network them together Multi-VM

Slide 181

Slide 181 text

Multi-VM Config 1 config.vm.box = "lucid64" 2 3 config.vm.define :web do |web| 4 web.vm.network :hostonly, "10.1.40.5" 5 end 6 7 config.vm.define :db do |db| 8 db.vm.network :hostonly, "10.1.40.6" 9 end

Slide 182

Slide 182 text

• vagrant up • vagrant status • vagrant reload • vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle

Slide 183

Slide 183 text

• vagrant up • vagrant status • vagrant reload • vagrant suspend • vagrant halt • vagrant destroy VM Lifecycle + Multi-VM

Slide 184

Slide 184 text

~ ! vagrant up web [web] Importing base box 'lucid64'... [web] Matching MAC address for NAT networking... [web] Clearing any previously set forwarded ports... [web] Forwarding ports... [web] -- 22 => 2222 (adapter 1) [web] Creating shared folders metadata... [web] Clearing any previously set network interfaces... [web] Preparing network interfaces based on configuration... [web] Booting VM... [web] Waiting for VM to boot. This can take a few minutes. [web] VM booted and ready for use! [web] Configuring and enabling network interfaces... [web] Mounting shared folders... [web] -- v-root: /vagrant

Slide 185

Slide 185 text

~ ! vagrant ssh web Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$

Slide 186

Slide 186 text

• Web, DB, queue, etc. on separate machines. • Allow developers to work on single service in SOA • Distributed system testing (fault tolerance, network partition, etc.) Use Cases

Slide 187

Slide 187 text

Packaging http://vagrantup.com

Slide 188

Slide 188 text

Provisioning takes a long time! The Problem

Slide 189

Slide 189 text

vagrant package

Slide 190

Slide 190 text

• Exports currently created Vagrant environment as a box. vagrant package

Slide 191

Slide 191 text

~ ! vagrant package [default] Attempting graceful shutdown of VM... [default] Clearing any previously set forwarded ports... [default] Creating temporary directory for export... [default] Exporting VM... [default] Compressing package to: /Users/mitchellh/Dropbox/ Documents/presentations/2012_velocity/package.box

Slide 192

Slide 192 text

~ ! vagrant box add my_site package.box [vagrant] Downloading with Vagrant::Downloaders::File... [vagrant] Copying box to temporary location... [vagrant] Extracting box... [vagrant] Verifying box... [vagrant] Cleaning up downloaded box...

Slide 193

Slide 193 text

Packaged Box 1 config.vm.box = “my_site”

Slide 194

Slide 194 text

~ ! vagrant up [default] Importing base box 'my_site'... [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant

Slide 195

Slide 195 text

Best Practices • Package up installed software • Use provisioners for configuration and managing services

Slide 196

Slide 196 text

Modern WebOps with Vagrant http://vagrantup.com

Slide 197

Slide 197 text

1. Manually work in Vagrant 2. Automate a piece 3. Test in Vagrant, repeat. 4. Push to stage, then prod

Slide 198

Slide 198 text

“Automate the installation and configuration of Apache.”

Slide 199

Slide 199 text

1. Manually work in Vagrant

Slide 200

Slide 200 text

Slide 201

Slide 201 text

~ ! vagrant ssh Ubuntu 10.04.3 LTS Welcome to Ubuntu! vagrant@lucid32:~$ sudo apt-get install apache2 ... vagrant@lucid32:~$ vi /etc/apache2/apache2.conf ...

Slide 202

Slide 202 text

Get to the target system manually using Vagrant

Slide 203

Slide 203 text

2. Automate a piece

Slide 204

Slide 204 text

Slide 205

Slide 205 text

3. Test in Vagrant

Slide 206

Slide 206 text

Slide 207

Slide 207 text

Piece-by-piece get to target system previously reached in Vagrant

Slide 208

Slide 208 text

Repeat. Destroy VM if needed.

Slide 209

Slide 209 text

4. Push to stage/prod

Slide 210

Slide 210 text

~ ! knife cookbook upload apache2

Slide 211

Slide 211 text

Uses (Ops) • Developing Chef/Puppet/etc. scripts • Testing Chef/Puppet/etc. scripts • Using production-quality ops scripts to setup dev environments. • QA