Vagrant: Not just for developers - LOPSA-East 2014

Vagrant’s tag line is ‘Development environments made easy’, and it is indeed a tool that vastly improves working with developer virtual machines. However, it can also make the life of a Systems Administrator much easier, providing throwaway test environments to help with troubleshooting, deploying new infrastructure, and playing with new technologies.

This class will take you through the fundamentals of vagrant, what it provides over and above virtualbox/vmware alone, getting your first virtual machine running, setting upnetworking, provisioning with configuration management tools, and creating your own boxes.

1. Vagrant: Not just for developers Mark Harrison Chef
2. Who am I • Mark Harrison • @mivok on twitter • Senior Systems Administrator at Chef • Vagrant user
3. Why am I here? • Tell you why vagrant is awesome. • Teach you how to use Vagrant. • Teach you how to do awesome things with Vagrant.
4. Why are you here?
5. What is Vagrant? Vagrant is a tool for building complete development environments. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases development/production parity, and makes the "works on my machine" excuse a relic of the past. (From: http://www.vagrantup.com/about.html)
6. What is Vagrant Vagrant is some developer tool that devs use because they don’t know how to install Linux or use Virtualbox. It helps you make dev environments, deploy your code, capistrano your ruby gems and other dev stuff that I’ll never use. (From: Me, before I actually started using vagrant)
7. What is Vagrant? I was wrong!
8. What is Vagrant Vagrant is tool for making working with virtual machines not suck.
9. Normal VM workflow • Start virtualbox • Dismiss random error message • Click new • Enter a name for your VM • Pick memory, disk space, OS, favorite color • Start VM
10. Normal VM Workflow • Realize you don’t have an ISO • Download ISO • Attach CD • Install OS • Wait… • Look at cat pictures
11. http://www.flickr.com/photos/plizzba/239595759
12. http://www.flickr.com/photos/mathias-erhart/2562106102
14. Normal VM Workflow • Finish looking at cat pictures • Reboot • Configure networking, users, hostname, software • Forget why you wanted the stupid VM in the first place
15. Vagrant Workflow me@laptop:~$ vagrant init hashicorp/precise32 me@laptop:~$ vagrant up me@laptop:~$ vagrant ssh vagrant@my_awesome_vm:~$
16. What does vagrant get you? • Pre-built ‘gold master’ images (called boxes) • Runs virtualbox headless (no console window) • Automatically does port forwarding for SSH • vagrant user with ssh key and sudo • /vagrant shared folder with the host • Built in support for provisioning with chef, puppet, ansible, shell scripts
17. Vagrant use cases • Throwaway test server • Testing new software • Testing config changes • Build system • Isolated environment for software
18. Example use cases • ATS and OmniOS build machines • Installing/testing logstash configurations • Developing OmniOS zone migration procedure • Testing firewall failover procedures • Developing cookbooks/playbooks/manifests • SRE interview VM
19. Convinced? Good! • Install Virtualbox from http://www.virtualbox.org/ • Install vagrant from http://www.vagrantup.com/ downloads.html • Packages are provided for common linux distributions, and installers are available for Windows and Mac
20. Starting out Vagrant < 1.5: $ mkdir -p ~/vagrant/precise_test $ cd ~/vagrant/precise_test $ vagrant init hashicorp/precise32 \
 http://files.vagrantup.com/precise32.box
21. Starting out Vagrant >= 1.5: $ mkdir -p ~/vagrant/precise_test $ cd ~/vagrant/precise_test $ vagrant init hashicorp/precise32 (No URL required - uses boxes from vagrantcloud.com)
22. Starting out When you have been using vagrant for a while: $ mkdir -p ~/vagrant/precise_test $ cd ~/vagrant/precise_test $ vim Vagrantfile
23. The Vagrantfile Vagrant.configure(2) do |config| config.vm.box = “hashicorp/precise64” #config.vm.box_url = “http://…” end
24. Boxes • “Gold master” images • .box extension • Usually downloaded via http • List at http://www.vagrantbox.es/ (unofficial) • Also at https://vagrantcloud.com/ (newer, official) • Can create your own using Packer (http://packer.io)
25. vagrantcloud.com
26. Example - Starting the VM
27. Starting the VM Steps vagrant performs when starting up: • Import the base box, downloading if needed • Create the VM in virtualbox • Set up port forwarding • Set up folder sharing • Boot the VM • Provision the VM (chef, puppet, ansible, shell scripts)
28. Using the VM $ cd ~/vagrant/my_vm $ vagrant ssh • Vagrant automatically knows which port to connect to • Vagrant boxes automatically have a vagrant user with a common (insecure) ssh key • The insecure key isn’t normally an issue for test machines, but be aware of it
29. Example - SSH into VM
30. Managing your VM • vagrant status - view the current state of the VM • vagrant halt - shut down the VM • vagrant up - bring it back up (and create if needed) • vagrant reload - halt + up in one command • vagrant destroy - delete the VM • vagrant suspend/resume - save VM state
31. Example - Managing your VM
32. Managing boxes • Download/manage boxes without starting the VM • vagrant box add hashicorp/precise64 • vagrant box list • vagrant box remove BOXNAME • vagrant box outdated [--global] • vagrant box update
33. Providers • Vagrant’s way of abstracting away different VM software • Comes with out of the box support for Virtualbox • Supports other providers too, including • Vmware fusion • ec2 • hyper-v (windows 8.1+)
34. Vagrant configuration • Folder sharing • Port forwarding • Custom networking • Provisioning • Multi-VM projects
35. Folder sharing • config.vm.synced_folder “some/folder", “/var/foo” • Not normally needed - /vagrant is usually sufficient • Uses Virtualbox shared folders • Somewhat slow, but not usually a problem • If performance is a concern, NFS or other options available.
36. Example - Folder Sharing
37. Folder sharing - nfs • To use nfs, add the “type” option: config.vm.synced_folder ".", "/vagrant", type: "nfs" • If you’re on Virtualbox, you also need to set up private networking to make this work: config.vm.network “private_network”, ip: “192.168.50.2”
38. Example - Folder sharing with NFS
39. Folder sharing - rsync • Rsync is new in 1.5 config.vm.synced_folder ".", "/vagrant", type: "rsync" • Need to run vagrant rsync or vagrant rsync-auto to sync • Sync is one-way - host to guest • Warning: rsync also uses the --delete option
40. Example - Folder sharing with Rsync
41. Port forwarding config.vm.network "forwarded_port", guest: 80, host: 8080 config.vm.network "forwarded_port", guest: 80, host: 8080, auto_correct: true • This is done automatically for port 22 (ssh)
42. Example - Port forwarding
43. Custom networking • Port forwarding is usually enough, but sometimes you need something different • Multiple VMs that need to talk to each other (private networking) • VMs accessible to machines on the network other than the host (public networking)
44. Private network • Used for when you want VMs to talk to each other • Uses ‘Host only’ networking option in virtualbox - only the host and other VMs can see the VM config.vm.network “private_network”, ip: “192.168.50.2”
45. Public network • Used when you want the VM to be part of the same network as the host • Uses virtualbox ‘bridged’ networking config.vm.network “public_network” The VM is configured to use DHCP, and will get the address from your network’s DHCP server.
46. Provisioning • One of the more powerful features of vagrant • Base boxes provide the bare minimum needed for vagrant to run • Provisioning lets you install and configure the rest • Vagrant supports chef, puppet, cfengine, ansible, salt, docker, shell scripts • We’ll cover chef, shell and ansible here
47. Provisioning • Provisioning happens: • The initial run of ‘vagrant up’ that creates the VM • When you run ‘vagrant provision’ • Older versions of vagrant (before 1.3.0) would provision every time the VM was booted.
48. Provisioning • Provisioners: • Run inside the VM, as root • Require that the relevant tools be installed (e.g. chef/ puppet) • Some base boxes provide chef/puppet already
49. Provisioning: shell • Often used in conjunction with other provisioners, e.g. to install the chef package • Useful if your provisioning needs are simple • Other provisioners are more powerful and you should use those • If you do use shell, you should make the script idempotent
50. Provisioning: shell • Simple command, directly in Vagrantfile: ! config.vm.provision "shell", inline: "which python2 || (pacman -Syy; pacman -S --noconfirm python2)”
51. Provisioning: shell • Installing chef: config.vm.provision “shell”, inline: "which chef-client || (curl -L https://www.opscode.com/chef/install.sh | bash)"
52. Provisioning: shell • Run a shell script from a file config.vm.provision "shell", path: “myscript.sh" • The path is relative to the vagrant project directory
53. Example - Shell provisioning
54. Provisioning: chef solo • Place your cookbooks/roles inside the vagrant project directory: myproject/
 Vagrantfile
 cookbooks/
 users/
 apache/
 roles/
 base.rb
55. Provisioning: chef solo • Add the run list to your vagrant file: config.vm.provision "chef_solo" do |chef|
 chef.roles_path “roles” # You need to specify this
 chef.add_role "base"
 end
56. Provisioning: chef solo • You can also include recipes directly, and set node configuration: config.vm.provision "chef_solo" do |chef|
 chef.add_recipe(“users”)
 chef.json = {
 “users" => [
 {“name” => “mark”, “uid” => 1000, …}
 ]
 }
 end
57. Example - Chef provisioning
58. Provisioning - installing chef • You can use the shell provisioner • Alternative - vagrant-omnibus plugin • install with ‘vagrant plugin install vagrant-omnibus’ • Then add the following to your Vagrantfile: config.omnibus.chef_version = :latest
59. Provisioning - chef client • This requires you have a chef server • Provisioning is similar to chef-solo, with some changes: config.vm.provision "chef_client" do |chef|
 chef.chef_server_url = “https://...”
 chef.validation_key_path = "validation.pem"
 end
60. Provisioning - chef client • Vagrant doesn’t delete the chef client for you by default • To do this, add the following to your Vagrantfile config.vm.provision "chef_client" do |chef|
 …
 chef.delete_node = true
 chef.delete_client = true
 end
61. Provisioning: ansible • Like with chef, you put your configuration in the vagrant directory: myproject/
 Vagrantfile
 site.yml
 roles/
 …
62. Provisioning: ansible • Add the path to the playbook in the Vagrantfile: config.vm.provision "ansible" do |ansible|
 ansible.playbook = "site.yml"
 end • You don’t need to generate an inventory file, vagrant will create one for you.
63. Example - Ansible provisioning
64. Multi-VM Projects • A vagrant project doesn’t need to be just one VM, you can have multiple. • The usual example given is allowing you to have a web server and database server on separate machines. • But, this is a really useful for SAs in that you can easily test failover with two identical machines. • This is done using ‘config.vm.define’ in your Vagrantfile
65. Multi-VM Projects Vagrant.configure("2") do |config| config.vm.box = “omnios-151006c-r1" config.vm.provision :shell, :path => "provision-common.sh" config.vm.define :src do |src| src.vm.hostname = "src" src.vm.provision :shell, :path => "provision-src.sh" src.vm.network :private_network, ip: "192.168.50.3" end config.vm.define :dst do |dst| dst.vm.hostname = "dst" dst.vm.network :private_network, ip: "192.168.50.4" end end
66. Multi-VM projects • You need to append the VM name to most vagrant commands now: • vagrant up vmname • vagrant ssh vmname • vagrant destroy vmname • If you leave out the VM name, it will assume you want to work on all VMs (e.g. vagrant up brings up all VMs)
67. Recap • What makes vagrant better than virtualbox alone • How to create and use a simple VM • Networking and shared folder configuration • Provisioning • Multiple VMs
68. Bonus Section: Vagrant 1.6 • New release of vagrant in the next few weeks • Global status - show all your vagrant VMs • Global control - no need to cd into vagrant dir • Full windows guest support • Docker guest support (in a linux VM on OSX/Windows)
69. Bonus Section: Packer Build your own boxes
70. Building your own Boxes • You will nearly always use pre-built vagrant boxes • What if you need to build your own? • OS or OS version for which a box doesn’t exist already • Custom software preinstalled • Special configuration/disk layout
71. What’s in a box • VM image with some special properties • ssh server running on boot • vagrant user with ssh key and sudo access • provisioning software (optionally) installed • guest additions/vmware tools • Metadata • vagrant package command
72. Packer - automation • This process is error prone and tedious • Packer was created to automate the process • Uses a json template to build VMs • I’ll be walking through a simple example to make an Arch Linux box for Virtualbox.
73. Packer - installation • Written in go, distributed as a zip file containing binaries • Download from packer.io • unzip to ~/packer (or elsewhere) • Mac Homebew - binary tap • brew tap homebrew/binary • brew install packer
74. Packer - template.json From: https://github.com/mivok/arch_scripts/tree/master/packer { "builders": [ { "type": "virtualbox-iso", "guest_os_type": "ArchLinux", "iso_url": "http://mirror.umd.edu/archlinux/iso/2014.04.01/archlinux-2014.04.01-dual.iso", "iso_checksum": "ae8caf8cd356dfcc24a4b645fd815185", "iso_checksum_type": "md5", "http_directory": ".", "ssh_username": "root", "ssh_password": "vagrant",
75. Packer - template.json "boot_command": [ "<enter><wait10><wait10>", "curl -O http://{{ .HTTPIP }}:{{ .HTTPPort }}/install.sh<enter><wait5>", "bash ./install.sh<enter>" ], "vboxmanage": [ ["modifyvm", "{{.Name}}", "--memory", "384"]] } ], "provisioners": [{ "type": "shell", "script": "provision.sh" }], "post-processors": [{"type": “vagrant", "compression_level": 9}] }
76. Packer - template.json • 4 main sections: • description • builders - create the vm image • provisioners - install and configure additional software • post-processors - generate a box from the image
77. Packer - builders • The ‘meat’ of the VM building • Specifies which VM software to use (virtualbox-iso) • Specifies VM parameters • Commands needed to install the OS • Finishes with a rebooted machine and SSH available
78. Packer - builders {
 "type": "virtualbox-iso",
 "guest_os_type": "ArchLinux",
 "iso_url": "http://mirror.umd.edu/archlinux/iso/ 2014.04.01/archlinux-2014.04.01-dual.iso",
 "iso_checksum": "ae8caf8cd356dfcc24a4b645fd815185",
 "iso_checksum_type": "md5",
 "http_directory": “.",
 "ssh_username": "root",
 "ssh_password": "vagrant",

79. Packer - builders "shutdown_command": "shutdown -h now",
 "boot_command": [
 "<enter><wait10><wait10>",
 "curl -O http://{{ .HTTPIP }}:{{ .HTTPPort }}/ install.sh<enter><wait5>",
 "bash ./install.sh<enter>"
 ],
 "vboxmanage": [
 ["modifyvm", "{{.Name}}", "--memory", "384"]
 ]
 }
80. Packer - install script • See https://github.com/mivok/arch_scripts/blob/master/packer/ install.sh for specifics • Partitions drives • Installs packages • Sets up networking (dhcp) • Sets ssh to start on boot • Sets the root password • Reboots
81. Packer - provisioners • Identical to vagrant • Can use chef, puppet, ansible, salt, shell "provisioners": [
 {
 "type": "shell",
 "script": "provision.sh"
 }
 ]
82. Packer - provisioning tasks • Add a vagrant user • Add ssh key for vagrant user • Set up sudo • Disable root user (optional) • Install guest additions (packer drops an iso in place for you) • Install chef/puppet (optional)
83. Packer - post processors "post-processors": [ { "type": "vagrant", "compression_level": 9 } ]
84. Packer - running • packer build [template.json] • Downloads the iso • Creates a new VM • Installs the OS • Provisions the machine • Creates a box
85. Example - Packer
86. Packer - recommendations • boot-command covers much more than booting • do as little as possible in boot-command • install a base system • get ssh access • use provisioners for the rest • use chef/puppet if you have it already • shell if you have simple needs
87. Questions?