Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtualized Development Environments with Vagrant - PHPNE, 15th May 2012

ichilton
May 15, 2012
520

Virtualized Development Environments with Vagrant - PHPNE, 15th May 2012

A talk at PHP North East, 12th May 2012 - introduction to Virtualization, Vagrant, Configuration Management / Infrastructure as Code, Puppet and Chef...

ichilton

May 15, 2012
Tweet

Transcript

  1. Introduction Ian Chilton Better Brand Agency http://www.betterbrandagency.com http://www.bettercommunities.co.uk Twitter: @ichilton

    Web: http://www.ichilton.co.uk LinkedIn: http://uk.linkedin.com/in/ichilton Tuesday, 15 May 12
  2. Contents - Introduction to: - Development Environments - Virtualization -

    Why Virtualize your development environment? - Introduction to Vagrant - Using Vagrant - Vagrant Plugins - Automated Provisioning (Configuration Management) Tuesday, 15 May 12
  3. Development Environments We don’t want to be working on live

    sites. So we setup a development environment. Tuesday, 15 May 12
  4. Development Environments We don’t want to be working on live

    sites. So we setup a development environment. DEV Staging VCS LIVE Tuesday, 15 May 12
  5. Development Environments We don’t want to be working on live

    sites. So we setup a development environment. DEV Staging VCS LIVE DEV Staging DEV LIVE VCS DEV Tuesday, 15 May 12
  6. Development Environments We don’t want to be working on live

    sites. So we setup a development environment. DEV Staging VCS LIVE DEV Staging DEV LIVE VCS DEV We work on development, push/pull to/from version control and deploy to staging and live. Tuesday, 15 May 12
  7. Development Environments We don’t want to be working on live

    sites. So we setup a development environment. DEV Staging VCS LIVE DEV Staging DEV LIVE VCS DEV We work on development, push/pull to/from version control and deploy to staging and live. It’s DEV we’re going to be talking about here. Tuesday, 15 May 12
  8. Development Environments Number of (valid) ways: - Copy of site

    (on live server) - Can easily affect the live site. Tuesday, 15 May 12
  9. Development Environments Number of (valid) ways: - Copy of site

    (on live server) - Can easily affect the live site. - Shared development server - Local / Remote / Dedicated / Cloud / VPS - Cost - Remote = latency / connection failure - Setup / management / maintenance - All developers and all projects sharing same hardware, software, versions etc. Tuesday, 15 May 12
  10. Development Environments Number of (valid) ways: - continued - Develop

    Locally: - MAMP (mac) or WAMP (windows) - Homebrew (mac), apt-get etc (linux) to install services locally Tuesday, 15 May 12
  11. Development Environments Number of (valid) ways: - continued - Develop

    Locally: - MAMP (mac) or WAMP (windows) - Homebrew (mac), apt-get etc (linux) to install services locally - However... - Clutter on workstation - Still shared between projects, set versions etc... - Different OS / platform / architecture - Self sufficient - no remote latency / connection failure Tuesday, 15 May 12
  12. Development Environments Consider these scenarios (just some examples of many...)

    The methods on the previous slides could install a different (much older or newer) version of a service like MySQL or PHP than the production box. You could then inadvertently code with something only added in a later version, or use something which had bugs in earlier versions. You’d there for have code which works in development but not when deployed to production. Tuesday, 15 May 12
  13. Development Environments Consider these scenarios (just some examples of many...)

    Both Windows and OS X use case insensitive filesystems. Imagine writing some code which used MyFile.txt. You could call this file myfile.txt and it would work on Windows/Mac. On Linux that would either error or be a completely different file. Tuesday, 15 May 12
  14. Virtualization Such bugs can be very difficult and time consuming

    to track down. The lesson... We really need to develop on the same operating system, architecture, package versions and configuration as production. Tuesday, 15 May 12
  15. Virtualization Virtualization is a technology where by a number of

    “virtual machines” can be hosted on top of a single machine. Tuesday, 15 May 12
  16. Virtualization Virtualization is a technology where by a number of

    “virtual machines” can be hosted on top of a single machine. The software running on the virtual machines (guests) is separated from the software running on the “host”. Tuesday, 15 May 12
  17. Virtualization Virtualization is a technology where by a number of

    “virtual machines” can be hosted on top of a single machine. The software running on the virtual machines (guests) is separated from the software running on the “host”. The software used to make this possible is called a “hypervisor”. Tuesday, 15 May 12
  18. Virtualization Virtualization is a technology where by a number of

    “virtual machines” can be hosted on top of a single machine. The software running on the virtual machines (guests) is separated from the software running on the “host”. The software used to make this possible is called a “hypervisor”. This allows us to run different software on different virtual machines - for example you could run Windows and Linux virtual machines on a Mac. Tuesday, 15 May 12
  19. Types of Virtualization Full Virtualization Almost complete simulation of the

    actual hardware. This allows the guest operating systems to run unmodified. Eg: - Oracle Virtualbox - Free virtualization product for the desktop - Cross platform - Windows, OS X, Linux - Lots of features - VMWare - Commercial (££) - Workstation and Server products Tuesday, 15 May 12
  20. Types of Virtualization Paravirtualization The hardware environment is not simulated.

    The guest operating system is specially modified to allow the hypervisor to run the guests in their own isolated domain. Eg: - Xen which is used by: - Amazon EC2 - Rackspace Cloud - Some VPS providers Tuesday, 15 May 12
  21. Types of Virtualization Hardware Assisted Virtualization Modern Intel or AMD

    CPU’s contain native virtualization functionality on the CPU - Intel VT-x and AMD-V. Eg: - Linux KVM (Kernel Virtual Machine) which is used by: - Bytemark - Brightbox - VPS providers - Xen, Virtualbox & VMWare also support this as well, for hosts that have the hardware support. Tuesday, 15 May 12
  22. Types of Virtualization Operating System Level Virtualization (aka. containers) The

    virtualization is implemented in software, on the host operating system/kernel. The guest operating systems share the kernel of the host, but run the guest operating system independently of the host. This allows isolated virtual machines and different distributions, but is generally limited to Unix based operating systems. Eg: - OpenVZ - LXC on Linux - FreeBSD jails Often used by budget VPS providers as they can over-sell. Tuesday, 15 May 12
  23. Advantages of Using Virtualization - We can run the exact

    same operating system and setup as the production boxes, on top of any other machine, be it Mac, Windows or Linux. - We can install the same software / packages / versions. - Everything is run in a contained sandbox and resources can be managed. - We can create distributable images to share. - If we run the VM’s locally, we can develop offline. Tuesday, 15 May 12
  24. Disadvantages of Using Virtualization - Many of the products we’ve

    mentioned require Linux (or another UNIX operating system) and don’t run on Mac or Windows. - Require searching around for operating system images or installing the operating system on the VM manually. - Can be complicated/time consuming to learn, setup and configure. Tuesday, 15 May 12
  25. Disadvantages of Using Virtualization - Many of the products we’ve

    mentioned require Linux (or another UNIX operating system) and don’t run on Mac or Windows. - Require searching around for operating system images or installing the operating system on the VM manually. - Can be complicated/time consuming to learn, setup and configure. What we need is a simple and quick way of bringing up virtual machines with minimal fuss. Tuesday, 15 May 12
  26. Vagrant Created by Mitchell Hashimoto and John Bender Command line

    interface/wrapper around Oracle’s free VirtualBox software (support for other hypervisors are in the works). Allows virtual machines to be created, inside of a minute, from pre- provided images, with a few simple commands. http://vagrantup.com Tuesday, 15 May 12
  27. Vagrant We can use vagrant machines for: - Development -

    Testing - Experimenting new/different software/settings - Testing configuration management systems (explained later). http://vagrantup.com Tuesday, 15 May 12
  28. Apprehensions - Won’t it be slow? - Headless Linux VM’s

    aren’t as resource hungry as a Windows virtualized. - I want to use my local graphical editor... - Vagrant automatically creates a shared directory on the host. Tuesday, 15 May 12
  29. Install Vagrant - Install Oracle Virtualbox (free from: http://virtualbox.org) -

    versions for Windows, Mac OS X or Linux. • Plus the extension pack. - Install Vagrant (free from: http://downloads.vagrantup.com) Still available as a Ruby gem but as of the recent v1.0, installers are provided for Windows, Mac and Linux (deb and rpm). $ vagrant -v Vagrant version 1.0.3 Tuesday, 15 May 12
  30. Base Boxes Pre-packaged operating system images. Official base boxes -

    Ubuntu LTS (long term support) releases. Ubuntu 10.04 (Lucid Lynx) $ vagrant box add lucid32 http://files.vagrantup.com/lucid32.box $ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box Ubuntu 12.04 (Precise Pangolin): $ vagrant box add precise32 http://files.vagrantup.com/precise32.box $ vagrant box add precise64 http://files.vagrantup.com/precise64.box Tuesday, 15 May 12
  31. Basic Vagrant Usage $ mkdir vagrant $ cd vagrant $

    mkdir mydev $ cd mydev Create a new config file (VagrantFile) for this instance: $ vagrant init lucid32 Boot the machine (create from base box the 1st time): $ vagrant up SSH in: $ vagrant ssh Tuesday, 15 May 12
  32. Basic Vagrant Usage Vagrant automatically maps /vagrant inside the VM

    to the directory on the host: vagrant@vagrant:~$ ls /vagrant index.html Vagrantfile Ctrl-d exits the SSH session and takes us back to the host. Suspend the machine: $ vagrant suspend $ vagrant resume Shut down the machine: $ vagrant halt Destroy the machine: $ vagrant destroy Tuesday, 15 May 12
  33. Vagrant Configuration The “vagrant init” created a VagrantFile for this

    VM. Vagrant::Config.run do |config| config.vm.box = "lucid32" end We can edit this to add additional options. Tuesday, 15 May 12
  34. Vagrant Configuration GUI or Headless?: Vagrant::Config.run do |config| config.vm.box =

    "lucid32" # Boot with a GUI so you can see the screen. # (Default is headless) # config.vm.boot_mode = :gui end Tuesday, 15 May 12
  35. Vagrant Configuration Port Forwarding: Vagrant::Config.run do |config| config.vm.box = "lucid32"

    # Map port 8080 on the host to port 80 in the VM: config.vm.forward_port 80, 8080 end Tuesday, 15 May 12
  36. Vagrant Configuration Control VirtualBox: Vagrant::Config.run do |config| config.vm.box = "lucid32"

    # Make the VM have 1GB RAM: config.vm.customize ["modifyvm", :id, "--memory", 1024] end Tuesday, 15 May 12
  37. Vagrant Configuration Multiple Machines: Vagrant::Config.run do |config| # Web Server:

    config.vm.define :web do |web_config| web_config.vm.box = "web" web_config.vm.forward_port 80, 8080 end # Database Server: config.vm.define :db do |db_config| db_config.vm.box = "db" db_config.vm.forward_port 3306, 3306 end end Tuesday, 15 May 12
  38. Vagrant Configuration Networking: - Host Only - local communication between

    different VM’s. Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.network :hostonly, “192.168.50.1” end - Bridged - bridge the VM onto the host’s network so we can access anything on the network, and they can access the VM as if it was a normal machine on the network. Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.network :bridged end Tuesday, 15 May 12
  39. Vagrant Configuration Packaging: Use as a starting point, or distribute

    instances. $ vagrant package --vagrantfile Vagrantfile.pkg The --vagrantfile Vagrantfile.pkg option tells Vagrant to take the options placed in Vagrantfile.pkg and add them into the box so things like port forwarding will be done automatically. We can then use this as normal: $ vagrant box add my_box /path/to/the/package.box $ vagrant init my_box $ vagrant up Tuesday, 15 May 12
  40. Vagrant Plugins Vagrant Box.es - Gareth Rushgrove has created a

    site at: http://vagrantbox.es which is a list of links to base boxes that people have built and made public (currently 49 boxes). He has also created a vagrant plugin to search, view and install them: https://github.com/garethr/ruby-vagrantboxes Sahara - allow vagrant to run in a sandbox so any changes can be rolled back: https://github.com/jedi4ever/sahara Vagrant Snap - allow snapshots and supports a multi vm environment: https://github.com/t9md/vagrant-snap Vagrant Hosts - automatically update a dns server with vagrant hostnames: https://github.com/BerlinVagrant/vagrant-dns Vagrant plugin for Jenkins: http://unethicalblogger.com/2012/03/13/introducing-the-vagrant-plugin.html Tuesday, 15 May 12
  41. Vagrant Plugins VeeWee - https://github.com/jedi4ever/veewee Build custom base boxes from

    scratch. Includes templates for popular distributions - CentOS, Debian, Fedora, ArchLinux, Gentoo, openSUSE, Ubuntu and even FreeBSD. Tuesday, 15 May 12
  42. Vagrant Plugins VeeWee - https://github.com/jedi4ever/veewee $ veewee vbox define 'myubuntubox'

    'ubuntu-12.04-server-i386' The basebox 'myubuntubox' has been succesfully created from the template 'ubuntu-12.04-server-i386' You can now edit the definition files stored in definitions/ myubuntubox or build the box with: veewee vbox build 'myubuntubox' $ ls definitions/myubuntubox definition.rb postinstall.sh preseed.cfg Tuesday, 15 May 12
  43. Vagrant Plugins VeeWee - https://github.com/jedi4ever/veewee Here, we can customise the

    build. - definition.rb - contains the parameters for the box: cpu’s, memory, disk, O/S, ISO file and url, boot sequence, SSH details etc. - preseed.cfg - This is the file the Debian/Ubuntu installer uses to automate the installation. - postinstall.sh - This is a shell script which is ran on the VM after the installation. In this case, it will: •Upgrade the O/S (apt-get update && apt-get dist-upgrade) •Install the dependancies for Ruby. •Install (or build from source) Ruby. •Install Puppet and Chef •Create a vagrant user and add in the standard Vagrant SSH public key into ~/.ssh/ authorized_keys. Tuesday, 15 May 12
  44. Vagrant Plugins VeeWee - https://github.com/jedi4ever/veewee We now build the box:

    $ veewee vbox build 'myubuntubox' - Download’s ISO the first time. - Starts a mini web server to serve the preseed.cfg file. - Creates a VirtualBox VM and boots the ISO. - At the boot prompt, it enters a custom boot prompt which points it to the url for the preseed.cfg. - The install complete’s automatically (using the preseed.cfg file) and veewee just waits until it can successfully SSH to the VM. Tuesday, 15 May 12
  45. Vagrant Plugins VeeWee - https://github.com/jedi4ever/veewee Once the build is complete

    (takes some time), we can run some automated Cucumber tests against the box: $ veewee vbox validate 'myubuntubox' ... 7 scenarios (7 passed) 21 steps (21 passed) 0m14.810s This checks the box is working and that Ruby, Ruby Gems, Chef, Puppet etc is responding as intended. If so, we can then build the box and add it into Vagrant: $ vagrant basebox export 'myubuntubox' Tuesday, 15 May 12
  46. Automated Provisioning (Configuration Management) We’ve now got the ability to

    quickly spin up virtual machines as we require them. However, that only gets us a clean install - we’ve still got to start installing packages and configuring things how we like them. If we are looking to share environments with colleagues etc, the problem with pre-packaged boxes/images is that they get out of date so they either have to be periodically updated or extra steps added in whenever they are used. Tuesday, 15 May 12
  47. Automated Provisioning (Configuration Management) This also doesn’t just apply to

    development - what if we have tens or hundreds of production servers? - How can we ensure that every time we setup a new server, all of the correct setup steps are followed? - How can we ensure that the development, staging and production servers are all setup the same. - What if we have a failure and it takes someone hours of running manual commands to build a new server? - what if a whole platform of tens or hundreds of servers fail? (connectivity or business) - How boring is it going to get for someone manually setting up servers? - We could document it, but will it be kept up-to-date? - What about maintaining the boxes on an ongoing basis? - how do we keep them up to date? when someone new starts, how long will it take to go round a lot of servers and add their user and ssh key or to apply a firewall change to all servers? Tuesday, 15 May 12
  48. Automated Provisioning (Configuration Management) This is where something called “Infrastructure

    as code” comes in. What if we were to do our server setup and management as we do development: - Write code (which also acts as documentation) - Put the code into version control so we can: - See and analyse changes, who made them and when. - Have branches and tags. - Easily roll back to old versions. - Run automated tests with Rspec and Cucumber to ensure everything is working correctly without manually checking each server. - TDD - Test Driven Development This ensures that server setup is consistent, reproducible, fast, mistake free and automatable. Tuesday, 15 May 12
  49. Automated Provisioning (Configuration Management) In the past, people would write

    shell scripts to do this. However, this is difficult to manage and maintain. There are now a number of “configuration management” tools written specifically for this purpose. The most popular and widely used of these is Puppet and Chef. Tuesday, 15 May 12
  50. Automated Provisioning (Configuration Management) Puppet: Puppet Labs - founded by

    Luke Kanies in 2005. Uses a declarative language to describe system configuration. Widely used and trusted by a number of large enterprise users - Wikimedia, Dell, Rackspace, Zynga, Twitter, New York Stock Exchange, Disney, Citrix, Oracle, Stanford University, Google and GovUK. James Turnbull (works for Puppet Labs) has written two books about Puppet which are published through Apress: - Pulling strings with Puppet - Pro Puppet -“Puppet 2.7 Cookbook” by John Arundel (Packtpub) Tuesday, 15 May 12
  51. Automated Provisioning (Configuration Management) Chef: Opscode (2009) Recipes written in

    pure Ruby. Opscode provides a “Hosted Chef” service to avoid running a local Chef server (like Github is to Git). Cookbooks - sharable ! ! (although Puppet now has “Puppet Forge”) Knife - feature rich command line tool. Documentation improving: - Wiki - Stephen Nelson Smith, author of “Test Driven Infrastructure with Chef” is working on “Chef: The Definitive Guide” which is due to be published in September. - Podcast: http://www.foodfightshow.org/2012/03/episode-5-getting-started-with- chef.html Tuesday, 15 May 12
  52. Automated Provisioning (Configuration Management) Both products can run in both

    a standalone mode or a client-server model - Vagrant supports both of these. In the client/server model, the client will automatically “phone home” every so often to check for changes. Idempotence: Run multiple times without affecting the result. i.e: - Instead of saying “Install Apache”, we say “Apache should be installed”. - Instead of “chown root.root myfile” we say “myfile should have this owner / group and these permissions. Tuesday, 15 May 12
  53. Automated Provisioning (Configuration Management) Simple Puppet Example: $ mkdir manifests

    In the VagrantFile: config.vm.provision :puppet OR Vagrant::Config.run do |config| config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "my_manifest.pp" end end Tuesday, 15 May 12
  54. Automated Provisioning (Configuration Management) Simple Puppet Example: In manifests/default.pp: class

    apache { package { "apache2": ensure => present, } service { "apache2": ensure => running, require => Package["apache2"], } } include apache Tuesday, 15 May 12
  55. Automated Provisioning (Configuration Management) Simple Puppet Example: file { "/etc/apache/apache.conf":

    owner => root, group => root, mode => 644, source => "puppet:////vagrant/puppet/files/apache.conf", require => Package["apache2"], notify => Service["apache2"] } file { "/etc/apache2/sites-available/vagrantsite": owner => root, group => root, mode => 644, source => "puppet:////vagrant/puppet/files/vhost.conf", require => Package["apache2"], notify => Service["apache2"] } Tuesday, 15 May 12
  56. Automated Provisioning (Configuration Management) Simple Puppet Example: file { "/etc/apache2/sites-enabled/vagrantsite":

    ensure => symlink, target => "/etc/apache2/sites-available/vagrantsite", require => Package["apache2"], notify => Service["apache2"] } file { "/etc/apache2/sites-enabled/default": ensure => absent, require => Package["apache2"], notify => Service["apache2"] } Tuesday, 15 May 12
  57. More Information Vagrant: http://www.vagrantup.com Google Group: http://groups.google.com/group/vagrant-up #vagrant on irc.freenode.net

    Puppet: http://puppetlabs.com #puppet on irc.freenode.net http://www.puppetcookbook.com Chef: http://www.opscode.com #chef on irc.feenode.net Other: http://devopsweekly.com Tuesday, 15 May 12