Boosting teamwork using Vagrant

Talk given at DjangoCon 2012 (http://www.djangocon.us/).

The video: http://pyvideo.org/video/1408/boosting-teamwork-with-vagrant

Transcript

1. BOOSTING TEAMWORK
   USING VAGRANT
   Julien Phalip
   Odopod
   @julienphalip
   

   View Slide

2. CONNECTED PERSONAL OBJECTS
   5/2012
   Hullo, I’m Julien...
   ‣ Weird French
   ‣ Full-stack kinda guy
   ‣ Worked with Django since 2007
   ‣ Django core developer since 2011
   ‣ Tweet and stuff: @julienphalip
   

   View Slide

3. CONNECTED PERSONAL OBJECTS
   5/2012
   www.odopod.com
   

   View Slide

4. The Problems
   THAT DEVELOPERS COMMONLY FACE
   WHILE WORKING IN TEAMS
   

   View Slide

5. CONNECTED PERSONAL OBJECTS
   5/2012
   Recurring problems
   

   View Slide

6. CONNECTED PERSONAL OBJECTS
   5/2012
   Recurring problems
   ‣ On-boarding other developers to
   your project is hard and time-
   consuming.
   

   View Slide

7. CONNECTED PERSONAL OBJECTS
   5/2012
   Recurring problems
   ‣ On-boarding other developers to
   your project is hard and time-
   consuming.
   ‣ Inconsistencies between team
   members’ and the production’s
   environments cause things to
   unexpectedly break.
   

   View Slide

8. CONNECTED PERSONAL OBJECTS
   5/2012
   Lies, damn lies and statistics*
   

   View Slide

9. CONNECTED PERSONAL OBJECTS
   5/2012
   Lies, damn lies and statistics*
   Development
   platforms for Django
   developers
   

   View Slide

10. CONNECTED PERSONAL OBJECTS
    5/2012
    Lies, damn lies and statistics*
    * These statistics are 100% made up. They might be true, though.
    Development
    platforms for Django
    developers
    Mac
    Non-Mac Unix-like
    Windows
    

    View Slide

11. CONNECTED PERSONAL OBJECTS
    5/2012
    Lies, damn lies and statistics*
    Production
    platforms for Django
    websites
    * These statistics are 100% made up. They might be true, though.
    Development
    platforms for Django
    developers
    Mac
    Non-Mac Unix-like
    Windows
    

    View Slide

12. CONNECTED PERSONAL OBJECTS
    5/2012
    Lies, damn lies and statistics*
    Production
    platforms for Django
    websites
    * These statistics are 100% made up. They might be true, though.
    Development
    platforms for Django
    developers
    Mac
    Non-Mac Unix-like
    Windows
    

    View Slide

13. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    

    View Slide

14. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    ‣ Have you ever tried to install a
    MySQL or PostgreSQL server on a
    Mac or on Windows?
    

    View Slide

15. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    ‣ Have you ever tried to install a
    MySQL or PostgreSQL server on a
    Mac or on Windows?
    ‣ Have you ever messed up your
    development environment and
    then had to spend hours fixing it?
    

    View Slide

16. CONNECTED PERSONAL OBJECTS
    5/2012
    

    View Slide

17. CONNECTED PERSONAL OBJECTS
    5/2012
    Things that make for a bad
    developer experience
    

    View Slide

18. CONNECTED PERSONAL OBJECTS
    5/2012
    Things that make for a bad
    developer experience
    ‣ Installing a web stack on Windows/Mac is
    insane. Stop Doing This To Yourself™. You
    deserve better!
    

    View Slide

19. CONNECTED PERSONAL OBJECTS
    5/2012
    Things that make for a bad
    developer experience
    ‣ Installing a web stack on Windows/Mac is
    insane. Stop Doing This To Yourself™. You
    deserve better!
    ‣ Development environments accumulate
    cruft after working on multiple projects.
    

    View Slide

20. CONNECTED PERSONAL OBJECTS
    5/2012
    Things that make for a bad
    developer experience
    ‣ Installing a web stack on Windows/Mac is
    insane. Stop Doing This To Yourself™. You
    deserve better!
    ‣ Development environments accumulate
    cruft after working on multiple projects.
    ‣ Most developers generally dislike installing
    & configuring software. They’d rather spend
    time writing code and building features.
    

    View Slide

21. The solutions
    LET’S GET VIRTUAL!
    

    View Slide

22. CONNECTED PERSONAL OBJECTS
    5/2012
    Imagine an ideal world where...
    all developers
    worked on the
    same, pre-built
    platform.
    the development
    and production
    platforms shared
    the same specs.
    

    View Slide

23. CONNECTED PERSONAL OBJECTS
    5/2012
    This world already exists!
    And it’s called virtualization.
    

    View Slide

24. CONNECTED PERSONAL OBJECTS
    5/2012
    The idea was that developers are
    now working in these transient
    ‘boxes’ that are portable, move
    around, etc. No permanent
    residence. Like a vagrant.
    MITCHELL HASHIMOTO | CREATOR OF VAGRANT — on IRC
    “
    Meet Vagrant
    

    View Slide

25. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrant is...
    

    View Slide

26. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrant is...
    Vagrant
    DEVELOPER VIRTUALBOX
    ... a developer-friendly
    interface for Virtualbox
    ... and for VMWare soon, too.
    

    View Slide

27. CONNECTED PERSONAL OBJECTS
    5/2012
    The Vagrant workflow
    

    View Slide

28. CONNECTED PERSONAL OBJECTS
    5/2012
    The Vagrant workflow
    Vagrant
    DEVELOPER
    $ vagrant up
    Vagrantfile
    

    View Slide

29. CONNECTED PERSONAL OBJECTS
    5/2012
    The Vagrant workflow
    Vagrant
    DEVELOPER
    VIRTUAL MACHINE
    VIRTUALBOX
    $ vagrant up
    Vagrantfile
    

    View Slide

30. CONNECTED PERSONAL OBJECTS
    5/2012
    The Vagrant workflow
    Vagrant
    DEVELOPER
    VIRTUAL MACHINE
    VIRTUALBOX
    PROVISIONERS
    $ vagrant up
    Vagrantfile
    

    View Slide

31. CONNECTED PERSONAL OBJECTS
    5/2012
    The Vagrant workflow
    Vagrant
    DEVELOPER
    VIRTUAL MACHINE
    VIRTUALBOX
    PROVISIONERS
    SSH
    $ vagrant up
    Vagrantfile
    

    View Slide

32. CONNECTED PERSONAL OBJECTS
    5/2012
    Provisioners
    CHEF PUPPET SALT
    STACK
    GOOD OL’
    SHELL
    

    View Slide

33. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick examples: Chef (1/2)
    

    View Slide

34. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick examples: Chef (1/2)
    package "libjpeg" do
    action :install
    end
    Installing system packages:
    

    View Slide

35. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick examples: Chef (1/2)
    package "libjpeg" do
    action :install
    end
    Installing system packages:
    web_app "mysite" do
    application_name "mysite"
    template "apache.conf.erb"
    docroot "/vagrant/"
    end
    Configuring Apache:
    

    View Slide

36. CONNECTED PERSONAL OBJECTS
    5/2012
    
    Alias /media/ <%= @params[:docroot] %>media/
    media/">
    Order allow,deny
    Allow from all
    
    WSGIScriptAlias / <%= @params[:docroot] %>site.wsgi
    
    Quick examples: Chef (2/2)
    Templates:
    

    View Slide

37. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrantfile: Gluing it all together
    ‣ Base box (e.g. Ubuntu 10.04)
    ‣ List of folders to be shared between
    the host and the VM.
    ‣ List of provisioning scripts to run.
    

    View Slide

38. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    

    View Slide

39. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    

    View Slide

40. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    ‣ Clone project’s code repository, including
    the Vagrantfile and the provisioning scripts.
    

    View Slide

41. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    ‣ Clone project’s code repository, including
    the Vagrantfile and the provisioning scripts.
    ‣ Run ‘vagrant up’ to automatically create
    and provision your own reproducible virtual
    machine.
    

    View Slide

42. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    ‣ Clone project’s code repository, including
    the Vagrantfile and the provisioning scripts.
    ‣ Run ‘vagrant up’ to automatically create
    and provision your own reproducible virtual
    machine.
    ‣ Write code using your favorite IDE or editor.
    

    View Slide

43. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    ‣ Clone project’s code repository, including
    the Vagrantfile and the provisioning scripts.
    ‣ Run ‘vagrant up’ to automatically create
    and provision your own reproducible virtual
    machine.
    ‣ Write code using your favorite IDE or editor.
    ‣ Let the VM serve the site.
    

    View Slide

44. CONNECTED PERSONAL OBJECTS
    5/2012
    New team workflow
    ‣ Install Vagrant and VirtualBox.
    ‣ Clone project’s code repository, including
    the Vagrantfile and the provisioning scripts.
    ‣ Run ‘vagrant up’ to automatically create
    and provision your own reproducible virtual
    machine.
    ‣ Write code using your favorite IDE or editor.
    ‣ Let the VM serve the site.
    ‣ View the site with your favorite browser.
    

    View Slide

45. CASE STUDY #1:
    djangoproject.com
    

    View Slide

46. CONNECTED PERSONAL OBJECTS
    5/2012
    Requirements (the typicals)
    ‣ Create virtualenv
    ‣ Configure Django settings files
    ‣ Install DB server, create DB, load
    initial data.
    ‣ Install and configure web server
    

    View Slide

47. CONNECTED PERSONAL OBJECTS
    5/2012
    Requirements (the specifics)
    ‣ Build Sphinx documentation
    ‣ Serve 2 sites:
    www.djangoproject.com
    docs.djangoproject.com
    

    View Slide

48. CONNECTED PERSONAL OBJECTS
    5/2012
    Show the code or it doesn’t exist
    ‣ https://github.com/jphalip/
    djangoproject.com/tree/vagrant
    

    View Slide

49. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrantfile (1/2)
    Vagrant::Config.run do |config|
    config.vm.box = "lucid32"
    config.vm.box_url = "http://files.vagrantup.com/
    lucid32.box"
    config.vm.network :hostonly, "2.3.4.5"
    config.vm.share_folder(
    "v-djangoproject", "/djangoproject", ".", :nfs => true)
    

    View Slide

50. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrantfile (1/2)
    ...
    Vagrant::Config.run do |config|
    config.vm.box = "lucid32"
    config.vm.box_url = "http://files.vagrantup.com/
    lucid32.box"
    config.vm.network :hostonly, "2.3.4.5"
    config.vm.share_folder(
    "v-djangoproject", "/djangoproject", ".", :nfs => true)
    

    View Slide

51. CONNECTED PERSONAL OBJECTS
    5/2012
    Vagrantfile (2/2)
    config.vm.provision :chef_solo do |chef|
    chef.cookbooks_path = "provisioning/cookbooks"
    chef.run_list = [
    "recipe[python::pip]",
    "recipe[python::virtualenv]",
    "recipe[apache2]",
    "recipe[apache2::mod_wsgi]",
    "recipe[git]",
    "recipe[openssl]",
    "recipe[memcached]",
    "recipe[postgresql::client]",
    "recipe[postgresql::server]",
    "recipe[djangoproject]"]
    end
    

    View Slide

52. CONNECTED PERSONAL OBJECTS
    5/2012
    Create virtualenv
    python_virtualenv "/home/vagrant/.virtualenvs/djangoproject" do
    interpreter "python2.6"
    owner "vagrant"
    action :create
    not_if "test -d /home/vagrant/.virtualenvs/djangoproject"
    end
    bash "Initial loading of virtualenv requirements" do
    user "vagrant"
    code <<-EOH
    source /home/vagrant/.virtualenvs/djangoproject/bin/activate
    cd /djangoproject
    pip install -r deploy-requirements.txt
    pip install -r local-requirements.txt
    EOH
    not_if "test -d /home/vagrant/.virtualenvs/djangoproject/lib/
    python2.6/site-packages/django"
    end
    

    View Slide

53. CONNECTED PERSONAL OBJECTS
    5/2012
    Create database
    bash "Create database" do
    user "vagrant"
    code <<-EOH
    # Create the user and database
    echo "CREATE USER djangoproject WITH SUPERUSER PASSWORD
    'secret';" | sudo -u postgres psql
    sudo -u postgres createdb -O djangoproject djangoproject
    # Load initial data dump
    sudo -u postgres psql -d djangoproject < /djangoproject/
    provisioning/initial.sql
    EOH
    not_if "sudo -u postgres psql -l | grep djangoproject"
    end
    

    View Slide

54. CONNECTED PERSONAL OBJECTS
    5/2012
    Build the Sphinx docs
    bash "Clone Django" do
    user "vagrant"
    code <<-EOH
    sudo mkdir /django
    cd /django
    git clone http://github.com/django/django.git .
    EOH
    not_if "test -d /django"
    end
    bash "Build docs" do
    user "vagrant"
    code <<-EOH
    cd /django/docs
    source /home/vagrant/.virtualenvs/djangoproject/bin/activate
    make json
    EOH
    not_if "test -d /django/docs/_build/json"
    end
    

    View Slide

55. CONNECTED PERSONAL OBJECTS
    5/2012
    Create local settings files (1/2)
    template "/djangoproject/local_settings.py" do
    source "local_settings.py.erb"
    variables(
    :base_settings => "django_website.settings.www",
    :developer_name => configuration['developer_name'],
    :developer_email => configuration['developer_email'])
    not_if "test -e /djangoproject/local_settings.py"
    end
    template "/djangoproject/local_settings_docs.py" do
    source "local_settings.py.erb"
    variables(
    :base_settings => "django_website.settings.docs",
    :developer_name => configuration['developer_name'],
    :developer_email => configuration['developer_email'])
    not_if "test -e /djangoproject/local_settings_docs.py"
    end
    

    View Slide

56. CONNECTED PERSONAL OBJECTS
    5/2012
    Create local settings files (2/2)
    from <%= @base_settings %> import *
    DEBUG = True
    DATABASES['default']['PASSWORD'] = 'secret'
    DATABASES['default']['HOST'] = 'localhost'
    ADMINS = (
    ('<%= @developer_name %>', '<%= @developer_email %>'),
    )
    MANAGERS = ADMINS
    DEFAULT_FROM_EMAIL = '<%= @developer_email %>'
    local_settings.py.erb
    

    View Slide

57. CONNECTED PERSONAL OBJECTS
    5/2012
    Create WSGI files (1/2)
    template "/djangoproject/site-docs.wsgi" do
    source "site.wsgi.erb"
    variables(:settings_module => "local_settings_docs")
    end
    template "/djangoproject/site-www.wsgi" do
    source "site.wsgi.erb"
    variables(:settings_module => "local_settings")
    end
    

    View Slide

58. CONNECTED PERSONAL OBJECTS
    5/2012
    Create WSGI files (2/2)
    import sys, os, site
    if not os.path.dirname(__file__) in sys.path[:1]:
    sys.path.insert(0, os.path.dirname(__file__))
    os.environ['DJANGO_SETTINGS_MODULE'] = '<%= @settings_module %>'
    site.addsitedir('/home/vagrant/.virtualenvs/djangoproject/lib/
    python2.6/site-packages')
    from django.core.handlers.wsgi import WSGIHandler
    application = WSGIHandler()
    site.wsgi.erb
    

    View Slide

59. CONNECTED PERSONAL OBJECTS
    5/2012
    Configure Apache (1/3)
    web_app "djangoproject" do
    application_name djangoproject
    template "apache.conf.erb"
    docroot "/djangoproject"
    end
    service "apache2" do
    action :restart
    end
    

    View Slide

60. CONNECTED PERSONAL OBJECTS
    5/2012
    Configure Apache (2/3)
    
    Alias /admin_media/ /home/vagrant/.virtualenvs/djangoproject/
    lib/python2.6/site-packages/django/contrib/admin/media/
    packages/django/contrib/admin/media>
    Order allow,deny
    Allow from all
    
    Alias /media/ <%= @params[:docroot] %>/media/
    /media/">
    AllowOverride all
    Order allow,deny
    Allow from all
    
    WSGIScriptAlias / /djangoproject/site-www.wsgi
    
    apache.conf.erb
    

    View Slide

61. CONNECTED PERSONAL OBJECTS
    5/2012
    Configure Apache (2/3)
    Listen 8080
    
    Alias /media/ <%= @params[:docroot] %>/media/
    /media/">
    AllowOverride all
    Order allow,deny
    Allow from all
    
    WSGIScriptAlias / <%= @params[:docroot] %>/site-docs.wsgi
    
    apache.conf.erb (c’ed)
    

    View Slide

62. CONNECTED PERSONAL OBJECTS
    5/2012
    Demo...
    

    View Slide

63. CONNECTED PERSONAL OBJECTS
    5/2012
    http:/
    /2.3.4.5/
    (Main site)
    

    View Slide

64. CONNECTED PERSONAL OBJECTS
    5/2012
    http:/
    /2.3.4.5:8080/
    (Documentation site)
    

    View Slide

65. CASE STUDY #2:
    Django test suite
    

    View Slide

66. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    

    View Slide

67. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    ‣ Have you ever run the Django core
    test suite with SQLite? Easy peasy.
    

    View Slide

68. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    ‣ Have you ever run the Django core
    test suite with SQLite? Easy peasy.
    ‣ And with MySQL? PostgreSQL?
    Not easy, but not too hard.
    

    View Slide

69. CONNECTED PERSONAL OBJECTS
    5/2012
    Quick poll
    ‣ Have you ever run the Django core
    test suite with SQLite? Easy peasy.
    ‣ And with MySQL? PostgreSQL?
    Not easy, but not too hard.
    ‣ How about the GeoDjango tests?
    Hrmm...
    

    View Slide

70. CONNECTED PERSONAL OBJECTS
    5/2012
    

    View Slide

71. CONNECTED PERSONAL OBJECTS
    5/2012
    Meet the djangocore-box
    ‣ https://github.com/jphalip/djangocore-box
    ‣ All supported versions of Python: 2.4, 2.5,
    2.6, 2.7, 3.2.
    ‣ All supported DB backends: SQLite,
    SpatiaLite, MySQL, PostgreSQL and PostGIS
    (soon Oracle).
    ‣ GeoDjango dependencies: GEOS, PROJ.4,
    GDAL.
    ‣ Take 1 hour to build from scratch. Or
    download the 1 GB pre-built image.
    

    View Slide

72. CONNECTED PERSONAL OBJECTS
    5/2012
    Demo...
    

    View Slide

73. CONNECTED PERSONAL OBJECTS
    5/2012
    Try it out at the sprints!
    USB
    Vagrant
    VirtualBox
    djangocore-box
    

    View Slide

74. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    

    View Slide

75. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    ‣ There are some overheads:
    

    View Slide

76. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    ‣ There are some overheads:
    ‣ Memory usage
    

    View Slide

77. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    ‣ There are some overheads:
    ‣ Memory usage
    ‣ Learning a provisioner
    

    View Slide

78. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    ‣ There are some overheads:
    ‣ Memory usage
    ‣ Learning a provisioner
    ‣ But they are largely compensated
    by the benefits.
    

    View Slide

79. CONNECTED PERSONAL OBJECTS
    5/2012
    In closing
    ‣ There are some overheads:
    ‣ Memory usage
    ‣ Learning a provisioner
    ‣ But they are largely compensated
    by the benefits.
    ‣ Vagrant is awesome. Try it out!
    

    View Slide

80. Thank you!
    Reach me at:
    @julienphalip
    http:/
    /odopod.com
    http:/
    /julienphalip.com
    

    View Slide