Boosting teamwork using Vagrant

Talk given at DjangoCon 2012 (http://www.djangocon.us/).

The video: http://pyvideo.org/video/1408/boosting-teamwork-with-vagrant

Transcript

1. BOOSTING TEAMWORK USING VAGRANT Julien Phalip Odopod @julienphalip
2. CONNECTED PERSONAL OBJECTS 5/2012 Hullo, I’m Julien... ‣ Weird French ‣ Full-stack kinda guy ‣ Worked with Django since 2007 ‣ Django core developer since 2011 ‣ Tweet and stuff: @julienphalip
3. CONNECTED PERSONAL OBJECTS 5/2012 www.odopod.com
4. The Problems THAT DEVELOPERS COMMONLY FACE WHILE WORKING IN TEAMS
5. CONNECTED PERSONAL OBJECTS 5/2012 Recurring problems
6. CONNECTED PERSONAL OBJECTS 5/2012 Recurring problems ‣ On-boarding other developers to your project is hard and time- consuming.
7. CONNECTED PERSONAL OBJECTS 5/2012 Recurring problems ‣ On-boarding other developers to your project is hard and time- consuming. ‣ Inconsistencies between team members’ and the production’s environments cause things to unexpectedly break.
8. CONNECTED PERSONAL OBJECTS 5/2012 Lies, damn lies and statistics*
9. CONNECTED PERSONAL OBJECTS 5/2012 Lies, damn lies and statistics* Development platforms for Django developers
10. CONNECTED PERSONAL OBJECTS 5/2012 Lies, damn lies and statistics* * These statistics are 100% made up. They might be true, though. Development platforms for Django developers Mac Non-Mac Unix-like Windows
11. CONNECTED PERSONAL OBJECTS 5/2012 Lies, damn lies and statistics* Production platforms for Django websites * These statistics are 100% made up. They might be true, though. Development platforms for Django developers Mac Non-Mac Unix-like Windows
12. CONNECTED PERSONAL OBJECTS 5/2012 Lies, damn lies and statistics* Production platforms for Django websites * These statistics are 100% made up. They might be true, though. Development platforms for Django developers Mac Non-Mac Unix-like Windows
13. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll
14. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll ‣ Have you ever tried to install a MySQL or PostgreSQL server on a Mac or on Windows?
15. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll ‣ Have you ever tried to install a MySQL or PostgreSQL server on a Mac or on Windows? ‣ Have you ever messed up your development environment and then had to spend hours fixing it?
16. CONNECTED PERSONAL OBJECTS 5/2012
17. CONNECTED PERSONAL OBJECTS 5/2012 Things that make for a bad developer experience
18. CONNECTED PERSONAL OBJECTS 5/2012 Things that make for a bad developer experience ‣ Installing a web stack on Windows/Mac is insane. Stop Doing This To Yourself™. You deserve better!
19. CONNECTED PERSONAL OBJECTS 5/2012 Things that make for a bad developer experience ‣ Installing a web stack on Windows/Mac is insane. Stop Doing This To Yourself™. You deserve better! ‣ Development environments accumulate cruft after working on multiple projects.
20. CONNECTED PERSONAL OBJECTS 5/2012 Things that make for a bad developer experience ‣ Installing a web stack on Windows/Mac is insane. Stop Doing This To Yourself™. You deserve better! ‣ Development environments accumulate cruft after working on multiple projects. ‣ Most developers generally dislike installing & configuring software. They’d rather spend time writing code and building features.
21. The solutions LET’S GET VIRTUAL!
22. CONNECTED PERSONAL OBJECTS 5/2012 Imagine an ideal world where... all developers worked on the same, pre-built platform. the development and production platforms shared the same specs.
23. CONNECTED PERSONAL OBJECTS 5/2012 This world already exists! And it’s called virtualization.
24. CONNECTED PERSONAL OBJECTS 5/2012 The idea was that developers are now working in these transient ‘boxes’ that are portable, move around, etc. No permanent residence. Like a vagrant. MITCHELL HASHIMOTO | CREATOR OF VAGRANT — on IRC “ Meet Vagrant
25. CONNECTED PERSONAL OBJECTS 5/2012 Vagrant is...
26. CONNECTED PERSONAL OBJECTS 5/2012 Vagrant is... Vagrant DEVELOPER VIRTUALBOX ... a developer-friendly interface for Virtualbox ... and for VMWare soon, too.
27. CONNECTED PERSONAL OBJECTS 5/2012 The Vagrant workflow
28. CONNECTED PERSONAL OBJECTS 5/2012 The Vagrant workflow Vagrant DEVELOPER $ vagrant up Vagrantfile
29. CONNECTED PERSONAL OBJECTS 5/2012 The Vagrant workflow Vagrant DEVELOPER VIRTUAL MACHINE VIRTUALBOX $ vagrant up Vagrantfile
30. CONNECTED PERSONAL OBJECTS 5/2012 The Vagrant workflow Vagrant DEVELOPER VIRTUAL MACHINE VIRTUALBOX PROVISIONERS $ vagrant up Vagrantfile
31. CONNECTED PERSONAL OBJECTS 5/2012 The Vagrant workflow Vagrant DEVELOPER VIRTUAL MACHINE VIRTUALBOX PROVISIONERS SSH $ vagrant up Vagrantfile
32. CONNECTED PERSONAL OBJECTS 5/2012 Provisioners CHEF PUPPET SALT STACK GOOD OL’ SHELL
33. CONNECTED PERSONAL OBJECTS 5/2012 Quick examples: Chef (1/2)
34. CONNECTED PERSONAL OBJECTS 5/2012 Quick examples: Chef (1/2) package "libjpeg" do action :install end Installing system packages:
35. CONNECTED PERSONAL OBJECTS 5/2012 Quick examples: Chef (1/2) package "libjpeg" do action :install end Installing system packages: web_app "mysite" do application_name "mysite" template "apache.conf.erb" docroot "/vagrant/" end Configuring Apache:
36. CONNECTED PERSONAL OBJECTS 5/2012 <VirtualHost *:80> Alias /media/ <%= @params[:docroot] %>media/ <Directory "<%= @params[:docroot] %>media/"> Order allow,deny Allow from all </Directory> WSGIScriptAlias / <%= @params[:docroot] %>site.wsgi </VirtualHost> Quick examples: Chef (2/2) Templates:
37. CONNECTED PERSONAL OBJECTS 5/2012 Vagrantfile: Gluing it all together ‣ Base box (e.g. Ubuntu 10.04) ‣ List of folders to be shared between the host and the VM. ‣ List of provisioning scripts to run.
38. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow
39. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox.
40. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox. ‣ Clone project’s code repository, including the Vagrantfile and the provisioning scripts.
41. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox. ‣ Clone project’s code repository, including the Vagrantfile and the provisioning scripts. ‣ Run ‘vagrant up’ to automatically create and provision your own reproducible virtual machine.
42. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox. ‣ Clone project’s code repository, including the Vagrantfile and the provisioning scripts. ‣ Run ‘vagrant up’ to automatically create and provision your own reproducible virtual machine. ‣ Write code using your favorite IDE or editor.
43. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox. ‣ Clone project’s code repository, including the Vagrantfile and the provisioning scripts. ‣ Run ‘vagrant up’ to automatically create and provision your own reproducible virtual machine. ‣ Write code using your favorite IDE or editor. ‣ Let the VM serve the site.
44. CONNECTED PERSONAL OBJECTS 5/2012 New team workflow ‣ Install Vagrant and VirtualBox. ‣ Clone project’s code repository, including the Vagrantfile and the provisioning scripts. ‣ Run ‘vagrant up’ to automatically create and provision your own reproducible virtual machine. ‣ Write code using your favorite IDE or editor. ‣ Let the VM serve the site. ‣ View the site with your favorite browser.
45. CASE STUDY #1: djangoproject.com
46. CONNECTED PERSONAL OBJECTS 5/2012 Requirements (the typicals) ‣ Create virtualenv ‣ Configure Django settings files ‣ Install DB server, create DB, load initial data. ‣ Install and configure web server
47. CONNECTED PERSONAL OBJECTS 5/2012 Requirements (the specifics) ‣ Build Sphinx documentation ‣ Serve 2 sites: www.djangoproject.com docs.djangoproject.com
48. CONNECTED PERSONAL OBJECTS 5/2012 Show the code or it doesn’t exist ‣ https://github.com/jphalip/ djangoproject.com/tree/vagrant
49. CONNECTED PERSONAL OBJECTS 5/2012 Vagrantfile (1/2) Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.box_url = "http://files.vagrantup.com/ lucid32.box" config.vm.network :hostonly, "2.3.4.5" config.vm.share_folder( "v-djangoproject", "/djangoproject", ".", :nfs => true)
50. CONNECTED PERSONAL OBJECTS 5/2012 Vagrantfile (1/2) ... Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.box_url = "http://files.vagrantup.com/ lucid32.box" config.vm.network :hostonly, "2.3.4.5" config.vm.share_folder( "v-djangoproject", "/djangoproject", ".", :nfs => true)
51. CONNECTED PERSONAL OBJECTS 5/2012 Vagrantfile (2/2) config.vm.provision :chef_solo do |chef| chef.cookbooks_path = "provisioning/cookbooks" chef.run_list = [ "recipe[python::pip]", "recipe[python::virtualenv]", "recipe[apache2]", "recipe[apache2::mod_wsgi]", "recipe[git]", "recipe[openssl]", "recipe[memcached]", "recipe[postgresql::client]", "recipe[postgresql::server]", "recipe[djangoproject]"] end
52. CONNECTED PERSONAL OBJECTS 5/2012 Create virtualenv python_virtualenv "/home/vagrant/.virtualenvs/djangoproject" do interpreter "python2.6" owner "vagrant" action :create not_if "test -d /home/vagrant/.virtualenvs/djangoproject" end bash "Initial loading of virtualenv requirements" do user "vagrant" code <<-EOH source /home/vagrant/.virtualenvs/djangoproject/bin/activate cd /djangoproject pip install -r deploy-requirements.txt pip install -r local-requirements.txt EOH not_if "test -d /home/vagrant/.virtualenvs/djangoproject/lib/ python2.6/site-packages/django" end
53. CONNECTED PERSONAL OBJECTS 5/2012 Create database bash "Create database" do user "vagrant" code <<-EOH # Create the user and database echo "CREATE USER djangoproject WITH SUPERUSER PASSWORD 'secret';" | sudo -u postgres psql sudo -u postgres createdb -O djangoproject djangoproject # Load initial data dump sudo -u postgres psql -d djangoproject < /djangoproject/ provisioning/initial.sql EOH not_if "sudo -u postgres psql -l | grep djangoproject" end
54. CONNECTED PERSONAL OBJECTS 5/2012 Build the Sphinx docs bash "Clone Django" do user "vagrant" code <<-EOH sudo mkdir /django cd /django git clone http://github.com/django/django.git . EOH not_if "test -d /django" end bash "Build docs" do user "vagrant" code <<-EOH cd /django/docs source /home/vagrant/.virtualenvs/djangoproject/bin/activate make json EOH not_if "test -d /django/docs/_build/json" end
55. CONNECTED PERSONAL OBJECTS 5/2012 Create local settings files (1/2) template "/djangoproject/local_settings.py" do source "local_settings.py.erb" variables( :base_settings => "django_website.settings.www", :developer_name => configuration['developer_name'], :developer_email => configuration['developer_email']) not_if "test -e /djangoproject/local_settings.py" end template "/djangoproject/local_settings_docs.py" do source "local_settings.py.erb" variables( :base_settings => "django_website.settings.docs", :developer_name => configuration['developer_name'], :developer_email => configuration['developer_email']) not_if "test -e /djangoproject/local_settings_docs.py" end
56. CONNECTED PERSONAL OBJECTS 5/2012 Create local settings files (2/2) from <%= @base_settings %> import * DEBUG = True DATABASES['default']['PASSWORD'] = 'secret' DATABASES['default']['HOST'] = 'localhost' ADMINS = ( ('<%= @developer_name %>', '<%= @developer_email %>'), ) MANAGERS = ADMINS DEFAULT_FROM_EMAIL = '<%= @developer_email %>' local_settings.py.erb
57. CONNECTED PERSONAL OBJECTS 5/2012 Create WSGI files (1/2) template "/djangoproject/site-docs.wsgi" do source "site.wsgi.erb" variables(:settings_module => "local_settings_docs") end template "/djangoproject/site-www.wsgi" do source "site.wsgi.erb" variables(:settings_module => "local_settings") end
58. CONNECTED PERSONAL OBJECTS 5/2012 Create WSGI files (2/2) import sys, os, site if not os.path.dirname(__file__) in sys.path[:1]: sys.path.insert(0, os.path.dirname(__file__)) os.environ['DJANGO_SETTINGS_MODULE'] = '<%= @settings_module %>' site.addsitedir('/home/vagrant/.virtualenvs/djangoproject/lib/ python2.6/site-packages') from django.core.handlers.wsgi import WSGIHandler application = WSGIHandler() site.wsgi.erb
59. CONNECTED PERSONAL OBJECTS 5/2012 Configure Apache (1/3) web_app "djangoproject" do application_name djangoproject template "apache.conf.erb" docroot "/djangoproject" end service "apache2" do action :restart end
60. CONNECTED PERSONAL OBJECTS 5/2012 Configure Apache (2/3) <VirtualHost *:80> Alias /admin_media/ /home/vagrant/.virtualenvs/djangoproject/ lib/python2.6/site-packages/django/contrib/admin/media/ <Directory /home/vagrant/.virtualenvs/djangoproject/site- packages/django/contrib/admin/media> Order allow,deny Allow from all </Directory> Alias /media/ <%= @params[:docroot] %>/media/ <Directory "<%= @params[:docroot] %>/media/"> AllowOverride all Order allow,deny Allow from all </Directory> WSGIScriptAlias / /djangoproject/site-www.wsgi </VirtualHost> apache.conf.erb
61. CONNECTED PERSONAL OBJECTS 5/2012 Configure Apache (2/3) Listen 8080 <VirtualHost *:8080> Alias /media/ <%= @params[:docroot] %>/media/ <Directory "<%= @params[:docroot] %>/media/"> AllowOverride all Order allow,deny Allow from all </Directory> WSGIScriptAlias / <%= @params[:docroot] %>/site-docs.wsgi </VirtualHost> apache.conf.erb (c’ed)
62. CONNECTED PERSONAL OBJECTS 5/2012 Demo...
63. CONNECTED PERSONAL OBJECTS 5/2012 http:/ /2.3.4.5/ (Main site)
64. CONNECTED PERSONAL OBJECTS 5/2012 http:/ /2.3.4.5:8080/ (Documentation site)
65. CASE STUDY #2: Django test suite
66. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll
67. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll ‣ Have you ever run the Django core test suite with SQLite? Easy peasy.
68. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll ‣ Have you ever run the Django core test suite with SQLite? Easy peasy. ‣ And with MySQL? PostgreSQL? Not easy, but not too hard.
69. CONNECTED PERSONAL OBJECTS 5/2012 Quick poll ‣ Have you ever run the Django core test suite with SQLite? Easy peasy. ‣ And with MySQL? PostgreSQL? Not easy, but not too hard. ‣ How about the GeoDjango tests? Hrmm...
70. CONNECTED PERSONAL OBJECTS 5/2012
71. CONNECTED PERSONAL OBJECTS 5/2012 Meet the djangocore-box ‣ https://github.com/jphalip/djangocore-box ‣ All supported versions of Python: 2.4, 2.5, 2.6, 2.7, 3.2. ‣ All supported DB backends: SQLite, SpatiaLite, MySQL, PostgreSQL and PostGIS (soon Oracle). ‣ GeoDjango dependencies: GEOS, PROJ.4, GDAL. ‣ Take 1 hour to build from scratch. Or download the 1 GB pre-built image.
72. CONNECTED PERSONAL OBJECTS 5/2012 Demo...
73. CONNECTED PERSONAL OBJECTS 5/2012 Try it out at the sprints! USB Vagrant VirtualBox djangocore-box
74. CONNECTED PERSONAL OBJECTS 5/2012 In closing
75. CONNECTED PERSONAL OBJECTS 5/2012 In closing ‣ There are some overheads:
76. CONNECTED PERSONAL OBJECTS 5/2012 In closing ‣ There are some overheads: ‣ Memory usage
77. CONNECTED PERSONAL OBJECTS 5/2012 In closing ‣ There are some overheads: ‣ Memory usage ‣ Learning a provisioner
78. CONNECTED PERSONAL OBJECTS 5/2012 In closing ‣ There are some overheads: ‣ Memory usage ‣ Learning a provisioner ‣ But they are largely compensated by the benefits.
79. CONNECTED PERSONAL OBJECTS 5/2012 In closing ‣ There are some overheads: ‣ Memory usage ‣ Learning a provisioner ‣ But they are largely compensated by the benefits. ‣ Vagrant is awesome. Try it out!
80. Thank you! Reach me at: @julienphalip http:/ /odopod.com http:/ /julienphalip.com