Developing & Deploying "Large" Scale Web Applications

Developing & Deploying "Large" Scale Web Applications

June 12th, 2012 @ Urban Airship


Matt Robenolt

June 12, 2012


  1. 2.


  2. 4.

    + What this is not This is not a definitive

    guide to solving for x. This is not a solution to your problems. All problems are unique and must be approached pragmatically.
  3. 6.

    + What this is This is an exploration of potentially

    new tools. This is an introduction to new ideas and concepts. This is the start of a discussion. This is what worked for us.
  4. 8.

    + What do I mean by “Large”? Lots and lots

    of Moving pieces Django, Node.js, Redis, SSL, virtual hosts, MongoDB, CDN, etc.
  5. 9.

    + What do I mean by “Large”? Life Beyond python runserver http://localhost:8000/
  6. 16.

    + Story of Drund Our stack Today Django, MySQL, Redis,

    Node.js, MongoDB, RabbitMQ, CDN, SSL-only, LESS, Require.js, Hogan.js, background worker, etc.
  7. 20.

    + What’s the problem? Django Nothing wrong with Django per

    se. Conditioned us for no external dependencies. Makes getting up and started super simple. SQLite3, Dummy cache, local memory cache, etc.
  8. 21.

    + What’s the problem? Node.js An external process, so needs

    another port. Requires two Terminal windows to spawn Django + Node.js app. Cookies aren’t shared across different ports, must match domain:port combo.
  9. 22.

    + What’s the problem? Redis, MongoDB, RabbitMQ Can’t be mocked

    in Django realistically like MySQL <-> SQLite. Annoying to run on local machine, mismatched configuration/versions, blah blah.
  10. 23.

    + What’s the problem? Less, Require.js, Hogan.js Assets must be

    compiled before being loaded into the browser. Tedious tasks during rapid development.
  11. 27.

    + What’s the problem? drund.local www.drund.local api.drund.local admin.drund.local team.drund.local partners.drund.local matt.drund.local nick.drund.local kevin.drund.local mike.drund.local adam.drund.local lee.drund.local
  12. 28.

    + What’s the problem? Configuration Management Django uses Node.js

    uses settings.json. Both represent very similar data. Database, cache, etc.
  13. 29.

    + What’s the problem? Deployments git pull with Fabric. Every

    server in the cluster has different definitions of “update”. System dependencies, libxml, mysql-client, etc.
  14. 32.

    + Virtual Hosts Forget about /etc/hosts. You can point a

    real DNS entry to! Real DNS gives you wildcard and CNAME entries.
  15. 35.

    + Make & The Makefile Not just for “Real” compiling.

    It’s pretty much just /bin/sh. Simplify long or tedious commands. You’ll use it for a lot of things, I promise.
  16. 36.

    + Make & The Makefile Constructed of “Targets”. Targets can

    be real or not real. 98%* of your targets will be “PHONY”. Targets are essentially commands executed through `make`. make <target> * Not a real statistic.
  17. 37.

    + Make & The Makefile Targets support dependencies. “static depends

    on css and js first” targets == functions or commands static: @rm -rf ./static ./ collectstatic --noinput -v0 clean: find . -name "*.pyc" -delete
  18. 39.

    + Make & The Makefile ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less >

    ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o ./ collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:
  19. 40.

    + Make & The Makefile FFuuuuuu!!! ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less

    > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o ./ collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:
  20. 44.

    all: css js templates static css: @../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less

    > \ ./media/main/css/kovu.css @echo "css compiled." js: templates @node ../tools/r.js/dist/r.js -o @echo "js compiled <>" templates: @../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} \ $(shell find ./media -name *.mustache) @echo "Templates smushed successfully." static: css js templates @rm -rf ./static ./ collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 + Make & The Makefile Makefile
  21. 45.

    + Make & The Makefile Why not Fabric? Fabric is

    amazing for Python projects. We use it. Not the best for non-python. Makefiles are more generic. Choose what works for you, where and when it makes sense. * bitprophet: I do love Fabric. ;)
  22. 46.


  23. 47.

    + M4 Macro processor. Very “Dumb”. Basic string replacements. Small

    macros. Inheritance. Allows us to build configuration files for both Django and Node.js without duplication. Allows us to build configuration files for different environments.
  24. 50.

    + m4 define(`__DEBUG__', `True') define(`__DATABASE_ENGINE__', `mysql') define(`__DATABASE_NAME__', `drund') define(`__DATABASE_USER__', `root')

    define(`__DATABASE_PASSWORD__', `root') define(`__DATABASE_HOST__', `') define(`__DATABASE_PORT__', `3306') m4/base.m4 define(`__DEBUG__', `False') define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `$uper$ecret!') define(`__DATABASE_HOST__', `') m4/production.m4 define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `heywereonstaging!') m4/staging.m4
  25. 51.

    + m4 DEBUG = __DEBUG__ DATABASES = { 'default': {

    'ENGINE': 'django.db.backends.__DATABASE_ENGINE__', 'NAME': '__DATABASE_NAME__', 'USER': '__DATABASE_USER__', 'PASSWORD': '__DATABASE_PASSWORD__', 'HOST': '__DATABASE_HOST__', 'PORT': __DATABASE_PORT__', } } m4/ { "database": { "name": "__DATABASE_NAME__", "user": "__DATABASE_USER__", "password": "__DATABASE_PASSWORD__", "host": "__DATABASE_HOST__", "port": __DATABASE_PORT__ } } m4/settings.json.m4
  26. 52.

    + m4 m4 m4/base.m4 m4/production.m4 m4/ > m4 m4/base.m4

    m4/production.m4 m4/settings.json.m4 > settings.json settings: m4 m4/base.m4 m4/production.m4 m4/ > m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json Makefile Build it!
  27. 54.

    + Vagrant “Virtualized development made easily.” Headless, local virtual machine.

    port forwarding. Easy bootstrapping with Chef, Puppet, or Shell script. vagrant up Runs our “Utilities” without cluttering local machine. (MySQL, Redis, RabbitMQ, MongoDB)
  28. 55.

    + Vagrant do |config| = "precise64" config.vm.box_url =

    "" config.vm.forward_port 3306, 3306 # MySQL config.vm.forward_port 6379, 6379 # Redis config.vm.forward_port 5672, 5672 # RabbitMQ config.vm.forward_port 27017, 27017 # MongoDB config.vm.provision :shell, :path => "" end Vagrantfile
  29. 56.

    + Vagrant Downloads “box” for first use. Creates Virtual Machine

    using VirtualBox. Forwards necessary ports, so I can access via<port> Executes after build. vagrant ssh # SSH into the server vagrant halt # Shut down gracefully vagrant destroy # Delete the VM entirely
  30. 57.

    + Vagrant #!/bin/sh # First, let's check if everything is

    installed before we go installing shit again if [ -f /usr/sbin/mysqld ]; then \ exit 0 fi # MySQL root password echo mysql-server-5.1 mysql-server/root_password password root | debconf-set-selections echo mysql-server-5.1 mysql-server/root_password_again password root | debconf-set-selections apt-get update && apt-get install mysql-server -y echo "grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;"| mysql -u root -proot echo "Creating \"drund\" database..." echo "create database drund;" | mysql -u root -proot echo "Binding MySQL to" cp /etc/mysql/my.cnf /etc/mysql/my.cnf.old sed -e's/' /etc/mysql/my.cnf.old > /etc/mysql/my.cnf echo "Restarting MySQL..." service mysql restart exit 0
  31. 60.

    + Almost... but not quite good enough. # Only run

    once make settings make vm # Get the server running make static ./ runserver # Listening @ Current status
  32. 61.

    + Almost... but not quite good enough. What’s missing? make

    static needs to be run every time a file is changed. Node.js is still running on another port. No SSL. No way to simulate a CDN.
  33. 63.

    + Node-http-proxy & “node-testserver” Node.js... wut? Stand in as a

    dummy/fancier nginx. Proxy virtual host requests to different apps/ports. Pipe stdout/stderr like expected to console. Render Less and Mustache templates on the fly. “CDN” is just another virtual host. SSL.
  34. 64.

    + Node-http-proxy & “node-testserver” in a nutshell http-proxy spawns server

    on 443, terminates SSL, and processes vhosts as expected. -> localhost:9001 # A miscellaneous Node.js app * -> localhost:8000 # Main Django app -> localhost:9002 # Internal Express media server
  35. 65.

    + Node-http-proxy & “node-testserver” in a nutshell Requests to internal

    media server are resolved through Django’s findstatic command. ./ findstatic --first <path> LESS files are compiled into CSS when requested. Mustache templates are rendered on the fly. main.css -> main.less hello.template.js -> hello.template.mustache
  36. 66.

    + Node-http-proxy & “node-testserver” Added Bonus! Django is no longer

    serving up static media. Load assets concurrently. It actually makes a difference.
  37. 70.

    + Jenkins Post-commit hooks to run builds automatically. Executes a

    series of Shell commands. Builds new virtualenv for each build. Runs tests. Fail if tests fail. Notify team. Build deb package for source code. Push package to private apt repository. apt-get install drund-{component}-{branch} -y on servers with Fabric. Integrates nicely with IRC.
  38. 72.

    + Deb packaging Why a system package? Originally skeptical myself.

    Very easily manage dependencies outside of Python/pip. Post-install hooks. No compiling on production servers.
  39. 73.

    + Deb packaging Drund is broken up into many packages

    by code branch. Packages can be mixed and matched depending on roles. drund-src-master # Main Drund source code. Only. drund-www-master # Depends on drund-src-master, nginx config drund-worker-master # Depends on drund-src-master, starts background worker drund-activity-master # Node.js app drund-proxy-master # Node.js app drund-cron-master # Depends on drund-src-master, cron.d files.
  40. 74.

    + Deb packaging drund-src-master Virtual package for drund-src-r{build}. Essentially: drund-srcdepends

    on drund-src-r{build} drund-src-r100 and drund-src-r101 can be installed at the same time. post install manages a symlink. 100% new environment for each build. Easy to roll back. Just repoint symlink. No processes are spawned.
  41. 75.

    + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer:

    Drund <> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:
  42. 76.

    + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer:

    Drund <> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:
  43. 79.

    + Continuous Integration/Deployments Tests are run automatically before anything is

    deployed. Tests are run after every single push. It’s actually impossible to deploy code with broken tests. Jenkins can be configured to run a Fabric command to update after a successful build. It’s just another shell command. Annoying at first. You will make mistakes.
  44. 80.

    + Continuous Integration/Deployments Forced to write better code. Forced to

    write more tests. Once your environment is configured, everything is done correctly. Every time. Computers are pretty good at that. Easy to bring up staging or development servers.
  45. 82.

    + New & Improved Workflow master branch is 100% as

    stable as possible. Production only. Code is only merged in from develop. develop contains code ready to be merged into master at any time. Continuously deployed to dev/staging servers. Feature branches off of develop. If bad code is pushed to develop, Jenkins complains and posts in IRC. If all is good, code review, queue up for merge into master.
  46. 83.

    + New & Improved Workflow Hit the ground running in

    very little time. git clone git@... make init # Grabs submodules, create virtualenv, installs requirements
  47. 84.

    + New & Improved Workflow Run the project. sudo make

    testserver # sudo is needed for port 443.
  48. 85.

    + New & Improved Workflow Deploy code to staging. git

    checkout develop git merge feature-branch # Or Pull Request git push origin develop
  49. 86.

    + New & Improved Workflow Deploy to production. git checkout

    master git merge develop # Preferably a Pull Request git push origin master
  50. 87.