Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developing & Deploying "Large" Scale Web Applications

Developing & Deploying "Large" Scale Web Applications

June 12th, 2012 @ Urban Airship

Matt Robenolt

June 12, 2012
Tweet

More Decks by Matt Robenolt

Other Decks in Programming

Transcript

  1. Developing “large” Scale Web Applications
    Matt Robenolt // github.com/mattrobenolt OR @mattrobenolt
    s/developing/deploying/

    View Slide

  2. Hi.

    View Slide

  3. What this is not

    View Slide

  4. + What this is not
    This is not a definitive guide to solving for x.
    This is not a solution to your problems.
    All problems are unique and must be approached pragmatically.

    View Slide

  5. What this is

    View Slide

  6. + What this is
    This is an exploration of potentially new tools.
    This is an introduction to new ideas and concepts.
    This is the start of a discussion.
    This is what worked for us.

    View Slide

  7. What do I mean by “large”?

    View Slide

  8. + What do I mean by “Large”?
    Lots and lots of Moving pieces
    Django, Node.js, Redis, SSL, virtual hosts, MongoDB, CDN, etc.

    View Slide

  9. + What do I mean by “Large”?
    Life Beyond
    python manage.py runserver
    http://localhost:8000/

    View Slide

  10. The Story of Drund

    View Slide

  11. + Story of Drund
    Standard, out-of-the-box Django project
    First ~6 months

    View Slide

  12. + Story of Drund
    Things were...
    GREAT!

    View Slide

  13. + Story of Drund
    Developing locally
    ./manage.py runserver

    View Slide

  14. + Story of Drund
    Deploying to production
    ssh [email protected]… -- svn up; touch /tmp/reload;

    View Slide

  15. + Story of Drund
    Our stack Today

    View Slide

  16. + Story of Drund
    Our stack Today
    Django, MySQL, Redis, Node.js, MongoDB, RabbitMQ, CDN, SSL-only,
    LESS, Require.js, Hogan.js, background worker, etc.

    View Slide

  17. + Story of Drund
    Development
    is a bitch.

    View Slide

  18. + Story of Drund
    Deploying
    is a bitch.

    View Slide

  19. What’s the problem?

    View Slide

  20. + What’s the problem?
    Django
    Nothing wrong with Django per se.
    Conditioned us for no external dependencies.
    Makes getting up and started super simple.
    SQLite3, Dummy cache, local memory cache, etc.

    View Slide

  21. + What’s the problem?
    Node.js
    An external process, so needs another port.
    Requires two Terminal windows to spawn Django + Node.js app.
    Cookies aren’t shared across different ports, must match domain:port combo.

    View Slide

  22. + What’s the problem?
    Redis, MongoDB, RabbitMQ
    Can’t be mocked in Django realistically like MySQL SQLite.
    Annoying to run on local machine, mismatched configuration/versions, blah blah.

    View Slide

  23. + What’s the problem?
    Less, Require.js, Hogan.js
    Assets must be compiled before being loaded into the browser.
    Tedious tasks during rapid development.

    View Slide

  24. + What’s the problem?
    CDN
    …wut?
    Introduces occasional cross-domain issues. File uploads.

    View Slide

  25. + What’s the problem?
    SSL
    Django can’t handle SSL. Real web server required.

    View Slide

  26. + What’s the problem?
    Virtual hosts
    /etc/hosts becomes unwieldy.
    No wildcards.

    View Slide

  27. + What’s the problem?
    127.0.0.1 drund.local
    127.0.0.1 www.drund.local
    127.0.0.1 api.drund.local
    127.0.0.1 admin.drund.local
    127.0.0.1 team.drund.local
    127.0.0.1 partners.drund.local
    127.0.0.1 matt.drund.local
    127.0.0.1 nick.drund.local
    127.0.0.1 kevin.drund.local
    127.0.0.1 mike.drund.local
    127.0.0.1 adam.drund.local
    127.0.0.1 lee.drund.local

    View Slide

  28. + What’s the problem?
    Configuration Management
    Django uses settings.py.
    Node.js uses settings.json.
    Both represent very similar data. Database, cache, etc.

    View Slide

  29. + What’s the problem?
    Deployments
    git pull with Fabric.
    Every server in the cluster has different definitions of “update”.
    System dependencies, libxml, mysql-client, etc.

    View Slide

  30. Re-architect
    ALL THE THINGS!

    View Slide

  31. Virtual Hosts

    View Slide

  32. + Virtual Hosts
    Forget about /etc/hosts.
    You can point a real DNS entry to 127.0.0.1!
    Real DNS gives you wildcard and CNAME entries.

    View Slide

  33. + Virtual Hosts
    local.drund.net. A 127.0.0.1
    *.local.drund.net. CNAME local.drund.net
    See Also: http://xip.io

    View Slide

  34. Make & The Makefile

    View Slide

  35. + Make & The Makefile
    Not just for “Real” compiling.
    It’s pretty much just /bin/sh.
    Simplify long or tedious commands.
    You’ll use it for a lot of things, I promise.

    View Slide

  36. + Make & The Makefile
    Constructed of “Targets”.
    Targets can be real or not real.
    98%* of your targets will be “PHONY”.
    Targets are essentially commands executed
    through `make`.
    make
    * Not a real statistic.

    View Slide

  37. + Make & The Makefile
    Targets support dependencies.
    “static depends on css and js first”
    targets == functions or commands
    static:
    @rm -rf ./static
    ./manage.py collectstatic --noinput -v0
    clean:
    find . -name "*.pyc" -delete

    View Slide

  38. + Make & The Makefile
    How we compile our static assets before:

    View Slide

  39. + Make & The Makefile
    ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css
    ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache)
    ../tools/r.js/dist/r.js -o kovu.build.js
    ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0
    How we compile our static assets before:

    View Slide

  40. + Make & The Makefile
    FFuuuuuu!!!
    ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css
    ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache)
    ../tools/r.js/dist/r.js -o kovu.build.js
    ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0
    How we compile our static assets before:

    View Slide

  41. + Make & The Makefile
    How we compile our assets with make:

    View Slide

  42. + Make & The Makefile
    make static
    How we compile our assets with make:

    View Slide

  43. + Make & The Makefile
    make static
    How we compile our assets with make:

    View Slide

  44. all: css js templates static
    css:
    @../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > \
    ./media/main/css/kovu.css
    @echo "css compiled."
    js: templates
    @node ../tools/r.js/dist/r.js -o kovu.build.js
    @echo "js compiled "
    templates:
    @../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} \
    $(shell find ./media -name *.mustache)
    @echo "Templates smushed successfully."
    static: css js templates
    @rm -rf ./static
    ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0
    + Make & The Makefile
    Makefile

    View Slide

  45. + Make & The Makefile
    Why not Fabric?
    Fabric is amazing for Python projects. We use it.
    Not the best for non-python.
    Makefiles are more generic.
    Choose what works for you, where and when it
    makes sense.
    * bitprophet: I do love Fabric. ;)

    View Slide

  46. m4

    View Slide

  47. + M4
    Macro processor.
    Very “Dumb”.
    Basic string replacements.
    Small macros.
    Inheritance.
    Allows us to build configuration files for
    both Django and Node.js without duplication.
    Allows us to build configuration files for
    different environments.

    View Slide

  48. + M4
    C-ish define statements.
    define(`__NAME__', `Matt')
    Hello __NAME__

    View Slide

  49. + M4
    Odd syntax.
    Surprisingly does a lot.

    View Slide

  50. + m4
    define(`__DEBUG__', `True')
    define(`__DATABASE_ENGINE__', `mysql')
    define(`__DATABASE_NAME__', `drund')
    define(`__DATABASE_USER__', `root')
    define(`__DATABASE_PASSWORD__', `root')
    define(`__DATABASE_HOST__', `127.0.0.1')
    define(`__DATABASE_PORT__', `3306')
    m4/base.m4
    define(`__DEBUG__', `False')
    define(`__DATABASE_USER__', `drund')
    define(`__DATABASE_PASSWORD__', `$uper$ecret!')
    define(`__DATABASE_HOST__', `10.0.0.0')
    m4/production.m4
    define(`__DATABASE_USER__', `drund')
    define(`__DATABASE_PASSWORD__', `heywereonstaging!')
    m4/staging.m4

    View Slide

  51. + m4
    DEBUG = __DEBUG__
    DATABASES = {
    'default': {
    'ENGINE': 'django.db.backends.__DATABASE_ENGINE__',
    'NAME': '__DATABASE_NAME__',
    'USER': '__DATABASE_USER__',
    'PASSWORD': '__DATABASE_PASSWORD__',
    'HOST': '__DATABASE_HOST__',
    'PORT': __DATABASE_PORT__',
    }
    }
    m4/settings.py.m4
    {
    "database": {
    "name": "__DATABASE_NAME__",
    "user": "__DATABASE_USER__",
    "password": "__DATABASE_PASSWORD__",
    "host": "__DATABASE_HOST__",
    "port": __DATABASE_PORT__
    }
    }
    m4/settings.json.m4

    View Slide

  52. + m4
    m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py
    m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json
    settings:
    m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py
    m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json
    Makefile
    Build it!

    View Slide

  53. Vagrant
    http://vagrantup.com

    View Slide

  54. + Vagrant
    “Virtualized development made easily.”
    Headless, local virtual machine.
    port forwarding.
    Easy bootstrapping with Chef, Puppet, or Shell script.
    vagrant up
    Runs our “Utilities” without cluttering local machine.
    (MySQL, Redis, RabbitMQ, MongoDB)

    View Slide

  55. + Vagrant
    Vagrant::Config.run do |config|
    config.vm.box = "precise64"
    config.vm.box_url = "http://files.vagrantup.com/precise64.box"
    config.vm.forward_port 3306, 3306 # MySQL
    config.vm.forward_port 6379, 6379 # Redis
    config.vm.forward_port 5672, 5672 # RabbitMQ
    config.vm.forward_port 27017, 27017 # MongoDB
    config.vm.provision :shell, :path => "bootstrap.sh"
    end
    Vagrantfile

    View Slide

  56. + Vagrant
    Downloads “box” for first use.
    Creates Virtual Machine using VirtualBox.
    Forwards necessary ports, so I can access via 127.0.0.1:
    Executes bootstrap.sh after build.
    vagrant ssh # SSH into the server
    vagrant halt # Shut down gracefully
    vagrant destroy # Delete the VM entirely

    View Slide

  57. + Vagrant
    #!/bin/sh
    # First, let's check if everything is installed before we go installing shit again
    if [ -f /usr/sbin/mysqld ]; then \
    exit 0
    fi
    # MySQL root password
    echo mysql-server-5.1 mysql-server/root_password password root | debconf-set-selections
    echo mysql-server-5.1 mysql-server/root_password_again password root | debconf-set-selections
    apt-get update && apt-get install mysql-server -y
    echo "grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;"| mysql -u root -proot
    echo "Creating \"drund\" database..."
    echo "create database drund;" | mysql -u root -proot
    echo "Binding MySQL to 0.0.0.0..."
    cp /etc/mysql/my.cnf /etc/mysql/my.cnf.old
    sed -e's/127.0.0.1/0.0.0.0/' /etc/mysql/my.cnf.old > /etc/mysql/my.cnf
    echo "Restarting MySQL..."
    service mysql restart
    exit 0
    bootstrap.sh

    View Slide

  58. + Vagrant
    vm:
    vagrant up
    ./manage.py syncdb --migrate --noinput
    Makefile

    View Slide

  59. Almost... but not quite good enough.

    View Slide

  60. + Almost... but not quite good enough.
    # Only run once
    make settings
    make vm
    # Get the server running
    make static
    ./manage.py runserver
    # Listening @ http://www.local.drund.net:8000/
    Current status

    View Slide

  61. + Almost... but not quite good enough.
    What’s missing?
    make static needs to be run every time a file is changed.
    Node.js is still running on another port.
    No SSL.
    No way to simulate a CDN.

    View Slide

  62. Node-HTTP-Proxy & “Node-testserver”
    https://github.com/nodejitsu/node-http-proxy

    View Slide

  63. + Node-http-proxy & “node-testserver”
    Node.js... wut?
    Stand in as a dummy/fancier nginx.
    Proxy virtual host requests to different apps/ports.
    Pipe stdout/stderr like expected to console.
    Render Less and Mustache templates on the fly.
    “CDN” is just another virtual host.
    SSL.

    View Slide

  64. + Node-http-proxy & “node-testserver”
    in a nutshell
    http-proxy spawns server on 443, terminates SSL, and
    processes vhosts as expected.
    node0.local.drund.net -> localhost:9001 # A miscellaneous Node.js app
    *.local.drund.net -> localhost:8000 # Main Django app
    testcdn.local.azork.com -> localhost:9002 # Internal Express media server

    View Slide

  65. + Node-http-proxy & “node-testserver”
    in a nutshell
    Requests to internal media server are resolved through Django’s
    findstatic command. ./manage.py findstatic --first
    LESS files are compiled into CSS when requested.
    Mustache templates are rendered on the fly.
    main.css -> main.less
    hello.template.js -> hello.template.mustache

    View Slide

  66. + Node-http-proxy & “node-testserver”
    Added Bonus!
    Django is no longer serving up static media.
    Load assets concurrently. It actually makes a difference.

    View Slide

  67. sudo make testserver
    https://www.local.drund.net/

    View Slide

  68. Jenkins
    http://jenkins-ci.org/

    View Slide

  69. + Jenkins
    Continuous Integration & Continuous Deployment Nirvana

    View Slide

  70. + Jenkins
    Post-commit hooks to run builds automatically.
    Executes a series of Shell commands.
    Builds new virtualenv for each build.
    Runs tests.
    Fail if tests fail. Notify team.
    Build deb package for source code.
    Push package to private apt repository.
    apt-get install drund-{component}-{branch} -y on servers with Fabric.
    Integrates nicely with IRC.

    View Slide

  71. Deb packaging

    View Slide

  72. + Deb packaging
    Why a system package?
    Originally skeptical myself.
    Very easily manage dependencies outside of Python/pip.
    Post-install hooks.
    No compiling on production servers.

    View Slide

  73. + Deb packaging
    Drund is broken up into many packages by code branch.
    Packages can be mixed and matched depending on roles.
    drund-src-master # Main Drund source code. Only.
    drund-www-master # Depends on drund-src-master, nginx config
    drund-worker-master # Depends on drund-src-master, starts background worker
    drund-activity-master # Node.js app
    drund-proxy-master # Node.js app
    drund-cron-master # Depends on drund-src-master, cron.d files.

    View Slide

  74. + Deb packaging
    drund-src-master
    Virtual package for drund-src-r{build}.
    Essentially: drund-srcdepends on drund-src-r{build}
    drund-src-r100 and drund-src-r101 can be installed at the same time.
    post install manages a symlink.
    100% new environment for each build.
    Easy to roll back. Just repoint symlink.
    No processes are spawned.

    View Slide

  75. + Deb packaging
    Source: drund-src-master-r100
    Section: web
    Priority: optional
    Maintainer: Drund
    Build-Depends: debhelper (>= 5)
    Standards-Version: 3.8.4
    Homepage: https://www.drund.com
    Package: drund-src-kovu
    Architecture: all
    Depends: drund-src-master-r100
    Package: drund-src-master-r100
    Architecture: all
    Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ...
    Drund-src control file:

    View Slide

  76. + Deb packaging
    Source: drund-src-master-r100
    Section: web
    Priority: optional
    Maintainer: Drund
    Build-Depends: debhelper (>= 5)
    Standards-Version: 3.8.4
    Homepage: https://www.drund.com
    Package: drund-src-kovu
    Architecture: all
    Depends: drund-src-master-r100
    Package: drund-src-master-r100
    Architecture: all
    Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ...
    Drund-src control file:

    View Slide

  77. + Deb packaging
    Install via Apt-get
    apt-get install drund-src-master

    View Slide

  78. Continuous Integration/deployments

    View Slide

  79. + Continuous Integration/Deployments
    Tests are run automatically before anything is deployed.
    Tests are run after every single push.
    It’s actually impossible to deploy code with broken tests.
    Jenkins can be configured to run a Fabric command to update
    after a successful build. It’s just another shell command.
    Annoying at first.
    You will make mistakes.

    View Slide

  80. + Continuous Integration/Deployments
    Forced to write better code.
    Forced to write more tests.
    Once your environment is configured, everything is done
    correctly. Every time. Computers are pretty good at that.
    Easy to bring up staging or development servers.

    View Slide

  81. New & improved workflow

    View Slide

  82. + New & Improved Workflow
    master branch is 100% as stable as possible. Production only. Code is
    only merged in from develop.
    develop contains code ready to be merged into master at any time.
    Continuously deployed to dev/staging servers.
    Feature branches off of develop.
    If bad code is pushed to develop, Jenkins complains and posts in IRC.
    If all is good, code review, queue up for merge into master.

    View Slide

  83. + New & Improved Workflow
    Hit the ground running in very little time.
    git clone [email protected]
    make init # Grabs submodules, create virtualenv, installs requirements

    View Slide

  84. + New & Improved Workflow
    Run the project.
    sudo make testserver # sudo is needed for port 443.

    View Slide

  85. + New & Improved Workflow
    Deploy code to staging.
    git checkout develop
    git merge feature-branch # Or Pull Request
    git push origin develop

    View Slide

  86. + New & Improved Workflow
    Deploy to production.
    git checkout master
    git merge develop # Preferably a Pull Request
    git push origin master

    View Slide

  87. View Slide

  88. Thanks.
    @Mattrobenolt
    github.com/mattrobenolt
    mattrobenolt.com

    View Slide