Tweet
Tweet

Slide 1

Slide 1 text

Developing “large” Scale Web Applications Matt Robenolt // github.com/mattrobenolt OR @mattrobenolt s/developing/deploying/

Slide 2

Slide 2 text

Hi.

Slide 3

Slide 3 text

What this is not

Slide 4

Slide 4 text

+ What this is not This is not a definitive guide to solving for x. This is not a solution to your problems. All problems are unique and must be approached pragmatically.

Slide 5

Slide 5 text

What this is

Slide 6

Slide 6 text

+ What this is This is an exploration of potentially new tools. This is an introduction to new ideas and concepts. This is the start of a discussion. This is what worked for us.

Slide 7

Slide 7 text

What do I mean by “large”?

Slide 8

Slide 8 text

+ What do I mean by “Large”? Lots and lots of Moving pieces Django, Node.js, Redis, SSL, virtual hosts, MongoDB, CDN, etc.

Slide 9

Slide 9 text

+ What do I mean by “Large”? Life Beyond python manage.py runserver http://localhost:8000/

Slide 10

Slide 10 text

The Story of Drund

Slide 11

Slide 11 text

+ Story of Drund Standard, out-of-the-box Django project First ~6 months

Slide 12

Slide 12 text

+ Story of Drund Things were... GREAT!

Slide 13

Slide 13 text

+ Story of Drund Developing locally ./manage.py runserver

Slide 14

Slide 14 text

+ Story of Drund Deploying to production ssh matt@… -- svn up; touch /tmp/reload;

Slide 15

Slide 15 text

+ Story of Drund Our stack Today

Slide 16

Slide 16 text

+ Story of Drund Our stack Today Django, MySQL, Redis, Node.js, MongoDB, RabbitMQ, CDN, SSL-only, LESS, Require.js, Hogan.js, background worker, etc.

Slide 17

Slide 17 text

+ Story of Drund Development is a bitch.

Slide 18

Slide 18 text

+ Story of Drund Deploying is a bitch.

Slide 19

Slide 19 text

What’s the problem?

Slide 20

Slide 20 text

+ What’s the problem? Django Nothing wrong with Django per se. Conditioned us for no external dependencies. Makes getting up and started super simple. SQLite3, Dummy cache, local memory cache, etc.

Slide 21

Slide 21 text

+ What’s the problem? Node.js An external process, so needs another port. Requires two Terminal windows to spawn Django + Node.js app. Cookies aren’t shared across different ports, must match domain:port combo.

Slide 22

Slide 22 text

+ What’s the problem? Redis, MongoDB, RabbitMQ Can’t be mocked in Django realistically like MySQL <-> SQLite. Annoying to run on local machine, mismatched configuration/versions, blah blah.

Slide 23

Slide 23 text

+ What’s the problem? Less, Require.js, Hogan.js Assets must be compiled before being loaded into the browser. Tedious tasks during rapid development.

Slide 24

Slide 24 text

+ What’s the problem? CDN …wut? Introduces occasional cross-domain issues. File uploads.

Slide 25

Slide 25 text

+ What’s the problem? SSL Django can’t handle SSL. Real web server required.

Slide 26

Slide 26 text

+ What’s the problem? Virtual hosts /etc/hosts becomes unwieldy. No wildcards.

Slide 27

Slide 27 text

+ What’s the problem? 127.0.0.1 drund.local 127.0.0.1 www.drund.local 127.0.0.1 api.drund.local 127.0.0.1 admin.drund.local 127.0.0.1 team.drund.local 127.0.0.1 partners.drund.local 127.0.0.1 matt.drund.local 127.0.0.1 nick.drund.local 127.0.0.1 kevin.drund.local 127.0.0.1 mike.drund.local 127.0.0.1 adam.drund.local 127.0.0.1 lee.drund.local

Slide 28

Slide 28 text

+ What’s the problem? Configuration Management Django uses settings.py. Node.js uses settings.json. Both represent very similar data. Database, cache, etc.

Slide 29

Slide 29 text

+ What’s the problem? Deployments git pull with Fabric. Every server in the cluster has different definitions of “update”. System dependencies, libxml, mysql-client, etc.

Slide 30

Slide 30 text

Re-architect ALL THE THINGS!

Slide 31

Slide 31 text

Virtual Hosts

Slide 32

Slide 32 text

+ Virtual Hosts Forget about /etc/hosts. You can point a real DNS entry to 127.0.0.1! Real DNS gives you wildcard and CNAME entries.

Slide 33

Slide 33 text

+ Virtual Hosts local.drund.net. A 127.0.0.1 *.local.drund.net. CNAME local.drund.net See Also: http://xip.io

Slide 34

Slide 34 text

Make & The Makefile

Slide 35

Slide 35 text

+ Make & The Makefile Not just for “Real” compiling. It’s pretty much just /bin/sh. Simplify long or tedious commands. You’ll use it for a lot of things, I promise.

Slide 36

Slide 36 text

+ Make & The Makefile Constructed of “Targets”. Targets can be real or not real. 98%* of your targets will be “PHONY”. Targets are essentially commands executed through `make`. make * Not a real statistic.

Slide 37

Slide 37 text

+ Make & The Makefile Targets support dependencies. “static depends on css and js first” targets == functions or commands static: @rm -rf ./static ./manage.py collectstatic --noinput -v0 clean: find . -name "*.pyc" -delete

Slide 38

Slide 38 text

+ Make & The Makefile How we compile our static assets before:

Slide 39

Slide 39 text

+ Make & The Makefile ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:

Slide 40

Slide 40 text

+ Make & The Makefile FFuuuuuu!!! ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:

Slide 41

Slide 41 text

+ Make & The Makefile How we compile our assets with make:

Slide 42

Slide 42 text

+ Make & The Makefile make static How we compile our assets with make:

Slide 43

Slide 43 text

+ Make & The Makefile make static How we compile our assets with make:

Slide 44

Slide 44 text

all: css js templates static css: @../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > \ ./media/main/css/kovu.css @echo "css compiled." js: templates @node ../tools/r.js/dist/r.js -o kovu.build.js @echo "js compiled " templates: @../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} \ $(shell find ./media -name *.mustache) @echo "Templates smushed successfully." static: css js templates @rm -rf ./static ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 + Make & The Makefile Makefile

Slide 45

Slide 45 text

+ Make & The Makefile Why not Fabric? Fabric is amazing for Python projects. We use it. Not the best for non-python. Makefiles are more generic. Choose what works for you, where and when it makes sense. * bitprophet: I do love Fabric. ;)

Slide 46

Slide 46 text

m4

Slide 47

Slide 47 text

+ M4 Macro processor. Very “Dumb”. Basic string replacements. Small macros. Inheritance. Allows us to build configuration files for both Django and Node.js without duplication. Allows us to build configuration files for different environments.

Slide 48

Slide 48 text

+ M4 C-ish define statements. define(`__NAME__', `Matt') Hello __NAME__

Slide 49

Slide 49 text

+ M4 Odd syntax. Surprisingly does a lot.

Slide 50

Slide 50 text

+ m4 define(`__DEBUG__', `True') define(`__DATABASE_ENGINE__', `mysql') define(`__DATABASE_NAME__', `drund') define(`__DATABASE_USER__', `root') define(`__DATABASE_PASSWORD__', `root') define(`__DATABASE_HOST__', `127.0.0.1') define(`__DATABASE_PORT__', `3306') m4/base.m4 define(`__DEBUG__', `False') define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `$uper$ecret!') define(`__DATABASE_HOST__', `10.0.0.0') m4/production.m4 define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `heywereonstaging!') m4/staging.m4

Slide 51

Slide 51 text

+ m4 DEBUG = __DEBUG__ DATABASES = { 'default': { 'ENGINE': 'django.db.backends.__DATABASE_ENGINE__', 'NAME': '__DATABASE_NAME__', 'USER': '__DATABASE_USER__', 'PASSWORD': '__DATABASE_PASSWORD__', 'HOST': '__DATABASE_HOST__', 'PORT': __DATABASE_PORT__', } } m4/settings.py.m4 { "database": { "name": "__DATABASE_NAME__", "user": "__DATABASE_USER__", "password": "__DATABASE_PASSWORD__", "host": "__DATABASE_HOST__", "port": __DATABASE_PORT__ } } m4/settings.json.m4

Slide 52

Slide 52 text

+ m4 m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json settings: m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json Makefile Build it!

Slide 53

Slide 53 text

Vagrant http://vagrantup.com

Slide 54

Slide 54 text

+ Vagrant “Virtualized development made easily.” Headless, local virtual machine. port forwarding. Easy bootstrapping with Chef, Puppet, or Shell script. vagrant up Runs our “Utilities” without cluttering local machine. (MySQL, Redis, RabbitMQ, MongoDB)

Slide 55

Slide 55 text

+ Vagrant Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box" config.vm.forward_port 3306, 3306 # MySQL config.vm.forward_port 6379, 6379 # Redis config.vm.forward_port 5672, 5672 # RabbitMQ config.vm.forward_port 27017, 27017 # MongoDB config.vm.provision :shell, :path => "bootstrap.sh" end Vagrantfile

Slide 56

Slide 56 text

+ Vagrant Downloads “box” for first use. Creates Virtual Machine using VirtualBox. Forwards necessary ports, so I can access via 127.0.0.1: Executes bootstrap.sh after build. vagrant ssh # SSH into the server vagrant halt # Shut down gracefully vagrant destroy # Delete the VM entirely

Slide 57

Slide 57 text

+ Vagrant #!/bin/sh # First, let's check if everything is installed before we go installing shit again if [ -f /usr/sbin/mysqld ]; then \ exit 0 fi # MySQL root password echo mysql-server-5.1 mysql-server/root_password password root | debconf-set-selections echo mysql-server-5.1 mysql-server/root_password_again password root | debconf-set-selections apt-get update && apt-get install mysql-server -y echo "grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;"| mysql -u root -proot echo "Creating \"drund\" database..." echo "create database drund;" | mysql -u root -proot echo "Binding MySQL to 0.0.0.0..." cp /etc/mysql/my.cnf /etc/mysql/my.cnf.old sed -e's/127.0.0.1/0.0.0.0/' /etc/mysql/my.cnf.old > /etc/mysql/my.cnf echo "Restarting MySQL..." service mysql restart exit 0 bootstrap.sh

Slide 58

Slide 58 text

+ Vagrant vm: vagrant up ./manage.py syncdb --migrate --noinput Makefile

Slide 59

Slide 59 text

Almost... but not quite good enough.

Slide 60

Slide 60 text

+ Almost... but not quite good enough. # Only run once make settings make vm # Get the server running make static ./manage.py runserver # Listening @ http://www.local.drund.net:8000/ Current status

Slide 61

Slide 61 text

+ Almost... but not quite good enough. What’s missing? make static needs to be run every time a file is changed. Node.js is still running on another port. No SSL. No way to simulate a CDN.

Slide 62

Slide 62 text

Node-HTTP-Proxy & “Node-testserver” https://github.com/nodejitsu/node-http-proxy

Slide 63

Slide 63 text

+ Node-http-proxy & “node-testserver” Node.js... wut? Stand in as a dummy/fancier nginx. Proxy virtual host requests to different apps/ports. Pipe stdout/stderr like expected to console. Render Less and Mustache templates on the fly. “CDN” is just another virtual host. SSL.

Slide 64

Slide 64 text

+ Node-http-proxy & “node-testserver” in a nutshell http-proxy spawns server on 443, terminates SSL, and processes vhosts as expected. node0.local.drund.net -> localhost:9001 # A miscellaneous Node.js app *.local.drund.net -> localhost:8000 # Main Django app testcdn.local.azork.com -> localhost:9002 # Internal Express media server

Slide 65

Slide 65 text

+ Node-http-proxy & “node-testserver” in a nutshell Requests to internal media server are resolved through Django’s findstatic command. ./manage.py findstatic --first LESS files are compiled into CSS when requested. Mustache templates are rendered on the fly. main.css -> main.less hello.template.js -> hello.template.mustache

Slide 66

Slide 66 text

+ Node-http-proxy & “node-testserver” Added Bonus! Django is no longer serving up static media. Load assets concurrently. It actually makes a difference.

Slide 67

Slide 67 text

sudo make testserver https://www.local.drund.net/

Slide 68

Slide 68 text

Jenkins http://jenkins-ci.org/

Slide 69

Slide 69 text

+ Jenkins Continuous Integration & Continuous Deployment Nirvana

Slide 70

Slide 70 text

+ Jenkins Post-commit hooks to run builds automatically. Executes a series of Shell commands. Builds new virtualenv for each build. Runs tests. Fail if tests fail. Notify team. Build deb package for source code. Push package to private apt repository. apt-get install drund-{component}-{branch} -y on servers with Fabric. Integrates nicely with IRC.

Slide 71

Slide 71 text

Deb packaging

Slide 72

Slide 72 text

+ Deb packaging Why a system package? Originally skeptical myself. Very easily manage dependencies outside of Python/pip. Post-install hooks. No compiling on production servers.

Slide 73

Slide 73 text

+ Deb packaging Drund is broken up into many packages by code branch. Packages can be mixed and matched depending on roles. drund-src-master # Main Drund source code. Only. drund-www-master # Depends on drund-src-master, nginx config drund-worker-master # Depends on drund-src-master, starts background worker drund-activity-master # Node.js app drund-proxy-master # Node.js app drund-cron-master # Depends on drund-src-master, cron.d files.

Slide 74

Slide 74 text

+ Deb packaging drund-src-master Virtual package for drund-src-r{build}. Essentially: drund-srcdepends on drund-src-r{build} drund-src-r100 and drund-src-r101 can be installed at the same time. post install manages a symlink. 100% new environment for each build. Easy to roll back. Just repoint symlink. No processes are spawned.

Slide 75

Slide 75 text

+ Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer: Drund Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:

Slide 76

Slide 76 text

+ Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer: Drund Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:

Slide 77

Slide 77 text

+ Deb packaging Install via Apt-get apt-get install drund-src-master

Slide 78

Slide 78 text

Continuous Integration/deployments

Slide 79

Slide 79 text

+ Continuous Integration/Deployments Tests are run automatically before anything is deployed. Tests are run after every single push. It’s actually impossible to deploy code with broken tests. Jenkins can be configured to run a Fabric command to update after a successful build. It’s just another shell command. Annoying at first. You will make mistakes.

Slide 80

Slide 80 text

+ Continuous Integration/Deployments Forced to write better code. Forced to write more tests. Once your environment is configured, everything is done correctly. Every time. Computers are pretty good at that. Easy to bring up staging or development servers.

Slide 81

Slide 81 text

New & improved workflow

Slide 82

Slide 82 text

+ New & Improved Workflow master branch is 100% as stable as possible. Production only. Code is only merged in from develop. develop contains code ready to be merged into master at any time. Continuously deployed to dev/staging servers. Feature branches off of develop. If bad code is pushed to develop, Jenkins complains and posts in IRC. If all is good, code review, queue up for merge into master.

Slide 83

Slide 83 text

+ New & Improved Workflow Hit the ground running in very little time. git clone git@... make init # Grabs submodules, create virtualenv, installs requirements

Slide 84

Slide 84 text

+ New & Improved Workflow Run the project. sudo make testserver # sudo is needed for port 443.

Slide 85

Slide 85 text

+ New & Improved Workflow Deploy code to staging. git checkout develop git merge feature-branch # Or Pull Request git push origin develop

Slide 86

Slide 86 text

+ New & Improved Workflow Deploy to production. git checkout master git merge develop # Preferably a Pull Request git push origin master

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

Thanks. @Mattrobenolt github.com/mattrobenolt mattrobenolt.com