Developing & Deploying "Large" Scale Web Applications

Transcript

1. Developing “large” Scale Web Applications Matt Robenolt // github.com/mattrobenolt OR

   @mattrobenolt s/developing/deploying/

2. Hi.

3. What this is not

4. + What this is not This is not a definitive

   guide to solving for x. This is not a solution to your problems. All problems are unique and must be approached pragmatically.

5. What this is

6. + What this is This is an exploration of potentially

   new tools. This is an introduction to new ideas and concepts. This is the start of a discussion. This is what worked for us.

7. What do I mean by “large”?

8. + What do I mean by “Large”? Lots and lots

   of Moving pieces Django, Node.js, Redis, SSL, virtual hosts, MongoDB, CDN, etc.

9. + What do I mean by “Large”? Life Beyond python

   manage.py runserver http://localhost:8000/

10. The Story of Drund

11. + Story of Drund Standard, out-of-the-box Django project First ~6

    months

12. + Story of Drund Things were... GREAT!

13. + Story of Drund Developing locally ./manage.py runserver

14. + Story of Drund Deploying to production ssh matt@… --

    svn up; touch /tmp/reload;

15. + Story of Drund Our stack Today

16. + Story of Drund Our stack Today Django, MySQL, Redis,

    Node.js, MongoDB, RabbitMQ, CDN, SSL-only, LESS, Require.js, Hogan.js, background worker, etc.

17. + Story of Drund Development is a bitch.

18. + Story of Drund Deploying is a bitch.

19. What’s the problem?

20. + What’s the problem? Django Nothing wrong with Django per

    se. Conditioned us for no external dependencies. Makes getting up and started super simple. SQLite3, Dummy cache, local memory cache, etc.

21. + What’s the problem? Node.js An external process, so needs

    another port. Requires two Terminal windows to spawn Django + Node.js app. Cookies aren’t shared across different ports, must match domain:port combo.

22. + What’s the problem? Redis, MongoDB, RabbitMQ Can’t be mocked

    in Django realistically like MySQL <-> SQLite. Annoying to run on local machine, mismatched configuration/versions, blah blah.

23. + What’s the problem? Less, Require.js, Hogan.js Assets must be

    compiled before being loaded into the browser. Tedious tasks during rapid development.

24. + What’s the problem? CDN …wut? Introduces occasional cross-domain issues.

    File uploads.

25. + What’s the problem? SSL Django can’t handle SSL. Real

    web server required.

26. + What’s the problem? Virtual hosts /etc/hosts becomes unwieldy. No

    wildcards.

27. + What’s the problem? 127.0.0.1 drund.local 127.0.0.1 www.drund.local 127.0.0.1 api.drund.local

    127.0.0.1 admin.drund.local 127.0.0.1 team.drund.local 127.0.0.1 partners.drund.local 127.0.0.1 matt.drund.local 127.0.0.1 nick.drund.local 127.0.0.1 kevin.drund.local 127.0.0.1 mike.drund.local 127.0.0.1 adam.drund.local 127.0.0.1 lee.drund.local

28. + What’s the problem? Configuration Management Django uses settings.py. Node.js

    uses settings.json. Both represent very similar data. Database, cache, etc.

29. + What’s the problem? Deployments git pull with Fabric. Every

    server in the cluster has different definitions of “update”. System dependencies, libxml, mysql-client, etc.

30. Re-architect ALL THE THINGS!

31. Virtual Hosts

32. + Virtual Hosts Forget about /etc/hosts. You can point a

    real DNS entry to 127.0.0.1! Real DNS gives you wildcard and CNAME entries.

33. + Virtual Hosts local.drund.net. A 127.0.0.1 *.local.drund.net. CNAME local.drund.net See

    Also: http://xip.io

34. Make & The Makefile

35. + Make & The Makefile Not just for “Real” compiling.

    It’s pretty much just /bin/sh. Simplify long or tedious commands. You’ll use it for a lot of things, I promise.

36. + Make & The Makefile Constructed of “Targets”. Targets can

    be real or not real. 98%* of your targets will be “PHONY”. Targets are essentially commands executed through `make`. make <target> * Not a real statistic.

37. + Make & The Makefile Targets support dependencies. “static depends

    on css and js first” targets == functions or commands static: @rm -rf ./static ./manage.py collectstatic --noinput -v0 clean: find . -name "*.pyc" -delete

38. + Make & The Makefile How we compile our static

    assets before:

39. + Make & The Makefile ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less >

    ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:

40. + Make & The Makefile FFuuuuuu!!! ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less

    > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:

41. + Make & The Makefile How we compile our assets

    with make:

42. + Make & The Makefile make static How we compile

    our assets with make:

43. + Make & The Makefile make static How we compile

    our assets with make:

44. all: css js templates static css: @../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less

    > \ ./media/main/css/kovu.css @echo "css compiled." js: templates @node ../tools/r.js/dist/r.js -o kovu.build.js @echo "js compiled <kovu.build.js>" templates: @../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} \ $(shell find ./media -name *.mustache) @echo "Templates smushed successfully." static: css js templates @rm -rf ./static ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 + Make & The Makefile Makefile

45. + Make & The Makefile Why not Fabric? Fabric is

    amazing for Python projects. We use it. Not the best for non-python. Makefiles are more generic. Choose what works for you, where and when it makes sense. * bitprophet: I do love Fabric. ;)

46. m4

47. + M4 Macro processor. Very “Dumb”. Basic string replacements. Small

    macros. Inheritance. Allows us to build configuration files for both Django and Node.js without duplication. Allows us to build configuration files for different environments.

48. + M4 C-ish define statements. define(`__NAME__', `Matt') Hello __NAME__

49. + M4 Odd syntax. Surprisingly does a lot.

50. + m4 define(`__DEBUG__', `True') define(`__DATABASE_ENGINE__', `mysql') define(`__DATABASE_NAME__', `drund') define(`__DATABASE_USER__', `root')

    define(`__DATABASE_PASSWORD__', `root') define(`__DATABASE_HOST__', `127.0.0.1') define(`__DATABASE_PORT__', `3306') m4/base.m4 define(`__DEBUG__', `False') define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `$uper$ecret!') define(`__DATABASE_HOST__', `10.0.0.0') m4/production.m4 define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `heywereonstaging!') m4/staging.m4

51. + m4 DEBUG = __DEBUG__ DATABASES = { 'default': {

    'ENGINE': 'django.db.backends.__DATABASE_ENGINE__', 'NAME': '__DATABASE_NAME__', 'USER': '__DATABASE_USER__', 'PASSWORD': '__DATABASE_PASSWORD__', 'HOST': '__DATABASE_HOST__', 'PORT': __DATABASE_PORT__', } } m4/settings.py.m4 { "database": { "name": "__DATABASE_NAME__", "user": "__DATABASE_USER__", "password": "__DATABASE_PASSWORD__", "host": "__DATABASE_HOST__", "port": __DATABASE_PORT__ } } m4/settings.json.m4

52. + m4 m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4

    m4/production.m4 m4/settings.json.m4 > settings.json settings: m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json Makefile Build it!

53. Vagrant http://vagrantup.com

54. + Vagrant “Virtualized development made easily.” Headless, local virtual machine.

    port forwarding. Easy bootstrapping with Chef, Puppet, or Shell script. vagrant up Runs our “Utilities” without cluttering local machine. (MySQL, Redis, RabbitMQ, MongoDB)

55. + Vagrant Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url =

    "http://files.vagrantup.com/precise64.box" config.vm.forward_port 3306, 3306 # MySQL config.vm.forward_port 6379, 6379 # Redis config.vm.forward_port 5672, 5672 # RabbitMQ config.vm.forward_port 27017, 27017 # MongoDB config.vm.provision :shell, :path => "bootstrap.sh" end Vagrantfile

56. + Vagrant Downloads “box” for first use. Creates Virtual Machine

    using VirtualBox. Forwards necessary ports, so I can access via 127.0.0.1:<port> Executes bootstrap.sh after build. vagrant ssh # SSH into the server vagrant halt # Shut down gracefully vagrant destroy # Delete the VM entirely

57. + Vagrant #!/bin/sh # First, let's check if everything is

    installed before we go installing shit again if [ -f /usr/sbin/mysqld ]; then \ exit 0 fi # MySQL root password echo mysql-server-5.1 mysql-server/root_password password root | debconf-set-selections echo mysql-server-5.1 mysql-server/root_password_again password root | debconf-set-selections apt-get update && apt-get install mysql-server -y echo "grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;"| mysql -u root -proot echo "Creating \"drund\" database..." echo "create database drund;" | mysql -u root -proot echo "Binding MySQL to 0.0.0.0..." cp /etc/mysql/my.cnf /etc/mysql/my.cnf.old sed -e's/127.0.0.1/0.0.0.0/' /etc/mysql/my.cnf.old > /etc/mysql/my.cnf echo "Restarting MySQL..." service mysql restart exit 0 bootstrap.sh

58. + Vagrant vm: vagrant up ./manage.py syncdb --migrate --noinput Makefile

59. Almost... but not quite good enough.

60. + Almost... but not quite good enough. # Only run

    once make settings make vm # Get the server running make static ./manage.py runserver # Listening @ http://www.local.drund.net:8000/ Current status

61. + Almost... but not quite good enough. What’s missing? make

    static needs to be run every time a file is changed. Node.js is still running on another port. No SSL. No way to simulate a CDN.

62. Node-HTTP-Proxy & “Node-testserver” https://github.com/nodejitsu/node-http-proxy

63. + Node-http-proxy & “node-testserver” Node.js... wut? Stand in as a

    dummy/fancier nginx. Proxy virtual host requests to different apps/ports. Pipe stdout/stderr like expected to console. Render Less and Mustache templates on the fly. “CDN” is just another virtual host. SSL.

64. + Node-http-proxy & “node-testserver” in a nutshell http-proxy spawns server

    on 443, terminates SSL, and processes vhosts as expected. node0.local.drund.net -> localhost:9001 # A miscellaneous Node.js app *.local.drund.net -> localhost:8000 # Main Django app testcdn.local.azork.com -> localhost:9002 # Internal Express media server

65. + Node-http-proxy & “node-testserver” in a nutshell Requests to internal

    media server are resolved through Django’s findstatic command. ./manage.py findstatic --first <path> LESS files are compiled into CSS when requested. Mustache templates are rendered on the fly. main.css -> main.less hello.template.js -> hello.template.mustache

66. + Node-http-proxy & “node-testserver” Added Bonus! Django is no longer

    serving up static media. Load assets concurrently. It actually makes a difference.

67. sudo make testserver https://www.local.drund.net/

68. Jenkins http://jenkins-ci.org/

69. + Jenkins Continuous Integration & Continuous Deployment Nirvana

70. + Jenkins Post-commit hooks to run builds automatically. Executes a

    series of Shell commands. Builds new virtualenv for each build. Runs tests. Fail if tests fail. Notify team. Build deb package for source code. Push package to private apt repository. apt-get install drund-{component}-{branch} -y on servers with Fabric. Integrates nicely with IRC.

71. Deb packaging

72. + Deb packaging Why a system package? Originally skeptical myself.

    Very easily manage dependencies outside of Python/pip. Post-install hooks. No compiling on production servers.

73. + Deb packaging Drund is broken up into many packages

    by code branch. Packages can be mixed and matched depending on roles. drund-src-master # Main Drund source code. Only. drund-www-master # Depends on drund-src-master, nginx config drund-worker-master # Depends on drund-src-master, starts background worker drund-activity-master # Node.js app drund-proxy-master # Node.js app drund-cron-master # Depends on drund-src-master, cron.d files.

74. + Deb packaging drund-src-master Virtual package for drund-src-r{build}. Essentially: drund-srcdepends

    on drund-src-r{build} drund-src-r100 and drund-src-r101 can be installed at the same time. post install manages a symlink. 100% new environment for each build. Easy to roll back. Just repoint symlink. No processes are spawned.

75. + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer:

    Drund <[email protected]> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:

76. + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer:

    Drund <[email protected]> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:

77. + Deb packaging Install via Apt-get apt-get install drund-src-master

78. Continuous Integration/deployments

79. + Continuous Integration/Deployments Tests are run automatically before anything is

    deployed. Tests are run after every single push. It’s actually impossible to deploy code with broken tests. Jenkins can be configured to run a Fabric command to update after a successful build. It’s just another shell command. Annoying at first. You will make mistakes.

80. + Continuous Integration/Deployments Forced to write better code. Forced to

    write more tests. Once your environment is configured, everything is done correctly. Every time. Computers are pretty good at that. Easy to bring up staging or development servers.

81. New & improved workflow

82. + New & Improved Workflow master branch is 100% as

    stable as possible. Production only. Code is only merged in from develop. develop contains code ready to be merged into master at any time. Continuously deployed to dev/staging servers. Feature branches off of develop. If bad code is pushed to develop, Jenkins complains and posts in IRC. If all is good, code review, queue up for merge into master.

83. + New & Improved Workflow Hit the ground running in

    very little time. git clone git@... make init # Grabs submodules, create virtualenv, installs requirements

84. + New & Improved Workflow Run the project. sudo make

    testserver # sudo is needed for port 443.

85. + New & Improved Workflow Deploy code to staging. git

    checkout develop git merge feature-branch # Or Pull Request git push origin develop

86. + New & Improved Workflow Deploy to production. git checkout

    master git merge develop # Preferably a Pull Request git push origin master
87. None

88. Thanks. @Mattrobenolt github.com/mattrobenolt mattrobenolt.com