Single Artifact Deployments (Jason Filipe)

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

Transcript

1. SINGLE ARTIFACT DEPLOYMENTS For Python Applications Jason Filipe @jfilipe
4. DEVELOPER ON THE PLATFORM TEAM
5. WHAT WE’LL COVER • Typical deployment process • Single artifact deployment, obstacles encountered and tools used
6. TYPICAL DEPLOYMENT Fabric script $  fab  -­‐-­‐set  target=staging  deploy command
7. FABRIC SCRIPT On each app server + worker I. Download tarball of git tag II. Install requirements in a new virtualenv III. Static asset compilation (css/js) IV. Run database migrations (South) V. Swap symlinks (code and virtualenv) and restart processes
8. SYMLINK BASED source code /srv/releases/2012-12-31--00-00-00 /srv/releases/2013-01-01--00-00-00 /srv/releases/current -> 2013-01-01--00-00-00 /srv/releases/previous -> 2012-12-31-00-00-00
9. SYMLINK BASED virtualenv /srv/virtualenv/2012-12-31--00-00-00 /srv/virtualenv/2013-01-01--00-00-00 /srv/virtualenv/default -> 2013-01-01--00-00-00 /srv/virtualenv/previous -> 2012-12-31-00-00-00
10. WHY IT SUCKS
11. DEVELOPER MACHINE DEPENDENCY • Spotty internet connectivity • Addition local deploy dependencies • Fear
12. DUPLICATION • Download tarball of git tag • Install requirements • Static asset minification • etc.. app-01 app-*
13. SLOW 18+ minutes
14. WE WANTED SOMETHING BETTER
15. SINGLE ARTIFACT DEPLOYS
16. Create a single tarball that contains the source code, compiled virtualenv and all static assets minified and ready to go.
17. TOOLS
18. HAPROXY
21. SUPERVISORD
23. COMPILED VIRTUALENV PAIN • Hardcoded paths in pyc files • https://github.com/fireteam/ virtualenv-tools • Doesn’t work with python dependencies that have C extensions (e.g. gevent)
24. Compile the virtualenv in the target directory.
25. BUILD PHASE • Jenkins job • Part of build pipeline • accounting-­‐unit accounting-­‐integration accounting-­‐build
31. #!/bin/bash cd  $WORKSPACE virtualenv_hash  =  `md5sum  requirements.txt` if  [  -­‐d  "/srv/virtualenv/$virtualenv_hash"  ];  then        echo  "Reusing  existing  virtualenv  ${virtualenv_hash}" else        echo  "Cached  virtualenv  not  found,  creating  new..."        virtualenv  "/srv/virtualenv/$virtualenv_hash"        pip  install  -­‐r  requirements.txt fi cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv" #  static  asset  compilation  goes  here #  grunt  release #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput #  virtualenv/bin/python  manage.py  compress tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *
32. Build server must match production. python version, distro, etc
33. /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7 /srv/virtualenv/default -> ef332e984ecf9044f35da82adaa182f7
34. /srv/releases/1337 /srv/releases/current -> 1337 /srv/releases/1337/virtualenv -> /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7
35. NEW FABRIC SCRIPT $  fab  -­‐-­‐set  target=staging   deploy:accounting-­‐1337.tgz command
36. NEW FABRIC SCRIPT On each app server + worker I. Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
37. NEW FABRIC SCRIPT On each app server + worker I. Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
39. Nathan Duthoit The path to smoother database migrations
40. NEW FABRIC SCRIPT On each app server + worker I. Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
41. SUPERVISORD #  ... stopwaitsecs=35 stopsignal=QUIT #  .... config $  supervisorctl  stop  <gunicorn> command
42. FAST ~ 2 minutes majority is waiting for inflight requests
43. MINIMIZED SPOF • If things fail it’s during build time
44. ALL DONE? • Much better than initial deploy process • Still have developer machine dependency But we have Jenkins!
45. NEW DEPLOY JOB • accounting-­‐deploy • Executes Fabric script • Need to manage Jenkins credentials and click a button We can still do better!
46. +
47. DEPLOY ALL THE THINGS
48. DEPLOYS VIA HIPCHAT
49. ALTERNATIVES • Wheel PEP 427 • pip support as of 1.4 to build and install wheels command $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/ wheelhouse  -­‐r  requirements.txt $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐ links=/tmp/accounting/wheelhouse  -­‐r   requirements.txt
50. IMPROVEMENTS • Build native packages (rpm/deb) from single artifact • Optimizations during build phase
51. FUTURE • Docker and packer • Immutable servers
52. TL;DR • Typically longest step of deployments is installing requirements • Precompile your virtualenv in the same directory it will be used in production to avoid hardcoded paths in pyc files using hash of requirements as virtualenv name • Enjoy faster deploys!
53. THANKS! @jfilipe filipe.ca