Single Artifact Deployments

Single Artifact Deployments

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

Watch the talk: http://www.youtube.com/watch?v=uza3AMUOESk

6d3514e945147125d4504a8ae5806233?s=128

Jason Filipe

August 11, 2013
Tweet

Transcript

  1. SINGLE ARTIFACT DEPLOYMENTS For Python Applications Jason Filipe @jfilipe

  2. None
  3. None
  4. DEVELOPER ON THE PLATFORM TEAM

  5. WHAT WE’LL COVER • Typical deployment process • Single artifact

    deployment, obstacles encountered and tools used
  6. TYPICAL DEPLOYMENT Fabric script $  fab  -­‐-­‐set  target=staging  deploy command

  7. FABRIC SCRIPT On each app server + worker I. Download

    tarball of git tag II. Install requirements in a new virtualenv III. Static asset compilation (css/js) IV. Run database migrations (South) V. Swap symlinks (code and virtualenv) and restart processes
  8. SYMLINK BASED source code /srv/releases/2012-12-31--00-00-00 /srv/releases/2013-01-01--00-00-00 /srv/releases/current -> 2013-01-01--00-00-00 /srv/releases/previous

    -> 2012-12-31-00-00-00
  9. SYMLINK BASED virtualenv /srv/virtualenv/2012-12-31--00-00-00 /srv/virtualenv/2013-01-01--00-00-00 /srv/virtualenv/default -> 2013-01-01--00-00-00 /srv/virtualenv/previous ->

    2012-12-31-00-00-00
  10. WHY IT SUCKS

  11. DEVELOPER MACHINE DEPENDENCY • Spotty internet connectivity • Addition local

    deploy dependencies • Fear
  12. DUPLICATION • Download tarball of git tag • Install requirements

    • Static asset minification • etc.. app-01 app-*
  13. SLOW 18+ minutes

  14. WE WANTED SOMETHING BETTER

  15. SINGLE ARTIFACT DEPLOYS

  16. Create a single tarball that contains the source code, compiled

    virtualenv and all static assets minified and ready to go.
  17. TOOLS

  18. HAPROXY

  19. None
  20. None
  21. SUPERVISORD

  22. None
  23. COMPILED VIRTUALENV PAIN • Hardcoded paths in pyc files •

    https://github.com/fireteam/ virtualenv-tools • Doesn’t work with python dependencies that have C extensions (e.g. gevent)
  24. Compile the virtualenv in the target directory.

  25. BUILD PHASE • Jenkins job • Part of build pipeline

    • accounting-­‐unit accounting-­‐integration accounting-­‐build
  26. None
  27. None
  28. None
  29. None
  30. None
  31. #!/bin/bash cd  $WORKSPACE virtualenv_hash  =  `md5sum  requirements.txt` if  [  -­‐d

     "/srv/virtualenv/$virtualenv_hash"  ];  then        echo  "Reusing  existing  virtualenv  ${virtualenv_hash}" else        echo  "Cached  virtualenv  not  found,  creating  new..."        virtualenv  "/srv/virtualenv/$virtualenv_hash"        pip  install  -­‐r  requirements.txt fi cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv" #  static  asset  compilation  goes  here #  grunt  release #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput #  virtualenv/bin/python  manage.py  compress tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *
  32. Build server must match production. python version, distro, etc

  33. /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7 /srv/virtualenv/default -> ef332e984ecf9044f35da82adaa182f7

  34. /srv/releases/1337 /srv/releases/current -> 1337 /srv/releases/1337/virtualenv -> /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7

  35. NEW FABRIC SCRIPT $  fab  -­‐-­‐set  target=staging   deploy:accounting-­‐1337.tgz command

  36. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  37. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  38. None
  39. Nathan Duthoit The path to smoother database migrations

  40. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  41. SUPERVISORD #  ... stopwaitsecs=35 stopsignal=QUIT #  .... config $  supervisorctl

     stop  <gunicorn> command
  42. FAST ~ 2 minutes majority is waiting for inflight requests

  43. MINIMIZED SPOF • If things fail it’s during build time

  44. ALL DONE? • Much better than initial deploy process •

    Still have developer machine dependency But we have Jenkins!
  45. NEW DEPLOY JOB • accounting-­‐deploy • Executes Fabric script •

    Need to manage Jenkins credentials and click a button We can still do better!
  46. +

  47. DEPLOY ALL THE THINGS

  48. DEPLOYS VIA HIPCHAT

  49. ALTERNATIVES • Wheel PEP 427 • pip support as of

    1.4 to build and install wheels command $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/ wheelhouse  -­‐r  requirements.txt $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐ links=/tmp/accounting/wheelhouse  -­‐r   requirements.txt
  50. IMPROVEMENTS • Build native packages (rpm/deb) from single artifact •

    Optimizations during build phase
  51. FUTURE • Docker and packer • Immutable servers

  52. TL;DR • Typically longest step of deployments is installing requirements

    • Precompile your virtualenv in the same directory it will be used in production to avoid hardcoded paths in pyc files using hash of requirements as virtualenv name • Enjoy faster deploys!
  53. THANKS! @jfilipe filipe.ca