Single Artifact Deployments

Single Artifact Deployments

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

Watch the talk: http://www.youtube.com/watch?v=uza3AMUOESk

6d3514e945147125d4504a8ae5806233?s=128

Jason Filipe

August 11, 2013
Tweet

Transcript

  1. 2.
  2. 3.
  3. 5.

    WHAT WE’LL COVER • Typical deployment process • Single artifact

    deployment, obstacles encountered and tools used
  4. 7.

    FABRIC SCRIPT On each app server + worker I. Download

    tarball of git tag II. Install requirements in a new virtualenv III. Static asset compilation (css/js) IV. Run database migrations (South) V. Swap symlinks (code and virtualenv) and restart processes
  5. 12.

    DUPLICATION • Download tarball of git tag • Install requirements

    • Static asset minification • etc.. app-01 app-*
  6. 16.

    Create a single tarball that contains the source code, compiled

    virtualenv and all static assets minified and ready to go.
  7. 17.
  8. 18.
  9. 19.
  10. 20.
  11. 22.
  12. 23.

    COMPILED VIRTUALENV PAIN • Hardcoded paths in pyc files •

    https://github.com/fireteam/ virtualenv-tools • Doesn’t work with python dependencies that have C extensions (e.g. gevent)
  13. 25.

    BUILD PHASE • Jenkins job • Part of build pipeline

    • accounting-­‐unit accounting-­‐integration accounting-­‐build
  14. 26.
  15. 27.
  16. 28.
  17. 29.
  18. 30.
  19. 31.

    #!/bin/bash cd  $WORKSPACE virtualenv_hash  =  `md5sum  requirements.txt` if  [  -­‐d

     "/srv/virtualenv/$virtualenv_hash"  ];  then        echo  "Reusing  existing  virtualenv  ${virtualenv_hash}" else        echo  "Cached  virtualenv  not  found,  creating  new..."        virtualenv  "/srv/virtualenv/$virtualenv_hash"        pip  install  -­‐r  requirements.txt fi cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv" #  static  asset  compilation  goes  here #  grunt  release #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput #  virtualenv/bin/python  manage.py  compress tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *
  20. 36.

    NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  21. 37.

    NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  22. 38.
  23. 40.

    NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  24. 44.

    ALL DONE? • Much better than initial deploy process •

    Still have developer machine dependency But we have Jenkins!
  25. 45.

    NEW DEPLOY JOB • accounting-­‐deploy • Executes Fabric script •

    Need to manage Jenkins credentials and click a button We can still do better!
  26. 46.

    +

  27. 49.

    ALTERNATIVES • Wheel PEP 427 • pip support as of

    1.4 to build and install wheels command $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/ wheelhouse  -­‐r  requirements.txt $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐ links=/tmp/accounting/wheelhouse  -­‐r   requirements.txt
  28. 52.

    TL;DR • Typically longest step of deployments is installing requirements

    • Precompile your virtualenv in the same directory it will be used in production to avoid hardcoded paths in pyc files using hash of requirements as virtualenv name • Enjoy faster deploys!