Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Single Artifact Deployments (Jason Filipe)

Single Artifact Deployments (Jason Filipe)

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

PyCon Canada

August 11, 2013
Tweet

More Decks by PyCon Canada

Other Decks in Programming

Transcript

  1. WHAT WE’LL COVER • Typical deployment process • Single artifact

    deployment, obstacles encountered and tools used
  2. FABRIC SCRIPT On each app server + worker I. Download

    tarball of git tag II. Install requirements in a new virtualenv III. Static asset compilation (css/js) IV. Run database migrations (South) V. Swap symlinks (code and virtualenv) and restart processes
  3. DUPLICATION • Download tarball of git tag • Install requirements

    • Static asset minification • etc.. app-01 app-*
  4. Create a single tarball that contains the source code, compiled

    virtualenv and all static assets minified and ready to go.
  5. COMPILED VIRTUALENV PAIN • Hardcoded paths in pyc files •

    https://github.com/fireteam/ virtualenv-tools • Doesn’t work with python dependencies that have C extensions (e.g. gevent)
  6. BUILD PHASE • Jenkins job • Part of build pipeline

    • accounting-­‐unit accounting-­‐integration accounting-­‐build
  7. #!/bin/bash cd  $WORKSPACE virtualenv_hash  =  `md5sum  requirements.txt` if  [  -­‐d

     "/srv/virtualenv/$virtualenv_hash"  ];  then        echo  "Reusing  existing  virtualenv  ${virtualenv_hash}" else        echo  "Cached  virtualenv  not  found,  creating  new..."        virtualenv  "/srv/virtualenv/$virtualenv_hash"        pip  install  -­‐r  requirements.txt fi cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv" #  static  asset  compilation  goes  here #  grunt  release #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput #  virtualenv/bin/python  manage.py  compress tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *
  8. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  9. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  10. NEW FABRIC SCRIPT On each app server + worker I.

    Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued
  11. ALL DONE? • Much better than initial deploy process •

    Still have developer machine dependency But we have Jenkins!
  12. NEW DEPLOY JOB • accounting-­‐deploy • Executes Fabric script •

    Need to manage Jenkins credentials and click a button We can still do better!
  13. +

  14. ALTERNATIVES • Wheel PEP 427 • pip support as of

    1.4 to build and install wheels command $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/ wheelhouse  -­‐r  requirements.txt $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐ links=/tmp/accounting/wheelhouse  -­‐r   requirements.txt
  15. TL;DR • Typically longest step of deployments is installing requirements

    • Precompile your virtualenv in the same directory it will be used in production to avoid hardcoded paths in pyc files using hash of requirements as virtualenv name • Enjoy faster deploys!