Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Single Artifact Deployments

Single Artifact Deployments

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

Watch the talk: http://www.youtube.com/watch?v=uza3AMUOESk

Jason Filipe

August 11, 2013
Tweet

Other Decks in Programming

Transcript

  1. SINGLE
    ARTIFACT
    DEPLOYMENTS
    For Python Applications
    Jason Filipe
    @jfilipe

    View full-size slide

  2. DEVELOPER ON THE
    PLATFORM TEAM

    View full-size slide

  3. WHAT WE’LL COVER
    • Typical deployment process
    • Single artifact deployment,
    obstacles encountered and tools
    used

    View full-size slide

  4. TYPICAL DEPLOYMENT
    Fabric script
    $  fab  -­‐-­‐set  target=staging  deploy
    command

    View full-size slide

  5. FABRIC SCRIPT
    On each app server + worker
    I. Download tarball of git tag
    II. Install requirements in a new virtualenv
    III. Static asset compilation (css/js)
    IV. Run database migrations (South)
    V. Swap symlinks (code and virtualenv) and
    restart processes

    View full-size slide

  6. SYMLINK BASED
    source code
    /srv/releases/2012-12-31--00-00-00
    /srv/releases/2013-01-01--00-00-00
    /srv/releases/current -> 2013-01-01--00-00-00
    /srv/releases/previous -> 2012-12-31-00-00-00

    View full-size slide

  7. SYMLINK BASED
    virtualenv
    /srv/virtualenv/2012-12-31--00-00-00
    /srv/virtualenv/2013-01-01--00-00-00
    /srv/virtualenv/default -> 2013-01-01--00-00-00
    /srv/virtualenv/previous -> 2012-12-31-00-00-00

    View full-size slide

  8. DEVELOPER MACHINE
    DEPENDENCY
    • Spotty internet connectivity
    • Addition local deploy
    dependencies
    • Fear

    View full-size slide

  9. DUPLICATION
    • Download tarball of git tag
    • Install requirements
    • Static asset minification
    • etc..
    app-01
    app-*

    View full-size slide

  10. SLOW
    18+ minutes

    View full-size slide

  11. WE WANTED
    SOMETHING
    BETTER

    View full-size slide

  12. SINGLE ARTIFACT
    DEPLOYS

    View full-size slide

  13. Create a single tarball that
    contains the source code,
    compiled virtualenv and all static
    assets minified and ready to go.

    View full-size slide

  14. COMPILED
    VIRTUALENV PAIN
    • Hardcoded paths in pyc files
    • https://github.com/fireteam/
    virtualenv-tools
    • Doesn’t work with python
    dependencies that have C
    extensions (e.g. gevent)

    View full-size slide

  15. Compile the virtualenv in the
    target directory.

    View full-size slide

  16. BUILD PHASE
    • Jenkins job
    • Part of build pipeline
    • accounting-­‐unit
    accounting-­‐integration
    accounting-­‐build

    View full-size slide

  17. #!/bin/bash
    cd  $WORKSPACE
    virtualenv_hash  =  `md5sum  requirements.txt`
    if  [  -­‐d  "/srv/virtualenv/$virtualenv_hash"  ];  then
           echo  "Reusing  existing  virtualenv  ${virtualenv_hash}"
    else
           echo  "Cached  virtualenv  not  found,  creating  new..."
           virtualenv  "/srv/virtualenv/$virtualenv_hash"
           pip  install  -­‐r  requirements.txt
    fi
    cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv"
    #  static  asset  compilation  goes  here
    #  grunt  release
    #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput
    #  virtualenv/bin/python  manage.py  compress
    tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *

    View full-size slide

  18. Build server must match production.
    python version, distro, etc

    View full-size slide

  19. /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7
    /srv/virtualenv/default ->
    ef332e984ecf9044f35da82adaa182f7

    View full-size slide

  20. /srv/releases/1337
    /srv/releases/current -> 1337
    /srv/releases/1337/virtualenv ->
    /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7

    View full-size slide

  21. NEW FABRIC SCRIPT
    $  fab  -­‐-­‐set  target=staging  
    deploy:accounting-­‐1337.tgz
    command

    View full-size slide

  22. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued

    View full-size slide

  23. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued

    View full-size slide

  24. Nathan Duthoit
    The path to smoother database
    migrations

    View full-size slide

  25. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued

    View full-size slide

  26. SUPERVISORD
    #  ...
    stopwaitsecs=35
    stopsignal=QUIT
    #  ....
    config
    $  supervisorctl  stop  
    command

    View full-size slide

  27. FAST
    ~ 2 minutes
    majority is waiting for inflight requests

    View full-size slide

  28. MINIMIZED SPOF
    • If things fail it’s during build time

    View full-size slide

  29. ALL DONE?
    • Much better than initial deploy
    process
    • Still have developer machine
    dependency
    But we have Jenkins!

    View full-size slide

  30. NEW DEPLOY JOB
    • accounting-­‐deploy
    • Executes Fabric script
    • Need to manage Jenkins
    credentials and click a button
    We can still do better!

    View full-size slide

  31. DEPLOY ALL THE
    THINGS

    View full-size slide

  32. DEPLOYS VIA HIPCHAT

    View full-size slide

  33. ALTERNATIVES
    • Wheel PEP 427
    • pip support as of 1.4 to build and
    install wheels
    command
    $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/
    wheelhouse  -­‐r  requirements.txt
    $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐
    links=/tmp/accounting/wheelhouse  -­‐r  
    requirements.txt

    View full-size slide

  34. IMPROVEMENTS
    • Build native packages (rpm/deb)
    from single artifact
    • Optimizations during build phase

    View full-size slide

  35. FUTURE
    • Docker and packer
    • Immutable servers

    View full-size slide

  36. TL;DR
    • Typically longest step of deployments is
    installing requirements
    • Precompile your virtualenv in the same
    directory it will be used in production to avoid
    hardcoded paths in pyc files using hash of
    requirements as virtualenv name
    • Enjoy faster deploys!

    View full-size slide

  37. THANKS!
    @jfilipe
    filipe.ca

    View full-size slide