Single Artifact Deployments

Deploying python applications should be fast, safe and repeatable. In this talk I outline a new deployment process developed at Wave Accounting and show you the tips and tricks you'll need to build your own system.

Watch the talk: http://www.youtube.com/watch?v=uza3AMUOESk

Transcript

1. SINGLE
   ARTIFACT
   DEPLOYMENTS
   For Python Applications
   Jason Filipe
   @jfilipe
   

   View Slide

2. View Slide

3. View Slide

4. DEVELOPER ON THE
   PLATFORM TEAM
   

   View Slide

5. WHAT WE’LL COVER
   • Typical deployment process
   • Single artifact deployment,
   obstacles encountered and tools
   used
   

   View Slide

6. TYPICAL DEPLOYMENT
   Fabric script
   $  fab  -­‐-­‐set  target=staging  deploy
   command
   

   View Slide

7. FABRIC SCRIPT
   On each app server + worker
   I. Download tarball of git tag
   II. Install requirements in a new virtualenv
   III. Static asset compilation (css/js)
   IV. Run database migrations (South)
   V. Swap symlinks (code and virtualenv) and
   restart processes
   

   View Slide

8. SYMLINK BASED
   source code
   /srv/releases/2012-12-31--00-00-00
   /srv/releases/2013-01-01--00-00-00
   /srv/releases/current -> 2013-01-01--00-00-00
   /srv/releases/previous -> 2012-12-31-00-00-00
   

   View Slide

9. SYMLINK BASED
   virtualenv
   /srv/virtualenv/2012-12-31--00-00-00
   /srv/virtualenv/2013-01-01--00-00-00
   /srv/virtualenv/default -> 2013-01-01--00-00-00
   /srv/virtualenv/previous -> 2012-12-31-00-00-00
   

   View Slide

10. WHY
    IT SUCKS
    

    View Slide

11. DEVELOPER MACHINE
    DEPENDENCY
    • Spotty internet connectivity
    • Addition local deploy
    dependencies
    • Fear
    

    View Slide

12. DUPLICATION
    • Download tarball of git tag
    • Install requirements
    • Static asset minification
    • etc..
    app-01
    app-*
    

    View Slide

13. SLOW
    18+ minutes
    

    View Slide

14. WE WANTED
    SOMETHING
    BETTER
    

    View Slide

15. SINGLE ARTIFACT
    DEPLOYS
    

    View Slide

16. Create a single tarball that
    contains the source code,
    compiled virtualenv and all static
    assets minified and ready to go.
    

    View Slide

17. TOOLS
    

    View Slide

18. HAPROXY
    

    View Slide

19. View Slide

20. View Slide

21. SUPERVISORD
    

    View Slide

22. View Slide

23. COMPILED
    VIRTUALENV PAIN
    • Hardcoded paths in pyc files
    • https://github.com/fireteam/
    virtualenv-tools
    • Doesn’t work with python
    dependencies that have C
    extensions (e.g. gevent)
    

    View Slide

24. Compile the virtualenv in the
    target directory.
    

    View Slide

25. BUILD PHASE
    • Jenkins job
    • Part of build pipeline
    • accounting-­‐unit
    accounting-­‐integration
    accounting-­‐build
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. #!/bin/bash
    cd  $WORKSPACE
    virtualenv_hash  =  `md5sum  requirements.txt`
    if  [  -­‐d  "/srv/virtualenv/$virtualenv_hash"  ];  then
           echo  "Reusing  existing  virtualenv  ${virtualenv_hash}"
    else
           echo  "Cached  virtualenv  not  found,  creating  new..."
           virtualenv  "/srv/virtualenv/$virtualenv_hash"
           pip  install  -­‐r  requirements.txt
    fi
    cp  -­‐r  "/srv/virtualenv/$virtualenv_hash"  "$WORKSPACE/virtualenv"
    #  static  asset  compilation  goes  here
    #  grunt  release
    #  virtualenv/bin/python  python  manage.py  collectstatic  -­‐-­‐noinput
    #  virtualenv/bin/python  manage.py  compress
    tar  zcvf  accounting-­‐$BUILD_NUMBER.tgz  -­‐-­‐exclude='.git'  *
    

    View Slide

32. Build server must match production.
    python version, distro, etc
    

    View Slide

33. /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7
    /srv/virtualenv/default ->
    ef332e984ecf9044f35da82adaa182f7
    

    View Slide

34. /srv/releases/1337
    /srv/releases/current -> 1337
    /srv/releases/1337/virtualenv ->
    /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7
    

    View Slide

35. NEW FABRIC SCRIPT
    $  fab  -­‐-­‐set  target=staging  
    deploy:accounting-­‐1337.tgz
    command
    

    View Slide

36. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued
    

    View Slide

37. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued
    

    View Slide

38. View Slide

39. Nathan Duthoit
    The path to smoother database
    migrations
    

    View Slide

40. NEW FABRIC SCRIPT
    On each app server + worker
    I. Download single artifact tarball
    II. Untar artifact
    III. List out any pending db migrations
    IV. Swap symlinks (code and virtualenv) and
    restart
    continued
    

    View Slide

41. SUPERVISORD
    #  ...
    stopwaitsecs=35
    stopsignal=QUIT
    #  ....
    config
    $  supervisorctl  stop  
    command
    

    View Slide

42. FAST
    ~ 2 minutes
    majority is waiting for inflight requests
    

    View Slide

43. MINIMIZED SPOF
    • If things fail it’s during build time
    

    View Slide

44. ALL DONE?
    • Much better than initial deploy
    process
    • Still have developer machine
    dependency
    But we have Jenkins!
    

    View Slide

45. NEW DEPLOY JOB
    • accounting-­‐deploy
    • Executes Fabric script
    • Need to manage Jenkins
    credentials and click a button
    We can still do better!
    

    View Slide

46. +
    

    View Slide

47. DEPLOY ALL THE
    THINGS
    

    View Slide

48. DEPLOYS VIA HIPCHAT
    

    View Slide

49. ALTERNATIVES
    • Wheel PEP 427
    • pip support as of 1.4 to build and
    install wheels
    command
    $  pip  wheel  -­‐-­‐wheel-­‐dir=/tmp/accounting/
    wheelhouse  -­‐r  requirements.txt
    $  pip  install  -­‐-­‐use-­‐wheel  -­‐-­‐no-­‐index  -­‐-­‐find-­‐
    links=/tmp/accounting/wheelhouse  -­‐r  
    requirements.txt
    

    View Slide

50. IMPROVEMENTS
    • Build native packages (rpm/deb)
    from single artifact
    • Optimizations during build phase
    

    View Slide

51. FUTURE
    • Docker and packer
    • Immutable servers
    

    View Slide

52. TL;DR
    • Typically longest step of deployments is
    installing requirements
    • Precompile your virtualenv in the same
    directory it will be used in production to avoid
    hardcoded paths in pyc files using hash of
    requirements as virtualenv name
    • Enjoy faster deploys!
    

    View Slide

53. THANKS!
    @jfilipe
    filipe.ca
    

    View Slide