Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous-Delivery-Meetup-NYC-2013.pdf

 Continuous-Delivery-Meetup-NYC-2013.pdf

How we tackle deployments and keep them fast at Etsy.

Reprised version with some additions for the Continuous Delivery NYC meetup October 2013.

Daniel Schauenberg

October 09, 2013
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. Scaling Deployment at Etsy
    Daniel Schauenberg
    [email protected]
    @mrtazz
    Thursday, October 10, 13

    View full-size slide

  2. Thursday, October 10, 13

    View full-size slide

  3. August 2013
    •1.8 billion page views
    •5,483,399 items sold
    •$109.1 million of goods sold
    •> 30 million members
    •> 1 million active shops
    http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
    Thursday, October 10, 13

    View full-size slide

  4. LAMMP
    Item by TheBackPackShoppe
    Thursday, October 10, 13

    View full-size slide

  5. Item by FrankelPhotos
    Monolithic App
    Thursday, October 10, 13

    View full-size slide

  6. No Branching
    Item by NurseryWallArt
    Thursday, October 10, 13

    View full-size slide

  7. Deploy Frequency
    Thursday, October 10, 13

    View full-size slide

  8. First Day
    Item by flowersandfleurons
    Thursday, October 10, 13

    View full-size slide

  9. Thursday, October 10, 13

    View full-size slide

  10. IRC
    Thursday, October 10, 13

    View full-size slide

  11. Thursday, October 10, 13

    View full-size slide

  12. Developer VMs
    •KVM
    •Dev version of full Etsy stack
    •Chef
    •DevTools
    Thursday, October 10, 13

    View full-size slide

  13. Thursday, October 10, 13

    View full-size slide

  14. Item by codecards
    Thursday, October 10, 13

    View full-size slide

  15. % review -r dschauenberg
    Thursday, October 10, 13

    View full-size slide

  16. automatically assigned
    automatically assigned
    Thursday, October 10, 13

    View full-size slide

  17. Try
    Item by CSSDesign
    Thursday, October 10, 13

    View full-size slide

  18. Actually ...
    Thursday, October 10, 13

    View full-size slide

  19. The Bobs
    Item by Signz
    Thursday, October 10, 13

    View full-size slide

  20. The Bobs
    •LXC containers on buildtests
    •Multiple SSDs
    •Labels for heavy/any execution
    •One heavy executor per disk
    Thursday, October 10, 13

    View full-size slide

  21. CI/Try
    •~260 Bobs
    •Mostly for try
    •Constant monitoring for slow tests
    Thursday, October 10, 13

    View full-size slide

  22. push train
    Item by decomodwalls
    Thursday, October 10, 13

    View full-size slide

  23. #push
    •IRC channel to organize push trains
    •Join a train if you want to deploy changes
    •Schedule is planned via the channel topic
    •First in the train is the driver (controls the deploy)
    •Opening hours: 7am - 10pm NYC time
    Thursday, October 10, 13

    View full-size slide

  24. #push
    kseever* + jameslee | jpaul | DanielConvissor (c)
    Thursday, October 10, 13

    View full-size slide

  25. #push
    bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
    + jlaster (c) | dawa + corey + sandosh + jklein + magera
    + seth_home + mpascual + nathan | bateman | russp (c)
    Thursday, October 10, 13

    View full-size slide

  26. pushbot
    •.join
    •.in
    •.good
    •.done
    Thursday, October 10, 13

    View full-size slide

  27. pushbot
    Thursday, October 10, 13

    View full-size slide

  28. Item by EsalonPhotography
    Thursday, October 10, 13

    View full-size slide

  29. Deployinator
    Thursday, October 10, 13

    View full-size slide

  30. Thursday, October 10, 13

    View full-size slide

  31. Thursday, October 10, 13

    View full-size slide

  32. stale commits
    Thursday, October 10, 13

    View full-size slide

  33. version checks
    buttons disabled
    buttons disabled
    Thursday, October 10, 13

    View full-size slide

  34. version checks
    Thursday, October 10, 13

    View full-size slide

  35. lock down deploys
    Thursday, October 10, 13

    View full-size slide

  36. https://github.com/etsy/deployinator
    Thursday, October 10, 13

    View full-size slide

  37. Downsides
    •Deploys not atomic on the request level
    •Limbo during the time of the local rsync
    •Common strategy was to split commits into 3 deploys
    Thursday, October 10, 13

    View full-size slide

  38. Item by Geographicsart
    Thursday, October 10, 13

    View full-size slide

  39. Atomic Deploys
    Thursday, October 10, 13

    View full-size slide

  40. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13

    View full-size slide

  41. Basic Idea
    Yin Yang
    Active Docroot
    rsync
    Thursday, October 10, 13

    View full-size slide

  42. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13

    View full-size slide

  43. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13

    View full-size slide

  44. Problems
    •Symlink swap during requests
    •Code needs to be guaranteed to finish on the docroot
    it started
    •Code inclusion mid request
    Thursday, October 10, 13

    View full-size slide

  45. etsy/mod_realdoc
    •Apache post_read_request hook
    •Whole request works on realpath of docroot
    •Caches realpath for 2s
    Thursday, October 10, 13

    View full-size slide

  46. ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');
    Thursday, October 10, 13

    View full-size slide

  47. etsy/incpath
    •PHP module to set the incpath
    •Gets docroot from Apache or realpath() itself
    •Looks for a pattern to replace in include_path
    •Restores include_path at the end of the request
    Thursday, October 10, 13

    View full-size slide

  48. What did we get?
    •Remove functions and call site in same deploy
    •No restarts necessary
    •Opcode caches stay warm for files that don’t change
    between 2 deploys
    Thursday, October 10, 13

    View full-size slide

  49. Things to watch out for
    •Code that uses full path names to scripts
    •Atomic symlink swapping with `mv -T`
    •Realpath caching to not stress the filesystem
    •Opcode cache needs to fit 2x code size
    •Only request atomicity
    Thursday, October 10, 13

    View full-size slide

  50. The Plateau
    Item by finandfancy
    Thursday, October 10, 13

    View full-size slide

  51. The Plateau
    •Regular deploys took ~15 mins
    •Config deploys about half
    •10am - 6pm => ~ 32 deploys
    •Long waiting times
    Thursday, October 10, 13

    View full-size slide

  52. Item by KlaireWarren
    Thursday, October 10, 13

    View full-size slide

  53. Waiting for push queue
    Waiting for push queue
    Thursday, October 10, 13

    View full-size slide

  54. Split The Queues
    Item by KlaireWarren
    Thursday, October 10, 13

    View full-size slide

  55. HELLO SPLIT QUEUES
    HELLO SPLIT QUEUES
    Thursday, October 10, 13

    View full-size slide

  56. Dashboards
    deploy lines
    deploy lines
    Thursday, October 10, 13

    View full-size slide

  57. Supergrep
    Thursday, October 10, 13

    View full-size slide

  58. Thursday, October 10, 13

    View full-size slide

  59. Summary
    •Current setup has scaled to ~150 people
    •Constantly trying to improve the speed of deployment
    •Find weak parts in the process and make them more
    robust/faster
    •Bring Dev closer to Prod
    •Not being able to deploy has the same status as the site
    being down
    Thursday, October 10, 13

    View full-size slide

  60. codeascraft.etsy.com
    www.etsy.com/codeascraft/talks
    etsy.github.com
    www.etsy.com/careers
    Thursday, October 10, 13

    View full-size slide

  61. Questions?
    Thursday, October 10, 13

    View full-size slide

  62. Scaling Deployment at Etsy
    Daniel Schauenberg
    [email protected]
    @mrtazz
    Thursday, October 10, 13

    View full-size slide