Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Deployment at Etsy

Scaling Deployment at Etsy

How we keep deployment fast at Etsy, presented at Scaleconf 2013

Daniel Schauenberg

April 18, 2013
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. Scaling Deployment at Etsy
    Daniel Schauenberg
    [email protected]
    @mrtazz
    Thursday, April 18, 13

    View Slide

  2. Thursday, April 18, 13

    View Slide

  3. the marketplace we make
    together
    Thursday, April 18, 13

    View Slide

  4. February 2013
    •1.49 billion page views
    •4,215,169 items sold
    •$94.7 million of goods sold
    •22+ million members
    •800,000+ active shops
    http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
    Thursday, April 18, 13

    View Slide

  5. LAMMP
    Item by TheBackPackShoppe
    Thursday, April 18, 13

    View Slide

  6. Item by FrankelPhotos
    Monolithic App
    Thursday, April 18, 13

    View Slide

  7. No Branching
    Item by NurseryWallArt
    Thursday, April 18, 13

    View Slide

  8. Deploy Frequency
    Thursday, April 18, 13

    View Slide

  9. First Day
    Item by flowersandfleurons
    Thursday, April 18, 13

    View Slide

  10. Thursday, April 18, 13

    View Slide

  11. IRC
    Thursday, April 18, 13

    View Slide

  12. Thursday, April 18, 13

    View Slide

  13. Developer VMs
    •KVM
    •Dev version of full Etsy stack
    •Chef
    •DevTools
    Thursday, April 18, 13

    View Slide

  14. Thursday, April 18, 13

    View Slide

  15. Item by codecards
    Thursday, April 18, 13

    View Slide

  16. % review -r dschauenberg
    Thursday, April 18, 13

    View Slide

  17. automatically assigned
    automatically assigned
    Thursday, April 18, 13

    View Slide

  18. Try
    Item by CSSDesign
    Thursday, April 18, 13

    View Slide

  19. Actually ...
    Thursday, April 18, 13

    View Slide

  20. CI - The Old Way
    •~10 buildtest servers, single SSD
    •running multiple Jenkins executors
    •tests pegging IO
    •Jenkins is much better at running single executors
    Thursday, April 18, 13

    View Slide

  21. The Bobs
    Item by Signz
    Thursday, April 18, 13

    View Slide

  22. The Bobs
    •LXC containers on buildtests
    •Multiple SSDs
    •Labels for heavy/any execution
    •One heavy executor per disk
    Thursday, April 18, 13

    View Slide

  23. CI/Try
    •~200 Bobs
    •Mostly for try
    •Constant monitoring for slow tests
    Thursday, April 18, 13

    View Slide

  24. push train
    Item by decomodwalls
    Thursday, April 18, 13

    View Slide

  25. #push
    •IRC channel to organize push trains
    •Join a train if you want to deploy changes
    •Schedule is planned via the channel topic
    •First in the train is the driver (controls the deploy)
    •Opening hours: 7am - 10pm NYC time
    Thursday, April 18, 13

    View Slide

  26. #push
    kseever* + jameslee | jpaul | DanielConvissor (c)
    Thursday, April 18, 13

    View Slide

  27. #push
    bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
    + jlaster (c) | dawa + corey + sandosh + jklein + magera
    + seth_home + mpascual + nathan | bateman | russp (c)
    Thursday, April 18, 13

    View Slide

  28. pushbot
    •.join
    •.in
    •.good
    •.done
    Thursday, April 18, 13

    View Slide

  29. pushbot
    Thursday, April 18, 13

    View Slide

  30. https://github.com/etsy/PushBot
    Thursday, April 18, 13

    View Slide

  31. Item by EsalonPhotography
    Thursday, April 18, 13

    View Slide

  32. Deployinator
    Thursday, April 18, 13

    View Slide

  33. Thursday, April 18, 13

    View Slide

  34. Thursday, April 18, 13

    View Slide

  35. stale commits
    Thursday, April 18, 13

    View Slide

  36. version checks
    buttons disabled
    buttons disabled
    Thursday, April 18, 13

    View Slide

  37. version checks
    Thursday, April 18, 13

    View Slide

  38. lock down deploys
    Thursday, April 18, 13

    View Slide

  39. https://github.com/etsy/deployinator
    Thursday, April 18, 13

    View Slide

  40. Dashboards
    deploy lines
    deploy lines
    Thursday, April 18, 13

    View Slide

  41. https://github.com/etsy/dashboard
    Thursday, April 18, 13

    View Slide

  42. Supergrep
    Thursday, April 18, 13

    View Slide

  43. https://github.com/etsy/supergrep
    Thursday, April 18, 13

    View Slide

  44. Thursday, April 18, 13

    View Slide

  45. Summary
    •Current setup has scaled to ~150 people
    •Constantly trying to improve the speed of deployment
    •Find weak parts in the process and make them more
    robust/faster
    •Bring Dev closer to Prod
    •Not being able to deploy has the same status as the site
    being down
    Thursday, April 18, 13

    View Slide

  46. codeascraft.etsy.com
    www.etsy.com/codeascraft/talks
    etsy.github.com
    www.etsy.com/careers
    Thursday, April 18, 13

    View Slide

  47. Questions?
    Thursday, April 18, 13

    View Slide

  48. Scaling Deployment at Etsy
    Daniel Schauenberg
    [email protected]
    @mrtazz
    Thursday, April 18, 13

    View Slide