Scaling Deployment at Etsy

How we keep deployment fast at Etsy, presented at Scaleconf 2013

Transcript

1. Scaling Deployment at Etsy
   Daniel Schauenberg
   dschauenberg@etsy.com
   @mrtazz
   Thursday, April 18, 13
   

   View Slide

2. Thursday, April 18, 13
   

   View Slide

3. the marketplace we make
   together
   Thursday, April 18, 13
   

   View Slide

4. February 2013
   •1.49 billion page views
   •4,215,169 items sold
   •$94.7 million of goods sold
   •22+ million members
   •800,000+ active shops
   http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
   Thursday, April 18, 13
   

   View Slide

5. LAMMP
   Item by TheBackPackShoppe
   Thursday, April 18, 13
   

   View Slide

6. Item by FrankelPhotos
   Monolithic App
   Thursday, April 18, 13
   

   View Slide

7. No Branching
   Item by NurseryWallArt
   Thursday, April 18, 13
   

   View Slide

8. Deploy Frequency
   Thursday, April 18, 13
   

   View Slide

9. First Day
   Item by flowersandfleurons
   Thursday, April 18, 13
   

   View Slide

10. Thursday, April 18, 13
    

    View Slide

11. IRC
    Thursday, April 18, 13
    

    View Slide

12. Thursday, April 18, 13
    

    View Slide

13. Developer VMs
    •KVM
    •Dev version of full Etsy stack
    •Chef
    •DevTools
    Thursday, April 18, 13
    

    View Slide

14. Thursday, April 18, 13
    

    View Slide

15. Item by codecards
    Thursday, April 18, 13
    

    View Slide

16. % review -r dschauenberg
    Thursday, April 18, 13
    

    View Slide

17. automatically assigned
    automatically assigned
    Thursday, April 18, 13
    

    View Slide

18. Try
    Item by CSSDesign
    Thursday, April 18, 13
    

    View Slide

19. Actually ...
    Thursday, April 18, 13
    

    View Slide

20. CI - The Old Way
    •~10 buildtest servers, single SSD
    •running multiple Jenkins executors
    •tests pegging IO
    •Jenkins is much better at running single executors
    Thursday, April 18, 13
    

    View Slide

21. The Bobs
    Item by Signz
    Thursday, April 18, 13
    

    View Slide

22. The Bobs
    •LXC containers on buildtests
    •Multiple SSDs
    •Labels for heavy/any execution
    •One heavy executor per disk
    Thursday, April 18, 13
    

    View Slide

23. CI/Try
    •~200 Bobs
    •Mostly for try
    •Constant monitoring for slow tests
    Thursday, April 18, 13
    

    View Slide

24. push train
    Item by decomodwalls
    Thursday, April 18, 13
    

    View Slide

25. #push
    •IRC channel to organize push trains
    •Join a train if you want to deploy changes
    •Schedule is planned via the channel topic
    •First in the train is the driver (controls the deploy)
    •Opening hours: 7am - 10pm NYC time
    Thursday, April 18, 13
    

    View Slide

26. #push
    kseever* + jameslee | jpaul | DanielConvissor (c)
    Thursday, April 18, 13
    

    View Slide

27. #push
    bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
    + jlaster (c) | dawa + corey + sandosh + jklein + magera
    + seth_home + mpascual + nathan | bateman | russp (c)
    Thursday, April 18, 13
    

    View Slide

28. pushbot
    •.join
    •.in
    •.good
    •.done
    Thursday, April 18, 13
    

    View Slide

29. pushbot
    Thursday, April 18, 13
    

    View Slide

30. https://github.com/etsy/PushBot
    Thursday, April 18, 13
    

    View Slide

31. Item by EsalonPhotography
    Thursday, April 18, 13
    

    View Slide

32. Deployinator
    Thursday, April 18, 13
    

    View Slide

33. Thursday, April 18, 13
    

    View Slide

34. Thursday, April 18, 13
    

    View Slide

35. stale commits
    Thursday, April 18, 13
    

    View Slide

36. version checks
    buttons disabled
    buttons disabled
    Thursday, April 18, 13
    

    View Slide

37. version checks
    Thursday, April 18, 13
    

    View Slide

38. lock down deploys
    Thursday, April 18, 13
    

    View Slide

39. https://github.com/etsy/deployinator
    Thursday, April 18, 13
    

    View Slide

40. Dashboards
    deploy lines
    deploy lines
    Thursday, April 18, 13
    

    View Slide

41. https://github.com/etsy/dashboard
    Thursday, April 18, 13
    

    View Slide

42. Supergrep
    Thursday, April 18, 13
    

    View Slide

43. https://github.com/etsy/supergrep
    Thursday, April 18, 13
    

    View Slide

44. Thursday, April 18, 13
    

    View Slide

45. Summary
    •Current setup has scaled to ~150 people
    •Constantly trying to improve the speed of deployment
    •Find weak parts in the process and make them more
    robust/faster
    •Bring Dev closer to Prod
    •Not being able to deploy has the same status as the site
    being down
    Thursday, April 18, 13
    

    View Slide

46. codeascraft.etsy.com
    www.etsy.com/codeascraft/talks
    etsy.github.com
    www.etsy.com/careers
    Thursday, April 18, 13
    

    View Slide

47. Questions?
    Thursday, April 18, 13
    

    View Slide

48. Scaling Deployment at Etsy
    Daniel Schauenberg
    dschauenberg@etsy.com
    @mrtazz
    Thursday, April 18, 13
    

    View Slide