Scaling Deployment at Etsy

How we keep deployment fast at Etsy, presented at Scaleconf 2013

Transcript

1. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, April 18, 13
2. Thursday, April 18, 13
3. the marketplace we make together Thursday, April 18, 13
4. February 2013 •1.49 billion page views •4,215,169 items sold •$94.7 million of goods sold •22+ million members •800,000+ active shops http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, April 18, 13
5. LAMMP Item by TheBackPackShoppe Thursday, April 18, 13
6. Item by FrankelPhotos Monolithic App Thursday, April 18, 13
7. No Branching Item by NurseryWallArt Thursday, April 18, 13
8. Deploy Frequency Thursday, April 18, 13
9. First Day Item by flowersandfleurons Thursday, April 18, 13
10. Thursday, April 18, 13
11. IRC Thursday, April 18, 13
12. Thursday, April 18, 13
13. Developer VMs •KVM •Dev version of full Etsy stack •Chef •DevTools Thursday, April 18, 13
14. Thursday, April 18, 13
15. Item by codecards Thursday, April 18, 13
16. % review -r dschauenberg Thursday, April 18, 13
17. automatically assigned automatically assigned Thursday, April 18, 13
18. Try Item by CSSDesign Thursday, April 18, 13
19. Actually ... Thursday, April 18, 13
20. CI - The Old Way •~10 buildtest servers, single SSD •running multiple Jenkins executors •tests pegging IO •Jenkins is much better at running single executors Thursday, April 18, 13
21. The Bobs Item by Signz Thursday, April 18, 13
22. The Bobs •LXC containers on buildtests •Multiple SSDs •Labels for heavy/any execution •One heavy executor per disk Thursday, April 18, 13
23. CI/Try •~200 Bobs •Mostly for try •Constant monitoring for slow tests Thursday, April 18, 13
24. push train Item by decomodwalls Thursday, April 18, 13
25. #push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, April 18, 13
26. #push <prod> kseever* + jameslee | jpaul | DanielConvissor (c) Thursday, April 18, 13
27. #push bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c) Thursday, April 18, 13
28. pushbot •.join •.in •.good •.done Thursday, April 18, 13
29. pushbot Thursday, April 18, 13
30. https://github.com/etsy/PushBot Thursday, April 18, 13
31. Item by EsalonPhotography Thursday, April 18, 13
32. Deployinator Thursday, April 18, 13
33. Thursday, April 18, 13
34. Thursday, April 18, 13
35. stale commits Thursday, April 18, 13
36. version checks buttons disabled buttons disabled Thursday, April 18, 13
37. version checks Thursday, April 18, 13
38. lock down deploys Thursday, April 18, 13
39. https://github.com/etsy/deployinator Thursday, April 18, 13
40. Dashboards deploy lines deploy lines Thursday, April 18, 13
41. https://github.com/etsy/dashboard Thursday, April 18, 13
42. Supergrep Thursday, April 18, 13
43. https://github.com/etsy/supergrep Thursday, April 18, 13
44. Thursday, April 18, 13
45. Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, April 18, 13
46. codeascraft.etsy.com www.etsy.com/codeascraft/talks etsy.github.com www.etsy.com/careers Thursday, April 18, 13
47. Questions? Thursday, April 18, 13
48. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, April 18, 13