Development, Deployment and Collaboration at Etsy

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

Transcript

1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz
3. @mrtazz
4. @mrtazz
5. @mrtazz Item by TheBackPackShoppe
6. http://www.flickr.com/photos/brianglanz/1095706242
7. avg 50 deploys/ day
8. avg n > m deploys/ day
9. How comfortable are you deploying a change right now?
10. @mrtazz http://www.flickr.com/photos/renaissancechambara/2349811492 small change
11. Config Flags Item by RocajoStudio
13. “If this is your first day at Etsy, you deploy the site”
14. Developer VMs
15. @mrtazz Developer VMs • KVM • Every engineer has one • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles
17. Continuous Integration
19. @mrtazz Continuous Integration • Run set of tests before each deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one)
20. http://www.flickr.com/photos/egfocus/6962179321
21. @mrtazz The Bobs • LXC virtualized hosts • 14/physical hosts • Spread over 3 SSDs • Most of them attached to try
23. Item by decomodwalls
24. Deployinator
25. @mrtazz Deployinator • 2 Buttons, no ambiguity • Overview of current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy
26. http://www.flickr.com/photos/jbgeronimi/6363087361
28. Monitoring
29. @mrtazz shouldigraphit.com
30. @mrtazz Monitoring • Devs do their feature monitoring • Everybody can access all the graphs • Dashboard All The Things! • Stream All The Logs!
34. On Call
35. If you are writing code, you are on-call
36. @mrtazz On-Call Schedules • ops on-call • dev on-call • payments on-call • support on-call
37. @mrtazz Dev On-Call • Scheduled for 6 months • On-call roughly every 4 weeks for 1 week • L1 and L2 escalations • L1 if it’s your first time
38. Incident Response
39. @mrtazz Incident Response • “This graph looks funny” • “Hey I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!”
41. #warroom
42. @mrtazz #warroom • only outage related conversations • coordinate investigation, communication, countermeasures and monitoring • good place to lurk for new engineers
43. Post Mortems
44. blameless
45. Everybody’s invited
46. Learning Opportunity
47. Summary
48. @mrtazz Summary • These are things that work for *us* • Culture is an on-going effort • Share everything • Encourage learning/teaching
49. @mrtazz Summary • Lunch ’n learns • DC visits • On-call for a day • Bootcamps/Senior rotations
50. codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers
51. Questions?
52. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com