Development, Deployment and Collaboration at Etsy

Development, Deployment and Collaboration at Etsy

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

89e0ad1229121f46047977ac547bd7b4?s=128

Daniel Schauenberg

March 05, 2014
Tweet

Transcript

  1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz

  2. None
  3. @mrtazz

  4. @mrtazz

  5. @mrtazz Item by TheBackPackShoppe

  6. http://www.flickr.com/photos/brianglanz/1095706242

  7. avg 50 deploys/ day

  8. avg n > m deploys/ day

  9. How comfortable are you deploying a change right now?

  10. @mrtazz http://www.flickr.com/photos/renaissancechambara/2349811492 small change

  11. Config Flags Item by RocajoStudio

  12. None
  13. “If this is your first day at Etsy, you deploy

    the site”
  14. Developer VMs

  15. @mrtazz Developer VMs • KVM • Every engineer has one

    • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles
  16. None
  17. Continuous Integration

  18. None
  19. @mrtazz Continuous Integration • Run set of tests before each

    deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one)
  20. http://www.flickr.com/photos/egfocus/6962179321

  21. @mrtazz The Bobs • LXC virtualized hosts • 14/physical hosts

    • Spread over 3 SSDs • Most of them attached to try
  22. None
  23. Item by decomodwalls

  24. Deployinator

  25. @mrtazz Deployinator • 2 Buttons, no ambiguity • Overview of

    current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy
  26. http://www.flickr.com/photos/jbgeronimi/6363087361

  27. None
  28. Monitoring

  29. @mrtazz shouldigraphit.com

  30. @mrtazz Monitoring • Devs do their feature monitoring • Everybody

    can access all the graphs • Dashboard All The Things! • Stream All The Logs!
  31. None
  32. None
  33. None
  34. On Call

  35. If you are writing code, you are on-call

  36. @mrtazz On-Call Schedules • ops on-call • dev on-call •

    payments on-call • support on-call
  37. @mrtazz Dev On-Call • Scheduled for 6 months • On-call

    roughly every 4 weeks for 1 week • L1 and L2 escalations • L1 if it’s your first time
  38. Incident Response

  39. @mrtazz Incident Response • “This graph looks funny” • “Hey

    I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!”
  40. None
  41. #warroom

  42. @mrtazz #warroom • only outage related conversations • coordinate investigation,

    communication, countermeasures and monitoring • good place to lurk for new engineers
  43. Post Mortems

  44. blameless

  45. Everybody’s invited

  46. Learning Opportunity

  47. Summary

  48. @mrtazz Summary • These are things that work for *us*

    • Culture is an on-going effort • Share everything • Encourage learning/teaching
  49. @mrtazz Summary • Lunch ’n learns • DC visits •

    On-call for a day • Bootcamps/Senior rotations
  50. codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers

  51. Questions?

  52. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com