Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development, Deployment and Collaboration at Etsy

Development, Deployment and Collaboration at Etsy

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

Daniel Schauenberg

March 05, 2014

More Decks by Daniel Schauenberg

Other Decks in Technology


  1. @mrtazz Developer VMs • KVM • Every engineer has one

    • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles
  2. @mrtazz Continuous Integration • Run set of tests before each

    deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one)
  3. @mrtazz The Bobs • LXC virtualized hosts • 14/physical hosts

    • Spread over 3 SSDs • Most of them attached to try
  4. @mrtazz Deployinator • 2 Buttons, no ambiguity • Overview of

    current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy
  5. @mrtazz Monitoring • Devs do their feature monitoring • Everybody

    can access all the graphs • Dashboard All The Things! • Stream All The Logs!
  6. @mrtazz Dev On-Call • Scheduled for 6 months • On-call

    roughly every 4 weeks for 1 week • L1 and L2 escalations • L1 if it’s your first time
  7. @mrtazz Incident Response • “This graph looks funny” • “Hey

    I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!”
  8. @mrtazz #warroom • only outage related conversations • coordinate investigation,

    communication, countermeasures and monitoring • good place to lurk for new engineers
  9. @mrtazz Summary • These are things that work for *us*

    • Culture is an on-going effort • Share everything • Encourage learning/teaching
  10. @mrtazz Summary • Lunch ’n learns • DC visits •

    On-call for a day • Bootcamps/Senior rotations