Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development, Deployment and Collaboration at Etsy

Development, Deployment and Collaboration at Etsy

At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.

Daniel Schauenberg

March 05, 2014
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. Development, Deployment
    and Collaboration at Etsy
    Daniel Schauenberg
    [email protected]
    @mrtazz

    View Slide

  2. View Slide

  3. @mrtazz

    View Slide

  4. @mrtazz

    View Slide

  5. @mrtazz
    Item by TheBackPackShoppe

    View Slide

  6. http://www.flickr.com/photos/brianglanz/1095706242

    View Slide

  7. avg 50 deploys/
    day

    View Slide

  8. avg n > m deploys/
    day

    View Slide

  9. How comfortable
    are you deploying
    a change right
    now?

    View Slide

  10. @mrtazz
    http://www.flickr.com/photos/renaissancechambara/2349811492
    small change

    View Slide

  11. Config
    Flags
    Item by RocajoStudio

    View Slide

  12. View Slide

  13. “If this is your first
    day at Etsy, you
    deploy the site”

    View Slide

  14. Developer VMs

    View Slide

  15. @mrtazz
    Developer VMs
    • KVM
    • Every engineer has one
    • Fully Chef’d with the Etsy Stack
    • Different sizes and Chef roles

    View Slide

  16. View Slide

  17. Continuous
    Integration

    View Slide

  18. View Slide

  19. @mrtazz
    Continuous Integration
    • Run set of tests before each deploy
    • Full QA suite
    • Princess/Production smoker tests
    • Try (yup, there is one)

    View Slide

  20. http://www.flickr.com/photos/egfocus/6962179321

    View Slide

  21. @mrtazz
    The Bobs
    • LXC virtualized hosts
    • 14/physical hosts
    • Spread over 3 SSDs
    • Most of them attached to try

    View Slide

  22. View Slide

  23. Item by decomodwalls

    View Slide

  24. Deployinator

    View Slide

  25. @mrtazz
    Deployinator
    • 2 Buttons, no ambiguity
    • Overview of current state of deploy
    • Links to Logwatcher and Dashboards
    • Easy to add stacks for new tools to deploy

    View Slide

  26. http://www.flickr.com/photos/jbgeronimi/6363087361

    View Slide

  27. View Slide

  28. Monitoring

    View Slide

  29. @mrtazz
    shouldigraphit.com

    View Slide

  30. @mrtazz
    Monitoring
    • Devs do their feature monitoring
    • Everybody can access all the graphs
    • Dashboard All The Things!
    • Stream All The Logs!

    View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. On Call

    View Slide

  35. If you are writing
    code, you are
    on-call

    View Slide

  36. @mrtazz
    On-Call Schedules
    • ops on-call
    • dev on-call
    • payments on-call
    • support on-call

    View Slide

  37. @mrtazz
    Dev On-Call
    • Scheduled for 6 months
    • On-call roughly every 4 weeks for 1 week
    • L1 and L2 escalations
    • L1 if it’s your first time

    View Slide

  38. Incident Response

    View Slide

  39. @mrtazz
    Incident Response
    • “This graph looks funny”
    • “Hey I just got paged for elevated error rate
    after deploys”
    • “Supergrep is going crazy!!”

    View Slide

  40. View Slide

  41. #warroom

    View Slide

  42. @mrtazz
    #warroom
    • only outage related conversations
    • coordinate investigation, communication,
    countermeasures and monitoring
    • good place to lurk for new engineers

    View Slide

  43. Post Mortems

    View Slide

  44. blameless

    View Slide

  45. Everybody’s invited

    View Slide

  46. Learning Opportunity

    View Slide

  47. Summary

    View Slide

  48. @mrtazz
    Summary
    • These are things that work for *us*
    • Culture is an on-going effort
    • Share everything
    • Encourage learning/teaching

    View Slide

  49. @mrtazz
    Summary
    • Lunch ’n learns
    • DC visits
    • On-call for a day
    • Bootcamps/Senior rotations

    View Slide

  50. codeascraft.com

    etsy.com/codeascraft/talks

    etsy.github.com

    etsy.com/careers

    View Slide

  51. Questions?

    View Slide

  52. Development,
    Deployment and
    Collaboration at Etsy
    Daniel Schauenberg
    [email protected]

    View Slide