Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploy, Collaborate and Listen

Deploy, Collaborate and Listen

At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this?

We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.

Daniel Schauenberg

November 06, 2014
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. DEPLOY
    COLLABORATE & LISTEN
    Daniel Schauenberg
    [email protected]
    @mrtazz

    View Slide

  2. ETSY

    View Slide

  3. THE MONOLITH

    View Slide

  4. LAMP

    View Slide

  5. View Slide

  6. HOW COMFORTABLE
    ARE YOU
    DEPLOYING A CHANGE
    RIGHT NOW

    View Slide

  7. MTTR
    TRUMPS
    MTBF

    View Slide

  8. IF THIS IS YOUR
    FIRST DAY AT ETSY
    YOU DEPLOY THE SITE

    View Slide

  9. View Slide

  10. THE DARK PAST

    View Slide


  11. View Slide

  12. - HINDSIGHT IS 20/20
    - THERE WOULD BE NO ETSY
    - I WASN'T AROUND FOR THIS
    (the grain of salt disclaimer)

    View Slide

  13. DARK
    LESS FUN

    View Slide

  14. ARCHITECTURE OVERVIEW
    - UBUNTU
    - POSTGRESQL
    - LIGHTTPD
    - PHP/PYTHON

    View Slide

  15. SINGLE BIG
    DATABASE

    View Slide

  16. BUSINESS LOGIC
    IN STORED
    PROCEDURES

    View Slide

  17. View Slide

  18. SILOS

    View Slide

  19. DEV ! DBA ! OPS

    View Slide

  20. SPROUTER

    View Slide

  21. A SOFTWARE
    MANIFESTATION
    OF SILOS

    View Slide

  22. SITE UPTIME
    WASN'T GOOD

    View Slide

  23. MORE UPTIME > LESS UPTIME

    View Slide

  24. View Slide

  25. Trust
    THE
    PEOPLE

    View Slide

  26. View Slide

  27. SPROUTER

    View Slide

  28. NO MORE
    MAINFRAME
    MASTER

    View Slide

  29. HORIZONTAL
    SCALING
    (A SINGLE BOX ONLY SCALES SO FAR)

    View Slide

  30. MASTER-MASTER
    REPLICATED
    MYSQL SHARDS

    View Slide

  31. FLICKR DNA

    View Slide

  32. DEPLOYINATOR

    View Slide

  33. DEPLOY
    !=
    RELEASE

    View Slide

  34. CONFIG FLAGS

    View Slide

  35. if Feature::isEnabled($feature) {
    // new hawtness
    } else {
    // nothing to see here
    }

    View Slide

  36. METRICS!

    View Slide

  37. View Slide

  38. STATSD
    "HOW DO YOU KNOW THIS WORKS IN
    PRODUCTION?"

    View Slide

  39. CHEF

    View Slide

  40. <3 CHEF

    View Slide

  41. DEVELOPER VMS
    - KVM
    - EVERY ENGINEER HAS ONE
    - FULLY CHEF’D WITH THE ETSY STACK
    - DIFFERENT SIZES AND CHEF ROLES

    View Slide

  42. Try

    View Slide

  43. BLAMELESS
    POST MORTEMS

    View Slide

  44. View Slide

  45. View Slide

  46. OVERHAULS
    &
    ITERATIONS

    View Slide

  47. CULTURE & TOOLS

    View Slide

  48. HUMANS ARE
    AWESOME

    View Slide

  49. NOBODY COMES
    TO WORK TO DO A
    BAD JOB

    View Slide

  50. Trust YOUR CO-
    WORKERS

    View Slide

  51. THERE IS A LOT OF KNOWLEDGE IN YOUR
    ENGINEERING TEAM

    View Slide

  52. DEPLOY
    (AS OFTEN AS IT MAKES SENSE)

    View Slide

  53. COLLABORATE
    (EVEN IF YOU THINK YOU DON'T HAVE TO)

    View Slide

  54. View Slide

  55. LISTEN
    (TO PROBLEMS AND EXPERIENCES OF YOUR COWORKERS)

    View Slide

  56. CODEASCRAFT.COM
    ETSY.COM/CODEASCRAFT/TALKS
    ETSY.COM/CAREERS

    View Slide

  57. THANK YOU!

    View Slide

  58. DEPLOY
    COLLABORATE & LISTEN
    Daniel Schauenberg
    [email protected]
    @mrtazz

    View Slide