Deploy, Collaborate and Listen

Deploy, Collaborate and Listen

At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this?

We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.

89e0ad1229121f46047977ac547bd7b4?s=128

Daniel Schauenberg

November 06, 2014
Tweet

Transcript

  1. DEPLOY COLLABORATE & LISTEN Daniel Schauenberg d@etsy.com @mrtazz

  2. ETSY

  3. THE MONOLITH

  4. LAMP

  5. None
  6. HOW COMFORTABLE ARE YOU DEPLOYING A CHANGE RIGHT NOW

  7. MTTR TRUMPS MTBF

  8. IF THIS IS YOUR FIRST DAY AT ETSY YOU DEPLOY

    THE SITE
  9. None
  10. THE DARK PAST

  11. - HINDSIGHT IS 20/20 - THERE WOULD BE NO ETSY

    - I WASN'T AROUND FOR THIS (the grain of salt disclaimer)
  12. DARK LESS FUN

  13. ARCHITECTURE OVERVIEW - UBUNTU - POSTGRESQL - LIGHTTPD - PHP/PYTHON

  14. SINGLE BIG DATABASE

  15. BUSINESS LOGIC IN STORED PROCEDURES

  16. None
  17. SILOS

  18. DEV ! DBA ! OPS

  19. SPROUTER

  20. A SOFTWARE MANIFESTATION OF SILOS

  21. SITE UPTIME WASN'T GOOD

  22. MORE UPTIME > LESS UPTIME

  23. None
  24. Trust THE PEOPLE

  25. None
  26. SPROUTER

  27. NO MORE MAINFRAME MASTER

  28. HORIZONTAL SCALING (A SINGLE BOX ONLY SCALES SO FAR)

  29. MASTER-MASTER REPLICATED MYSQL SHARDS

  30. FLICKR DNA

  31. DEPLOYINATOR

  32. DEPLOY != RELEASE

  33. CONFIG FLAGS

  34. if Feature::isEnabled($feature) { // new hawtness } else { //

    nothing to see here }
  35. METRICS!

  36. None
  37. STATSD "HOW DO YOU KNOW THIS WORKS IN PRODUCTION?"

  38. CHEF

  39. <3 CHEF

  40. DEVELOPER VMS - KVM - EVERY ENGINEER HAS ONE -

    FULLY CHEF’D WITH THE ETSY STACK - DIFFERENT SIZES AND CHEF ROLES
  41. Try

  42. BLAMELESS POST MORTEMS

  43. None
  44. None
  45. OVERHAULS & ITERATIONS

  46. CULTURE & TOOLS

  47. HUMANS ARE AWESOME

  48. NOBODY COMES TO WORK TO DO A BAD JOB

  49. Trust YOUR CO- WORKERS

  50. THERE IS A LOT OF KNOWLEDGE IN YOUR ENGINEERING TEAM

  51. DEPLOY (AS OFTEN AS IT MAKES SENSE)

  52. COLLABORATE (EVEN IF YOU THINK YOU DON'T HAVE TO)

  53. None
  54. LISTEN (TO PROBLEMS AND EXPERIENCES OF YOUR COWORKERS)

  55. CODEASCRAFT.COM ETSY.COM/CODEASCRAFT/TALKS ETSY.COM/CAREERS

  56. THANK YOU!

  57. DEPLOY COLLABORATE & LISTEN Daniel Schauenberg d@etsy.com @mrtazz