Put a Button on it: Removing Barriers to Going Fast.

Bfd6b681ec6e8c9bef82ff0521c364f7?s=47 kastner
September 26, 2011

Put a Button on it: Removing Barriers to Going Fast.

For most companies, pushing code to production is a slow and painful process that can span hours or days, and is done very infrequently because of this and other general anxieties behind public facing launches. At Etsy, we built a tool called Deployinator to make it a painless process for everyone to push code at will. We pair this with a suite of tests, graphs, and metrics to make high velocity change safe and transparent.

Bfd6b681ec6e8c9bef82ff0521c364f7?s=128

kastner

September 26, 2011
Tweet

Transcript

  1. kastner@gmail.com / @kastner jgoulah@gmail.com / @johngoulah Put a Button on

    it Removing Barriers to Going Fast
  2. None
  3. 1+ Billion Pageviews Per Month

  4. Speed

  5. ~80 Engineers

  6. Barriers

  7. Deployinator

  8. None
  9. We’re Hiring http://etsy.com/jobs

  10. None
  11. Optimization

  12. Happiness

  13. Sad Engineers are Bad Engineers

  14. Barriers

  15. Process

  16. “Process is an embedded reaction to prior stupidity” – Clay

    Shirky
  17. Fear

  18. None
  19. Correctness

  20. Text "My strategy has always been: be wrong as fast

    as we can, Which basically means, we're gonna screw up, … Let's not be afraid of that." – Andrew Stanton
  21. Legacy

  22. How?

  23. Communication

  24. Trust

  25. Openness

  26. Constant Improvement

  27. Then vs. Now

  28. 2009

  29. 2009 2011

  30. 2009 Single Deploy Master

  31. 2009 2011 Single Deploy Master Anyone Can Deploy

  32. 2009 Rollback in Fear

  33. 2009 2011 Rollback in Fear Roll Forward to Victory

  34. 2009 All Day Deploys

  35. 2009 2011 All Day Deploys Deploy All Day

  36. How… for reals

  37. So, what does Etsy optimize for?

  38. MTTR vs. MTBF

  39. Default to Open

  40. Culture First

  41. If it’s your first day at Etsy, you must deploy!

  42. None
  43. Doing the dumbest thing that can possibly work

  44. Committing to Trunk

  45. Branching in Code

  46. if ($cfg["new_search"]) { // new hotness $resp = search_solr(); }

    else { // old busted $resp = search_grep(); }
  47. Configuration is Code

  48. $cfg["new_search"] = true;

  49. Blameless Post-mortems

  50. Databases are Different

  51. Does it Work?

  52. None
  53. None
  54. 6 Change-related incidents in 2010

  55. November code-freeze

  56. November code-freeze 721 Deploys

  57. Deployinator

  58. None
  59. None
  60. None
  61. Capistrano

  62. Puppet Capistrano

  63. Fabric Puppet Capistrano

  64. Fabric Chef Puppet Capistrano

  65. None
  66. None
  67. None
  68. None
  69. None
  70. “Framework”

  71. STACK=search rake new_stack

  72. run_cmd %Q{cap deploy}

  73. log_and_stream "Starting Web deploy<br>"

  74. log_and_shout({ :old_build => old_build, :build => build, :send_email => true

    })
  75. Communication

  76. None
  77. None
  78. None
  79. None
  80. None
  81. None
  82. Dashboard Driven Development

  83. None
  84. 5:27 < devbot> PRODUCTION deployed by vthommeret version: 7f08b7a-20110722-191739-UTC took:

    131.042598 seconds diff: http://deployinator.etsycorp.com/diff/web/0cbf2f0/7f08b7a
  85. Etsy Confidential

  86. Deployinator Deploy Host Web 1 Web 2 Web N

  87. Deployinator Deploy Host Web 1 Web 2 Web N ssh

  88. Deployinator Deploy Host Web 1 Web 2 Web N ssh

    dsh
  89. Deployinator Deploy Host Web 1 Web 2 Web N ssh

    dsh
  90. Deployinator Deploy Host Web 1 Web 2 Web N ssh

    dsh Rsync
  91. What’s Stopping You?

  92. Know what you’re optimizing for

  93. Deployinator is Open Source Software!

  94. Deployinator is Open Source Software! http://github.com/etsy/deployinator

  95. Deployinator is Open Source Software! http://github.com/etsy/deployinator (right after this talk)

  96. Thank you

  97. Thank you http://github.com/etsy/deployinator