How Etsy deploys confidently 50+ times a day

Etsy is an international marketplace which have around 1000 employees around the world.
Every 15 min, engineers deploys the website and keep it stable.
Like many other companies at their early stages, deploys at Etsy needed an army of engineers during 6 to 14 hours, and ended badly most of the time.

Over time, Etsy managed to put in place processes and tools to improve the platform thanks to Continuous Delivery philosophy.

Today, every employee can deploy with confidence, even pets!

In this talk, I will show how Etsy managed to reach their objectives through the philosophy that engineers put in the continuous delivery pipeline, the tools & integrations we've set up (pushbot on chat room, feature flags, etc.) and how we're assuring stability in our code changes, from the developer's machine to the production and its monitoring.

Transcript

1. How Etsy deploys confidently 50+ times a day Sylvain Mauduit @swop Senior Software Engineer, Etsy JOIND-IN http://bit.ly/2eHyfZN LANYRD http://bit.ly/2dY34Lf SLIDES http://bit.ly/2dY1mcW
2. 2 2005 FOUNDED 921 EMPLOYEES 27M ACTIVE USERS AND PEOPLE BUYING OR SELLING FROM Nearly every country in the world 40M ITEMS FOR SALE About Etsy
3. About Etsy 3 HEADQUARTERED IN Brooklyn, NY Paris Berlin Dublin Hudson WITH ADDITIONAL OFFICES IN London Melbourne San Francisco Tokyo Toronto
4. About 4 2014 ACQUISITION 40 EMPLOYEES 2008 FOUNDED 2nd LARGEST ETSY ENGINEERING OFFICE AFTER BROOKLYN
5. 5 Image source: pexels.com
7. 7 Image source: pexels.com
8. 8 The wrong path Large change sets Infrequent deploys Weak confidence in deploy success Significant time spent deploying Low ability to experiment/iterate/react Developer stress
9. Failure is inevitable … BUT EXPENSIVE FAILURE IS NOT !
10. THE RIGHT PATH Reduce the cost of failure … to reduce fear during deployment
11. Continuous delivery
12. Continuous delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. 
 It aims at building, testing, and releasing software faster and more frequently. 
 The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. with —, Wikipedia
13. Continuous delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. 
 It aims at building, testing, and releasing software faster and more frequently. 
 The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. with —, Wikipedia
14. Building Ok, time to code! Deploying We're ready! Ship this! Failing If we still fail... Monitoring It's alive!! Oh wait...
15. 1. Building 2. Deploying 4. Failing 3. Monitoring Today’s schedule | Show me the menu… Recap Tips for small companies Analyse
16. 1. Building Ok, time to code!
17. Small chunks Image source: pexels.com
18. ~100 loc
19. Easier code reviews Image source: pexels.com
20. Small company | Tip YOU DON’T HAVE A CODE REVIEW TOOL? Try pair-programming! (and continue to do it even after…)
21. Easy to integrate & share code Image source: permacultureprinciples.com
22. Manual testing with QA Security check with Sec team Image source: pexels.com Image source: pexels.com
23. Automated tests Image source: pexels.com
25. Try Image source: androidheadlines.com
26. $ try
 Executing DEFAULT sub-jobs: try-php-cs3 try-js-integration try-php-cs4 try-orm-tests try-scss-compile try-php-cs1 try-runkit-tests try-php- cs2 try-composer-lint try-build-js try-phpunit-code-sniffer try-scss- lint try-validate-model-schemas try-replication-tests try-gci-tests try-phan try-extraction-sniffs try-rodeo-sniffer try-api-check try- styleguide-compile try-feature-config-lint try-smarty-lint try- mustache-lint try-build-mustache try-api-compile-routes try-build- templates try-integration-tests-part1 try-fontlint try-eslint try- cache-tests try-file-tests try-validate-crons try-integration-tests- part6 try-js-unit-tests try-unit-tests-part1 try-integration-tests- part2 try-unit-tests-part2 try-integration-tests-part3 try- integration-tests-part4 try-integration-tests-part5
 ......... try ( http://try.etsycorp.com/job/try/220033/ )
28. Small company | Tip SAAS TOOLS ARE YOUR FRIENDS TravisCI : Run tests when pushing code / open a PR
29. Get feedback early Image source: williamsalternative.com
30. Features flags etsy/feature
31. Soak testing A B
32. Internal staff testing A ADMINS B
33. Beta release / public prototypes A BETA TESTERS B
34. Progressive deployments A B 90% 10%
35. Switch off features on emergencies
36. A/B testing CTRL 50% A 50% ?
37. Small company | Tip FLAGS: KEEP IT SIMPLE AT START Config : Store flag values in an array Usage : Just use <if> statements, using config array values
38. Building | Let’s recap…
39. Building | Let’s recap… AGILITY
 Prepare small chunks of code.
 TESTS
 Implicate QA early.
 Find an easy way for engineers to run the tests fast & easy.
 EXPERIMENTS
 Shipping features progressively give you more control.
40. 2. Deploying We're ready! Ship this!
41. Deploy smooth workflow: Let people keep focus on the quality of their code instead of the reliability/stability of the deploy platform. Image source: pexels.com
45. Integrate several systems using chat as a glue both humains & bots ORCHESTRATORS +
46. Deploy train 1 to 8 engineers/train PROD CYCLE
47. AUTOMATIC TESTS WITH SMAUDUIT & RLERDORF CODE CODE PUSH .in .join .join smauduit rlerdorf : You're up smauduit rlerdorf : everyone is ready Starting build for job : qa Project qa build SUCCESS in 3 min 44 sec (link) NEW CYCLE .in .join
48. MANUAL CHECKS ON PRINCESS AUTOMATIC TESTS ON PRINCESS DEPLOY ON PRINCESS smauduit rlerdorf : qa tests have passed Starting build for job : princess Project princess build SUCCESS in 1 min 59 sec (link) and everyone has run Try recently. w00t! smauduit rlerdorf : Your code is on princess .good .good
49. SMOKE TESTS ON PROD DEPLOY ON PROD smauduit rlerdorf : princess tests have passed Starting build for job : prod Project prod build SUCCESS in 28 sec (link) smauduit rlerdorf : Your code is live. Time to watch graphs: (link)
50. MANUAL CHECKS ON PROD smauduit rlerdorf : prod tests have passed ogarcia : You're up .good NEW CYCLE .good .done .in Total deploy time:
 16 min
51. 51 Confidence Optimal queue size Normalised communication Improved visibility Historical records (post-mortems <3)
52. Small company | Tip PUSH TRAIN & DEPLOYMENT PIPELINE:
 SAAS TOOLS ARE STILL YOUR FRIENDS! SlashDeploy.io : Slack bot (deploy & locking) Dockbit : Deployment + push queue pipeline GitLab : Continuous delivery pipeline
53. Speed Image source: pexels.com
55. Duration: 3 min 44 sec ~22.000 tests, 81.000 assertions PHPUNIT TEST SUITE
56. Long test cases SOLUTION 1
 THROW MORE MONEY AT IT SOLUTION 2
 SPOT THEM, TRASH THEM
57. Deploy often IT WILL BRING YOU SPEED & FLEXIBILITY
59. Trust Image source: pexels.com
60. Deploying | Let’s recap…
61. Deploying | Let’s recap… SIMPLICITY
 Automate things & make deploys easy
 but let humans keep control.
 SPEED
 Keep your deploy time low.
 It will allow you to deploy often, and quickly in prod.
 TESTS
 Drop long or unstable tests.
 Run automatic tests at deploy time.
62. 3. Monitoring It's alive!! Oh wait...
63. Graphs If something goes wrong, graphs will move
66. Dog fooding etsy/statsd graphite-project grafana/grafana etsy/dashboard
67. Small company | Tip SAAS TOOLS ARE YOUR FRIENDS NewRelic : Basic tech metrics monitoring Hosted Graphite : Business metrics
68. Alerting Because you’ll forget to watch the ONE graph Image source: istockphoto.com
69. etsy/411
70. Logs Easily spot which code is related to a specific error Image source: pexels.com
72. No rollback. We only roll forward Image source: pexels.com
73. Monitoring | Let’s recap…
74. Monitoring | Let’s recap… LOGS & GRAPHS
 Adding logs or graphs is cheap
 but engineers will thank you if you do it.
 CONTEXT
 Decorate them with context info as much as possible.
 DON’T LOOK BACK
 Always move forward, forget rollbacks.
75. 4. Failing If we still fail...
76. Despite all of this you will still fail Image source: pexels.com
77. Learn from failure with blameless postmortems Image source: pexels.com
78. Celebrate failure Image source: pexels.com
82. Failing | Let’s recap…
83. Failing | Let’s recap… KEEP LEARNING
 Despite all efforts, you will still fail
 so learn from your failures.
 CELEBRATE
 It will remind you that you shouldn’t be afraid to make mistakes.
84. 84 START SMALL (we did…)
 Take one step at a time, experiments small things.
 If something don’t work, you can always go back. Conclusion | Some last few tips…
85. 85 Have a story around maintaining quality. DON’T COPY Find a process which works for you & your team. Conclusion | Some last few tips…
86. Sylvain Mauduit @swop Senior Software Engineer, Etsy JOIND-IN http://bit.ly/2eHyfZN LANYRD http://bit.ly/2dY34Lf SLIDES http://bit.ly/2dY1mcW We’re hiring!