Shipping with Confidence

How does Shopify change their production systems - both Hadoop and web tier - without breaking them? This discusses a spectrum of techniques: ring deploys, Docker based isolation, staging environments, logging strategies, and online verifiers to test changes and determine whether a solution is valid before exposing it to the world.

Transcript

1. Shipping with Confidence Harry Brundage harry@shopify.com
2. We’re in the business of changing things without breaking them. - Your goal is to ship something, and once it’s shipped, ship the next thing. The old thing just sticks around. - product devs, infrastructure devs, hadoop devs, whatever
3. Things don’t usually break on their own. We’re pretty good at managing the risk of stuff we already know is gonna break. And, once a thing ships and matures, it usually stays fine until something changes. If the load on the system isn’t changing and you aren’t changing how it works, it’s usually fine. We have things like RAID and software redundancy and whatnot to mitigate failures we can’t predict, and other than that, the thing just stays fine.
4. - So if things don’t break on their own, what makes them break? - Graph of errors in our big data deployment called Starscream over December and January of this year
5. - So if things don’t break on their own, what makes them break? - Graph of errors in our big data deployment called Starscream over December and January of this year
6. We are the biggest threat to our systems’ stability. Yeah you. Me too. If you ask me, its us.
7. Speed of Change Stability of System - So the way people usually talk about managing this risk is as a tension.
8. The faster you change things the more you break things • Less time spent testing • Harder to isolate issues or understand root causes • Harder to keep context and not break things inadvertently • Harder to identify structural improvements that should be made
9. Speed of Change Stability of System I think this is all true, but, there’s a way to win. I think there is another dimension to this whole
10. Speed of Change Stability of System I think this is all true, but, there’s a way to win. I think there is another dimension to this whole
11. Speed of Change Stability of System ??? I think this is all true, but, there’s a way to win. I think there is another dimension to this whole
12. Speed of Change Stability of System Tools I think this is all true, but, there’s a way to win. I think there is another dimension to this whole
13. You can invest in your toolchain to be able to ship quickly and with confidence.
14. • Type systems • Unit testing • Integration testing • Continuous Integration • Quality Assurance teams • Staging environments So these are a bunch of tools you can use to move up on the scale and improve speed without sacrificing stability. All done! JUST KIDDING — BORING
15. • Type systems • Unit testing • Integration testing • Continuous Integration • Quality Assurance teams • Staging environments So these are a bunch of tools you can use to move up on the scale and improve speed without sacrificing stability. All done! JUST KIDDING — BORING
16. For big spooky changes, there’s often that big spooky moment when deploying them.
17. • New backend state store (database, queue) • New dependency versions • Code paths you don’t/can’t understand • Data you don’t/can’t expect • Untested/untestable code paths • Code paths where any errors are extremely costly
18. Some stuff is just really hard to build confidence in before shipping.
19. Log Before You Leap
20. Log Before You Leap
21. If you aren’t sure if a proposed change is going to work, ship it along side the existing implementation without engaging it’s effects, and then verify it does what you expect.
23. require "scientist" module Authenticator include Scientist def allows?(user) return science "example-permissions" do |e| e.use { model.check_user(user).valid? } # old way e.try { user.can?(:read, model) } # new way end end end https://github.com/github/scientist
25. Other Good things • Can verify 100% production correctness before any customer impact. Confidence! • Simultaneous execution forces you to think about intermediate, simultaneous state that was going to happen anyways • Decouples when you ship with when you engage with when you ship next with when you back out
26. Deploy So, at the end of the day, there’s still always this deploy button. Even if we are doing a good job of isolating changes such that they are deployed in isolation
27. ~1.0 % chance that a master commit breaks production enough to fix it immediately. 0.012 reverts / commit 0.045 reverts / merged PR fix forward happens probably as often if not more over counting reverts though, sometimes valid 1.2% revert commit rate 4.5% revert PR rate fix forward happens probably as often if not more over counting reverts though, sometimes valid
28. Commits per day 0 30 60 90 120 2016-01-11 2016-01-18 2016-01-25 2016-02-01 2016-02-08 2016-02-15 Commit Count Observed starscream commit count over the past 6 weeks
29. Estimated Failures over time 0 12.5 25 37.5 50 2016-01-11 2016-01-18 2016-01-25 2016-02-01 2016-02-08 2016-02-15 # Failures at 1 % # Failures at 2.5 % 43 failures 17 failures
30. Undeployed Git Commits over time 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15
31. Undeployed Git Commits over time If we deploy every 6 weeks 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 1716 1640 1580 1522 1498 1494 1468 1372 1256 1156 1062 972 970 968 920 859 801 736 687 669 652 619 618 610 608 573 568 565 505 448 397 326 277 274 259 211 174 135 64
32. Undeployed Git Commits over time If we eploy every 14 days 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 744 668 608 550 526 522 496 400 284 184 90 399 397 395 347 286 228 163 114 96 79 46 45 37 35 573 568 565 505 448 397 326 277 274 259 211 174 135 64
33. Undeployed Git Commits over time If we deploy every 7 days 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 218 142 82 24 526 522 496 400 284 184 90 285 283 281 233 172 114 49 114 96 79 46 45 37 35 296 291 288 228 171 120 49 277 274 259 211 174 135 64
34. Undeployed Git Commits over time If we deploy every 3 days 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 194 118 58 54 30 26 312 216 100 186 92 2 111 109 61 172 114 49 68 50 33 11 10 2 43 8 3 168 108 51 123 52 3 100 85 37 174 135 64
35. Undeployed Git Commits over time If we deploy every 2 days 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 76 118 58 28 4 122 96 216 100 184 90 4 2 109 61 123 65 67 18 50 33 9 8 37 35 8 3 117 57 122 71 52 3 63 48 76 39 135 64
36. Undeployed Git Commits over time If we deploy every day 0 450 900 1350 1800 2016-01-11 2016-01-16 2016-01-21 2016-01-26 2016-01-31 2016-02-05 2016-02-10 2016-02-15 76 60 58 24 4 26 96 116 100 94 90 2 2 48 61 58 65 49 18 17 33 1 8 2 35 5 3 60 57 51 71 49 3 15 48 37 39 71 64
37. Estimated Failures per Deploy @ 1.5 % failure rate 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16
38. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy once 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 26 25 24 23 22 22 22 21 19 17 16 15 15 15 14 13 12 11 10 10 10 9 9 9 9 9 9 8 8 7 6 5 4 4 4 3 3 2 1
39. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy every 14 days 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 11 10 9 8 8 8 7 6 4 3 1 6 6 6 5 4 3 2 2 1 1 1 1 1 1 9 9 8 8 7 6 5 4 4 4 3 3 2 1
40. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy every 7 days 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 3 2 1 0 8 8 7 6 4 3 1 4 4 4 4 3 2 1 2 1 1 1 1 1 1 4 4 4 3 3 2 1 4 4 4 3 3 2 1
41. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy every 3 days 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 3 2 1 1 0 0 5 3 2 3 1 0 2 2 1 3 2 1 1 1 1 0 0 0 1 0 0 3 2 1 2 1 0 2 1 1 3 2 1
42. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy every 2 days 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 1 2 1 0 0 2 1 3 2 3 1 0 0 2 1 2 1 1 0 1 1 0 0 1 1 0 0 2 1 2 1 1 0 1 1 1 1 2 1
43. Estimated Failures per Deploy @ 1.5 % failure rate If we deploy every day 0 7.5 15 22.5 30 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 1 1 1 0 0 0 1 2 2 1 1 0 0 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 1 1 0 0 1 1 1 1 1
44. Continuous Deployment
45. Continuous Deployment
46. continuous delivery [kuh n-tin-yoo-uh s dih-liv-uh-ree] noun, plural
47. continuous delivery [kuh n-tin-yoo-uh s dih-liv-uh-ree] noun, plural 1. Deploying code as often and in as small increments as possible, as soon as the code is deemed ready.
48. continuous delivery [kuh n-tin-yoo-uh s dih-liv-uh-ree] noun, plural 1. Deploying code as often and in as small increments as possible, as soon as the code is deemed ready. 2. Pretty fucking terrifying.
49. Deploys per day 0 30 60 90 120 2016-01-11 2016-01-17 2016-01-23 2016-01-29 2016-02-04 2016-02-10 2016-02-16 Commit Count Deploy Count
50. Observed commits per deploy 0 4 8 12 16 2016-01-11 2016-01-18 2016-01-25 2016-02-01 2016-02-08 2016-02-15
51. Failure rate per deploy 0 0.075 0.15 0.225 0.3 2016-01-11 2016-01-18 2016-01-25 2016-02-01 2016-02-08 2016-02-15
53. https://github.com/Shopify/shipit-engine
54. The easiest way to know if something is going to work in production is to … run it in production.
55. We can’t run half- finished code in production though.
56. We can’t run half- finished code in production though.
57. Developer Production Frontends Backends Database Database Queue Deploy pipeline
58. Developer Production Sort of close to production Frontends Backends Database Database Queue Frontends Backends Dat aba D at Deploy pipeline
59. Developer Production Frontends Backends Database Database Queue Deploy pipeline
60. Developer Production Frontends Backends Database Database Queue
61. Developer Production Frontends Backends Database Database Queue Workload A Workload B
62. Developer Production Frontends Backends Database Database Queue Workload A Workload B
63. Hue Filebrowser + SQL Console HDFS Kafka data Relational data Sched- uler Developer Workload A Workload B Docker build & run infrastructure Shopify Data Platform Production
64. Prodaging
65. Prodaging
66. If you make it safe, you can test in production. Leaving it alone isn’t a viable strategy, it will just get exposed to the real world eventually, may as well embrace that and make it awesome.
67. Risk Profiles if we have time
68. Risk Profiles if we have time
69. OS Libraries Frameworks Applications Features Languages Pressure Users
70. OS Libraries Frameworks Applications Features Languages Pressure Users
71. Thanks! harry@shopify.com We’re hiring! http://www.shopify.com/careers, or just email me!