Move Fast and Break Nothing

Move fast and break things. Scale later. Ship it. They're the rallying cries of our industry. While they've helped propel us forward, they're still crude solutions. We're moving fast, but what specifically is okay for us to break, and how much can we break it? How do we quickly ship really difficult changes? The next step is being smarter about the answers to these questions. Turns out, there are some really cool approaches we can take these days to keep us fast while your company grows.

You can also read the full text accompaniment to this talk at: http://zachholman.com/talk/move-fast-break-nothing/

Transcript

1. break nothing move fast &
3. Move fast and break things — Mark Zuckerberg, Facebook “ ”
4. Move fast and fix things — Mark Zuckerberg, Facebook “ ”
5. Move fast with stable infra — Mark Boringberg, Facebook “ ”
6. Move fast and be bold “ — Miguel Velazquez, Facebook ”
7. Drop the ‘the’ — it’s cleaner. “ — JUSTIN TIMBERLAKE SWOOOOOON ”
8. good philosophy moving quickly and safely is
9. what does it mean in practice?
10. @ holman
11. @ holman !
14. got me thinking about building environments where success is a byproduct
15. code 3 areas , process& talk
16. code
17. move fast
18. people like fast better product
19. people like fast see what works
20. people like fast momentum
21. break things and
22. break things fix later and… stuff and
23. some things you can’t break
24. billing· permissions · upgrades· migrations
25. how do you move fast in dicey code?
26. we redid github’s permissions code
27. hella scary.
28. .01% errors unacceptable =
29. 0 errors mandatory =
30. how do we ship this fast + safe?
31. tests are great, of course
32. production can differ from what your tests expect
33. production can differ from what you even expect
34. ensure that your code doesn’t change production behavior
35. parallel codepaths
36. request
37. user.old_code always gets run request
38. user.new_code also gets run request user.old_code always gets run
39. user.new_code also gets run request user.old_code always gets run response
40. run both, compare
41. /github/dat-science p
42. science “new-auth” do | e | e.control { user.slow_auth } e.candidate { user.fast_auth } end
43. science “new-auth” do | e | e.control { user.slow_auth } e.candidate { user.fast_auth } end always runs and returns your result
44. can be run as a percentage science “new-auth” do | e | e.control { user.slow_auth } e.candidate { user.fast_auth } end
45. collects the two results science “new-auth” do | e | e.control { user.slow_auth } e.candidate { user.fast_auth } end
46. attempts vs. mismatches
47. 75th and 99th percentile performance
48. quickly iterate until you can prove safety
49. existing process build into your
50. what do you have today that you can already use?
51. each layer of process added is expensive
52. instead grow process laterally
53. tests rule everything around me
54. ci as code maintenance
55. failing test: removing html class without cleaning css
56. failing test: removing css without removing html class
57. failing test: adding <img> that’s not on our cdn
58. failing test: invalid css (scss linting)
59. automates gruntwork out of code review
60. we test blog posts, too
61. failing test: images off our cdn
62. failing test: image non-retina or too big
63. failing test: using an oft-used phrase
64. pro- cess
65. what’s important?
66. feature launches need: · designers developers · product managers · qa · lawyers · marketing· ops
67. text files are rad
68. 
69.  anpp (apple new product process)
70.  (apple new product process) giant-ass checklist anpp
71.  (apple new product process) lists at tim-level and small team-level anpp
72.  (apple new product process) happens as very first step anpp
73. d
74. d ship lists
75. d ship lists happens often as last step
76. d ship lists all the todos for teams for their +1
78. simple todo lists make you think less
79. ownership
80. don’t throw code over the wall
81. tie code to writers d
82. tie code to writers d notifies you if you screw up
84. authors know their own code
85. authors should fix their bugs
86. how can your process help out?
87. tie code to writers
88. tie code to writers
89. tie code to writers chrome has owner files
90. tie code to writers dictates responsibility
91. tie code to writers makes mentors visible
92. tie code to writers d
93. tie code to writers d per-file ownership
94. tie code to writers d class BranchesController areas_of_responsibility :git end
97. tie code to writers d issues get assigned to the appropriate team
98. talk
99. more communication! yayyyyyyyycrap
100. email meetings chat issues im on-call video al
101. more is good better is better
102. most things aren’t emergencies
103. most things aren’t emergencies be mindful of time
104. measure impact of your actions be mindful of time
105. imagine:
106. 1. page a coworker for help 2. they get woken up 3. their phone takes a selfie of them 4. selfie gets posted into chat
108. would this make you respect your coworker’s time more?
109. add empathy to your process
110. empathy stems from being exposedto real pain
111. everyone should do some support
112. there’s a difference between reading about it helping with it &
113. institutional teaching
114. [we] have a responsibility to be teachers—that this should be a central part of [our] jobs “ — Ed Catmull, Creativity, Inc.
115. …it’s just logic that someday we won’t be here. — Ed Catmull, Creativity, Inc. ”
116. more productive if workers are smart
117. how do you teach without making it lame?
118. chatops: learning by osmosis
119. app crashes. here’s a normal flow:
120. 1. employee gets paged 2. they ssh into… something 3. they fix it… somehow app crashes. here’s a normal flow:
121. no visibility means no teaching
122. no visibility means no improvement
123. d HUBOT
124. 1. employee gets paged 2. they manage it in a chat room 3. they fix it and people can watch here’s a better flow: app crashes.
125. visibility means teaching
126. visibility means improvement
127. real-life debugging
128. github’s wifi is down floor two, san francisco (right now!)
133. debug in the open
134. feedback
135. blue angels the feedback on
137. two-way street
138. two-way street how well do you receive feedback?
139. two-way street how well do you give feedback?
140. improving feedbackacross an org
142. caution with a degree of moving fast
143. need to be fast
144. need to be safe
145. takes effort to achieve both
146. @ holman