SRE: the good, the bad, and the ouch

SRE sounds like a plan with no drawbacks … but making it work in practice can be trickier than the theory says. This talk shares stories of SRE wins and SRE accidents.

Transcript

1. SRE: the good, the bad, and the ouch Holly Cummins Red Hat WTF is SRE April 28, 2022
2. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins
3. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins
4. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession:
5. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession: i am not an SRE
6. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession: i am not an SRE Robert Barron Cansu Kavılı Örnek but some of my good friends are
7. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs
8. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs
9. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs
10. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs
11. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE poll: who are you? sli.do #886041
12. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is SRE?
13. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is SRE?
14. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE no really, WTF is SRE?
15. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE what ops would be like if it was done by software engineers
16. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what was wrong with the old way of doing ops?
17. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE old ops
18. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual old ops
19. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive old ops
20. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed old ops
21. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed not aligned to business goals old ops
22. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed not aligned to business goals unable to handle complexity of cloud native old ops
23. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE eliminate repetitive tasks
24. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE eliminate toil
25. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE aligned incentives
26. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE failure is a symptom, not a cause
27. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE devops?
28. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE automate everything DevOps SRE
29. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE holistic + collaborative DevOps SRE
30. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what could possibly go wrong?
31. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE poll: where did you come from? sli.do #886041
32. @holly_cummins IBM Garage true story “we’re SRE now” the cunning rebrand
33. @holly_cummins IBM Garage true story “we’re SRE now” the cunning rebrand
34. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers software engineer
35. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers software engineer
36. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers ops software engineer
37. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ops SRE: what ops would be like if it was done by ops?
38. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ops SRE: what ops would be like if it was done by ops?
39. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE we’ve read the book!
40. @holly_cummins IBM Garage true story “we are just as good: we have scripts” the cunning rebrand
41. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what triggers the scripts?
42. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev how much contact do SRE have with dev? SRE
43. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev how much contact do SRE have with dev? SRE
44. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev are there any SRE NFRs in the dev backlog? SRE
45. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE uh … we never thought of speaking to the developers
46. @holly_cummins IBM Garage true story “we do SRE … in silos” the cunning rebrand
47. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
48. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
49. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
50. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
51. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE I am not designed for this.
52. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE two war rooms
53. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile
54. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile we’re responsible for stability of the mainframe we’re responsible for stability of the front end
55. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile the ambassador we’re responsible for stability of the mainframe we’re responsible for stability of the front end we’re responsible for stability of the mainframe … as long as it’s used correctly
56. @holly_cummins IBM Garage true story “we have a ticket per team, not per incident” dots aren’t connected
57. “we want to do SRE but we don’t have enough permissions on our systems”
58. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “the DBAs don’t trust us”
59. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it takes us 15 minutes just to get permission to run a standard set of SQL diagnostic statements”
60. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it takes us 15 minutes just to get permission to run a standard set of SQL diagnostic statements”
61. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE silos cost
62. @holly_cummins IBM Garage true story “we do post-mortems after every incident … maybe” the gap between intent and reality
63. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
64. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents
65. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents measure the number of post-mortems
66. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents measure the number of post-mortems see if they match
67. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
68. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics:
69. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem?
70. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem? does it include more than the people directly involved?
71. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem? does it include more than the people directly involved? did we invite more than our own team?
72. @holly_cummins IBM Garage true story “no one says anything in our blameless post-mortems”
73. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ‘blameless’ post-mortem
74. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE if involvement in an incident is punished, people will avoid engaging with systems
75. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “great idea, go build that!” if ideas are punished with extra work, people will try not to have ideas
76. @holly_cummins IBM Garage true story “we have success metrics” the perverse incentive
77. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE metrics are good
78. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SREs are data-driven
79. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE but …
80. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE as senior leaders, be careful what you incentivise
81. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE be careful what behaviours you discourage
82. @holly_cummins IBM Garage true story “we count how many incidents we have; if the number goes down, it means we are working better” the perverse incentive
83. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE during the holidays, quality is outstanding!
84. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE delivery excellence!
85. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE fewer people working → fewer incidents
86. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE new release → more incidents
87. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE are you measuring the right things?
88. “management has weaponised the metrics”
89. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure things, not people
90. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what should you measure?
91. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
92. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE make work visible
93. @holly_cummins IBM Garage true story “we never seem to complete the work we planned” the email timesink
94. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint
95. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint 50% story points
96. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint 50% story points 50% unplanned work (tickets)
97. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 50% unplanned work (tickets)
98. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets)
99. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets
100. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets 40% ??
101. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets 40% ??
102. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
103. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “
104. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?”
105. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?” “where is this documented?”
106. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?” “where is this documented?”
107. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
108. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure
109. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure it was a data quality issue
110. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure it was a data quality issue it was a process issue
111. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE track work use the data to eliminate toil
112. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure blockers
113. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE mean time to failure? mean time to detect problems?
114. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is failure in a complex system? if a system goes down but user experience is fine, does that count?
115. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure “what have I learned” measure “have I made sure it won’t happen again”
116. @holly_cummins IBM Garage true client story “we can’t actually release this.” value on the shelf
117. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE are our incentives hurting us?
118. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what’s stopping more frequent deploys?
119. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it costs too much to release”
120. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it costs too much to release” you can fix that
121. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “we can’t ship until we have more confidence in the quality”
122. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “we can’t ship until we have more confidence in the quality” you can fix that
123. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE deferred wiring
124. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE feature flags
125. @holly_cummins IBM Garage true client story “we can’t release this microservice… we deploy all our microservices at the same time… because otherwise nothing works.” the monolithic microservices
126. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE let’s talk about microservices
127. @holly_cummins IBM Garage true client story “every time we change code, something breaks” the peril of microservices
128. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE just because a system runs across 6 containers doesn’t mean it’s decoupled
129. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
130. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE mars climate explorer
131. @holly_cummins #IBMGarage Courtesy NASA/ JPL-Caltech
134. distributing did not help
135. metric units distributing did not help
136. metric units imperial units distributing did not help
137. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE testing
138. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE $370 million loss https://en.wikipedia.org/wiki/Cluster_(spacecraft) Cluster + Ariane 5
139. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
140. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it …
141. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it … but stubbed out one component.
142. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it … but stubbed out one component. that component was the one that broke.
143. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE you can’t a/b test a $370 million rocket the ariane failed in 36 seconds
144. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE testing will always be incomplete aim for recoverability
145. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
146. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE resilience
147. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE resilience recoverability
148. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE observability
149. they often couldn’t see the explorer
150. feedback is good engineering
151. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE when SRE is right it is great
152. bank
153. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile remember this bank?
154. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile we’re responsible for stability of the front end remember this bank?
155. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile the ambassador we’re responsible for stability of the front end we’re responsible for stability of the mainframe … as long as it’s used correctly remember this bank?
156. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE another department … web front-end back-end one team
157. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE another department … web front-end back-end one team mobile front-end
158. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE one team, range of techniques canary deploys CI/CD pipelines one team CI/CD pipelines big-bang deploys onto AIX
159. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE by the way …
160. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE big bang deploys
161. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE big bang deploys 50% failure rate
162. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE canary deploys big bang deploys 50% failure rate
163. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE 10% failure rate canary deploys big bang deploys 50% failure rate
164. industrial
165. remember the suspicious DBAs?
166. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE two root problems: • automation • trust and transparency
167. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE trigger automation via slack
168. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE because it was transparent, DBAs were happy and automated more things
169. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what happens when things go wrong?
170. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE
171. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE leadership need to provide a safety net.
172. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE celebrate success celebrate failure
173. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE celebrate success celebrate learning
174. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE remember the why
175. @holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE better, safer, faster, happier
176. thank you! #RedHat @holly_cummins