“Finding your soul
metric can take time.
But in the end, it boils
down to what
matters most to your
users.”
Mathias Meyer, CEO Travis CI
Slide 79
Slide 79 text
Find the subset of
metrics people
should look at after
a deploy
Slide 80
Slide 80 text
Stick them on a
central place
Slide 81
Slide 81 text
Logs!
Slide 82
Slide 82 text
Syslog works really
well
Slide 83
Slide 83 text
Decide on a log format
Slide 84
Slide 84 text
No content
Slide 85
Slide 85 text
No content
Slide 86
Slide 86 text
No content
Slide 87
Slide 87 text
Awesome
Tools
Slide 88
Slide 88 text
Empower the
Individual
Slide 89
Slide 89 text
Information
Overload
Slide 90
Slide 90 text
Tools
need
attention
Slide 91
Slide 91 text
Culture
Slide 92
Slide 92 text
No content
Slide 93
Slide 93 text
No content
Slide 94
Slide 94 text
No content
Slide 95
Slide 95 text
No content
Slide 96
Slide 96 text
No content
Slide 97
Slide 97 text
No content
Slide 98
Slide 98 text
realtalk:
things break
Slide 99
Slide 99 text
New View
Slide 100
Slide 100 text
Complex
Socio-Technical
Systems
Slide 101
Slide 101 text
“Erkenntnis und
Irrtum fließen aus
denselben
psychischen Quellen;
nur der Erfolg vermag
beide zu scheiden.”
Ernst Mach, Erkenntnis und Irrtum (p. 116)
Slide 102
Slide 102 text
Success and failure
can only be
determined
a posteriori
Slide 103
Slide 103 text
Things made sense at
the time
Slide 104
Slide 104 text
People don't come to
work to do a bad job
Slide 105
Slide 105 text
Nietzschean
Anxiety
Slide 106
Slide 106 text
So I always get off the hook
whatever I do?
Slide 107
Slide 107 text
depends
Slide 108
Slide 108 text
“There is a difference
between explaining
and excusing human
performance.”
Sidney Dekker, The Field Guide to Understanding
Human Error (p. 196)
Slide 109
Slide 109 text
Blameless
Postmortems
Slide 110
Slide 110 text
Open
Meeting
Slide 111
Slide 111 text
Everybody
is Invited
Slide 112
Slide 112 text
What
happened?
Slide 113
Slide 113 text
Timeline
Slide 114
Slide 114 text
Describe the past
Don't excuse it away
Slide 115
Slide 115 text
The Facilitator
Slide 116
Slide 116 text
Guide the Discussion
Slide 117
Slide 117 text
Look out for indicators of
Old View thinking
Slide 118
Slide 118 text
Counterfactuals
Slide 119
Slide 119 text
she should have
if they just had
if he would have
you failed to
Slide 120
Slide 120 text
Biases
Slide 121
Slide 121 text
Hindsight Bias
Confirmation Bias
Outcome Bias
Slide 122
Slide 122 text
there are many
more
Slide 123
Slide 123 text
Who is
in charge?
Slide 124
Slide 124 text
Etsy School
Slide 125
Slide 125 text
Taught Facilitator
Course
Slide 126
Slide 126 text
3 x 90 minutes
Slide 127
Slide 127 text
Remediation
Items
Slide 128
Slide 128 text
incorporate learning and
takeaways from the
meeting
Slide 129
Slide 129 text
No content
Slide 130
Slide 130 text
turn surprises into
known factors
Slide 131
Slide 131 text
MORGUE
Slide 132
Slide 132 text
No content
Slide 133
Slide 133 text
No content
Slide 134
Slide 134 text
No content
Slide 135
Slide 135 text
No content
Slide 136
Slide 136 text
No content
Slide 137
Slide 137 text
No content
Slide 138
Slide 138 text
No content
Slide 139
Slide 139 text
https://github.com/etsy/morgue
Slide 140
Slide 140 text
Near Miss
Slide 141
Slide 141 text
"Hey all, I just ran rm -rf
$DIR/ and since the variable
was empty I deleted my
whole VM. This would have
been bad in production.
Don't do that."
Slide 142
Slide 142 text
Pre Mortem
Slide 143
Slide 143 text
Architecture
Reviews
Slide 144
Slide 144 text
Operability
Reviews
Slide 145
Slide 145 text
“It is also worth pointing
out that the bias
towards investigating
failures rather than
success itself
represents a trade-off.”
Erik Hollnagel, The ETTO Principle: Efficiency-Thoroughness
Trade-Off
Slide 146
Slide 146 text
Investigate
Success
Slide 147
Slide 147 text
Why did it work?
Slide 148
Slide 148 text
Human Error is
where you
stopped looking
Slide 149
Slide 149 text
Learning
>
Blaming
Slide 150
Slide 150 text
No content
Slide 151
Slide 151 text
No content
Slide 152
Slide 152 text
How did we end up here?
Slide 153
Slide 153 text
No content
Slide 154
Slide 154 text
Overhauls
&
Iterations
Slide 155
Slide 155 text
Culture
&
Tools
(you can't really have one without the other)
Slide 156
Slide 156 text
Humans are
AWESOME
Slide 157
Slide 157 text
Nobody comes
to work to do
a bad job
Slide 158
Slide 158 text
Trust
your
Co-Workers
Slide 159
Slide 159 text
There is a lot of
knowledge in your
engineering team
Slide 160
Slide 160 text
Deploy
(as often as it makes sense)
Slide 161
Slide 161 text
Collaborate
(even if you think you don't have to)
Slide 162
Slide 162 text
No content
Slide 163
Slide 163 text
Listen
(to problems and experiences
of your coworkers)
Slide 164
Slide 164 text
Thank you!
Slide 165
Slide 165 text
codeascraft.com
etsy.com/codeascraft/talks
Slide 166
Slide 166 text
The Road to Success
is paved with
Small Improvements
Daniel Schauenberg • [email protected] • @mrtazz