It's 3AM, Do You Know Why You Got Paged?

It’s 3AM, Do You Know Why You Got Paged?

@Ryan_Frantz #me

@Ryan_Frantz 3AM Is the Worst • Takeaways • What makes
for good context? • Computers should augment our work

Why add context?

@Ryan_Frantz There are no “good” alerts

@Ryan_Frantz Alert Fatigue • High volume, low value alerts •
Heightened anxiety, distraction • Sleep loss • Lack of context

@Ryan_Frantz Worst Case: You’re Woken Up... ...and have no idea
what’s happening.

Good Context

@Ryan_Frantz Easy to Implement • Current value vs threshold •
Impact on site availability/application function • Downstream effects • Author’s intentions

@Ryan_Frantz Common Patterns

@Ryan_Frantz Color: It’s Not for Everyone

@Ryan_Frantz Alert History • Backstories are hidden in your alerts!

Computers as Augmenters

@Ryan_Frantz Focus

@Ryan_Frantz Augmentation “...increasing the capability of a [person] to approach
a complex problem situation, to gain comprehension to suit [one’s] particular needs, and to derive solutions to problems.” - Doug Englebart “Augmenting Human Intellect: A Conceptual Framework”

OODA Loops

@Ryan_Frantz OODA Patrick Edwin Moran - Own work. Licensed under
Creative Commons Attribution 3.0 via Wikimedia Commons

@Ryan_Frantz Orientation “The second O, orientation – as the repository
of our genetic heritage, cultural tradition, and previous experiences – is the most important part of the O-O-D- A loop since it shapes the way we observe, the way we decide, the way we act.” Col. Boyd “Organic Design for Command and Control”

Augmenting Orientation

@Ryan_Frantz Nagios

@Ryan_Frantz You, Operator

@Ryan_Frantz BOOM

@Ryan_Frantz Adding Context • Within Nagios checks • Always do
this! • Before alert notification • nagios-herald

nagios-herald

@Ryan_Frantz

Failed Check nagios-herald Formatter Helpers Graphite Ganglia Logstash Message

@Ryan_Frantz

@Ryan_Frantz nagios-herald is Calling • Write formatters • Need tests
• Need more docs • https://github.com/etsy/nagios-herald

@Ryan_Frantz Office Hours Today 5:00PM - 5:40PM Table 2 in
the Sponsor Pavillion Let’s chat!

@Ryan_Frantz Resources and Reads • Project: github.com/etsy/nagios-herald • Blog: bit.ly/alert-design
• Color contrast ratio: bit.ly/contrastratio • Englebart: bit.ly/augmenthumans • OODA Loop: bit.ly/oodaloops

[email protected] @Ryan_Frantz

It's 3AM, Do You Know Why You Got Paged?

It's 3AM, Do You Know Why You Got Paged?

More Decks by RyanFrantz

Other Decks in Technology

Featured

Transcript