Designing Alerts to Direct Attention

Many of the alerts monitoring systems send us are primitive capture events. They lack useful context that can help quickly and accurately orient and direct the attention of on-call engineers. The poor quality of these alerts exacerbates the sense that we are increasingly overwhelmed by the complexity of our systems, unable to effectively respond to events.

We must imagine them as teammates actively participating – with us – to achieve our goals. In cognitive systems engineering, this relationship is referred to as a joint cognitive system, “a whole comprising people and technology acting together.” We can, and should, design alerts so that when computers generate them, the experience is more cooperative and helps direct our attention in meaningful ways.

Transcript

1. @Ryan_Frantz Designing Alerts To Direct Attention
2. @Ryan_Frantz
3. @Ryan_Frantz Takeaways Computers as Teammates Directed Attention
4. @Ryan_Frantz
5. @Ryan_Frantz
6. @Ryan_Frantz
7. @Ryan_Frantz https://www.hbo.com/chernobyl
8. @Ryan_Frantz Monitor Service
9. @Ryan_Frantz Monitor Service “The System”
10. @Ryan_Frantz Monitor Service Joint Cognitive System
11. @Ryan_Frantz https://www.pbs.org/newshour/science/3-brain-technologies-to-watch-in-2018
12. @Ryan_Frantz Executive Functions Shift Initiation Working Memory Planning/Organization
13. @Ryan_Frantz Shift The ability to move freely from one situation to another and to think flexibly in order to respond appropriately to the situation.
14. @Ryan_Frantz Initiation The ability to begin a task or activity and to independently generate ideas, responses, or problem-solving strategies.
15. @Ryan_Frantz Working Memory The capacity to hold information in mind for the purpose of completing a task.
16. @Ryan_Frantz Planning/Organization The ability to manage current and future-oriented task demands.
17. @Ryan_Frantz https://www.scmp.com/news/asia/south-asia/article/2125922/moustached-indian-traffic-cop-moonwalks-michael-jackson
18. @Ryan_Frantz Directed Attention [A] cognitive function that inherently involves the coordination of multiple agents through the use of external media.
19. @Ryan_Frantz Alert Design Monitor ≈ Service ≈
20. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CPU Utilization for dbdata01 is 85% CPU utilization for host dbdata01 is >= 85%!
21. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CPU Utilization for dbdata01 is 85% CPU utilization for host dbdata01 is >= 85%! NOTE: Heavy CPU utilization corresponds with increased response latency which may impact customer service levels. Current perc90 response time <= 500ms (SLA == 600ms).
22. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CPU Utilization for dbdata01 is 85% CPU utilization for host dbdata01 is >= 85%! NOTE: Heavy CPU utilization corresponds with increased response latency which may impact customer service levels. Current perc90 response time <= 500ms (SLA == 600ms). Product/Service: This host stores customer data and is a dependency for the Search and Payments services.
23. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CPU Utilization for dbdata01 is 85% CPU utilization for host dbdata01 is >= 85%! NOTE: Heavy CPU utilization corresponds with increased response latency which may impact customer service levels. Current perc90 response time <= 500ms (SLA == 600ms). Product/Service: This host stores customer data and is a dependency for the Search and Payments services.
 
 dbdata01 is half of a pair of hosts storing customer data.
24. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CPU Utilization for dbdata01 is 85% CPU utilization for host dbdata01 is >= 85%! NOTE: Heavy CPU utilization corresponds with increased response latency which may impact customer service levels. Current perc90 response time <= 500ms (SLA == 600ms). Product/Service: This host stores customer data and is a dependency for the Search and Payments services.
 
 dbdata01 is half of a pair of hosts storing customer data. Frequency: This alert has been triggered 12 times in the past 12 hours.
25. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CC Processing is Delayed 75% of CC payments are still in the authorization phase
26. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CC Processing is Delayed 75% of CC payments are still in the authorization phase after 1 hour (auth typically averages 5 minutes).
27. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CC Processing is Delayed 75% of CC payments are still in the authorization phase after 1 hour (auth typically averages 5 minutes). Payfriend error rates have increased over the past hour.
28. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CC Processing is Delayed 75% of CC payments are still in the authorization phase after 1 hour (auth typically averages 5 minutes). Payfriend error rates have increased over the past hour. If needed, contact our Payfriend rep at 800-PAY-FREN (Acct # 1234).
29. @Ryan_Frantz To: From: Subject: ryan agent@monitoringlove.com CC Processing is Delayed 75% of CC payments are still in the authorization phase after 1 hour (auth typically averages 5 minutes). Payfriend error rates have increased over the past hour. If needed, contact our Payfriend rep at 800-PAY-FREN (Acct # 1234). Consider preparing to flip CC processing to PayAttentionCo.
30. @Ryan_Frantz
31. @Ryan_Frantz Service
32. @Ryan_Frantz
33. @Ryan_Frantz bit.ly/designing-alerts-refs