Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue

With information bombarding us every minute of our lives, it can be tough to know what warrants triggering a page. Alert fatigue is a real danger, but ignoring real problems is dangerous too. What lessons can we learn from other fields - such as public policy, public health, or clinical medicine - to reduce the risk of alert fatigue while also keeping our systems as healthy as possible?

With the STAT framework - Supported, Trustworthy, Actionable, and Triaged - we have a rapid diagnostic test we can use to identify the portions of our alerting systems that cause alert fatigue, and some strategies for reducing it.

Transcript

1. Warning: This Talk Contains Content
   Known to the State of California to Reduce
   Alert Fatigue
   Aditya Mukerjee
   Observability Engineer at Stripe
   @chimeracoder
   

   View Slide

2. @chimeracoder
   

   View Slide

3. Why we can learn from clinical healthcare
   •Direct personal contact
   •Visibly high-stakes
   •Systems which are difficult to control
   @chimeracoder
   

   View Slide

4. Alert Fatigue and Decision Fatigue
   @chimeracoder
   

   View Slide

5. When the frequency or severity of alerts causes the responder
   either to ignore important alerts or make mistakes more frequently
   @chimeracoder
   Alert Fatigue
   

   View Slide

6. When the frequency or complexity of decision points causes a
   person to avoid decisions or make mistakes more frequently.
   @chimeracoder
   Decision Fatigue
   

   View Slide

7. Alert Fatigue deals with the observability of systems
   @chimeracoder
   Decision Fatigue deals with the controllability of systems
   

   View Slide

8. 72-99% of clinical alarms are false positives
   @chimeracoder
   …but certain patterns of alerts and decisions contribute
   disproportionately to fatigue!
   

   View Slide

9. Four Steps to Reducing Alert Fatigue: STAT
   @chimeracoder
   (Supported, Trustworthy, Actionable, Triaged)
   

   View Slide

10. Supported
    •Who owns this monitor?
    •Who has the right or authority to change it?
    @chimeracoder
    

    View Slide

11. @chimeracoder
    An alerting system includes the people who participate in
    responding to alerts, not just the software that generates alerts
    

    View Slide

12. The person responding to an alert always has the right to
    change it, whether we realize it or not
    @chimeracoder
    

    View Slide

13. Responders must feel ownership over the end result
    @chimeracoder
    

    View Slide

14. Trustworthy
    • Do I trust this alert to notify me when a problem happens?
    • Do I trust this alert to stay silent when all is well?
    • Do I trust this alert to give me sufficient information to diagnose problems?
    @chimeracoder
    

    View Slide

15. Anomaly detection and opaque algorithms
    If you don’t understand why an alert is firing, you don’t understand
    whether it’s real or not
    @chimeracoder
    

    View Slide

16. When to use modeling for monitors
    •Does the model represent the interconnectedness of your systems?
    •Can the thresholds be adjusted?
    •Are the model parameters and outputs human-interpretable?
    @chimeracoder
    

    View Slide

17. Actionable
    •At most one decision required to respond
    •Alerts that are difficult to action become alerts that are ignored
    @chimeracoder
    

    View Slide

18. Making alerts more actionable
    “investigate”, “something”, “somewhere”, “someone”
    @chimeracoder
    Decision trees, interactive tooling, making the alerts specific
    

    View Slide

19. If it’s unclear who should be taking action, the alert is not actionable
    @chimeracoder
    

    View Slide

20. Triaged
    •Meticulously triage alerts
    •Alert type should reflect urgency
    •Urgency of alerts can change
    @chimeracoder
    

    View Slide

21. Steps for triaging
    • Commonly-understood tiers
    • Regular, periodic re-evaluation process
    @chimeracoder
    

    View Slide

22. What’s wrong with Prop 65 warnings?
    @chimeracoder
    

    View Slide

23. STAT is just the beginning
    @chimeracoder
    

    View Slide

24. Takeaways
    •Alert fatigue and decision fatigue deplete executive function
    •Tackle alert fatigue and decision fatigue in tandem
    •Use STAT as a quick check to evaluate alerting systems
    •Regularly re-evaluate your alerts and alerting systems
    @chimeracoder
    

    View Slide

25. Thank you!
    Aditya Mukerjee
    @chimeracoder
    

    View Slide