$30 off During Our Annual Pro Sale. View Details »

Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue

Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue

With information bombarding us every minute of our lives, it can be tough to know what warrants triggering a page. Alert fatigue is a real danger, but ignoring real problems is dangerous too. What lessons can we learn from other fields - such as public policy, public health, or clinical medicine - to reduce the risk of alert fatigue while also keeping our systems as healthy as possible?

With the STAT framework - Supported, Trustworthy, Actionable, and Triaged - we have a rapid diagnostic test we can use to identify the portions of our alerting systems that cause alert fatigue, and some strategies for reducing it.

Aditya Mukerjee

June 04, 2018
Tweet

More Decks by Aditya Mukerjee

Other Decks in Technology

Transcript

  1. Warning: This Talk Contains Content
    Known to the State of California to Reduce
    Alert Fatigue
    Aditya Mukerjee
    Observability Engineer at Stripe
    @chimeracoder

    View Slide

  2. @chimeracoder

    View Slide

  3. Why we can learn from clinical healthcare
    •Direct personal contact
    •Visibly high-stakes
    •Systems which are difficult to control
    @chimeracoder

    View Slide

  4. Alert Fatigue and Decision Fatigue
    @chimeracoder

    View Slide

  5. When the frequency or severity of alerts causes the responder
    either to ignore important alerts or make mistakes more frequently
    @chimeracoder
    Alert Fatigue

    View Slide

  6. When the frequency or complexity of decision points causes a
    person to avoid decisions or make mistakes more frequently.
    @chimeracoder
    Decision Fatigue

    View Slide

  7. Alert Fatigue deals with the observability of systems
    @chimeracoder
    Decision Fatigue deals with the controllability of systems

    View Slide

  8. 72-99% of clinical alarms are false positives
    @chimeracoder
    …but certain patterns of alerts and decisions contribute
    disproportionately to fatigue!

    View Slide

  9. Four Steps to Reducing Alert Fatigue: STAT
    @chimeracoder
    (Supported, Trustworthy, Actionable, Triaged)

    View Slide

  10. Supported
    •Who owns this monitor?
    •Who has the right or authority to change it?
    @chimeracoder

    View Slide

  11. @chimeracoder
    An alerting system includes the people who participate in
    responding to alerts, not just the software that generates alerts

    View Slide

  12. The person responding to an alert always has the right to
    change it, whether we realize it or not
    @chimeracoder

    View Slide

  13. Responders must feel ownership over the end result
    @chimeracoder

    View Slide

  14. Trustworthy
    • Do I trust this alert to notify me when a problem happens?
    • Do I trust this alert to stay silent when all is well?
    • Do I trust this alert to give me sufficient information to diagnose problems?
    @chimeracoder

    View Slide

  15. Anomaly detection and opaque algorithms
    If you don’t understand why an alert is firing, you don’t understand
    whether it’s real or not
    @chimeracoder

    View Slide

  16. When to use modeling for monitors
    •Does the model represent the interconnectedness of your systems?
    •Can the thresholds be adjusted?
    •Are the model parameters and outputs human-interpretable?
    @chimeracoder

    View Slide

  17. Actionable
    •At most one decision required to respond
    •Alerts that are difficult to action become alerts that are ignored
    @chimeracoder

    View Slide

  18. Making alerts more actionable
    “investigate”, “something”, “somewhere”, “someone”
    @chimeracoder
    Decision trees, interactive tooling, making the alerts specific

    View Slide

  19. If it’s unclear who should be taking action, the alert is not actionable
    @chimeracoder

    View Slide

  20. Triaged
    •Meticulously triage alerts
    •Alert type should reflect urgency
    •Urgency of alerts can change
    @chimeracoder

    View Slide

  21. Steps for triaging
    • Commonly-understood tiers
    • Regular, periodic re-evaluation process
    @chimeracoder

    View Slide

  22. What’s wrong with Prop 65 warnings?
    @chimeracoder

    View Slide

  23. STAT is just the beginning
    @chimeracoder

    View Slide

  24. Takeaways
    •Alert fatigue and decision fatigue deplete executive function
    •Tackle alert fatigue and decision fatigue in tandem
    •Use STAT as a quick check to evaluate alerting systems
    •Regularly re-evaluate your alerts and alerting systems
    @chimeracoder

    View Slide

  25. Thank you!
    Aditya Mukerjee
    @chimeracoder

    View Slide