Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue

Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue

With information bombarding us every minute of our lives, it can be tough to know what warrants triggering a page. Alert fatigue is a real danger, but ignoring real problems is dangerous too. What lessons can we learn from other fields - such as public policy, public health, or clinical medicine - to reduce the risk of alert fatigue while also keeping our systems as healthy as possible?

With the STAT framework - Supported, Trustworthy, Actionable, and Triaged - we have a rapid diagnostic test we can use to identify the portions of our alerting systems that cause alert fatigue, and some strategies for reducing it.

94dcff33cbdf74b5d785369ac54bc1a8?s=128

Aditya Mukerjee

June 04, 2018
Tweet

Transcript

  1. 1.

    Warning: This Talk Contains Content Known to the State of

    California to Reduce Alert Fatigue Aditya Mukerjee Observability Engineer at Stripe @chimeracoder
  2. 3.

    Why we can learn from clinical healthcare •Direct personal contact

    •Visibly high-stakes •Systems which are difficult to control @chimeracoder
  3. 5.

    When the frequency or severity of alerts causes the responder

    either to ignore important alerts or make mistakes more frequently @chimeracoder Alert Fatigue
  4. 6.

    When the frequency or complexity of decision points causes a

    person to avoid decisions or make mistakes more frequently. @chimeracoder Decision Fatigue
  5. 7.

    Alert Fatigue deals with the observability of systems @chimeracoder Decision

    Fatigue deals with the controllability of systems
  6. 8.

    72-99% of clinical alarms are false positives @chimeracoder …but certain

    patterns of alerts and decisions contribute disproportionately to fatigue!
  7. 10.

    Supported •Who owns this monitor? •Who has the right or

    authority to change it? @chimeracoder
  8. 11.

    @chimeracoder An alerting system includes the people who participate in

    responding to alerts, not just the software that generates alerts
  9. 12.

    The person responding to an alert always has the right

    to change it, whether we realize it or not @chimeracoder
  10. 14.

    Trustworthy • Do I trust this alert to notify me

    when a problem happens? • Do I trust this alert to stay silent when all is well? • Do I trust this alert to give me sufficient information to diagnose problems? @chimeracoder
  11. 15.

    Anomaly detection and opaque algorithms If you don’t understand why

    an alert is firing, you don’t understand whether it’s real or not @chimeracoder
  12. 16.

    When to use modeling for monitors •Does the model represent

    the interconnectedness of your systems? •Can the thresholds be adjusted? •Are the model parameters and outputs human-interpretable? @chimeracoder
  13. 17.

    Actionable •At most one decision required to respond •Alerts that

    are difficult to action become alerts that are ignored @chimeracoder
  14. 19.
  15. 24.

    Takeaways •Alert fatigue and decision fatigue deplete executive function •Tackle

    alert fatigue and decision fatigue in tandem •Use STAT as a quick check to evaluate alerting systems •Regularly re-evaluate your alerts and alerting systems @chimeracoder