$30 off During Our Annual Pro Sale. View Details »

Tackling Alert Fatigue

Tackling Alert Fatigue

Caitie McCaffrey

June 26, 2016
Tweet

More Decks by Caitie McCaffrey

Other Decks in Technology

Transcript

  1. “When alerts are more often false than true, the on-call’s

    sense of urgency in responding to alerts is diminished … the simple burden of alerts desensitizes the on-call to alerts.”
  2. “When alarms are more often false than true, the nursing

    staff’s sense of urgency in responding to alarms is diminished … the simple burden of alerts desensitizes caregivers to alarms.” Novel Approach to Cardiac Alarm Management on Telemetry Units
  3. The High Cost of: Alert Fatigue Unplanned Work Inability to

    Complete Planned Work Less Time to Focus on Core Business
  4. Tackling Alert Fatigue Increase thresholds for patient vitals Only Crisis

    Alarms would emit audible alerts Nursing staff required to tune false positive alerts in hospitals Novel Approach to Cardiac Alarm Management on Telemetry Units
  5. Cmd Line Tool Viz / Dashboad Alerting Svc Cuckoo-Read Cuckoo-Write

    Indexing Svc Relay Svc Twitter Front End Twitter Svc Twitter Statsite Twitter Svc Twitter Svc Scribe Collection Agent HDFS Manhattan Database Public Cloud Observability at Twitter
  6. Weekly On-Call Retro Handoff on going issues Review alerts fired

    in the previous week Schedule work to improve on-call or reliability
  7. –Astrid Atkinson “The goal is not to never get paged,

    the goal is to never get paged for the same thing twice” Engineering for the Long Game
  8. On-call slept through the night More time to do scheduled

    work while on-call Faster to ramp up new teammates
  9. The goal is to build systems that can scale linearly

    with machines & sub-linearly with people