Reducing Alert Fatigue and Noise

How to reducing alert fatigue and noise in BBM.
So no Issue is missed but everyone still get their sleep.

Transcript

1. Reducing Alert Fatigue and Noise Daniel Fablius - Sysops
2. Alert Fatigue Alarm fatigue or alert fatigue occurs when one is exposed to a large number of frequent alarms (alerts) and consequently becomes desensitized to them -Wikipedia
3. Current Condition On BBM ➔ Amount of alerts 1000-2000 alerts per week. ➔ Amount of incident tracked by alerts Less than 3 per month. ➔ Amount of existing alert in DD 1202 alerts..
4. How we treat our alert? - All alert treated the same - All alert has P3 priority - All alert escalated
5. How we set our Alert. - Most our alert use “average” condition. That lead to, our alert often getting triggered and auto closed less than 3 mins. - We set up the alert based on pillar info they provide. We mostly know only the threshold, but don’t know what this alert means, or what condition that can trigger this alert - We have escalation for every alert. Dev as L1, Devops L2 and then Management.
6. ➔ Some People becomes Desensitized to alert. ➔ Missing important alert ➔ People get tired of alerts Impact
7. What we do to solve this ➔ Refactoring alert so it won’t get triggered for spike ➔ Removing un useful alert ➔ Set priority for alert ➔ Treat different kind of alert with different notification
8. Thanks