What alert fatigue is and why it is dangerous
Alert fatigue is what happens when people are notified so often, about things that so rarely need them, that they stop reacting. It is not a morale problem first — it is a reliability problem. The engineer who has dismissed forty harmless pages this week will be slower to open the forty-first, and that may be the real outage.
The cost is larger than the minutes spent looking at each alert. Every interruption breaks concentration, and getting back into deep work takes far longer than acknowledging the page. A night-time alert costs the following day as well.
Noise ratio
The share of alerts that needed no human action. Above 50% the pager is training people to ignore it.
Alerts per shift
A widely used guideline is no more than two actionable incidents per on-call shift. Much more, and there is no time to fix causes.
Night pages
The most expensive kind. Anything that can wait until morning should be a ticket, not a page.
Repeat offenders
A handful of alert rules usually produce most of the noise. Fixing the top five beats tuning fifty.
Frequently asked questions
How many alerts are too many?
When a single on-call engineer gets more than a few actionable alerts per shift, or when more than half of all alerts need no action, the rotation is unsustainable and important alerts are likely to be missed.
What is an actionable alert?
One that requires a person to do something now and gives them enough information to start. If the right response is to wait and see, it should not have paged anyone.
How do I calculate the signal-to-noise ratio of my alerts?
Divide the alerts that led to human action by the total number of alerts in the same period. 30 actionable alerts out of 120 is a 25% signal ratio and 75% noise.
Why count time to regain focus?
Because the interruption costs more than the alert. After a context switch it takes many minutes to get back into the task, so ten two-minute alerts can consume an afternoon.