Every alert in your system was added by someone who had a reason. That’s the whole problem.
Alert sets only ever grow. An incident happens, and the retrospective action is “add monitoring for this.” Nobody’s retrospective action is “remove the four alerts that fired during this incident and told us nothing,” even though that would often have helped more. There’s no natural pressure in the other direction, so the set expands until responders are filtering it mentally — which is the point at which the alerting system has stopped working, regardless of how comprehensive it looks.
Alert fatigue is a rational adaptation#
It’s tempting to frame this as a discipline problem: responders should investigate every page. They shouldn’t, and they won’t.
If an alert has fired forty times and thirty-eight of those were noise, the correct Bayesian response to the forty-first is to deprioritize it. Responders aren’t being careless. They’re behaving optimally given the signal quality you’ve given them.
Which means the fix isn’t training or process. It’s improving signal quality, and the fastest route is subtraction.
The compounding version of this is worse. Noisy alerts slow response to real ones. Slow response means longer incidents. Longer incidents produce retrospectives. Retrospectives produce more alerts. The standard remedy makes the problem worse, which is why teams can work hard at reliability for a year and end up with more pages and slower response.
The three-part test#
Every page should satisfy all three of these. Not two.
Actionable. There is something a human can do about it right now. If the response is “watch it,” it’s a dashboard. If the response is “file a ticket,” it’s a ticket.
Urgent. It cannot wait until business hours. A large fraction of alerts fail here — they describe a real problem that would be better handled on Tuesday morning by someone rested.
Requires a human. If the remediation is deterministic, automate it. An alert whose runbook is three mechanical steps is a script that hasn’t been written yet.
Apply this to an existing alert set honestly and it typically removes a substantial fraction. That’s not a sign the alerts were badly written. Most were reasonable when added, under conditions that no longer hold.
What to look at first#
You don’t need to reason about every alert individually. A few queries do most of the work:
Alerts that have never fired. They’re either unnecessary or broken, and you can’t tell which. An alert that has never fired is untested code in the most critical path you have.
Alerts that fire constantly. If something pages daily and the system is fine, the threshold is wrong or the condition isn’t meaningful. This is the category responders have already learned to ignore, so deleting it changes nothing operationally — it only makes the ignoring official.
Alerts with no action rate. This requires tracking outcomes, which most teams don’t. It’s worth adding: for each page, did the responder do anything? Alerts with a near-zero action rate are pure cost.
Duplicate alerts on one failure mode. A single dependency failure that generates eleven pages across four services isn’t eleven signals. Alert on the symptom, and let the causes be diagnostic detail.
Alerts on causes rather than symptoms. High CPU is not an incident. Users failing is an incident. High CPU while users are fine is a capacity planning input.
Symptom-based alerting, and its honest cost#
The standard advice is to alert on symptoms — user-visible impact — rather than causes. I agree with it, and I want to be straight about what it costs.
Cause-based alerts can fire before users are affected. Disk filling, memory climbing, a queue growing. Symptom-based alerts, by definition, fire once impact has begun. You are trading detection lead time for a smaller, more trustworthy alert set.
For most systems that’s the right trade, because the lead time was theoretical anyway — buried in noise nobody was reading. But it isn’t universal. If a failure builds slowly and takes a long time to remediate, a predictive alert genuinely earns its place. Disk exhaustion is the classic example: by the time users notice, you have a much worse problem, and the fix isn’t instant.
The rule I’d defend is narrower than the usual formulation: page on symptoms, plus a small deliberate set of predictive alerts where lead time is genuinely actionable. Keep that second set small and justify each member individually.
What you’re actually risking#
Here’s the part that makes people hesitate, and they’re right to.
Some alert you delete will eventually have been the one that would have caught something. Not hypothetically — it will happen, and there will be a retrospective, and someone will note that this was monitored until recently.
That’s a real cost, and the argument for accepting it is comparative, not absolute: chronic alert fatigue causes more missed and slow-handled incidents than the deleted alert would have caught. But you should make that trade deliberately, with agreement, and preferably in writing beforehand — because after an incident is a terrible time to be litigating it.
Two things make it defensible:
Delete in batches, with review. A documented decision by several people is a different thing from an individual quietly removing pages.
Keep the underlying signal. Deleting an alert doesn’t mean deleting the metric. Move it to a dashboard. The data remains available for diagnosis; what’s removed is the interruption. Almost all the value with almost none of the cost.
Replace the removed alerts with two things#
Deletion alone leaves a gap. Fill it structurally rather than by re-adding.
Error budget burn rate alerting. Instead of alerting on many possible causes, alert on the rate at which you’re consuming your reliability budget. Multi-window — a short window for acute failures, a longer one for slow erosion. This catches novel failure modes, which cause-based alerting fundamentally cannot, because you can only alert on failures you anticipated.
Runbooks attached to what remains. Every surviving alert should link to what to check next. This is the difference between a five-minute and a fifty-minute response for anyone who isn’t the system’s author — and it’s also how on-call stops depending on the two people who can respond from memory.
Make the alert set an object with an owner#
The reason alert sets decay is that no one owns them. Individual alerts have authors; the set has nobody.
The practice that works is unexciting: a recurring review of what fired, what led to action, and what didn’t. Delete accordingly. It takes under an hour and it’s the highest-return recurring meeting on an on-call team’s calendar.
Track alert quality as a metric in its own right — fire count, action rate, false-positive rate per alert. Without measurement you’re arguing from impressions, and the person who added the alert always has a more vivid story than the person who wants it gone.
The reframe that makes this land#
Reliability work is usually pitched as adding capability. This is subtraction, which is a harder sell, so frame it in terms responders recognize:
We currently page seventeen times a week. Fourteen of those require no action. Our real detection latency is not our alerting latency — it’s alerting latency plus the time it takes someone to believe the page.
That last number is the one that matters, and it’s the one nobody measures. It improves through deletion, not addition.
The counterintuitive part isn’t really counterintuitive once stated: an alerting system’s value is set by how much responders trust it, and trust is destroyed by volume. Every unnecessary page is a small withdrawal from the account you’ll need during a real incident.