Alerting that someone will actually answer

Why this matters

Monitoring collects. Alerting interrupts a human being. That difference is the whole subject: an alert is a claim that something is worth waking someone for, and an organisation that makes that claim carelessly ends up with people who ignore alerts.

Alert fatigue is not a soft problem. It is the mechanism by which a real incident goes unnoticed for two hours inside a system that detected it immediately and said so, into a channel nobody reads any more. Every rule in this lesson exists to keep alerts credible.

The lesson

Alerting on symptoms the user feels rather than on every resource that twitches

The most useful principle in the topic: alert on symptoms, not causes.

A symptom is something the user experiences — requests failing, requests slow, work not being processed, data not arriving. A cause is an internal condition — high CPU, a full queue, a restarted process.

Why symptoms are the right trigger:

• Causes are not reliably problems. High CPU on an autoscaling group is the system working. A restarted container is self-healing doing its job. Alerting on those produces noise proportional to how elastic your system is.
• Symptoms are always problems. If users are getting errors, something is wrong regardless of which internal metric explains it.
• Symptom alerts survive re-architecture. Cause alerts are tied to today's implementation and go stale silently.
• You cannot enumerate all the causes. You can enumerate the handful of ways the service fails its users.

This is the reasoning behind service level indicators and objectives: pick a small number of measures that represent the user's experience — availability, latency, error rate, freshness — set an objective for each, and alert when you are at risk of missing it.

Causes still get monitored and put on dashboards; they are what you look at after the symptom alert fires, and they make diagnosis fast. The distinction is between what is collected and what interrupts someone.

The exceptions that legitimately alert on a cause are predictive: a disk filling at a rate that will exhaust it in four hours, a certificate expiring in a week, a quota approaching its limit, a backup that has not succeeded in three days. These warn before a symptom exists, which is the point.

Thresholds, durations and hysteresis, and the flapping alert that gets muted forever

The mechanics of an alert that does not cry wolf.

Thresholds should come from the baseline (previous lesson), not from a round number. 80% is a habit, not a measurement. Where the metric is user-facing, derive the threshold from the objective instead: alert when the error budget is being consumed too fast.

Duration is the most underused setting. Requiring a condition to persist — five minutes rather than one sample — removes the great majority of spurious alerts, because most transients resolve themselves. The cost is detection latency, so it is a deliberate trade: short durations for outright failure, longer ones for degradation.

Hysteresis means using different thresholds to fire and to clear — fire above 90, clear below 70. Without it, a metric hovering at the threshold produces an alert that opens and closes repeatedly. That is flapping, and flapping is how an alert gets muted by an exhausted human, after which it protects nothing and everyone believes it does. A muted alert is worse than no alert, because it appears on the list of controls.

Other mechanics that keep the signal clean:

• Grouping and deduplication. One database failure should produce one notification, not forty from every service that depends on it.
• Dependency suppression. If the network is down, suppress the alerts for everything behind it. Sending the symptom of the cause you already reported is noise.
• Maintenance windows. Silence alerts for planned work — and set an expiry, because a silence with no end date is how an alert disappears permanently.
• Auto-resolution. Alerts should clear themselves when the condition ends. Manual acknowledgement of resolved alerts is a backlog nobody clears.

Alert fatigue as an operational risk with a measurable cause

Alert fatigue deserves naming as a risk with a cause and a metric, because that is what makes it addressable rather than a complaint.

The mechanism is simple: people can sustain attention on a low-volume, high-signal channel. Raise the volume and lower the signal and they stop reading it — not through carelessness but because the channel stopped being informative.

It is measurable, and these are the numbers to track:

• Alerts per shift, especially out of hours. A person who is woken three times a night is not effective the next day, and this is a staffing and safety issue as much as a technical one.
• The proportion that required action. If most alerts required none, most alerts should not exist. This is the single most revealing metric here.
• The proportion auto-resolved before anyone looked, which says the condition was transient and the alert premature.
• The repeat offenders. A small number of alerts usually generate most of the volume, so fixing a handful fixes the problem.

Treat it as a maintained system. Review alerts on a schedule — monthly is reasonable — and for each noisy one decide: raise the threshold, add duration, fix the underlying flakiness, downgrade it from a page to a ticket, or delete it. Deleting an alert is a legitimate and underused outcome.

A useful standard: every alert that pages a human should have a documented action. If the answer to "what do I do when this fires?" is "look at it and usually nothing", it is not a page. It is a dashboard.

Routing, escalation and on-call, so an alert reaches a person who can act

An alert that fires into an unwatched channel has not alerted anyone.

Severity tiers, and it is worth having few:

• Page. Wake someone now. Reserved for user-visible impact or imminent impact. Should be rare.
• Ticket. Needs attention in working hours. The majority of legitimate alerts.
• Informational. Logged, dashboarded, never notified.

Routing sends each alert to the team that owns the affected service — which requires ownership to be recorded, and is another use for the tagging discipline from objective 1.8. A central team receiving every alert for every service cannot act on most of them and becomes a relay.

Escalation covers the case where the first responder does not acknowledge: after a defined interval, notify the next person, then a manager. Without it, a page to someone whose phone is silent is a page to nobody.

On-call is the human system underneath, and it has requirements that are easy to skip: a published rota, a handover, compensation, a documented scope of what the on-call person is expected to handle, and a limit on how often anyone is on it. A rota that burns people out produces slower responses, which is the opposite of the intention.

Two supporting practices: a runbook link in every alert, so the responder starts with instructions rather than a search; and a blameless post-incident review for significant incidents, whose actions should include alerting improvements — the alert that should have existed, the one that fired too late, the one that was ignored because it always fires.

Dashboards for understanding, alerts for waking someone, and not mixing the two

The closing distinction, and a genuinely useful discipline.

Dashboards are for looking at deliberately — during an investigation, after a deployment, in a capacity review. They can be dense, detailed and full of cause-level metrics. Nobody is watching them continuously, and designing as though somebody is leads to the "wall of screens" that is impressive and unread.

Alerts are for interrupting. They must be few, specific and actionable.

Mixing them produces both failure modes: dashboards that only show what is already alerting, and therefore cannot help diagnose; and alerts for everything on the dashboard, and therefore fatigue.

Practical dashboard design, since this is where the diagnosis actually happens:

• A service overview showing the handful of user-facing indicators — the same ones the alerts watch, so the responder immediately sees the alert in context.
• A drill-down per component with cause-level detail, arranged along the request path so it can be followed.
• Deployment and change markers overlaid on the graphs. Being able to see that a change coincided with a metric moving answers the first question of most incidents in seconds.
• Consistent layout across services, so a responder unfamiliar with a service still knows where to look.

The test of the pair: when a page fires at 3am, does the responder have — in one link — enough to know what is broken, how badly, what changed recently, and what to try first? If yes, the observability is doing its job. If they have to go hunting, the work belongs to this lesson rather than to them.

What to take into the exam

• Alert on symptoms users feel, not on internal causes. Causes go on dashboards. The legitimate cause-alerts are predictive — disk filling, certificate expiring, quota approaching, backup failing.
• Duration and hysteresis remove most noise. Flapping alerts get muted, and a muted alert is worse than none because it still appears to be a control.
• Suppress dependent alerts and group duplicates.
• Alert fatigue is measurable: alerts per shift, and the share that required action. Deleting an alert is a valid fix.
• Every page needs a documented action. No action means it is not a page.
• Route by ownership, escalate on non-acknowledgement, and give on-call a real rota and a runbook link in the alert.
• Dashboards explain; alerts interrupt. Overlay deployment markers — "what changed?" is the first question of most incidents.