False positives, false negatives, and tuning without going blind

Why this matters

Tuning is where detection programmes are won and lost, and the failure is asymmetric in a way that makes it dangerous.

A false positive is visible, immediate and annoying: an analyst investigates something benign and loses time. The pressure to suppress it is constant and comes from the people doing the work.

A false negative is invisible. Nothing indicates that a rule failed to fire, and the cost arrives months later as an incident. Nobody feels pressure to fix something they cannot see.

So the natural drift of every detection programme is toward fewer alerts and more blindness, and every step of that drift is individually justified. CAS-005 tests whether you can describe the discipline that resists it: narrowing rules rather than deleting them, suppressing with expiry, and measuring coverage against a framework rather than counting alerts.

The lesson

The two error types stated in operational terms, with the cost of each

• True positive — malicious activity, alerted. The system working.
• False positive — benign activity, alerted. Cost: analyst time, and cumulatively, credibility.
• True negative — benign activity, no alert. The bulk of everything.
• False negative — malicious activity, no alert. Cost: the incident.

The costs differ in kind, not only in degree, and the difference is the whole lesson.

False positive cost is linear, visible and shared. Each one costs some minutes. It is felt by the team continuously, and it is measurable.

False negative cost is lumpy, invisible and deferred. Most cost nothing because most missed events were not part of a successful attack. One is a breach. It is felt by the organisation, not the team, and it is unmeasurable in advance.

Two consequences follow, and both are examinable.

Alert fatigue is a real failure mode, not a complaint. A team receiving more alerts than it can investigate is effectively running a random sampling process, and a high-fidelity alert arriving in a flood of noise is missed exactly as thoroughly as if the rule had not existed. So excessive false positives cause false negatives, which is why tuning is necessary rather than merely comfortable.

The correct target is not zero false positives. A detection tuned until it never fires on anything benign has almost certainly been narrowed until it misses variants of the real thing. Some false positive rate is the price of coverage, and a team reporting near-zero should be asked what it is missing.

The precision-recall trade is the formal statement of this, and the practical version is: tune for a rate the team can sustain, and know what the narrowing cost you.

Tuning a noisy rule by narrowing the condition rather than deleting the rule

The method, and the difference between the two options is large.

Deleting a rule removes all coverage of that behaviour. Nothing records that it once existed, and the gap is invisible from that moment.

Narrowing a rule keeps the detection and excludes the specific benign case, and the exclusion is documented and reviewable.

The narrowing sequence, in order of preference:

1. Understand why it fires. A rule firing 200 times a day on one administrative tool is different from one firing on 200 different hosts. The investigation is the work; the exclusion is the easy part.
2. Exclude the specific cause, as narrowly as possible. This process, from this parent, on these hosts, under this account, within this window. Not "this process name", which excludes the attacker using the same name.
3. Add a condition that distinguishes rather than excludes. If the benign case always has a property the malicious case would not — a signed binary from a specific path, a parent that is the management agent — condition on it.
4. Split the rule. One version high-fidelity and alerting, one broader version logging to a lower-priority queue or feeding a hunt. This preserves the coverage while removing it from the queue that must be worked.
5. Change the response, not the detection. Some rules should never have been alerts; they are context. Route them to enrichment rather than to an analyst.

Two rules about exclusions that scenarios test. Exclusions must be specific to what was observed, because broad exclusions are what attackers live in — an exclusion by process name alone is an instruction for how to evade the rule. And every exclusion is recorded with a reason and an author, so that a future reader can evaluate it; an unexplained exclusion list is indistinguishable from an attacker's persistence mechanism, and that is not a hypothetical.

Suppression with an expiry date, so a temporary exception cannot become permanent

Suppression silences an alert for a period — during a migration, a penetration test, a known maintenance window, or while a noisy system is fixed. It is legitimate and it is the mechanism most likely to become permanent by accident.

The governance is lesson one's exception machinery, applied here:

• Every suppression has an expiry, and expiry is enforced by the platform rather than by a calendar reminder. This is the single control that matters.
• Every suppression names an owner and a reason.
• Suppressions are reviewed as a set, periodically, and the review looks at the whole list rather than at new additions.
• Suppressed events are still recorded, just not alerted. This preserves the data for hunting and for retrospective investigation, and it means a suppression is a decision about the queue rather than about collection.
• Suppression scope is as narrow as possible — this rule, this host, this window — not the rule globally.

The failure to recognise, and it is a common scenario: a suppression added during a migration two years ago, still active, covering the exact behaviour that was later used in an intrusion. Nobody did anything wrong at any point except omit an expiry date.

A related trap worth naming: suppression during a penetration test. Teams frequently suppress alerts to avoid noise during an authorised test, which both defeats the purpose of the test — you learn nothing about whether you would have detected it — and creates a window in which real activity is unmonitored. The better practice is to run the test without suppression, recording which detections fired, because that record is the most valuable output the test produces.

Measuring detection coverage against ATT&CK instead of counting alerts

Programmes report alert volume, cases closed and mean time to respond. All are activity measures, and none answers the question that matters: what can we detect, and what can we not?

Mapping detections to a behaviour framework answers it, and it is lesson nine's material consumed operationally. For each technique relevant to your environment, record whether a detection exists, what data source it depends on, what proportion of the estate that source covers, and when it was last validated.

That produces a coverage map with honest gaps, and three things follow:

• Gaps become a plan. A technique with no detection is a work item with a known data-source dependency, which turns a vague request for budget into a specific one.
• Ingestion decisions become justifiable, per lesson forty-one — this source buys these techniques.
• Coverage claims acquire a denominator, which stops "we have 400 detections" being read as a measure of anything.

Two cautions, because a coverage map is itself a claim that can be wrong:

• A technique marked covered by one brittle rule is not covered. Techniques have many implementations, and a rule matching one tool's default behaviour fails against the same technique performed differently. Depth of coverage matters, not presence.
• Coverage is weighted by the data source's estate coverage. A detection depending on endpoint telemetry present on 60% of hosts is 60% covered, and recording it as covered repeats the inventory error from lesson five in a new place.

The measure worth reporting alongside it: validated coverage — techniques where a detection exists and has been shown to fire on a test of that behaviour. Which is, once again, the canary discipline: a detection nobody has watched fire is a detection nobody has evidence about.

The tuned-out alert that would have caught the incident, and how to avoid creating it

The closing idea, and it is the failure this lesson exists to prevent.

The sequence is always the same. A rule is noisy. An analyst excludes the noisy case broadly, because the narrow exclusion is more work and the broad one solves today's problem. Months later an intrusion uses behaviour inside that exclusion. The rule was present, enabled, and blind.

Five practices that prevent it, in order of effect:

• Exclude the observation, not the category. The narrowest exclusion that removes the observed noise, every time.
• Review exclusions in the retrospective of every incident. For each incident, ask whether any exclusion or suppression covered part of the attack path. This turns exclusions into something inspected rather than accumulated, and it is the practice most likely to actually find one.
• Expire everything. Suppressions expire automatically; exclusions are reviewed on a cycle with an owner.
• Keep the broad version alive somewhere. When narrowing a rule, retain the broad version as a hunt query or a low-priority feed. The coverage is not lost, it is relocated, and the hunt in lesson forty-six can consume it.
• Validate after tuning. Re-run the test that the rule was written for. A rule narrowed until it no longer fires on the behaviour it was built to catch is the specific defect, and a validation test is the only thing that detects it.

The principle underneath all five, and it is the one to carry: every tuning decision is a decision about what you will not see, and it should be recorded as such. A change log of exclusions reads as bureaucratic until the day somebody needs to know why a rule did not fire — at which point it is the only document that can answer.