Metrics that change behaviour

Why this matters

The last lesson of the course, and it closes the loop: everything measured in Domain 1 and Domain 2 becomes a number, the numbers become a report, and the report either changes what the organisation does or does not.

Metrics are examinable — the standard ones have names and definitions worth knowing exactly. They are also the place where a security programme most easily deceives itself, because every metric in this lesson can be improved by detecting less, and none of them says so on its face.

The lesson

Detection time, response time, dwell time

The core operational metrics, and worth defining precisely because the exam does.

• Mean time to detect (MTTD) — from the start of malicious activity to detection. The crucial detail is the start point: it comes from the timeline reconstructed afterwards (lesson 30), not from the first alert. Measuring from the alert measures nothing, because the alert is the detection.
• Mean time to respond or acknowledge (MTTR/MTTA) — from detection to someone actually beginning work. Measures queue and staffing, not skill.
• Mean time to contain — from detection to the spread being stopped. The most operationally meaningful of the set, because it bounds the damage.
• Mean time to remediate or recover — to full eradication and restoration.
• Dwell time — total attacker presence, from first activity to eradication. The headline number in industry reporting, and the one executives recognise.

How to use them honestly:

• Report the median and a percentile, not the mean. One long incident destroys an average. The median says what a typical incident looks like; the 90th percentile says what the bad ones look like, and both are needed.
• Segment by incident type. Phishing and a hands-on intrusion are not comparable, and mixing them produces a number describing neither.
• Beware small denominators. Most organisations do not have enough incidents for these to be statistics. Four data points are a narrative with dates, and presenting them as a trend line invites conclusions the data cannot carry.
• Watch the perverse incentive, which is severe here. MTTD only counts incidents you detected. Detect fewer, and specifically miss the long quiet ones, and your MTTD improves. The metric rewards exactly the failure it is supposed to reveal. Every one of these numbers needs the coverage measures below beside it, or it is worse than nothing.

Coverage and detection efficacy

The counterweight. These measure how hard you are looking, which is the denominator the timing metrics silently assume.

Coverage metrics:

• Telemetry coverage — the proportion of assets sending the logs you rely on, measured against an independent inventory. Measured against the tool's own list it is circular and always high, which is lesson 18's point.
• Technique coverage — how much of the relevant part of ATT&CK you can detect, from lesson 27, counted in tested detections rather than log sources that could theoretically support one.
• Asset coverage of scanning, from Domain 2, against the same independent inventory.
• Log source health — sources currently reporting against sources expected. A source that silently stopped is the most common invisible failure in a SOC, and it makes everything downstream look calm.

Efficacy metrics:

• Detections validated by testing — how many rules have actually been proven to fire, per lesson 14. An untested rule is a hypothesis.
• Purple team or emulation results: of N techniques executed, how many alerted, how many were logged but not alerted, how many left no trace. This is the highest-quality evidence available about whether your detection works, because it is the only measurement that produces a real negative.
• Detection source distribution — internal tooling, internal person, external party. A rising external share is a serious finding, and it is the metric that catches a detection capability quietly degrading while every timing number looks fine.

The reason this section exists: timing metrics measure the incidents you found; coverage metrics estimate the ones you would find. Reporting the first without the second is the single most common way a security dashboard is confidently wrong.

False positive rate as a tuning signal

A metric that is routinely mistaken for a quality score, when it is a control knob.

The definitions to know: a true positive is real and alerted, a false positive alerted and was not real, a false negative was real and did not alert, and a true negative correctly stayed quiet.

Why a low false positive rate is not automatically good: you can reach zero by alerting on nothing. Detection is a trade, and moving the threshold trades false positives against false negatives. The right position depends on what is being detected — a high false positive rate is acceptable for a high-impact technique that is rare, and unacceptable for a common event, because at high volume the whole class gets ignored.

What the number is actually telling you:

• A rule with a persistently high false positive rate needs tuning, an enrichment step, or retirement. Left alone, it trains analysts to dismiss that alert type without looking, which turns it into a guaranteed miss.
• A rule that has never fired at all is not a low false positive rate. It is an untested rule, and it should be tested before it is trusted.
• The real cost is analyst attention, which is finite. Alert volume per analyst per shift is the metric that predicts burnout and missed detections, and it belongs in the report.
• False negatives are the number that matters and the number you cannot measure directly. You only learn about the ones you eventually discover. Purple teaming and hunting are how you generate that evidence deliberately — which is why they are the only real answer to "are we missing things".

Reporting cadence and audience

Metrics have to arrive somewhere they can be acted on, which makes cadence and audience part of the design rather than logistics.

A workable structure:

• Operational, weekly, to the SOC and its manager: alert volumes, tuning actions, detections added and tested, log source health, open incidents. Content the team can act on this week.
• Tactical, monthly, to security leadership and system owners: vulnerability trends and SLA compliance from lesson 36, incident summary, coverage movements, action completion.
• Strategic, quarterly, to executives and risk committees: are we improving, where is the material exposure, what investment decisions are being asked for, how do we stand against obligations.
• Ad hoc, immediately, when something material changes — a major incident, a critical exposure, a capability that has broken.

What makes the cadence work:

• Same format each time, so trends are visible and the reader knows where to look.
• Matched to the decision cycle. A quarterly report to a board that sets budget annually arrives too late to change anything; align it to when the decision is actually made.
• Every metric has an owner and an action. A number that nobody can move should not be in the report — it is decoration, and decoration teaches people the report is decoration.
• Say what changed and why, which is the part readers actually use. A chart with no annotation gets glanced at; a sentence explaining the step change gets read.

Gaming: metrics that get met instead of improved

The closing idea of the course, because it is the failure mode that undoes everything else in it.

Any metric that carries consequences will be optimised, and the cheapest way to improve a number is rarely the thing the number was meant to encourage. This is not dishonesty; it is what measurement does to people under pressure.

Worked examples, all of which occur:

• MTTD improves by not detecting the slow, quiet intrusions — the ones that matter most.
• Incident counts fall by classifying more things as "events" rather than incidents, or by declaring later.
• Vulnerability counts fall by narrowing scan scope, dropping a credential so the scan goes uncredentialed and finds less, or excluding a noisy asset group.
• SLA compliance improves by starting the clock at ticket creation and creating tickets late, or by reclassifying severity downward.
• Remediation rates improve by closing tickets on assertion rather than on a verified rescan — lesson 24's point, as a metric failure.
• False positive rates improve by disabling noisy detections rather than tuning them.
• Patch compliance improves by removing hosts from the inventory.

How to build metrics that resist it:

• Pair every metric with its counterweight. Timing with coverage. Findings with scan scope. Incident count with detection source. Alert volume with purple team results. A single number can always be gamed; a pair is much harder, because improving one by cheating visibly degrades the other.
• Show the denominator, always.
• Define the clock start where it cannot be manipulated — discovery, not ticket creation.
• Close on evidence, not on assertion.
• Use metrics to ask questions, not to judge people. The moment a number determines someone's performance review, you are measuring their reporting behaviour rather than the world. This is the whole mechanism, and it is why blameless review from lesson 35 and honest metrics are the same idea.
• Include a metric that can only get worse when you look harder, such as techniques tested and failed, or hunt hypotheses that found something. It is the only kind of number that cannot be improved by looking away, and a programme that reports one is telling you something real.

And so the course ends where it began, with the sentence it has approached from every direction. An organisation with excellent security and one that has simply not noticed its intrusions produce the same clean reports, the same green dashboards and the same quiet quarters. Every artefact is identical. Only a deliberate test — a hunt, a purple team, a restore, an exercise, a scan you were afraid to run — tells you which one you are. Being the analyst who insists on that test, and who reports its result accurately when it is unflattering, is the job.

Topics this lesson owns

• [x] Detection time, response time, dwell time
• [x] Coverage and detection efficacy
• [x] False positive rate as a tuning signal
• [x] Reporting cadence and audience
• [x] Gaming: metrics that get met instead of improved