Compute your metrics, then game them on purpose

short · 50 min · Objective 4.2

Task

Calculate the standard operational metrics from your own incident data, then deliberately improve every one of them without improving security at all. The second half is what makes the first half safe to report.

Steps

  1. From your staged incidents, compute: time to detect (from the first attacker action in the timeline, not from the alert), time to respond, time to contain, and dwell time.
  2. Report the median and the highest value rather than the mean, and say how many incidents the figures are drawn from.
  3. Compute the coverage counterweights: telemetry coverage against the independent inventory, techniques tested and alerting, and log source health.
  4. Compute detection source distribution: how many of your incidents were found by a rule, by you looking, or only because you knew where to look.
  5. Now game them. Recompute each metric after making one change that improves the number and nothing else: exclude the host with the worst data, start the clock at the alert instead of the first action, reclassify the slowest incident as an event, narrow the scan scope, and disable your noisiest rule.
  6. Put the honest and gamed figures side by side, and for each pair write the counterweight metric that would have exposed the change.

Verify

awk -F'\t' 'NR>1{print $2-$1}' /tmp/incident-times.tsv | sort -n | awk '{a[NR]=$1} END{print "median", a[int((NR+1)/2)], "max", a[NR], "n", NR}'
comm -23 /tmp/active.txt /tmp/agents.txt | wc -l
awk -F'\t' '$3=="alerted"' /tmp/coverage.tsv | wc -l
awk -F'\t' '$3=="invisible"' /tmp/coverage.tsv | wc -l
awk -F'\t' '{print $4}' /tmp/incident-times.tsv | sort | uniq -c

The median with its n beside it -- a median over three incidents is a narrative with dates, not a statistic, and printing n is what stops it being read as one. The uncovered host count and the alerted/invisible split are the counterweights. The last is detection source distribution, and a rising share of "found because I looked" is the finding no timing metric shows.

Notes

Every timing metric in step 1 improved in step 5 without a single security improvement. That is not a flaw in the metrics; it is what measurement does to anything with consequences attached, and the only defence is pairing.

Add one metric that can only get worse when you look harder -- techniques tested and failed, or hunts that found something. It is the only kind of number that cannot be improved by looking away, and a programme that reports one is telling you something real.

This is an independent study companion for CompTIA CySA+ CS0-004 and is not produced by or endorsed by CompTIA.