Write an alert, then tune it until it is worth reading

short · 45 min · Objective 4.4

Task

Write a detection that fires on every occurrence, observe how much noise it produces, then tune it with context until it fires only on the thing you care about — and confirm it still catches that thing.

Steps

  1. Write version one: /tmp/alert1.sh alerts on every Failed password line in the auth log.
  2. Generate a realistic day: a handful of genuine typos across several accounts over ten minutes, plus one burst of fifteen failures against a single account.
  3. Run version one and count the alerts. Note that the real event is buried among the ordinary ones.
  4. Write version two: alert only when one source produces more than N failures against one account within a time window. Choose N from what you observed rather than from a default.
  5. Run it and confirm it fires once, on the burst, and not on the typos.
  6. Now prove it did not overfit: generate a DIFFERENT malicious pattern — a spray of one password across many accounts — and confirm version two misses it. Write the second rule you would need, and record both in /tmp/tuning.md.

Verify

bash /tmp/alert1.sh | wc -l
bash /tmp/alert2.sh | wc -l
python3 - <<'PY'
import subprocess
def n(s): return len([l for l in subprocess.run(['bash',s],capture_output=True,text=True).stdout.splitlines() if l.strip()])
a,b=n('/tmp/alert1.sh'),n('/tmp/alert2.sh')
print('untuned alerts: %d | tuned alerts: %d' % (a,b))
assert a>b, 'tuning did not reduce the noise'
assert b>=1, 'the tuned rule fires on nothing - it has been tuned past the signal'
PY
grep -ciE "spray|second rule|miss" /tmp/tuning.md

Both assertions matter and they pull in opposite directions, which is the whole craft: the tuned rule must be quieter than the untuned one AND must still fire on the real event. A rule tuned until it never fires is the failure mode that looks like success on a dashboard.

Notes

The last step is the one that generalises. Your tuned rule is correct and narrow, and a different attack walks straight past it — which is why detection is a portfolio rather than a rule, and why the threat hunting lesson insists a hunt should leave a new rule behind.

This is an independent study companion for CompTIA Security+ SY0-701 and is not produced by or endorsed by CompTIA.