Pit a signature rule against an anomaly detector on synthetic gateway traffic

short · 45 min · Objective 3.1

Task

Generate a week of synthetic gateway traffic with four planted events, then run a signature detector and a baseline anomaly detector over it. Each one catches something the other misses, and one of the anomaly detector's hits is a legitimate change rather than an attack. Measure all of that instead of asserting it.

Steps

  1. Write lab/make_synthetic_log.py, seeded with random.seed(7), that writes lab/synthetic-gateway.jsonl. It covers seven days of traffic for five users (alice, bob, carol, dan and svc-report), with the gateway's fields plus id and planted, decision set to allow and chunk ids in the path#n form. People work office hours at their own steady rates, svc-report calls every hour around the clock, and prompts are drawn from a list of ordinary questions about your documents. planted is an empty string on every normal record.
  2. Plant four events on day seven and label each one in planted. burst: 120 extra requests from dan in one hour. exact: replace the prompt text of three existing carol records with prompts containing the phrase ignore previous instructions. paraphrase: do the same to three more, with the same request in other words, such as disregard what you were told earlier. newuser: a new user erin working at an ordinary rate, standing in for a legitimate starter.
  3. Write lab/detect_traffic.py (lesson 15's lab/detect.py is a different detector; keep it). It must never read planted. The signature detector flags any prompt containing the exact phrase. The anomaly detector learns each user's requests per working hour from days one to five only, a fixed reference baseline rather than a rolling window. On days six and seven it flags every record in a user-hour whose count is more than three standard deviations above that user's mean, using a floor of 1 for the standard deviation so svc-report does not divide by zero. A user with no baseline is flagged, and the reason says so.
  4. Write every alert to lab/alerts.csv with header id,user,detector,score,reason. Give the script --threshold and --out options, then run it a second time with --threshold 2 --out lab/alerts-2sd.csv.
  5. Write lab/detect-findings.md with one line per planted label. Say which detector caught it, which missed it, and whether the hit is malicious or merely unusual. For newuser, say what context, such as an HR starter record, would have suppressed the alert before an analyst saw it.
  6. Under a heading ## Threshold, record how the number of alerts on unplanted records changed between three and two standard deviations, and which setting you would run if an analyst had ten minutes per alert.

Verify

cd ~/secai-lab
python - <<'EOF'
import csv, json
label = {}
for line in open('lab/synthetic-gateway.jsonl'):
    r = json.loads(line)
    label[r['id']] = r['planted']
for name in ('lab/alerts.csv', 'lab/alerts-2sd.csv'):
    hits, fp = {}, 0
    for a in csv.DictReader(open(name)):
        p = label.get(a['id'], '')
        if p:
            hits.setdefault(p, set()).add(a['detector'])
        else:
            fp += 1
    print(name)
    for p in ('burst', 'exact', 'paraphrase', 'newuser'):
        print('   ', p, sorted(hits.get(p, set())) or 'missed')
    print('    alerts on unplanted records:', fp)
EOF
grep -c planted lab/detect_traffic.py

At three standard deviations, burst is caught by anomaly only, exact by signature only, paraphrase is missed by both, and newuser is caught by anomaly, which is a false positive caused by a legitimate change. The two-standard-deviation file shows more alerts on unplanted records. The grep must print 0. A detector that mentions planted has seen the answer key, and its results mean nothing.

Notes

The paraphrase row is the lesson's argument for layering, in one line. The signature misses a reworded instruction, and the anomaly detector cannot see it either, because one changed prompt does not change anyone's request rate. Catching it needs a third technique, a classifier trained on labelled examples of the pattern, and lesson 10's prompt firewall is where that would live.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.