Check an AI-drafted incident timeline against its source log

short · 40 min · Objective 3.1

Task

Ask your lab model to turn a synthetic proxy log into an incident timeline, then check every claim in the draft against the log line it rests on. The log reproduces the lesson's scenario: two internal hosts contacted the same command-and-control domain, the second only once and among routine traffic. The question is whether the draft says so, and whether your checking would have noticed if it did not.

Steps

  1. Write lab/incident-proxy.log, by hand or with a short script. Make it about 60 lines, one event per line, each starting with an ISO timestamp followed by the source host, the destination and the action. Most lines are ordinary traffic from 10.20.0.0/24 hosts to internal services. Include a phishing link click from 10.20.0.15 at 09:12, then 10.20.0.15 contacting cdn-update.example.net (203.0.113.45) at 09:14 and every five minutes after that, and 10.20.0.31 contacting the same domain once, at 11:40, among routine lines.
  2. Send the whole log through your gateway (POST $GW/ask) as user ir-draft, with the instruction Summarise this proxy log as an incident timeline with timestamps and affected hosts and "retrieve": false in the body, since the log needs no documents. Lesson 10's limits apply: if the gateway answers 413 block:input-tokens, raise MAX_INPUT_TOKENS, and if the draft stops mid-sentence, raise MAX_OUTPUT_TOKENS, for this lab only; restart the gateway, and put them back afterwards. Save the response unchanged as lab/model-summary.md.
  3. Turn the draft into lab/timeline.csv with header entry,ts,host,claim,source_line,verified. For each claim the model made, find the log line that supports it and record its line number. Copy ts and host from that line, never from the summary. If no line supports the claim, set source_line to 0 and verified to no.
  4. Add every event the draft omitted that belongs in the timeline, with its source line, and start its claim with OMITTED: so the gap stays visible to the next reader.
  5. In lab/timeline-review.md, list each omission, fabrication and wrong timestamp you found, and next to each the rule from the lesson that would have caught it: citations to source, indicators copied from the original, or numbers and dates checked every time.

Verify

cd ~/secai-lab
python - <<'EOF'
import csv
src = open('lab/incident-proxy.log').read().splitlines()
rows = list(csv.DictReader(open('lab/timeline.csv')))
bad = 0
for r in rows:
    n = int(r['source_line'])
    line = src[n - 1] if 0 < n <= len(src) else ''
    if r['verified'] == 'yes' and not (r['ts'] in line and r['host'] in line):
        bad += 1
        print('entry', r['entry'], 'is marked verified but line', n, 'does not show it')
print(len(rows), 'entries;', sum(r['verified'] == 'yes' for r in rows), 'verified;', bad, 'wrongly verified')
print('second host in the timeline:', any(r['host'] == '10.20.0.31' for r in rows))
EOF
grep -c '10.20.0.31' lab/model-summary.md
python -c "import json; print(sum(json.loads(l)['user'] == 'ir-draft' for l in open('logs/gateway.jsonl')), 'ir-draft request(s) in the gateway log')"

Zero wrongly verified entries, and True for the second host in the timeline, whatever the model wrote. The count of 10.20.0.31 in the model's summary tells you whether the omission happened in your run. If it is 0, your timeline must contain an OMITTED: row for it. The gateway log must show at least one ir-draft request, which proves the draft really came from the model and was not written by hand.

Notes

Run step 2 three times and compare the drafts. A small model will usually mention the second host in some runs and drop it in others, and that variation is the point. A check that only works on the run where the model happened to be right is not a check.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.