Check a model's summary against the raw log

short · 45 min · Objective 1.6

Task

Take an AI-written summary of an incident and verify every factual claim in it against the underlying data, counting the ones that are unsupported. This is the discipline that makes model output usable: it is a lead, and a lead is checked.

Steps

  1. Take the log window from one of your earlier labs -- the identity attack patterns are ideal -- and have a model summarise what happened in a paragraph, or write that paragraph yourself from memory.
  2. Split the summary into individual claims, one per line. A claim is anything that could be true or false: an account name, a count, a time, an ordering, a causal link.
  3. For each claim, go to the raw log and mark it: supported (you found the evidence), unsupported (you could not), or contradicted.
  4. Pay particular attention to counts, times and orderings, which is where confident-sounding text most often drifts.
  5. Compute the ratio. Then ask the important question: would you have noticed any of the unsupported claims if you had not checked?

Verify

grep -c "Failed password" /var/log/auth.log
grep "Failed password" /var/log/auth.log | awk '{print $(NF-5)}' | sort -u | wc -l
grep "Accepted" /var/log/auth.log | head -1 | cut -c1-15
grep "Accepted" /var/log/auth.log | tail -1 | cut -c1-15

These four give you the ground truth for the claims summaries most often get wrong: how many failures, across how many accounts, and the real first and last successful login. Compare each against the corresponding sentence in the summary. A summary that names a number not in this output is asserting something it does not know.

Notes

The useful output is the count of unsupported claims, and it is usually not zero. Nothing about the writing quality signals which sentences they are -- fluency and accuracy are independent, and that is precisely why model output is a lead rather than a finding.

Do this once and keep the number. It is the evidence you will need the first time someone proposes putting model output straight into a report.

This is an independent study companion for CompTIA CySA+ CS0-004 and is not produced by or endorsed by CompTIA.