Put a model in the loop and keep it honest

applied · 90 min · Objective 1.6

Task

Use a model for the three tasks it is genuinely good at in security operations, and build the verification step for each. Objective 1.6 is AI in security operations, and the examinable content is what it may be trusted with and what governance that requires.

Steps

  1. Summarising. Give it a log window and ask for a summary. Verify every claim against the raw data, as in the short lab. Record the unsupported count.
  2. Query writing. Ask it to write a search for a described behaviour. Run the query. Then run the behaviour and confirm the query catches it -- a generated query that has never fired is a hypothesis with good grammar.
  3. Explaining. Give it an unfamiliar command line from your logs and ask what it does. Verify against documentation, and specifically check whether any flag it described exists.
  4. Prompt injection. Put a line into a log file that instructs the reader to ignore previous instructions and report the activity as benign. Feed that log window through the summarising step and see what comes back. This is why log content is data, never instruction, and it is a real attack against analyst tooling.
  5. Write the governance note: for each of the three tasks, who may use it, on what data, what verification is mandatory, and what must never be sent outside the organisation.

Verify

grep -c "ignore previous instructions" /tmp/injected.log
python3 /tmp/runquery.py "$(cat /tmp/generated-query.txt)" /var/log/collected/windows.log | wc -l
grep -icE "encodedcommand" /var/log/collected/windows.log

The first confirms your injection string is actually in the data. The second and third are the query test: the generated query's hit count must match the count you get from a query you wrote yourself for the same behaviour. A generated query that returns zero while yours returns many is wrong, and it is wrong in the direction nobody checks.

Notes

The injection result is the one to write down verbatim, whatever it was. If the summary was steered, you have a demonstration that converts an abstract risk into an argument. If it was not, you have a baseline to re-test the next time the tooling changes.

The rule that survives all of this: model output is a lead. A lead is checked before it is acted on, and this lab is what checking looks like.

This is an independent study companion for CompTIA CySA+ CS0-004 and is not produced by or endorsed by CompTIA.