Prove an AI-written test suite detects anything, with mutation testing

short · 40 min · Objective 3.3

Task

Ask your lab model to write unit tests for your RAG app's access filter, then measure whether those tests catch five planted faults. Next, write tests from the access-control requirement rather than from the code, and measure again. The gap between the two results is the lesson's warning about tests that assert what the code does rather than what it should do.

Steps

  1. Write lab/access.py containing can_read(meta, role). It returns True for documents classified public or internal, True for restricted ones only when role == 'hr', and False for any unknown classification. If your app's filter differs, write this one to that specification anyway, because the requirement is what the second set of tests is written from.
  2. Through the gateway as user test-gen, with "retrieve": false, give the model the function's source and ask for unittest tests. Raise lesson 10's MAX_OUTPUT_TOKENS for this request if the tests come back cut off, and put it back afterwards. Save them unchanged as lab/tests_ai/test_access.py.
  3. Write lab/mutate.py, which takes a test folder as its argument and makes five mutants of access.py: == changed to != in the role check; the restricted check removed; the unknown-classification default flipped to True; 'hr' changed to 'HR'; and the function returning True outright. It runs the folder's tests against each mutant in a temporary directory and prints one line per mutant ending KILLED or SURVIVED.
  4. Run it against tests_ai and record each result in lab/mutation.csv with header mutant,change,suite,result.
  5. Now write lab/tests_req/test_access.py from the requirement, not the code: an hr reader can open a restricted document, every other role cannot, an unknown classification is refused, and the role comparison is exact. Run mutate.py against it and add its rows.
  6. In lab/mutation-notes.md, name the mutants the AI suite let through and say why: did its assertions describe what the code does, or what it should do?

Verify

cd ~/secai-lab/lab
python -m unittest discover -s tests_req -q; echo "requirement suite on the real code exit=$?"
python mutate.py tests_req | grep -c SURVIVED
python mutate.py tests_ai | grep -c SURVIVED
python - <<'EOF'
import csv
rows = list(csv.DictReader(open('mutation.csv')))
for suite in ('tests_ai', 'tests_req'):
    mine = [r for r in rows if r['suite'] == suite]
    print(suite, sum(r['result'] == 'KILLED' for r in mine), 'of', len(mine), 'mutants killed')
EOF

The requirement suite passes on the real code (exit 0) and lets no mutant survive (0), and the CSV shows tests_req killing 5 of 5. The AI suite's survivor count is whatever your run produced, usually 1 or more. If the requirement suite also lets a mutant survive, the requirement test it should have tripped is missing, and that is the finding to fix.

Notes

A suite with no survivors is also the only kind worth gating a pipeline on. The applied lab for this lesson runs mutate.py as a pipeline stage for that reason: coverage tells you which lines ran, while mutation testing tells you whether anything would have noticed them being wrong.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.