Review an AI-generated clean-up script before it is allowed to run

short · 40 min · Objective 3.3

Task

Ask your lab model to write a stale-account clean-up script, then put it through the lesson's review before it touches anything. Read every line, check every import, test the script against a known answer you worked out first, make dry-run the default, and leave privileged accounts for a person to decide.

Steps

  1. Write lab/accounts.csv with header user,last_login,enabled,privileged for 30 invented accounts. About a third should have a last login more than 90 days before a fixed reference date, including two privileged accounts, one already disabled, and one exactly 90 days old to test the boundary. Write the reference date to lab/as-of.txt.
  2. Work out the right answer by hand before the model sees anything: the enabled, non-privileged accounts idle for more than 90 days. Write them one per line to lab/expected-stale.txt. A known answer written first is what gives the test its meaning.
  3. Through your gateway, as user script-gen, ask the model for a Python script that disables accounts in that CSV not used for 90 days, with "retrieve": false in the body. Lesson 10's MAX_OUTPUT_TOKENS of 256 can cut a script off part-way, so raise it for this request and put it back afterwards. Save it unchanged as lab/stale_accounts.generated.py and do not run it.
  4. Review it into lab/script-review.md, one line per check, each starting - : what every line does; every import, and whether it is in the standard library; whether privileged accounts are protected; whether it writes in place with no backup; how it parses dates and treats the 90-day boundary; and any shell call, hard-coded path or credential.
  5. Write the reviewed version, lab/stale_accounts.py. It takes --as-of, defaults to a dry run that prints the usernames it would disable and changes nothing, and only with an explicit --apply flag writes a new file, lab/accounts.updated.csv, rather than editing in place. Privileged accounts are never disabled; they are listed in lab/for-review.txt instead. It uses standard library imports only.
  6. Run the dry run, compare it with your known answer, and fix the script until they match. Only then run --apply once and check the new file.

Verify

cd ~/secai-lab
sha256sum lab/accounts.csv > lab/accounts.sha256
python lab/stale_accounts.py --as-of "$(head -n 1 lab/as-of.txt)" | sort > lab/dry-run.txt
sha256sum -c lab/accounts.sha256
sort lab/expected-stale.txt | diff - lab/dry-run.txt && echo "dry run matches the known answer"
python - <<'EOF'
import ast, sys
tree = ast.parse(open('lab/stale_accounts.py').read())
mods = set()
for n in ast.walk(tree):
    if isinstance(n, ast.Import):
        mods.update(a.name.split('.')[0] for a in n.names)
    elif isinstance(n, ast.ImportFrom) and n.module:
        mods.add(n.module.split('.')[0])
print('non-standard imports:', sorted(m for m in mods if m not in sys.stdlib_module_names) or 'none')
EOF
grep -c '^- ' lab/script-review.md

The hash check reports OK, which proves the dry run changed nothing. diff prints nothing and the match message appears, so the script found exactly the accounts you identified by hand. Non-standard imports read none (this check needs Python 3.10 or later), and the review has at least six lines. A difference involving the account exactly 90 days old means the script and your answer disagree about the boundary, so decide which one matches the policy and fix the other.

Notes

Compare the generated script with your reviewed one. The model's version probably edited the CSV in place, treated privileged accounts like any other, and had no dry run, because the request did not ask for those things. Nothing in it was malicious. It was the plausible but unsafe default that the lesson warns about, and it would have run with whatever rights you had.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.