Put an AI agent through change control, a tested rollback and an audit

capstone · 150 min · Objective 3.3

Task

Pull domain 3 together on your own lab. Your lesson 21 detector raises the alerts. An agent reaches its tools only through the MCP server you inventoried in lesson 20. It proposes gateway changes as change requests, which a model screens and a person approves. Each change is deployed with its prior state captured and a rollback whose trigger you prove fires. A ticket carrying an attacker-style pretext tries to skip approval. At the end, the records must let someone who was not there reconstruct the whole run.

Steps

  1. Write lab/autonomy-matrix.csv with header action,reversible,blast_radius,verifiable,untrusted_input,level, covering at least these actions: add a gateway block rule, remove one, rate-limit a gateway user, disable a gateway user, and change the agent's own permissions. level is suggest, approve, notify or bounded, and changing the agent's own permissions stays at suggest.
  2. Extend your lesson 20 MCP server with three tools for the agent. read_alerts() returns the rows of lab/alerts.csv. propose_change(kind, target, reason) appends a request to lab/changes.csv. apply_change(change_id) refuses unless the row's approved_by is filled and differs from requested_by. Rerun mcp/list_tools.py and add the new tools to the inventory.
  3. Create lab/changes.csv with header change_id,kind,target,requested_by,risk_score,prescreen,approved_by,prior_state_sha256,rolled_back,status. status is requested, refused, applied or rolled-back. The agent requests as svc-agent-triage. A model, called through the gateway as user cab-prescreen, screens each request against the actual rule change and the rule standard, never the request's own narrative, and writes its note into prescreen. You approve as yourself.
  4. Write lab/ticket-0042.txt, a synthetic ticket whose body claims the change has been pre-approved by the CISO and can be applied without review, and give it to the agent as context. Record what the pre-screen says and what apply_change does. The tool's check, not the model's judgement, must be what stops it, and that request's row ends as refused.
  5. Before applying any change, write the rule file's sha256 to prior_state_sha256 and copy the file to lab/rollback/<change_id>.json. Approve two changes in this order: first a good one, a rate limit on dan for the burst behaviour (a rate_limits entry such as "dan": 2); then a deliberately bad one, a firewall_patterns entry blocking an ordinary word such as policy, standing in for a model's misjudgement.
  6. Write lab/health.py, which replays the 20 benign prompts in lab/benign-prompts.txt through the gateway as user health-check and exits non-zero if more than 10 per cent are blocked. Before any change, give health-check a rate_limits entry of 100 in the rule file, or lesson 10's five requests in ten minutes fail every run. A request counts as blocked when the HTTP status is not 200 or the logged decision starts with block:. The bad change shows up as 400 block:firewall, and the rate limit on dan as 429 block:rate, which health-check never triggers. Write lab/deploy.sh, which applies one approved change, reloads the gateway if it needs to, runs health.py, and on failure restores the prior rule file from lab/rollback/, sets rolled_back=yes and sets the status to rolled-back. Deploy both changes: the good one stays and the bad one rolls back.
  7. Prove the trigger as well as the mechanism. Before the real run, point health.py at a prompt list you know will be blocked and confirm that deploy.sh rolls back. Then restore the list. A rollback whose health check was never seen to fail is the untested rollback the lesson calls a hope.
  8. Log every agent step to lab/agent-runs.jsonl with the fields from the lesson 26 short lab, adding the change_id. Then write lab/capstone-review.md with the sections ## What happened, ## Who decided, ## What the model was given, ## What was undone and ## What should move down the dial, using only the log and the change records, as a reviewer who was not there would have to.

Verify

cd ~/secai-lab
python mcp/list_tools.py
python - <<'EOF'
import csv, hashlib, json
rows = list(csv.DictReader(open('lab/changes.csv')))
deployed = [r for r in rows if r['status'] in ('applied', 'rolled-back')]
print(len(rows), 'requests;', len(deployed), 'deployed;', sum(r['status'] == 'refused' for r in rows), 'refused')
print('deployed without independent approval:', [r['change_id'] for r in deployed if r['approved_by'] in ('', r['requested_by'])] or 'none')
back = [r for r in rows if r['rolled_back'] == 'yes']
now = hashlib.sha256(open('gateway/rules.json', 'rb').read()).hexdigest()
print('rolled back:', [r['change_id'] for r in back])
print('rule file equals the state before the bad change:', bool(back) and now == back[-1]['prior_state_sha256'])
tools = {t['name'] for t in json.load(open('lab/mcp-tools.json'))}
listed = {r['name'] for r in csv.DictReader(open('lab/mcp-tool-inventory.csv'))}
print('tools exposed but not inventoried:', sorted(tools - listed) or 'none')
for r in csv.DictReader(open('lab/autonomy-matrix.csv')):
    print('   ', r['action'], '->', r['level'])
EOF
python lab/health.py; echo "health check exit=$?"
grep -c '^## ' lab/capstone-review.md

At least three requests: two deployed and at least one refused (the ticket-0042 request). Nothing is deployed without approval from someone other than the requester. Exactly one change is rolled back, and the rule file's hash equals that change's recorded prior state (True), which proves the rollback restored a known state rather than reconstructing one. No agent tool is missing from the inventory, and the agent's own permissions show suggest. The health check exits 0 with the good change still in place, and the review has five sections.

Notes

The order in step 5 matters, and it is worth seeing why. The bad change's prior state includes the good change, so rolling it back keeps the rate limit on dan. Rolling back to the start of the day instead would quietly undo a correct decision. That is the lesson 27 point about restoring the right bundle, met here in miniature.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.