Assemble the governance pack for your lab system and audit it

capstone · 140 min · Objective 4.3

Task

Pull domain 4 together. Assemble every governance artefact you have written into one pack with a manifest and a cover note, carry one real material change through all of it, write a script that re-checks the whole pack and show that it fails when it should, and then act as the independent auditor: re-perform tests against the running lab and record findings owned by someone other than yourself-as-auditor.

Steps

  1. Create lab/governance-pack/manifest.csv with header artefact,path,owner,last_reviewed,next_review,sha256, one row per artefact: ai-policy, ai-standard, procedure-new-ai-use, ai-use-register, integration-map, raci, rai-controls, ai-risk-register, autonomy, crosswalk, applicability, allow-list, data-class-map, vendor-assessment, data-flows. path is relative to ~/secai-lab; owners are RACI roles; dates are YYYY-MM-DD. Leave the hashes until last.
  2. Write lab/governance-pack/cover.md with six level-two headings: ## System described, ## Scope and exclusions, ## Status of this pack, ## Top risks, ## Open actions, ## Sign-off. The status section states that the pack documents a study lab and is not legal compliance advice, in those words. Under ## Top risks, list the five highest-scoring risks from your register, one per line as - R-03: ....
  3. Make one material change for real and carry it through: intake and a new register row superseding the old; the risk register rescored where it moved; the regression set from lesson 30 rerun and the results recorded; a provenance row if the weights changed. Log every artefact you touched in lab/governance-pack/change-log.csv with header change_id,date,description,artefact,action.
  4. Write lab/governance-pack/pack-check.py: it re-runs the structural checks from the earlier verify blocks (policy sections, one A per RACI row, register scores, crosswalk enumerations, allow-list against data-class map, canary coverage of the data flows) and exits non-zero if any fails. Run it and get exit 0.
  5. Prove it detects: delete the ## Exceptions heading from the policy, run pack-check.py, see a non-zero exit, and restore the heading. Repeat with one RACI row given a second A.
  6. Audit as the independent role. Create lab/governance-pack/audit-findings.csv with header finding_id,artefact,test,result,severity,owner,due and at least five tests you re-perform rather than read: rerun the register against log users, recount the canary, rerun two regression questions, re-hash the weights against the provenance record, re-prove the runner's loopback binding. result is pass or fail. A failure is owned by the first-line role that fixes it, never by ai_auditor.
  7. Finally, fill in every manifest hash with sha256sum, set each next_review in the future, and run the verify block.

Verify

cd ~/secai-lab
python lab/governance-pack/pack-check.py; echo "pack-check exit=$?"
python - <<'EOF'
import csv, hashlib, datetime
P = "lab/governance-pack/"
need = {"ai-policy", "ai-standard", "procedure-new-ai-use", "ai-use-register",
        "integration-map", "raci", "rai-controls", "ai-risk-register", "autonomy",
        "crosswalk", "applicability", "allow-list", "data-class-map",
        "vendor-assessment", "data-flows"}
man = list(csv.DictReader(open(P + "manifest.csv", encoding="utf-8")))
print("artefacts missing from the manifest:", sorted(need - {r["artefact"] for r in man}))
print("hashes that do not match the file:",
      [r["artefact"] for r in man if hashlib.sha256(open(r["path"], "rb").read()).hexdigest() != r["sha256"]])
today = datetime.date.today().isoformat()
print("next review not in the future:", [r["artefact"] for r in man if r["next_review"] <= today])
reg = list(csv.DictReader(open("lab/ai-risk-register.csv", encoding="utf-8")))
scores = {r["id"]: int(r["score"]) for r in reg}
fifth = sorted(scores.values(), reverse=True)[4]
cover = open(P + "cover.md", encoding="utf-8").read()
top = cover.split("## Top risks", 1)[-1].split("## ", 1)[0]
ids = [l[2:].split()[0].rstrip(":") for l in top.splitlines() if l.startswith("- ")]
print(len(ids), "top risks listed; below the fifth-highest score:",
      [i for i in ids if scores.get(i, -1) < fifth])
print(len([l for l in cover.splitlines() if l.startswith("## ")]), "cover sections")
print("states it is not legal compliance advice:", "not legal compliance advice" in cover.lower())
ch = list(csv.DictReader(open(P + "change-log.csv", encoding="utf-8")))
print("artefacts touched by the change:", len({r["artefact"] for r in ch}))
au = list(csv.DictReader(open(P + "audit-findings.csv", encoding="utf-8")))
print(len(au), "audit tests,", sum(r["result"] == "fail" for r in au), "failed;",
      "bad results:", sum(r["result"] not in ("pass", "fail") for r in au), ";",
      "owned by the auditor:", sum(r["owner"] == "ai_auditor" for r in au))
EOF

pack-check exit=0; nothing missing from the manifest; no hash mismatches and no past review dates; five top risks listed with none below the fifth-highest score; six cover sections; True for the status statement; five or more artefacts touched by the change; and five or more audit tests with zero bad results and zero owned by the auditor. You must also have seen pack-check.py exit non-zero for both faults in step 5. A pack check you have never seen fail has not shown that it checks anything.

Notes

Hash the manifest last, after the change and the audit, or the first edit afterwards will turn it red, which is correct behaviour and a nuisance. The pack is what an AI governance engineer from lesson 29 would build: the evidence collection runs by itself and no longer depends on anyone remembering it. Keep it outside the snapshot chain, or copy it off the VM by a route you have approved, before reverting.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.