Sanitise the prompt log at write time and prove it with seeds

short · 40 min · Objective 2.5

Task

Lesson 2 left the canary quoted verbatim in your prompt log. Change the gateway's log writer so sensitive values are removed before the line is written, then prove the change by planting fake values and searching every destination a prompt can reach -- including the runner's own output.

Steps

  1. Count the problem first: grep -c CANARY-7Q4X-SECAI logs/gateway.jsonl. It should be above zero. If it is zero, send a question as carol that retrieves salaries.md, so the before state is a measured fact.
  2. Write lab/seed-values.txt with one value per line: the canary, a fake card number from the test range, an example.com email address and a QQ-prefixed NINO.
  3. Change the log writer only -- not what is sent to the model. Before a record is written, pass prompt and response through redact() and replace every match of the canary's shape, the regular expression CANARY-[A-Z0-9]{4}-[A-Z0-9]+, with [CANARY]. That shape leaves lesson 17's CANARY-REQUEST marker alone, and every later count of leaks matches either CANARY-7Q4X-SECAI or [CANARY], so it works whichever side of this change a record was written. Keep retrieved_ids; never copy retrieved chunk text into a separate field. Add prompt_sha256, so two identical prompts can still be matched without keeping either.
  4. Restart the gateway and write the time to lab/sanitise-cutover.txt in the gateway's ts format.
  5. Send at least five questions: carol asks about salary bands, and alice sends three questions that each contain one of the planted fake values, plus one ordinary question.
  6. Find every other place a prompt can land. For a systemd service save journalctl -u $RUNNER_UNIT --since "<cut-over time>" to lab/runner-since-cutover.txt, writing the time in a form journalctl accepts, such as 2026-10-13 14:05:00; otherwise copy the runner's log from the cut-over on. If the runner was started with debug logging, switch it off and repeat the step.
  7. Write lab/log-policy.md in three lines: what content is kept, how long, and who may read it. If you decide you still need raw content, it goes to a separate file with mode 600 and a deletion job, and that file is listed here.

Verify

python - <<'EOF'
import json
cut = open("lab/sanitise-cutover.txt").read().strip()
def late(ts):
    return float(ts) >= float(cut) if isinstance(ts, (int, float)) else ts >= cut
seeds = [l.strip() for l in open("lab/seed-values.txt") if l.strip()]
before = after = leak_before = leak_after = hashed = 0
for line in open("logs/gateway.jsonl"):
    r = json.loads(line)
    hit = any(s in line for s in seeds)
    if late(r["ts"]):
        after += 1
        leak_after += hit
        hashed += bool(r.get("prompt_sha256"))
    else:
        before += 1
        leak_before += hit
print("before:", before, "records,", leak_before, "leaking; after:", after, "records,", leak_after, "leaking,", hashed, "hashed")
print("PASS" if leak_before >= 1 and after >= 5 and leak_after == 0 and hashed == after else "FAIL")
EOF
grep -c -F -f lab/seed-values.txt lab/runner-since-cutover.txt
grep -c . lab/log-policy.md

The script reports at least one leaking record before the cut-over, five or more after it with none leaking and every one hashed, and prints PASS. The runner extract shows 0 seed matches, and the policy has three lines. A zero before the cut-over means the check never saw a leak, so its zero afterwards proves nothing.

Notes

Sanitising at write time matters because a nightly clean-up leaves the raw line in every replica and backup for a day. The [CANARY] replacement keeps the evidence that a canary was returned -- later labs alert on it -- without keeping the value. Encryption of this file at rest (lesson 13) is still needed; sanitisation decides what an authorised reader sees, encryption what a stolen disk shows.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.