Minimise, label, mask and redact the RAG pipeline end to end

applied · 85 min · Objective 2.4

Task

Apply the whole of objective 2.4's data-protection half to your own app: carry classification labels onto every chunk, cut retrieval to what the questions need, redact values that must never be answerable at index time, mask a value dynamically by role, redact what users type before it reaches the model or the log, and test a "released" synthetic dataset for re-identification by measuring k.

Steps

  1. Give every document in docs/ front-matter fields classification (public, internal or restricted), owner and department if it lacks them. Make sure ingestion stores all three on every chunk as vector store metadata, as lesson 2's ingest.py should already do (Chroma takes a metadata dictionary per chunk; with FAISS keep them in your own chunk table), and make it write lab/chunk-manifest.jsonl with chunk_id, doc, classification and owner for every chunk.
  2. Write lab/known-questions.csv with header question,expected_doc and ten questions. Lower the retrieval top-k until one more step down would lose an expected document, and record the chosen k and the old one in lab/minimisation.md. Remove from the prompt template any field the answer never needs, such as the owner line.
  3. Add a synthetic NINO (QQ prefix) to the HR policy, then make ingestion apply redact()'s NINO rule to every chunk, so the value is never indexed and never answerable by anyone. Apply that rule only: the full redact() would also strip the rota's phone numbers from the index, and the next step needs them there so it can mask them by role.
  4. Put two numbers from the drama range in the on-call rota. At prompt assembly, mask them to their last three digits as 07700 900*** for every user except dave, the on-call lead, who sees them in full: do it in the gateway's check_prompt() hook, which sees rec["user"] and the assembled rec["prompt"] before the model does. Drive the rule from a role table, not from the user name in code.
  5. Call redact() in the gateway on the user's question before it is forwarded AND before it is logged: in check(), replace body["question"] with its redacted form (add ~/secai-lab/lab to sys.path to import it), which the gateway reads only after check() returns, and set rec["decision"] to redact whenever it changed anything, returning None so the request goes on. Write the time this went live with date +%Y-%m-%dT%H:%M:%S%z > lab/a14-cutover.txt, the same form as the ts field of each logs/gateway.jsonl record: the Verify compares every record's ts with it and reads only the records logged since.
  6. Write lab/a14-seeds.csv with header type,value listing the NINO (type NINO), both full phone numbers (type PHONE), and a fake card number and email you will type into a question (type INBOUND). Then send at least six questions: alice asks about the rota and the HR policy, dave asks about the rota, and alice asks one question containing the INBOUND values. The Verify reads type and value from this file and searches each gateway record since the cut-over for every value: a NINO or INBOUND value anywhere is a failure, and so is a full PHONE number in a record whose user is not dave, which it counts as PHONE_TO_OTHERS.
  7. Write lab/staff-synthetic.csv with 60 invented rows and header name,postcode,birth_year,sex,department,salary_band. Produce lab/staff-release.csv with header district,birth_decade,sex,department,salary_band: drop the name, cut the postcode to its outward code, round the year down to its decade, and suppress any row whose district, decade and sex combination has fewer than three members.
  8. Write lab/control-choice.md with one line for each of the salary bands, the NINO, the rota numbers, user prompts and the staff dataset, naming the control chosen -- minimise, anonymise, pseudonymise, mask, redact or label -- and why, using the lesson's order of questions.

Verify

python - <<'EOF'
import csv, json
chunks = [json.loads(l) for l in open("lab/chunk-manifest.jsonl") if l.strip()]
unlabelled = sum(1 for c in chunks if not c.get("classification"))
print(len(chunks), "chunks;", len({c["doc"] for c in chunks}), "documents;", unlabelled, "unlabelled")
cut = open("lab/a14-cutover.txt").read().strip()
def late(ts):
    return float(ts) >= float(cut) if isinstance(ts, (int, float)) else ts >= cut
seeds = list(csv.DictReader(open("lab/a14-seeds.csv")))
bad = {"NINO": 0, "INBOUND": 0, "PHONE_TO_OTHERS": 0}
after = redacted = phone_to_lead = 0
for line in open("logs/gateway.jsonl"):
    r = json.loads(line)
    if not late(r["ts"]):
        continue
    after += 1
    redacted += r.get("decision") == "redact"
    for s in seeds:
        if s["value"] in line:
            if s["type"] != "PHONE":
                bad[s["type"]] += 1
            elif r.get("user") == "dave":
                phone_to_lead += 1
            else:
                bad["PHONE_TO_OTHERS"] += 1
print(after, "records since cut-over;", redacted, "redacted;", phone_to_lead, "with full numbers for the on-call lead;", bad)
ok = chunks and unlabelled == 0 and after >= 6 and redacted >= 1 and phone_to_lead >= 1 and not any(bad.values())
print("PASS" if ok else "FAIL")
EOF
python - <<'EOF'
import csv, collections
rows = list(csv.DictReader(open("lab/staff-release.csv")))
groups = collections.Counter((r["district"], r["birth_decade"], r["sex"]) for r in rows)
print(len(rows), "rows released; k =", min(groups.values()) if groups else 0, "; name column:", "name" in (rows[0] if rows else {}))
EOF
grep -c . lab/control-choice.md

The first script prints zero unlabelled chunks, at least six records since the cut-over, at least one redact decision, at least one record carrying the full numbers for dave, every bad count at zero, and PASS. The release shows k of 3 or more with no name column, and the control-choice file has at least five lines. If dave's count is zero, your masking applies to everyone and has protected the screen rather than the data's intended reader.

Notes

The released dataset is still only as anonymous as its weakest join: k of 3 on district, decade and sex says nothing about a fourth column an attacker holds. Treat the k you measured as the start of the re-identification test the lesson asks for, not the end of it, and remember that a tokenised copy of the same data would be pseudonymised, not anonymised.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.