Make retrieval respect the reader and stop the canary leaking

short · 45 min · Objective 2.3

Task

Move the access decision in your retrieval app from the application to the person asking: capture each document's classification at ingestion, filter by the signed-in user's own permissions before ranking, prove an ordinary user can no longer retrieve the restricted document, and prove a revocation takes effect without rebuilding the index. Then write down who may invoke, configure, change, administer and export the lab's model.

Steps

  1. Check what lesson 2's build already gives you: every document has a classification in its front matter -- public, internal or restricted -- with docs/restricted/salaries.md restricted; ingestion stores that value with every chunk; and chunk ids start with the source path (for example restricted/salaries.md#0). Fix anything missing, then re-index.
  2. Add a second HR user to app/users.json, the file lesson 5's lab wrote: hana, who also gets public, internal and restricted. Leave carol's entry alone; this lab revokes hana's access, not carol's.
  3. Filter at query time BEFORE ranking. With Chroma, lesson 5's metadata filter on the query (its where argument, using $in over the user's classifications) already does this; with FAISS, replace lesson 5's over-fetch-and-drop with one index per classification and search only the indexes the user may read. Do not filter the top results afterwards, which leaks through counts and ordering. Read users.json on every request, not once at start-up.
  4. Record the start with wc -l < logs/gateway.jsonl > lab/access-start.txt. Ask three salary questions as alice and three as hana. Lesson 10's rate limit allows five requests per user in ten minutes, so leave ten minutes before hana's second round in the next step.
  5. Revoke: remove restricted from hana in users.json without re-indexing. Record lab/access-revoked.txt the same way and ask hana's three questions again.
  6. Make sure no file under app/ -- the store, users.json, the ingestion script -- is readable by other accounts.
  7. Create lab/model-access.csv with header identity,invoke,configure,change_model,administer,export and a row for each identity in the lab: you, the gateway's account, the app's account, alice and hana. Fill each cell yes or no as things stand.

Verify

Run from ~/secai-lab.

python - <<'EOF'
import json
lines = open("logs/gateway.jsonl").readlines()
s = int(open("lab/access-start.txt").read())
v = int(open("lab/access-revoked.txt").read())
def restricted(r):
    return sum(str(i).startswith("restricted/") for i in (r.get("retrieved_ids") or []))
for label, seg in (("before revocation", lines[s:v]), ("after revocation", lines[v:])):
    recs = [json.loads(l) for l in seg]
    for u in ("alice", "hana"):
        mine = [r for r in recs if r["user"] == u]
        if mine:
            print(label, u, len(mine), "request(s),",
                  sum(restricted(r) for r in mine), "restricted chunk(s),",
                  sum(any(m in (r.get("response") or "") for m in ("CANARY-7Q4X-SECAI", "[CANARY]")) for r in mine),
                  "canary in response")
EOF
find app -type f -perm -o=r | wc -l
awk -F, 'NR>1 {n++; if ($6=="yes") e++} END {print n" identities, "e+0" may export"}' lab/model-access.csv

Before revocation: alice with zero restricted chunks and zero canary, hana with restricted chunks. After revocation: hana with zero restricted chunks, with no re-index in between. 0 world-readable files under app/, five or more identities, and no more than one that may export. If alice still retrieves a restricted chunk, the filter is being applied after ranking or not at all.

Notes

Notice what fixed the leak and what did not. Lesson 9's output guardrail caught the canary on the way out; lesson 5's filter, checked and hardened here, means the restricted document is never handed to alice's prompt at all, and this lab shows a revocation biting without a re-index, which is the lesson's point about deciding access for the person rather than the application. Check the model-access file against the lesson's five permissions: in most home labs every account can read the model weights on disk, so export is yes far more widely than it should be -- a finding worth writing down even in a lab.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.