Treat the vector store as a data store, show vectors are not anonymous, and test what a watermark proves

applied · 90 min · Objective 1.2

Task

Apply the whole of objective 1.2's retrieval half to your lab. Show that the store is another copy of your documents and lock it down. Show that vectors handed over "without the text" can still be matched to the people they describe. Reproduce a withdrawn document being served from a stale index, then fix synchronisation. Finish by judging six watermark claims and four fine-tune-or-retrieve decisions against the lesson's rules.

Steps

  1. The store is a copy. Run grep -rla CANARY-7Q4X-SECAI app/store and note in ~/secai-lab/lab/store-review.md which file holds the chunk text in clear. (Chroma keeps it in its own database file; with FAISS it is in whatever metadata file you wrote.) Then restrict it as you would any restricted store: chmod -R go-rwx app/store logs.
  2. Vectors are not anonymous. Write ten synthetic HR case sentences, each naming an invented person and a condition or grievance, to ~/secai-lab/lab/hr-cases.txt as case_id|text. Write ~/secai-lab/lab/guesses.txt holding a reworded version of each case under the same case_id, plus ten decoys with case_id decoy.
  3. Write ~/secai-lab/lab/embed-match.py. It embeds every case with your local embedding model, keeps only the vectors (this is what would be handed to a vendor), embeds every guess, matches each vector to the guess with the highest cosine similarity, and writes ~/secai-lab/lab/embed-match.csv with header vector_id,true_case,best_guess_case,similarity.
  4. Stale content. Create docs/withdrawn-memo.md (internal) containing the marker WITHDRAWN-MEMO-5K2 and a parking rule, ingest, and ask about parking as alice. Delete the file and ask again; the chunk is still served, because nothing removed it from the store.
  5. Fix synchronisation in ingest.py: after loading, delete from the store every chunk whose source no longer exists under docs/ (in Chroma, delete by id). Write the go-live time with date +%Y-%m-%dT%H:%M:%S%z > ~/secai-lab/lab/sync-on.txt, re-run ingestion and ask about parking as alice once more. The Verify splits logs/gateway.jsonl at this time by each record's ts (the same date form lesson 2's gateway writes) and counts the records whose retrieved_ids include a withdrawn-memo chunk on each side of it.
  6. Write ~/secai-lab/lab/watermark-claims.csv with header claim_id,verdict,reason, verdict being supported or not-supported, for: W1 a strong detection is evidence the text came from a system using that watermark; W2 this essay carries no watermark, so a person wrote it; W3 the watermark is intact, so the content is accurate; W4 records planted in our dataset appeared verbatim in a vendor model's output, which is evidence the model saw our data; W5 a one-sentence reply tested negative, so it was not generated; W6 the image's provenance metadata is missing, so no AI touched it.
  7. Write ~/secai-lab/lab/architecture-choices.csv with header scenario,choice,deciding_requirement, choice being retrieval, fine-tune or either-local, for: S1 different staff must see different documents; S2 the assistant must learn the house incident-report format from non-sensitive examples; S3 a person's data must be deletable on request; S4 the data must never leave the organisation.

Verify

cd ~/secai-lab
find app/store logs -perm /044 | wc -l
python - <<'EOF'
import csv, json
m = list(csv.DictReader(open("lab/embed-match.csv", encoding="utf-8")))
print(sum(r["true_case"] == r["best_guess_case"] for r in m), "of", len(m), "vectors matched to their person")
on = open("lab/sync-on.txt").read().strip()
rows = [json.loads(l) for l in open("logs/gateway.jsonl", encoding="utf-8") if l.strip()]
def memo(r):
    return any(i.startswith("withdrawn-memo") for i in r["retrieved_ids"])
print("memo served before the sync fix:", sum(memo(r) for r in rows if r["ts"] < on))
print("memo served after:", sum(memo(r) for r in rows if r["ts"] >= on))
w = {r["claim_id"]: r["verdict"] for r in csv.DictReader(open("lab/watermark-claims.csv", encoding="utf-8"))}
print("watermark claims supported:", sorted(k for k, v in w.items() if v == "supported"))
c = {r["scenario"]: r["choice"] for r in csv.DictReader(open("lab/architecture-choices.csv", encoding="utf-8"))}
print("choices:", [c.get(s) for s in ("S1", "S2", "S3", "S4")])
EOF

No files group- or world-readable (0). Most of the ten vectors matched to their person: chance is one in twenty, so anything above two already shows the vectors are not anonymous, and a working embedding model usually gets eight or more. The memo served before the fix at least twice (once before the deletion and once after it), and after the fix 0. Watermark claims supported: exactly ['W1', 'W4']. Choices: ['retrieval', 'fine-tune', 'retrieval', 'either-local'].

Notes

The embedding test is not inversion. It is the easier attack: someone who holds the vectors and a list of guesses can confirm which guess each vector encodes. That is enough to show that vectors of HR case notes must be handled as HR case notes. On S4, the lesson's point is that keeping data in-house is a hosting decision, which either architecture can meet with a local model.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.