Access control for models and the data behind them

Why this matters

Access control for AI systems uses the identity and authorisation machinery you already know. What changes is where the gaps open. A model is an unusual asset: its weights are intellectual property, they can retain fragments of the data they were trained on, and anyone allowed to query the model can try to draw those fragments out. Asking "who can read the training data?" is not enough when a second question, "who can query a model trained on it?", reaches the same data by another route.

AI applications also gather data in a way traditional applications rarely did. A retrieval-augmented assistant over the HR document library is useful precisely because it can read everything, and that is exactly the problem if it answers every employee with the authority of its own service account. The most common real-world failure in this area is not an exotic attack. It is an assistant that faithfully shows a user something they were never entitled to see, because the access decision was made for the application rather than for the person asking.

Scenario questions on this objective describe who was able to see or change something, and ask which access control was missing. This lesson covers the model itself and the data around it; lesson 12 covers agents and the APIs they call.

The lesson

Who may call a model, who may change it, and who may download its weights

Treat a model as a resource with several distinct permissions, not one, and grant each separately:

• Invoke: send requests and receive answers. This is what most users and applications need, and usually all they need.
• Configure: change the system prompt, generation parameters, guardrail settings or the retrieval sources attached to a deployment. These change behaviour as surely as code does, so they belong under change control, with review and a rerun of the evaluation suite from lesson 9.
• Change the model: fine-tune it, promote a new version into production, roll one back, or change which version an alias points to.
• Administer: create API keys, grant the permissions above, and manage the model registry.
• Export: download the weights or a copy of the model artefact.

Export deserves its own line because weights are both valuable and potentially sensitive. A copied model is stolen intellectual property, and it can be attacked offline at leisure, without rate limits or monitoring, for whatever it memorised (lesson 19 covers model inversion and membership inference). Export is a privileged operation: few holders, logged every time, and ideally needing approval.

The model registry is where these permissions live for self-hosted and fine-tuned models. Good practice is role-based access in the registry, signed artefacts so a tampered model is detectable, and promotion to production as an approved step rather than a file copy. For hosted models, the equivalent controls sit in the provider console: who may create keys, who may start fine-tuning jobs, who may change deployments. Those console roles are often granted generously during a pilot and never reviewed.

Data access for training, fine-tuning and retrieval, kept separate

Data reaches a model in three ways, and each has a different access profile.

Training and fine-tuning data becomes part of the weights. That is effectively irreversible: you cannot later revoke one person's access to one record inside a model. Every identity that can query the model is potentially able to elicit something of what it learned. The rule that follows is simple and heavily examinable: data used for training or fine-tuning must be acceptable for every identity that will be able to query the resulting model.

Retrieval data stays outside the weights. It is fetched at query time from an index, so access can be decided per request, per user, and revoked tomorrow.

The design consequence: per-user or restricted data belongs in retrieval with access control, not in fine-tuning. A support bot fine-tuned on raw support tickets may have memorised customers' addresses, and can repeat one to the wrong person. The same bot, fine-tuned on anonymised examples of good answers and retrieving the current customer's own tickets at query time, has the same usefulness and none of that exposure (lesson 14 covers anonymisation).

Keep the three uses separate in practice as well:

• Separate stores for curated training sets, fine-tuning sets and retrieval sources, each with its own permissions.
• Separate identities: the training pipeline reads curated data and nothing else; the inference service never reads training data at all.
• Tight write access. Write access to a training set or a retrieval corpus is the path for data poisoning (lesson 18), so it needs as much care as read access, which is the reverse of most people's instinct.
• Lineage records of which datasets, at which versions, produced which model version. Without them, the review in the last section of this lesson is guesswork.

Retrieval that respects the reader's own permissions, not the application's

The classic failure: a team builds an assistant over the HR document library. Ingestion runs as a service account that can read every folder, so it does. At query time, the assistant searches the whole index and answers. Any employee who asks about salaries now receives figures from documents their own account cannot open. No attack was needed; the application simply made its access decision with its own authority.

Permission-aware retrieval fixes this:

• Capture permissions at ingestion. Store each chunk with metadata naming the groups or users allowed to read the source document.
• Filter at query time by the signed-in user's identity, passed from the application to the retriever, ideally as a delegated token for that user rather than a field the application fills in. Filter before ranking, so the search only ever considers chunks the user may read; filtering after ranking can leak through counts, ordering or a too-helpful "some results were hidden".
• Keep permissions current. If a user loses access to a folder this morning, an index built last week still says they have it. Either resync permissions on a short schedule or check live permissions for the documents a query returns.
• Isolate tenants with separate indexes or enforced namespaces, so one customer's query cannot rank another customer's documents at all.
• Protect the vector store itself. Embeddings are derived from the source text and can reveal it, so direct access to the store needs the same protection as the documents. OWASP's 2025 list names this area Vector and Embedding Weaknesses (LLM08).

What does not fix it: a sentence in the system prompt asking the model to show users only their own documents. The model has already been given the content; the instruction only asks it to keep a secret it should never have been told.

Least privilege applied to service accounts that a model runs as

Every component of an AI system runs as some identity: the inference server, the retriever, the orchestration layer, the ingestion job, the evaluation pipeline. The convenient design gives them one shared identity with broad rights. The secure design gives each component its own:

• One identity per component, so the retriever's permissions are not the ingestion job's.
• Read-only where reading is the job. The retriever reads the index; it does not write to it.
• No standing administrative rights for anything that handles prompts.
• Short-lived credentials from workload identity in preference to static keys, and secrets held in a vault, never in environment files or prompts.
• Restricted egress, so a component that only talks to the index cannot talk to the internet.

The reason this matters more for AI than for an ordinary service: a model's behaviour can be steered by its input. Anyone who can successfully inject instructions into a model effectively borrows the permissions of the identity it runs as. That is a confused deputy, and the defence is to make sure the deputy has little worth borrowing. Where possible, act with the requesting user's delegated identity, so the system can never do more than that user could do directly.

Reviewing access when a model is retrained or replaced

A model version is a new asset, even when it carries the old name. Each of these events should trigger an access review:

• Retraining or fine-tuning on new data. If the new data is more sensitive than the old, the population allowed to query the model may now be too wide. Label a model with the highest classification of the data it was trained on, and let the label drive who may invoke it.
• Promotion of a replacement version. Permissions copied wholesale from the old deployment carry over its mistakes, and any exceptions granted for a pilot.
• Retirement. Old versions left callable on a forgotten endpoint, with keys still valid, are a frequent finding. Decommission the endpoint, revoke its keys, and secure or destroy its weights according to retention rules.
• A change of provider. New keys, new console roles, and new terms about how prompts and fine-tuning data are used and retained.
• A new retrieval source. Confirm its permissions are captured at ingestion before it goes live, not after the first complaint.

The review asks the same five questions every time: who can invoke, who can change, who can export, what data the model was trained on compared with who can query it, and whether retrieval still enforces each reader's permissions.

A worked scenario

After a support assistant is fine-tuned on a year of support tickets, a customer reports that it told them another customer's delivery address. Logs show no injection attempt; the customer asked an ordinary question.

The access-control reading: restricted per-customer data went into fine-tuning, where it became available to everyone who can query the model. The fix is to withdraw that model version, retrain on anonymised data, and serve per-customer details through permission-aware retrieval. An output filter for addresses is a reasonable compensating control in the meantime, but it is not the answer to "which control was missing".

What the exam is really asking

Find the identity that made the decision. If the application, a shared service account or the model itself decided what a person could see, the answer moves that decision back to the person's own permissions, enforced outside the model.