Model theft, model inversion, membership inference, model denial of service and overreliance

Why this matters

Lessons 17 and 18 covered attacks that change what a model does. The attacks in this lesson mostly change nothing. They take: the model itself, facts about the people in its training data, the secrets in its context, the compute it runs on, or the judgement of the people who rely on it. Several of them can be carried out entirely through the model's legitimate interface, by a user who is authenticated, within quota on any single request, and asking questions the application was built to answer.

That makes the evidence subtle. There is rarely one malicious request to point at. There is a pattern of requests — too many, too systematic, too similar, or too expensive — and the investigation is about recognising the pattern in the rate, cost and query monitoring of lessons 15 and 16.

The lesson closes by drawing together the compensating controls across all of objective 2.6, because the scenario questions in this objective frequently give you an attack and a list of controls, and the skill being tested is matching the two.

The lesson

Model theft and extraction through the model's own interface

Model theft (ML05; Model Theft on the 2023 OWASP LLM list) is obtaining a model the organisation paid to build. There are two routes, and they are different problems.

The first is exfiltration of the artefact: copying the weights from a model registry, a storage bucket, a training server or a developer laptop. This is ordinary data theft of an unusually valuable file, and its evidence is ordinary too: large reads or downloads of model artefacts, access from unusual identities or locations, and copies appearing outside the approved stores.

The second is extraction through the interface: querying the model at volume and using its answers to train a substitute that imitates it. The attacker never touches the weights; they reproduce the model's behaviour. The 2025 OWASP LLM list covers this under LLM10 Unbounded Consumption, which treats unrestricted querying as both a cost risk and a route to functional replication. Evidence:

• query volume far above what the account's purpose needs, often spread across several accounts or keys to stay under per-account limits;
• queries that look generated rather than written: systematic coverage of a topic, templated phrasing, or inputs that walk methodically across a range;
• interest in output detail beyond what the application displays, such as confidence scores or full probability distributions;
• many accounts created close together, with similar behaviour.

Watermarking model output (lesson 5) does not prevent extraction, but it can help show afterwards that a competitor's model was trained on yours.

Model inversion and membership inference: learning the training data from outside

These two attacks target the people and records the model learned from, and they work because models remember more about their training data than their designers intend, especially when they are overfitted.

Model inversion (ML03) uses a model's outputs to reconstruct information about its training data: an approximation of what a typical member of a class looks like, or sensitive attributes associated with a known input. A face recognition model that returns confidence scores can, in principle, be queried repeatedly to rebuild a recognisable image of someone it was trained on. Language models add a more direct version: prompting a model to reproduce memorised text from its training set.

Membership inference (ML04) asks a narrower question: was this specific record in the training data? A model often behaves measurably differently on data it was trained on, for example being more confident, and an attacker who holds a candidate record can compare. Answering yes can be a disclosure on its own: if the model was trained only on patients with a particular condition, membership reveals the condition.

Evidence for both is query-pattern evidence: many queries about one individual or one class; inputs that are small variations of each other, probing how the output changes; and heavy use of confidence scores. Specific mitigations sit in the closing section, but the most effective ones are in how the model is trained and what it returns: training with regularisation, or with differential privacy, which bounds how much any single record can influence the model; returning a label rather than raw scores, or rounded scores; and minimising what personal data enters training at all (lesson 14).

Sensitive information disclosure, and hallucinations that look like disclosure

Sensitive information disclosure (LLM02 in the 2025 list) is a model revealing data it should not: personal data, credentials, internal documents, the system prompt, or another user's conversation. The usual causes are architectural rather than exotic. A RAG application retrieves with the application's permissions instead of the reader's (lesson 11), so an employee's question pulls in documents they cannot open. Secrets are placed in the system prompt on the assumption that users cannot see it. Training data contained personal data that the model memorised. Session handling mixes contexts. Evidence: responses containing patterns that output guardrails and log monitoring are designed to flag, such as identifiers, keys and email addresses; retrieval logs showing documents outside the user's entitlement; and user reports.

Hallucinations — confident, fluent, false output — are on the attack list for two reasons. First, they can be exploited: if a code assistant regularly invents a plausible package name, an attacker can register that name and wait for developers to install it, which connects this lesson to the supply chain attacks in lesson 18. Second, they look like disclosure. A model asked for a colleague's salary or a customer's account details may produce a realistic answer it simply made up.

For an investigator this matters a great deal. Before declaring a data breach, verify the disclosed data against the source of record. If the "leaked" figures exist nowhere in the training data, the index or the context, the incident is a hallucination, with different notification duties and a different fix (grounding, citations and output checks) from a real disclosure (access control on retrieval, removal of secrets from prompts, data minimisation). Retrieval logs settle the question quickly when they exist.

Model denial of service, and why cost is part of availability

Model denial of service (Model Denial of Service on the 2023 list, now part of LLM10 Unbounded Consumption) is exhausting the resources a model needs. Model inference is expensive per request, and its cost varies enormously with input and output length, so a small number of requests can do what a classic flood needs thousands for. Patterns include very long inputs that fill the context window, requests engineered to produce very long outputs, inputs that make an agent loop or call tools repeatedly, and high volumes of ordinary requests.

On a hosted, pay-per-token model, the service may never fail at all. Instead it keeps answering and the bill rises, sometimes called denial of wallet. Either the organisation pays, or a spending cap cuts the service off for every user, which is a denial of service by another route. That is why cost is part of availability for AI systems, and why the cost monitoring of lesson 16 is a security control.

Evidence: rising latency and queue depth; token consumption per request well above the application's norm; a few accounts dominating spend; agent sessions with unusually many steps; and cost alerts before any availability alert.

Overreliance as an attack outcome, and the controls that keep a human in the decision

Overreliance (2023 list; merged into LLM09 Misinformation in 2025) is people or systems accepting model output without the checking its reliability warrants. It is less an attack than the condition that makes other attacks pay. A poisoned model, an injected summary or a hallucinated citation does damage only if someone acts on it unchecked. An attacker who knows that analysts approve whatever the triage assistant recommends needs only to influence the recommendation.

Evidence is behavioural: approval rates for AI recommendations close to total; review times too short for a real review; decisions with no record of the underlying evidence being opened; and errors that propagate because nobody compared the output with a source.

The controls keep a human in the decision in a way that is real rather than nominal, building on the human-centric design of lesson 6: require citations or sources for factual output and make them easy to check; show confidence and uncertainty; require human approval for high-impact actions (lesson 12); sample and audit accepted outputs against known answers (lesson 16); and train users that the model can be wrong in confident language.

Compensating controls across all of 2.6: rate limiting, encryption, access controls and data integrity controls

Objective 2.6 pairs a long list of attacks with a short list of controls. Prompt firewalls, prompt templates, model guardrails and least privilege were taught in lesson 17; data integrity controls and access controls on the pipeline in lesson 18. Two controls belong to this lesson's attacks.

Rate limiting is the primary control for extraction, inversion, membership inference and model denial of service, because each needs volume. Limit requests, tokens and spend per user, per key and per tenant (lesson 10), and detect the same actor spread across many accounts. Pair it with limits on output detail and with timeouts and step limits for agents.

Encryption protects weights, training data, vector stores and logs at rest and in transit (lesson 13), which addresses artefact theft and disclosure from storage. It does nothing against extraction through the interface, because the model decrypts to answer.

A map for scenario questions:

• Prompt injection, jailbreaking, guardrail circumvention: prompt firewalls, prompt templates, model guardrails.
• Excessive agency, insecure plug-in design, manipulated integrations: least privilege and access controls, plus human approval.
• Insecure output handling: output guardrails, encoding and validation.
• Poisoning, introduced bias, skewing, supply chain, transfer learning, output integrity: data integrity controls and access controls on the pipeline.
• Model theft, inversion, membership inference: rate limiting, access controls, encryption of artefacts, and limiting output detail.
• Sensitive information disclosure: access controls on retrieval, output guardrails, encryption and minimisation.
• Model DoS: rate limiting and token limits.
• Hallucinations and overreliance: guardrails, grounding and human validation.

When two answers both help, choose the one that acts at the point the evidence shows the attack succeeded.