Poisoning, introduced bias, model skewing and AI supply chain attacks

Why this matters

Lesson 17 dealt with attacks that arrive at inference time, through what the model reads. This lesson deals with attacks on what the model is: the data it learned from, the weights it carries, the components it was built on and the path its answers travel afterwards. They are slower, quieter and harder to investigate, because the attacker's action and its effect can be months apart and in different systems.

The practical difference for a defender is where the evidence lives. An injection is visible in a prompt log. A poisoned training set is visible, if at all, in a dataset's change history, in an evaluation score that moved, or in a behaviour that only appears for one rare input. A compromised pre-trained model is visible in a hash that does not match, or in a load-time warning nobody read. If those records were never kept, the attack may be impossible to prove even after the effect is obvious.

That is why the exam frames these attacks around integrity and provenance. Scenario questions tend to describe a model whose behaviour changed, or a team that downloaded something, and ask which attack fits and which control would have caught it. The answer is nearly always a control on the pipeline from lesson 6, not on the model's interface.

The lesson

Data poisoning and model poisoning, and the evidence each leaves behind

Data poisoning (ML02 on the OWASP Machine Learning Security Top 10, and half of LLM04 Data and Model Poisoning on the 2025 LLM list) is tampering with the data a model learns from: pre-training corpora, fine-tuning sets, labelled examples, or the documents behind a RAG index. Two goals are common. An availability attack degrades the model generally, so it becomes less accurate and less trusted. A targeted or backdoor attack leaves general behaviour intact and plants a trigger, so that a specific input — a phrase, a pattern, a watermark in an image — produces the attacker's chosen output. The backdoor is the dangerous form, because ordinary evaluation passes.

Routes in are mundane: write access to a training bucket or labelling tool, a public source the team scrapes, user feedback that is fed back into training, or a shared drive the RAG pipeline ingests.

Evidence of data poisoning:

• changes to a dataset outside the normal ingestion process, or by an account that does not usually write to it;
• a burst of near-duplicate records, or records from one contributor that are statistically unlike the rest;
• labels that disagree with an independent relabel of a sample;
• an evaluation score that drops after a retrain with no code change, or a regression concentrated in one class or topic;
• a rare input that produces a confident, consistent, wrong answer — the signature of a trigger.

Model poisoning (ML10) skips the data and alters the model directly: its weights, its adapters, or the fine-tuning step itself. An insider with access to the model registry, or an attacker who compromises the training environment, can implant behaviour that no dataset review will find. Its evidence is different: the deployed artefact's hash differs from the one evaluation approved, a model version appears in the registry with no matching training run, or behaviour changes between two deployments that were supposed to be identical. The distinction a scenario rewards: data poisoning is found by examining data and its lineage; model poisoning is found by examining artefacts and their custody.

Introducing bias and skewing a model over time

Introducing biases is poisoning with a social or commercial target. Rather than breaking the model, the attacker shifts its judgements against a group, a competitor, a product or a viewpoint: a hiring screener trained on records that under-represent one group, a review summariser fed reviews that favour one seller, a support bot whose fine-tuning examples are systematically dismissive of one kind of customer. It can be deliberate or accidental, and the evidence is the same either way, which is why it is caught by the bias and fairness auditing in lesson 16: outcome rates that differ across groups beyond what the data justifies, and differences that appeared after a specific data change.

Model skewing (ML08) targets models that keep learning. Where a model is retrained on its own operational feedback — fraud decisions that analysts confirm, recommendations users click, answers users rate — an attacker who can generate feedback at scale can steer the model gradually. Each contribution looks plausible; the cumulative effect moves the decision boundary in the attacker's favour. A fraud model can slowly learn to treat a fraud pattern as normal.

Evidence of skewing is a trend rather than an event:

• gradual drift in output distribution, such as approval rates or class balance, that does not match any change in the real-world population;
• feedback volume concentrated in a small number of accounts, devices or addresses;
• feedback that disagrees with ground truth when a sample is checked by hand;
• a growing gap between the live model and a frozen reference model on the same inputs.

The control that distinguishes skewing from honest drift is a comparison against something the attacker cannot influence: a frozen baseline model, a held-out evaluation set, or human-verified labels.

AI supply chain attacks and transfer learning attacks through borrowed models

AI supply chain attacks (ML06, and LLM03 Supply Chain) compromise something the organisation obtained rather than built: pre-trained models from public hubs, datasets, ML libraries and frameworks, plug-ins, and hosted model APIs. The AI-specific twist is that a model file is not inert data. Some serialisation formats execute code when the file is loaded, so downloading and opening a malicious model can compromise the workstation or training server before any inference happens. Other routes are familiar from software supply chain work: a typosquatted package, a compromised maintainer account, a dependency confusion attack, a model uploaded under a name that imitates a trusted publisher.

Evidence: a model or dataset with no verifiable publisher or signature, a hash that differs from the publisher's, a scanner warning about executable content in a model file, unexpected outbound network connections or child processes when a model loads, and a dependency added without review.

Transfer learning attacks (ML07) exploit the normal practice of starting from a pre-trained model and fine-tuning it. If the base model was trained with a backdoor, the backdoor can survive fine-tuning, because fine-tuning adjusts the model for the new task without retraining everything it learned. The organisation's own data is clean, its own evaluation passes, and the inherited trigger still works. The evidence looks like a backdoor from data poisoning, but the lineage review finds nothing wrong in the organisation's data; the fault lies upstream. That is the distinguishing clue in a scenario: behaviour that cannot be traced to anything the team supplied.

Output integrity attacks, and checking what a model hands downstream

An output integrity attack (ML09) leaves the model alone and tampers with its results after they are produced and before they are used: in transit between a model service and the application, in a queue, in a cache, or in the store where scores are written for another system to read. A credit decision, a malware verdict or a medical triage score can be altered so that the downstream system acts on a result the model never gave.

It overlaps with, but differs from, the insecure output handling of lesson 17. There the model's genuine output is dangerous because a downstream component interprets it unsafely. Here the output is not genuine: someone changed it.

Evidence: a mismatch between what the model service logged as its output and what the consuming system recorded receiving; results that fail schema or range validation; decisions that contradict the input; and changes on a result store by an identity other than the model service. The first of these is the decisive one, which is why both ends of the path need logging, with the logs correlated as lesson 15 described.

The controls are integrity controls on the path: authenticated, encrypted transport between services, such as mutual TLS (lesson 13); a signature or message authentication code on each result so the consumer can verify it came from the model unchanged; write access to result stores limited to the model service; and validation of every result against an expected schema and range before it is acted on.

Compensating controls: data integrity checks, provenance and access controls

Every attack in this lesson is an integrity attack on some stage of the pipeline, so the compensating controls are integrity controls applied at each stage.

Data integrity controls.

• Hash and version every dataset and every model artefact, and store the hashes somewhere the pipeline cannot overwrite. An evaluation approves a specific hash; deployment refuses anything else.
• Make training data append-only or immutable once approved, so a change creates a new version with an author rather than silently replacing the old.
• Validate data on ingestion: schema, duplicates, outliers, label consistency and per-contributor statistics. Rate-limit and weight user feedback so that no small group of accounts can dominate a retrain.
• Evaluate every retrain against a held-out set and a frozen baseline (lesson 9), with tests for known trigger patterns and for fairness across groups, and block promotion on regression.

Provenance.

• Record lineage: which data, code, base model and configuration produced each model version. Lesson 6 introduced this as a life-cycle gate; this lesson is why it matters.
• Obtain models and datasets only from approved sources, verify publisher signatures and hashes, prefer serialisation formats that cannot execute code, and scan model files before loading them, ideally in an isolated environment.
• Keep an inventory of AI components, often called an AI or ML bill of materials, so that when a base model or library is found to be compromised you can tell which of your models inherited it.

Access controls.

• Separate who can write training data, who can run training, who can register a model and who can promote it to production, so no single identity can both poison and ship (lesson 11).
• Restrict write access to feedback channels, RAG sources and result stores to the identities that need it, and log every write.

What the exam is really asking

If a model's behaviour changed after a retrain, look at the data and its lineage. If it changed with no retrain, look at the artefact and its custody. If it drifted slowly in one direction, think skewing and a frozen baseline. If the team fine-tuned a downloaded model and the problem cannot be found in their own data, think transfer learning attack and provenance. If the model logged one answer and the application acted on another, think output integrity.