AI attacks: prompt injection and model manipulation, explained

Objective 4.6 in this course covers AI attacks — and CompTIA's own verb is explaining prompt injection and model manipulation, against performing, executing, conducting, performing and exploiting for the other five attack objectives. This lesson matches that: it reasons about these attacks and assesses exposure, and it is the Domain 4 capstone. It is explain-only by design — see LAB-SAFETY.md — so its lab is analysis, not an attack against a live model.

Why this matters

CompTIA scoped this objective as understanding rather than execution, which is a deliberate and accurate choice: a candidate needs to reason about how an AI feature widens an application's attack surface, not to run jailbreaks. Getting this right on the exam is about the concepts and the assessment approach.

It is also new material that moves quickly, so the durable content is the shape of the risks and how to assess a feature's exposure — not any particular payload, which dates in months.

The lesson

Why CompTIA scopes this objective as explaining, not performing

The verb is the lesson. Every other Domain 4 objective is a doing verb; this one is explaining, and there are good reasons:

• The techniques are unstable. Specific attacks against specific models are patched and change constantly, so a performance skill would be obsolete before the exam refreshed.
• The durable skill is assessment. What a tester needs is the ability to look at an application with an AI feature and reason about how that feature changes its attack surface — which is stable even as the payloads change.
• The safe skill is assessment. Attacking a live model, especially a third-party one, runs into the same scope and ownership problems as any other target, plus the model provider's terms — so an explain-and-assess framing is both safer and truer to the job.

So this lesson teaches you to recognise and reason about these attacks and to assess a feature's exposure on paper, which is exactly what the objective asks.

Direct and indirect prompt injection, and what each requires

Prompt injection is getting an AI system to follow instructions it should not, by supplying those instructions where it reads input.

• Direct prompt injection is the user telling the model to ignore its instructions and do something else — supplying adversarial input directly in the conversation. It requires the attacker to be the one interacting with the model.
• Indirect prompt injection is the more serious and less obvious: the malicious instructions are placed in content the model will later read — a web page it retrieves, a document it summarises, an email it processes. The attacker never talks to the model; they plant instructions where the model will encounter them while acting for someone else.

The reason indirect is worse mirrors stored versus reflected XSS in lesson 31: indirect injection is persistent, hits whoever's session processes the poisoned content, and can reach a privileged context — an AI assistant acting on an administrator's behalf reading an attacker's document.

Model manipulation: poisoning, extraction and evasion, in outline

The broader families, in the outline the objective asks for:

• Data poisoning. Influencing what a model learns by contaminating its training data, so the model behaves as the attacker wants under certain conditions. It targets the model's construction rather than its use.
• Model extraction. Recovering a model's parameters or its training data through its outputs — stealing the model, or the sensitive data it memorised.
• Evasion. Crafting input that causes the model to misclassify — the adversarial-example family — so a detection model misses what it should catch or flags what it should not.
• Inference attacks. Determining whether specific data was in the training set, which is a privacy exposure when the training data was sensitive.

For assessment, the point is recognising which risks apply to a given deployment: a model the client trained on their own data has poisoning and extraction exposure; a detection model has evasion exposure; a feature that summarises untrusted content has injection exposure.

Where an LLM-backed feature widens an application's attack surface

The assessment skill, and the practical heart of the lesson. An AI feature adds attack surface in specific, recognisable ways:

• It processes untrusted input as if it were instructions. Any place the feature reads attacker-influenced content is a possible injection point — the injection version of lesson 30's "where does untrusted input reach the interpreter".
• It is often over-trusted downstream. If the model's output is used to make a decision, run an action, or build a query without validation, then injection into the model becomes injection into whatever the output drives. An AI feature wired to take actions is far more exposed than one that only produces text.
• It has access the attacker wants. A model connected to internal data, tools, or the ability to act inherits those as attack surface — the metadata service problem of lesson 33 in a new form: reach the feature, inherit its access.
• It blurs the trust boundary. The classic mistake is treating model output as trusted because it came from "your" system, when it was shaped by untrusted input.

So the assessment question is: what does this feature read, what does its output drive, and what can it reach — the same three questions as any other component, asked of a component that treats input as instructions by nature.

Assessing an AI feature's exposure without attacking a live model

The lab for objective 4.6, and it is analysis rather than execution by design.

Take an application with an AI feature — the client's, or one you build in the lab — and assess it on paper:

1. Map the inputs. Where does the feature read content, and how much of it is attacker-influenced (direct input, retrieved pages, uploaded documents)?
2. Map the outputs. What does the model's output do — displayed to a user, used in a decision, passed to a tool, built into a query or command?
3. Map the access. What data and capabilities does the feature have, and what would injection into it therefore reach?
4. Identify the trust boundary mistakes. Where is model output treated as trusted? Where is untrusted content fed in without isolation?
5. Rate the exposure from those, and recommend the mitigations below.

This produces a finding — "this feature summarises untrusted uploaded documents and its output triggers actions without validation, so indirect injection can drive those actions" — without ever crafting an attack against the model. That is the objective's intent, and it is a genuinely useful assessment.

Mitigations worth recommending

Remediation, at the level the assessment reveals:

• Treat model output as untrusted. Validate and constrain what the output can do exactly as you would any other untrusted input — the single most important fix, because it breaks the chain from injection to consequence.
• Isolate untrusted content the model reads, and do not let content the model retrieves carry the authority of a user instruction.
• Least privilege for the feature. A model wired to tools and data gets the minimum it needs, so injection reaches as little as possible — lesson 34's argument, applied to an AI feature.
• Keep a human in the loop for consequential actions, so injection cannot silently drive something irreversible.
• Monitor, because the field moves and today's safe deployment is tomorrow's finding.

The client message: an AI feature is a component that treats input as instructions by design, so the discipline is the same as the rest of the domain — do not trust input, do not over-trust the component's output, and give it least privilege.

What to take into the exam

• CompTIA scopes 4.6 as explaining, not performing: the skill is assessing how an AI feature changes attack surface, not running jailbreaks.
• Direct injection needs the attacker to interact with the model; indirect injection plants instructions in content the model later reads, and is worse — persistent, and reaches privileged contexts.
• Model manipulation families: poisoning (training), extraction (stealing the model or its data), evasion (adversarial misclassification), inference (membership/privacy).
• Assess a feature by its inputs, what its output drives, and what it can reach; the core fix is treating model output as untrusted and giving the feature least privilege.