AI in the CI/CD pipeline: code scanning, software composition analysis and model testing

Why this matters

A continuous integration and continuous deployment (CI/CD) pipeline is where security either happens automatically or mostly does not happen at all. Every change passes through it, so a check placed there runs on every change without anybody remembering to run it. That makes the pipeline the natural home for AI in security automation, and it makes AI a natural subject for the pipeline, because a team that ships an AI application ships a model, a prompt and a retrieval index alongside its code.

Both directions appear in this lesson. AI helps the pipeline: triaging scanner findings, prioritising vulnerable dependencies, writing tests, judging whether a deployment is healthy. And the pipeline has to test AI: a model is an artefact that can regress, be swapped or be poisoned like any other, but it cannot be tested the way a function is, because the same input does not always produce the same output.

There is a third thread. The pipeline holds the credentials that reach production, the keys that sign releases and write access to the model registry. An attacker who controls the pipeline controls everything it deploys, and an AI reviewer that reads untrusted pull-request text is a new way into it.

The exam angle: scenarios describe a pipeline where a check exists but does not detect, a model shipped without a gate it should have passed, or a pipeline component trusted with more than it needed. The answer names the missing gate or the excess trust.

The lesson

Code scanning and software composition analysis with AI assistance

Code scanning analyses source for vulnerable patterns: static application security testing (SAST) on the code itself, secrets scanning for committed keys, and infrastructure-as-code scanning for misconfigured cloud resources. Lesson 22 covered AI-assisted code review and vulnerability analysis as a task a person performs; here the same capability runs unattended on every commit.

AI changes three things in the scanning stage:

• Triage. Traditional scanners report many findings, a large share of them unexploitable in context. A model that reads the surrounding code can explain whether user input really reaches the dangerous call, and rank findings accordingly. This is the main gain, because an ignored scanner is worse than none.
• Explanation and fixes. A finding arrives with a plain explanation and a proposed patch. The patch is a code change like any other: it goes through review and the full test suite, because a fix that closes an injection flaw while breaking an authorisation check is a regression.
• Detection beyond rules. A model can spot logic flaws, such as a missing ownership check on an object lookup, that no pattern rule describes.

The trap is suppression. If the AI triage step can mark findings as false positives and remove them from the report, a model error, or a manipulated model, silently hides a true positive. Triage should re-rank and annotate; closing a finding should leave a record and, for high-severity findings, need a person.

Software composition analysis (SCA) inventories the third-party components a build pulls in, matches them against known vulnerabilities and checks licences. Its output is usually a software bill of materials (SBOM). AI helps by judging reachability, whether the application actually calls the vulnerable function in a library, which turns a list of hundreds of advisories into the few that matter.

Two AI-specific extensions belong in SCA:

• Hallucinated packages. Code assistants sometimes suggest dependency names that do not exist, and an attacker who registers such a name gets their code installed by anyone who trusts the suggestion. SCA should flag dependencies that are new to the organisation, very young or rarely used, and the pipeline should install only from an approved internal mirror.
• Models and datasets as components. A downloaded base model, a tokeniser or a fine-tuning dataset is a third-party component with provenance, a licence and possible tampering. Some model serialisation formats can execute code when a model is loaded, so the composition check should record the format, prefer ones that store weights only, and verify hashes against the publisher's. This is the supply-chain risk from lesson 18 handled as a pipeline gate.

Unit and regression testing, including tests an AI wrote

Unit tests check that individual functions behave as specified. Regression tests check that behaviour which used to be correct still is after a change. Both are standard CI/CD stages; AI changes who writes them and what they cover.

AI is good at generating tests quickly, and that creates a specific failure: tests that assert what the code currently does rather than what it should do. A model shown a function with an off-by-one error will cheerfully write a test that expects the wrong answer, and the bug is now protected by the suite. Other weak patterns are tests that cannot fail (asserting a value equals itself), tests that mock away the code under test, and high line coverage with assertions too loose to catch anything.

The controls:

• Review tests for intent. A reviewer reads the test names and assertions as a specification: is this what the function is supposed to do?
• Prove the tests detect. Mutation testing deliberately introduces small faults into the code and checks that some test fails. A suite that lets mutants survive is not testing what its coverage figure claims.
• Write security tests from requirements, not from code. "An unauthenticated request to this endpoint returns 401" is written from the access-control design, and should be written before or independently of the implementation.

For security, regression testing has a particular discipline: every fixed vulnerability gets a test that reproduces it, so the fix cannot be quietly undone by a later refactor. For an AI application the regression suite extends to behaviour: a library of known prompt-injection patterns (inert examples of the pattern, not working exploits) that must keep being refused, questions whose answers must keep coming only from permitted documents, and outputs that must keep passing the output-handling checks. When a prompt template, guardrail or model changes, that suite runs.

Model testing as a pipeline stage, alongside code tests

Model testing is the stage that decides whether a model, or a change to the prompt and retrieval around it, may be promoted. Lesson 9 introduced evaluation as a control; the pipeline is where it becomes automatic and blocking.

A model test stage typically measures:

• Task quality against a held-out evaluation set with known answers, with a minimum score.
• Safety and policy behaviour: refusal of disallowed requests, resistance to the injection and jailbreak patterns in the regression library, and no leakage of the system prompt.
• Bias and fairness metrics across the groups relevant to the use, where the model makes or informs decisions about people.
• Data leakage: checking that the model does not reproduce sensitive training records, for example by planting unique canary strings in training data and testing whether the model emits them.
• Robustness: performance on perturbed or noisy inputs, not only clean ones.
• Operational limits: latency, token use and cost per request, because a model that doubles cost is a regression too.

Three things make model testing different from code testing, and each is examinable:

• Non-determinism. The same prompt can give different answers. Tests run multiple samples and gate on rates ("refuses in at least the agreed proportion of runs") rather than single outcomes, and the generation settings used for testing should match production.
• Thresholds are decisions. A gate needs a number agreed before the run, with an owner. A score that is reviewed afterwards and found "close enough" is not a gate.
• The evaluation set is a secret. If evaluation data leaks into training or fine-tuning, scores rise while real quality does not. The set is access-controlled, versioned and kept separate from anything the training process can read, and refreshed so the model is not tuned to one fixed test.

Automated deployment and rollback for models as well as code

Lesson 26 covered automated deployment and rollback in change management. For an AI system the principle is the same, with one important difference: the deployable unit is a bundle. The behaviour users see comes from the model weights, the system prompt and templates, the guardrail and filter configuration, the retrieval index and its embedding model, and the application code. Changing any one changes behaviour.

So the release record names every element by version, and rollback restores the bundle, not just the part that changed. Rolling back the model while leaving a new prompt in place produces a combination that was never tested.

The deployment patterns, and what each offers for models:

• A model registry holding versioned, hashed and signed artefacts, with promotion between stages only through the pipeline.
• Shadow deployment: the new model receives a copy of live traffic and its answers are recorded but not shown, so it can be compared with the current model on real inputs before anyone depends on it.
• Canary release: a small share of users get the new version while quality, safety and cost signals are watched.
• Keeping the previous version deployable, so rollback is a routing change rather than a rebuild.

Rollback triggers for a model include the usual errors and latency but also behavioural signals: a rise in guardrail interventions, in user complaints, in refusals of legitimate requests, or a drift in output distributions. Some of these surface slowly, which is why monitoring from lesson 15 continues after the canary stage ends. And as with code, some things cannot be rolled back: data a model has already disclosed, or decisions it has already made about people. That is the argument for gating before release rather than relying on rollback after.

Keeping the pipeline itself from becoming the attack path

The pipeline is privileged by design. It reads source, holds secrets, signs artefacts and writes to production and the model registry. Compromising it is often easier than compromising production directly, and the result is the same.

The routes in, and the control for each:

• Over-privileged pipeline credentials. Use short-lived workload identity scoped to a repository and branch instead of long-lived keys, and separate the identity that builds from the identity that deploys.
• Untrusted contributions running with secrets. A pull request from an outside contributor should run in an isolated job with no access to secrets or deployment credentials.
• Unpinned dependencies and pipeline steps. Pin third-party actions and base images by digest, not by a tag that can be moved.
• Tampered artefacts. Sign build outputs and models, record provenance (what source and which build produced this artefact) and verify both at deployment. Frameworks such as SLSA describe graded levels of build provenance.
• Pipeline definitions changed quietly. Changes to the workflow files themselves get two-person review and branch protection, because editing the pipeline is a way to remove its checks.
• AI components reading attacker-controlled text. An AI reviewer or coding agent in the pipeline reads pull-request descriptions, issue text, code comments and dependency README files, any of which an outsider can write. Give it read-only access, no secrets, no ability to merge or approve, and treat its output as advice to a human reviewer. An AI review step that can approve its own suggestions removes the review.
• Training data entering through the pipeline. Where the pipeline retrains or fine-tunes, data sources need the integrity and provenance checks from lesson 18 before they reach training.

What the exam is really asking

Look for the check that cannot fail, the gate with no agreed threshold, the rollback that restores only half the bundle, or the pipeline component holding more trust than its job needs. The fix is usually a gate, a test that proves it detects, or a narrower identity.