Security across the AI life cycle, from business case to feedback loop

Why this matters

A conventional application has a life cycle security teams already know how to join: requirements, design review, code review, testing, release, operations. An AI system has the same stages and several more, and the extra ones are where the damage tends to start. A model's behaviour is decided by data nobody reviewed line by line, by a training or selection step that produces an artefact no one can read, and by an evaluation that measures performance on the cases somebody thought to include. None of that shows up in a code review.

The second difference is that an AI system keeps changing after release. A model is retrained on fresh data, a retrieval index is refreshed nightly, a vendor updates the hosted model behind a stable name, and user feedback is fed back in. Each of those is a change to production behaviour, and each can carry an attack, a regression or a quiet drift away from what was approved.

So the security professional's job is to treat every stage as a gate with an owner and evidence, from the decision to build the thing at all to the loop that feeds production back into training. A security review attached only to deployment arrives after the data was collected, the model was chosen and the risk was locked in.

Scenario questions on this topic usually describe a failure and ask where in the life cycle it should have been caught. The right answer is almost always the earliest stage at which the problem was visible, not the stage where it finally surfaced.

The lesson

Starting with the business use case and its alignment with corporate objectives, and the projects that should stop there

The life cycle starts before any data or model, with a business use case: what problem this solves, for whom, and how success will be measured. Its alignment with corporate objectives is a security matter as well as a strategic one, because an AI project that cannot name the objective it serves also cannot name the risk the organisation has agreed to accept for it.

A useful use-case record answers a short set of questions before anything is built:

• Purpose and owner. A named business owner who will accept the residual risk, and a measurable outcome.
• Data. Which data the system will see, at what classification, and whether the organisation may lawfully use it for this purpose.
• Agency. Whether the system only answers, or can also act: send email, change a ticket, approve a refund. Agency multiplies impact.
• Affected people. Who is affected by a wrong output, and whether the use falls into a regulated category (lesson 32 covers how the EU AI Act sorts uses by risk).
• Kill criteria. What evidence would stop the project.

Some projects should stop here, and recognising them is examinable. Stop, or redesign, when a deterministic rule would do the job as well; when the cost of a wrong answer is unacceptable and no human can check the output in time; when the required data cannot be used for this purpose; or when nobody will own the system once it is live. A support chatbot that answers from a published FAQ is a reasonable case. The same chatbot given authority to issue refunds without review is a different use case, and it needs a different decision.

Collecting data that is trustworthy and authentic, and proving it

Two properties of collected data are easy to confuse and both matter. Trustworthiness asks whether the data is fit to learn from: accurate, complete, representative of the cases the model will meet, and from a source with a reason to be right. Authenticity asks whether the data is what it claims to be, from whom it claims, unaltered since it was obtained.

A dataset can fail either test independently. A genuine export from a reputable source can still be skewed towards one customer segment, which is authentic but untrustworthy. An accurate-looking dataset downloaded from a mirror nobody can vouch for may have been altered, which is plausible but unauthenticated. Poisoning attacks (lesson 18) can exploit either gap.

Proving both needs records made at collection time, because they cannot be reconstructed later:

• Source and licence: where each dataset came from, under what terms, and with what consent where personal data is involved.
• Integrity evidence: a cryptographic hash of each file or record set as received, and a signature or checksum published by the source where one exists.
• Chain of custody: who handled the data and where it was stored before it reached the training environment.
• Allow-listed sources for anything collected automatically. Content at a URL can change after the URL was chosen, for example when a domain lapses and someone else registers it, so a list of addresses is not a record of content. A hash taken at collection is.

These records are the lineage and provenance of lesson 4, applied at the moment they are cheapest to capture.

Preparation, development or selection, evaluation, deployment and validation as security gates

Each middle stage is a gate: a point where evidence is checked before work moves on.

Data preparation. Cleansing, labelling, deduplication and balancing all change the data, so each step is logged and repeatable. Labelling is a particular exposure: whoever labels the data can teach the model a wrong answer, so labelling access is controlled and a sample of labels is reviewed. Sensitive fields that the use case does not need are removed here, which is data minimisation at the point where it does most good.

Model development or selection. Most organisations select a model rather than train one from scratch. Selection is a supply-chain decision: who published the model, under what licence, with what documentation of its training data and limits, and in what file format. Some model formats can execute code when loaded, so the format and a hash of the downloaded weights are checked before the file is opened anywhere that matters. Training in-house moves the same questions onto the training environment, which needs the access control and isolation of any system that produces a production artefact.

Model evaluation. Before release the model is measured against defined criteria: accuracy on held-out data, behaviour on the cases that matter most, bias across groups, and security tests such as resistance to prompt injection and refusal of out-of-scope requests. The examinable point is that the criteria and thresholds are set before results are seen, or the evaluation becomes a description of whatever the model happened to do. Lesson 9 treats evaluation as a control in its own right.

Deployment. The approved artefact, identified by hash, is what is deployed: no substitution between approval and release. The deployment carries its controls with it: access to the endpoint, a gateway, logging and limits.

Validation. Evaluation asks whether the model is good enough; validation asks whether the deployed system meets its requirements in its real environment, with real integrations, real users and real data flows. A model that passed evaluation can fail validation because the retrieval layer exposes documents the evaluator never saw, or because a tool integration gives it reach the test environment lacked. Validation is the acceptance gate that signs the system, not the model, into service.

Monitoring, maintenance, feedback and iteration once the model is live

Once live, three things change without anybody editing code. The world drifts, so the data the model meets stops resembling the data it learned from and accuracy falls. The attackers arrive, and probe with inputs the evaluation did not include. And the dependencies move: a hosted model is updated by its vendor, a library is patched, an index is rebuilt.

Monitoring watches for all three: prompts and responses, refusal and error rates, cost and volume, accuracy against a sample of known answers, and signs of abuse. Lessons 15 and 16 build that monitoring in detail.

Maintenance covers the deliberate changes: patching the serving stack, rotating credentials, refreshing the retrieval index, retiring a model. Each change that alters behaviour goes back through evaluation, because a model change is a production change even when no line of application code moved.

Feedback and iteration is where the life cycle becomes a loop, and where it is most exposed. Thumbs-up ratings, corrected answers and flagged outputs are valuable training signal, and they are also attacker-controlled input into the next version. A feedback channel that flows straight into retraining lets a determined group skew the model over time. The control is to treat feedback as untrusted data: collect it, review or sample it, check it for coordinated patterns, and admit it to training only through the same provenance and integrity gates as any other dataset.

Human-centric design: human-in-the-loop, human oversight and human validation

Human-centric AI design keeps people in a meaningful position relative to the system, and three distinct arrangements appear in scenarios. Telling them apart is most of what a question on this topic needs.

Human-in-the-loop. A person must act before the system's output takes effect: the model drafts, a human approves. Use it where an action is high-impact or cannot be undone, such as a refund over a threshold, a change to a firewall rule proposed by an agent, or a decision about an individual. Its cost is throughput, and its weakness is automation bias: a reviewer who approves hundreds of correct suggestions stops reading them. Effective designs show the reviewer the evidence behind the output and keep the approval volume low enough to be real.

Human oversight. People supervise the system as it runs and can intervene or stop it, without approving each output. It suits high-volume, lower-impact work: a triage model that routes tickets while an analyst watches the queues and can pause it. Oversight needs monitoring that tells the supervisor something useful, and a stop control that actually works. The EU AI Act makes human oversight a requirement for high-risk systems, which is one reason the term appears in governance scenarios.

Human validation. People check outputs for correctness, either before use or by sampling afterwards: reviewing generated code before merge, or auditing a sample of chatbot answers each week against the source documents. Validation is how hallucination and drift are caught when nothing has visibly failed.

The scenario pattern: match the arrangement to the impact and the volume. Irreversible and individual means in the loop. Continuous and high-volume means oversight with a stop control. Quality that degrades silently means validation by sampling. An answer that removes the human entirely from a high-impact decision, in the name of efficiency, is rarely the right one.