AI risks: bias, accidental leakage, reputation, accuracy, intellectual property, autonomy and shadow AI

Why this matters

Earlier lessons treated AI risk from the attacker's side: injection, poisoning, theft, abuse of agents. This lesson takes the risk register's view. Most of the risks here need no adversary at all. A model drifts, an employee pastes the wrong file, a chatbot invents a refund policy, an agent closes the wrong tickets, a team signs up for an AI tool on a credit card. Each one costs money, customers or legal exposure, and each one ends up in front of the people who run security and risk.

The skill being tested is translation. A security professional has to take a risk described in technical terms ("the model's precision on the new product line has fallen") and state it in business terms ("we are now refusing one in five legitimate claims from our fastest-growing segment"), because only the second version gets an owner, a budget and a decision about appetite.

Scenario questions in this area usually describe an outcome and ask which risk it represents, or describe a risk and ask for the control that reduces it most directly. The distractors are typically controls that address a neighbouring risk: an attacker-focused control offered for an accidental problem, or a technical fix offered for a missing owner.

The lesson

Introduced bias and accuracy or performance drift as business risks

Introduced bias is bias that enters a system through how it was built or how it is used, rather than through an attacker. The usual routes:

• Historical data that encodes past decisions, so the model learns to repeat them.
• Sampling that under-represents some customers, regions or languages.
• Labelling done inconsistently or by people with a shared blind spot.
• Proxies that stand in for a sensitive attribute, as lesson 30 described.
• Feedback loops, where the model's own decisions shape the data it is later retrained on. A fraud model that sends more investigators to one area finds more fraud there, which confirms the model.

Deliberate skewing by an adversary was covered in lesson 18; the business consequence is the same, and so is much of the evidence. The risk is discrimination claims, regulatory action, unfair treatment of customers and staff, and decisions that are simply worse because they are wrong about part of the population.

Accuracy and performance risk is the model being wrong more often than the business assumed. It has two forms. The model may never have been as good as believed, because evaluation used data that was cleaner or more familiar than production. Or it may degrade over time through data drift, where the inputs change (new products, new fraud patterns, a new customer segment), and concept drift, where the relationship between inputs and the right answer changes (what counted as suspicious last year is normal now). For generative systems, performance also includes hallucination rate, refusal rate and the quality of retrieved context.

The controls are measurement and ownership: an agreed performance threshold expressed in business terms, monitoring of live accuracy against it where ground truth eventually arrives, monitoring of input distributions where it does not, periodic re-evaluation, and a named owner who decides when the model is retrained or withdrawn.

Accidental data leakage through prompts, outputs and training

Accidental leakage is a common AI incident, and it has three paths.

Through prompts. Staff paste confidential material into an external service: source code for debugging, a customer list for formatting, a draft contract for summarising, meeting notes containing personal data. Whether that is a breach depends on the service's terms on retention, human review and use for training, which is why lesson 33 treats sanctioned tools and vendor terms as the main controls.

Through outputs. A model returns something the user should not see. In a RAG application over HR documents, a manager asks a general question and the answer quotes a colleague's disciplinary record, because retrieval ignored the access controls on the source documents. In a support chatbot, conversation history or another customer's ticket appears in a response. The control is enforcing the user's permissions at retrieval time, as lesson 11 described, plus output filtering for sensitive data patterns.

Through training. Data used to train or fine-tune a model can be memorised and later reproduced, or inferred through the attacks in lesson 19. Once in the weights it cannot reliably be selectively removed; the practical remedy is retraining without it, which is why minimisation before training (lesson 14) matters more than any later control.

Two places teams forget: logs, where full prompts and responses are kept indefinitely with broad access (lesson 15), and integrations, where an assistant with mailbox or file access can be asked to collect and forward material its user should not handle.

Reputational loss and intellectual property risk from what a model produces

Reputational loss comes from outputs the public sees: a customer-facing chatbot that is rude, offensive, politically partisan or simply wrong; a marketing image that misrepresents a product; a generated reply that commits the organisation to a refund or a price it never offered. Customers and regulators will generally treat what the organisation's chatbot says as something the organisation said. The damage is fast, because screenshots travel, and it is out of proportion to the technical severity, because one bad exchange can define the story.

Controls are scoping the assistant to its task, output guardrails tested against the topics most likely to embarrass, refusal and hand-off behaviour for anything that would create a commitment, disclosure that users are talking to an AI, monitoring for abusive conversation patterns, and a prepared communications response. The last is often missing and frequently the scenario's answer.

Intellectual property risk runs in both directions.

• Inbound: generated output may reproduce copyrighted text, images or licensed code closely enough to create infringement or licence obligations. A code assistant suggesting a block copied from a project under a copyleft licence can impose terms on the product that includes it.
• Outbound: the organisation's own trade secrets leave through prompts, or through fine-tuning a third party's model on proprietary data under terms that do not protect it.
• Ownership: whether generated output can be owned or protected at all varies by jurisdiction and is unsettled in places, so a business relying on exclusive rights to AI-generated material should get legal advice rather than assume.
• Model licences: openly downloadable models come under licences that may restrict commercial use, field of use or scale. "Open weights" is not the same as "free to use for anything".

Controls include code-scanning tools that flag licence matches (lesson 27), vendor terms covering output indemnity and training use, a register of models in use with their licences, and human review before generated material is published. Legal interpretation belongs to counsel; the security role is making sure the facts about what was used, where and under which terms are recorded.

Autonomous systems, and risk that grows with autonomy

An autonomous system is one that takes actions, not just produces text for a person to read. The risk of a model is roughly the risk of its worst output multiplied by what that output can do without anyone checking it. A summariser's error costs a misread paragraph. An agent with ticket and email tools that misreads an instruction can close a hundred customer tickets, email the wrong distribution list, or approve a change.

It helps to think in levels of autonomy:

1. Suggest: the model drafts, a person decides and acts.
2. Act with approval: the model prepares the action, a person confirms it.
3. Act and notify: the model acts, tells a person at once, and the person can reverse it.
4. Act within bounds: the model acts alone inside defined limits, with review after the event.

Risk rises with each step, and so does the importance of three properties: reversibility (can the action be undone?), blast radius (how many records, users or systems can one action touch?) and attributability (can we reconstruct which agent did what, on whose instruction?). Autonomy also creates an accountability gap: when an agent's action causes harm, someone must already be named as answerable for it.

The controls are those of lesson 12 and lesson 26: least-privilege tool permissions, approval gates for irreversible or high-impact actions, rate and volume limits, kill switches, full action logs, and starting low on the scale and moving up only on evidence. A scenario offering "give the agent broader permissions so it can finish the task" is almost always the wrong answer.

Shadow IT and shadow AI: finding the AI use nobody approved

Shadow IT is technology in use without the organisation's knowledge or approval. Shadow AI is the same pattern with AI services, and it is spreading faster for three reasons: the tools are free or cheap, they need no installation, and AI features are being switched on inside software the organisation already approved. A sanctioned note-taking or CRM product can acquire an AI assistant in an update, sending data to a new processor nobody assessed.

Finding it uses the familiar discovery tools pointed at a new category:

• Web proxy, secure web gateway and DNS logs for traffic to known AI services, with categories kept current.
• Cloud access security broker discovery of SaaS use, including uploads.
• Identity provider and OAuth consent records, showing which users have granted third-party AI apps access to mail, calendars or files.
• Browser extension and endpoint inventories, since many assistants run as extensions that read every page.
• Expense and procurement data for subscriptions bought on cards.
• Code repositories and cloud accounts for API keys to model providers.
• Vendor change notices for AI features added to existing products.

The response matters as much as the discovery. Blocking alone pushes use onto personal devices, where it is invisible. The effective pattern is to provide a sanctioned alternative that meets the need, block or restrict high-risk services, coach the users found, and feed the findings into the approval path from lesson 28. A scenario where discovery shows widespread use of one public chatbot is usually asking for the sanctioned alternative, not for punishment.

What the exam is really asking

Identify whether the risk is accidental or adversarial, and whether it is about data going in, output coming out, or actions being taken. Then choose the control that acts at that point. Leakage through prompts wants sanctioned tools and training; leakage through outputs wants retrieval-time access control; autonomy wants narrower permissions and approval gates; shadow AI wants discovery plus an alternative people will use.