AI-generated content: deepfakes, impersonation, misinformation and disinformation

Why this matters

For most of the history of security awareness, a voice on the phone that sounded like the finance director was reasonable evidence that it was the finance director, and a video call showing the faces of three colleagues was stronger evidence still. Generative models have ended both assumptions. A convincing synthetic voice can be produced from a modest amount of recorded speech, and live face and voice substitution on a video call is no longer a research demonstration. The cost of producing a plausible fake has fallen; the cost of being fooled by one has not.

The defender's difficulty is that most of these attacks contain nothing a technical control can block. There is no attachment, no malicious link and no exploit. A deepfaked call asking for a payment is, at the network level, an ordinary call. The controls that work are procedural: verification that does not depend on the channel the request arrived through, and processes designed so that one convincing conversation is never enough to move money, credentials or data.

This lesson treats AI-generated content as a threat to an organisation's decisions and reputation rather than to its machines; lesson 24 covers reconnaissance and attack automation. Both teach recognition and response, not production.

Scenario questions on this topic tend to describe a convincing request and offer a detection product, a training course and a process change as answers. The process change is usually the right one.

The lesson

Deepfakes of voice and video, and the verification steps that defeat them

A deepfake is synthetic media, whether audio, video or an image, produced or altered by a model so that it appears to show a real person saying or doing something they did not. For a defender the useful split is by channel:

• Voice clones. A model conditioned on samples of a person's speech produces new speech in that voice. Executives supply the samples themselves through interviews, results calls and conference talks. Voice is the most dangerous channel, because telephone audio is low-fidelity anyway and the artefacts that might betray a fake are masked by the line.
• Video. Pre-recorded fake video is used for disinformation and extortion. Live substitution on a video call is used for fraud, with the attacker appearing in real time as someone else.
• Images and documents. Fake identity documents, fake screenshots of an approval, fake photographs offered as evidence.

Deepfakes are often associated with generative adversarial networks, although the early face-swap tools used autoencoders, and diffusion models now do much of the work. The architecture does not change the defence; lesson 24 returns to adversarial networks.

The tempting defence is a detection tool that inspects media for artefacts such as lip-sync drift or spectral oddities in audio. Such tools help in forensic work after the event, but make a poor primary control for three reasons. They are in an arms race they structurally lose, because every detectable artefact becomes a training signal for the next generator. They give probabilistic answers, and a score of "likely synthetic" does not tell a payments clerk whether to release funds. And the media has to pass through them, which a call to a mobile phone does not.

The controls that defeat deepfakes do not try to decide whether the media is real. They make the decision independent of it:

• Out-of-band verification. A request received on one channel is confirmed on a different channel, initiated by the recipient, using a contact detail from a trusted record.
• Pre-agreed challenges for high-risk requests. A code phrase agreed in person between, for example, an executive and their assistant, and never stored where an attacker inside the mailbox could read it.
• Process controls that do not bend for seniority. Dual authorisation above a threshold, a waiting period on changes to supplier bank details, and no exception on one person's say-so, however senior they sound.
• Content provenance, where available. Standards such as C2PA, whose signed records are shown to users as Content Credentials, attach signed provenance metadata to media when it is created, and watermarking (lesson 5) can mark generated output. Both help show that genuine media is genuine. Neither proves a fake is fake, because a fake simply arrives without them, so missing provenance is a weak signal rather than a verdict.

Impersonation at scale, and why a call-back procedure still works

Impersonation predates AI; what AI changes is the economics. A human fraudster runs a handful of convincing pretexts a day. A model can draft thousands of individually tailored messages, in the target's language and in the house style of the organisation being impersonated, referring to real projects and colleagues found in public sources, and it can sustain a coherent back-and-forth with each victim. The giveaways that awareness training used to teach, such as clumsy grammar, generic greetings and an odd tone, are gone.

The forms that now scale include:

• Executive impersonation and business email compromise, followed up with a cloned voice when the victim hesitates.
• Supplier impersonation, asking accounts payable to update bank details.
• Help desk impersonation in both directions: an attacker posing as an employee to have MFA reset, or posing as IT support to an employee.
• Brand impersonation aimed at customers: fake support chatbots, fake apps and cloned websites with fluent copy.
• Candidate impersonation in remote hiring, where someone interviews under another person's identity, sometimes with live face substitution.

A call-back procedure still works against all of these because it does not depend on the quality of the fake. It has three rules. The recipient ends the inbound contact. They reach the claimed sender through a contact detail from a trusted source, such as the HR directory, the supplier master record created at onboarding or the number on the contract, and never one supplied in the request. And they confirm the specific request, not merely the person. An attacker with a perfect voice clone still cannot answer the real executive's real phone.

The procedure fails only if the trusted record has itself been altered, which is why changes to that record, such as a supplier's bank details or an employee's registered phone number, need their own verification and approval.

The help desk is where this matters most. A password or MFA reset is an identity decision, and verification for it should rest on something a voice cannot fake: a call back to the registered number, approval from the person's manager through an authenticated channel, or identity proofing for privileged accounts.

A scenario that turns on this usually includes a detail such as "the caller gave a number to ring back" or "the email included the supplier's new contact details". That is the trap: verification that uses the attacker's channel is not verification.

Misinformation versus disinformation, and the intent that separates them

Both are false or misleading information. Intent separates them.

• Misinformation is spread without intent to deceive; the person sharing it believes it. A colleague forwarding a fake breach notice they think is genuine spreads misinformation. So does a support chatbot that hallucinates a refund policy: the model has no intent and the organisation did not mean to mislead. LLM09 Misinformation in the 2025 OWASP Top 10 for LLM Applications covers this case for model output, and the 2023 list's Overreliance entry described the human side of it.
• Disinformation is created or spread deliberately to deceive, usually for financial, political or competitive gain. A fabricated video of a chief executive announcing a product recall, released to move a share price, is disinformation.

Some frameworks add malinformation: genuine information shared out of context to cause harm, such as a selectively leaked internal email.

Intent matters to a security team because it decides the response. Misinformation is corrected: publish the accurate information, fix the source by grounding the chatbot or briefing staff, and move on. Disinformation is an attack. It has an actor and a campaign, often coordinated amplification by networks of accounts, and it may be paired with other activity, such as a deepfaked announcement timed to coincide with a phishing wave that refers to it. It is handled as an incident, with evidence preservation, takedown requests to platforms, legal and communications involvement and threat intelligence on the source.

AI feeds both: hallucination produces misinformation by accident, and generation gives disinformation a volume and polish that once needed a well-funded operation.

On the exam, watch the cue words. "Accidentally", "believed" and "error" point to misinformation; "deliberately", "campaign", "coordinated" and "fabricated" point to disinformation.

AI-assisted social engineering, and the training that addresses it

Social engineering exploits trust, urgency, authority and helpfulness. AI adds no new levers; it pulls the existing ones with more precision. A model can turn a target's public footprint into a pretext, write in the voice of their manager, build rapport over a conversation lasting days and hand over to a cloned voice when a call becomes necessary. It also lets attackers work fluently in languages they do not speak, removing a filter many organisations relied on without realising it.

The consequence for training is that cue-based awareness is obsolete. Teaching people to spot spelling mistakes or a strange accent teaches them to trust messages without those features, which is now every message an attacker writes with a model. Training that still works is behaviour-based and centres on the request rather than the messenger:

• Recognise the high-risk request types whatever the channel: moving money, changing credentials or MFA, exporting data, changing payment details and urgent exceptions to process.
• Treat urgency, secrecy and authority pressure as a signal to slow down. "The chief executive needs this before the board meets, and keep it between us" is a description of an attack, not a reason to hurry.
• Know the verification step for each high-risk request type, and know that using it will never count against them, even when a genuine executive is mildly inconvenienced.
• Report quickly through one obvious mechanism.

Simulations should use AI-quality lures and, for finance, help desk and executive assistant roles, voice exercises. Watch the reporting rate and time to report as well as who was fooled, because a fast first report protects everyone after it. Executives need their own briefing: they are the most impersonated people in the organisation and the most likely to demand an exception to the procedure that protects them.

The cultural point is what scenarios test: staff who are afraid to question a senior request are a vulnerability no technical control fixes.

Building detection and response around content a filter cannot catch

Email gateways and content filters still matter, because they remove the bulk, and lesson 21 covers AI-assisted detection. But a well-written message with no link or attachment, a phone call and a video meeting all pass through filters by design. Detection for this class of threat moves from inspecting content to watching context and behaviour:

• Transaction and request anomalies. A payment to a new beneficiary, a bank detail change followed soon after by a large invoice, an MFA reset followed by a new device enrolment and a new mailbox forwarding rule. These are visible in the finance system and the identity provider however convincing the original request was.
• Identity signals. Newly registered lookalike domains, display-name spoofing, replies redirected to external addresses, a first-time sender claiming to be a known supplier.
• Brand and executive monitoring. Watching for impersonating domains, apps, social accounts and fabricated media about the organisation's leaders, so that takedown starts before a campaign peaks.
• Human reports as a sensor. Each staff report is telemetry, and several reports of the same pretext tell the security team a campaign is under way.

Response needs playbooks written for these events, because the malware playbook does not fit:

• Suspected deepfake fraud. Contact the bank at once to stop or recall the payment, preserve the media and call records, notify the person who was impersonated, and find out who else was approached.
• Disinformation about the organisation. A pre-agreed group drawn from security, communications, legal and the executive; a verified channel for official statements, established before it is needed; evidence preservation; and platform takedown requests.
• Abuse of an executive's likeness. A published statement of how the organisation genuinely communicates, such as "we never ask for payment changes by phone", gives staff, customers and suppliers something to check a fake against.

What the exam is really asking

Most questions here reduce to one principle. When the content cannot be reliably judged, stop judging it: judge the request, verify the requester through a channel the attacker does not control, and watch the systems where the harm would actually land. An answer that relies on a person or a tool spotting the fake is weaker than one that makes spotting it unnecessary.