AI for detection: signature matching, anomaly detection, pattern recognition and fraud

Why this matters

Detection is where machine learning has been doing security work the longest, long before anyone was talking about large language models. Spam filters, card fraud scoring and behavioural analytics in a SIEM are all models making decisions about whether something is bad. What has changed is the reach: the same techniques now sit in endpoint agents, email gateways, identity platforms and cloud consoles, and generative models have joined them as assistants that write and explain detection logic.

A security professional needs to know which technique a detection relies on, because each fails differently. A signature misses what it has never seen. An anomaly detector flags what is unusual, which is not the same as malicious, and it learns whatever "normal" it was shown, including an attacker who was already present. A classifier is only as good as the labels it learned from, and an adversary who knows it is there will shape their behaviour to pass it.

Scenario questions on this topic usually describe a detection goal and a constraint, such as a novel threat, a noisy environment or a need to explain each decision to an auditor, and ask which technique fits or why the deployed one is failing.

The lesson

Where machine learning helps signature matching, and where it does not

Signature matching compares an observation with a known indicator: a file hash, a byte pattern, a YARA rule, an intrusion detection signature, a known malicious domain. Its strengths are why it survives: it is deterministic, fast, explainable ("this rule matched these bytes") and has a very low false positive rate when the signature is good. Its weakness is that it only finds what someone has already described, and a trivially changed sample produces a different hash.

AI helps around the signature rather than replacing it:

• Drafting rules. A model can draft a YARA or IDS rule from a description of a sample's strings and behaviour, or convert a rule between formats. The draft is a starting point. It must be tested against the malicious samples it should match and a corpus of known-good files it must not, because a generated rule that looks plausible can be too broad to deploy or too narrow to catch anything.
• Grouping samples. Clustering and similarity techniques, including fuzzy hashing, group variants into families so one well-written signature covers many samples instead of one hash covering one.
• Explaining matches. An assistant can explain to a junior analyst what a rule detects and why it fired, which shortens triage.
• Prioritising. A model can rank which of thousands of indicators from a threat feed are relevant to your estate.

Where it does not help: when the requirement is a deterministic, auditable decision. A blocking control on a known-bad hash should stay a lookup, not become a probability. Mature detection stacks therefore run both: signatures for the known, models for the variant and the unknown.

Anomaly detection and pattern recognition, and the baseline they depend on

These two are often blurred, and a scenario may turn on the difference.

Anomaly detection asks whether something deviates from normal. It is typically unsupervised or semi-supervised: the model learns a baseline of ordinary behaviour, then scores how far new activity departs from it. User and entity behaviour analytics is the familiar form: a service account that starts authenticating interactively, an employee downloading ten times their usual volume, a server talking to a country it has never contacted. Its value is that it needs no prior knowledge of the attack, so it can surface novel techniques and insider activity.

Pattern recognition asks whether something resembles a known class. It is typically supervised: the model is trained on labelled examples and learns the features that separate them. Classifying phishing emails, spotting algorithmically generated domain names, recognising the regular timing of command-and-control beaconing and identifying malware families from behaviour are all pattern recognition. It is more precise than anomaly detection for threats it has learned, and blind to ones it has not.

Both depend on their training data, and anomaly detection depends on it in a specific way: the baseline is the model. That creates three failure modes worth recognising:

• A contaminated baseline. If an intruder was active during the learning period, their behaviour is learned as normal and never flagged.
• Gradual drift. An attacker who increases activity slowly can move the baseline with them. This is model skewing, covered in lesson 18, and the control is to keep a fixed reference baseline and compare against it, not only against a rolling window.
• Legitimate change. A reorganisation, a new application or a holiday period produces a flood of anomalies that are not attacks. Without context, analysts learn to ignore the detector.

The practical consequence: anomalies are leads, not verdicts. Their output should be enriched with context such as asset criticality, identity and recent change records before an analyst sees it.

Fraud detection as one of the oldest production uses of machine learning in security

Card issuers and payment processors have scored transactions with statistical and machine learning models for decades, and fraud detection remains the clearest example of a model making live security decisions at scale.

A fraud model scores each event, such as a payment, a login or an account change, in real time, using features such as transaction velocity, amount compared with the customer's history, device and browser fingerprint, geolocation and how far it is from the last location, and the age of the account or payee. The output is a risk score, and the score drives an action: allow, challenge with step-up authentication, hold for review or decline.

What makes fraud detection instructive is that it exposes every hard problem of ML in security at once:

• Imbalanced classes. Fraud is a tiny fraction of events, so a model that approves everything is almost always "accurate". Accuracy is the wrong measure; precision and recall on the fraud class are the right ones.
• Delayed and incomplete labels. The truth often arrives weeks later, as a chargeback or a customer complaint, and some fraud is never reported. The model trains on a partial picture.
• An adaptive adversary. Fraudsters probe the system with small transactions, learn which patterns pass and change their behaviour. This is input manipulation (ML01 in the OWASP Machine Learning Security Top 10) happening in production, which is why fraud models are retrained frequently and paired with rules that are harder to probe.
• Customer friction and fairness. Every false positive is a declined legitimate customer. If the features correlate with a protected characteristic or a region, the friction falls unevenly, which is a bias problem as well as a commercial one.

The same approach now extends beyond payments to account takeover, fake account creation, promotion abuse and insider fraud in internal systems.

False positives and false negatives when the detector is a model

A rule-based detector fails for a reason you can read. A model fails for reasons spread across thousands of learned weights, so managing its errors needs explicit design.

The threshold is a business decision. A model outputs a score; someone chooses where alerting begins. Lowering the threshold catches more true attacks and produces more false positives; raising it does the reverse. The right point depends on consequence. A model blocking payments can tolerate few false positives. A model flagging possible data exfiltration for review should tolerate more, because a missed incident costs more than an analyst's ten minutes.

The base rate dominates. When true attacks are rare, even a detector that is right almost every time produces mostly false alarms, because the number of benign events is so large. Scenario answers that celebrate a high accuracy figure without mentioning precision on the rare class are usually wrong.

False negatives are invisible. A false positive annoys someone and gets reported. A false negative does not appear anywhere. Measure them deliberately: red-team exercises, replaying known attack data and planting test events that the model should catch, so a degraded detector is noticed.

Confidence and explanation. Analysts work faster when an alert says why: which features drove the score. A calibrated confidence score lets the platform route high-confidence alerts to automated containment and uncertain ones to a person (lesson 15 covers monitoring response confidence).

Drift and retraining. Performance decays as the environment and the adversary change. Monitor precision and recall over time, retrain on recent labelled data, and treat retraining data as an attack surface, since poisoned feedback can teach the model to ignore an attacker.

Alert fatigue is a security failure. A detector that floods the queue is not conservative; it hides real alerts in noise. Tuning it is part of operating it.

Reading a detection scenario and picking the technique it calls for

Match the technique to what the scenario says is known about the threat:

Then check the failure mode the question is hinting at. "The model was trained during an ongoing compromise" points to a contaminated baseline. "Alerts rose after a reorganisation" points to legitimate change without context. "The model is 99.9% accurate but analysts are overwhelmed" points to the base rate and threshold. "Fraud losses rose slowly while scores looked normal" points to an adaptive adversary or drift, and the answer involves retraining, monitoring and layered rules.

What the exam is really asking

Detection questions reward the candidate who knows that no technique is complete on its own. The strongest answers layer signatures for the known, models for the unknown, and human review for anything consequential, and say how the model's errors will be measured.