Score synthetic card transactions and choose a threshold you can defend

applied · 90 min · Objective 3.1

Task

Apply the whole of the detection lesson to a fraud problem. Generate a synthetic, heavily imbalanced transaction set, then compare a rules baseline with an unsupervised anomaly model and a supervised classifier. Choose an alert threshold against a stated review capacity, and then watch an adaptive fraudster erode the rules while you measure what happens to the model.

Steps

  1. Write lab/make_transactions.py, seeded with 11, which writes lab/transactions.csv with header txn_id,ts,customer_id,amount,merchant_category,country,device_id,new_device,payee_age_days,txns_last_hour,is_fraud. Use ISO timestamps across 30 days, 500 customers, about 20,000 rows and a fraud rate near 0.5 per cent. Give each customer a home country, a usual device and a typical spend. Make fraud resemble the lesson's features (a new device, a young payee, a velocity burst, an amount far above the customer's history, a change of country), but never all of them at once, so that no single feature separates the classes.
  2. Split by time, not at random: days 1 to 21 for training, days 22 to 30 for testing, because in real life labels arrive late. Write the training IDs one per line to lab/train-ids.txt.
  3. Write the rules baseline, named rules: flag a transaction when the amount is over 500 and the device is new, or when txns_last_hour is above 5. Its score is 1 or 0.
  4. Train two models on the training days. The first, isoforest, is scikit-learn's IsolationForest fitted on the features without the label: anomaly detection. The second, forest, is RandomForestClassifier fitted on the labels: pattern recognition. Write a score for every test transaction under each model to lab/scores-test.csv with header txn_id,model,score, where a higher score means more suspicious. For the isolation forest, negate score_samples so that its sign agrees.
  5. Write lab/fraud-metrics.csv with header model,threshold,accuracy,precision,recall,alerts,chosen, measured on the test days, with several thresholds per model. Include an allow-all row that approves everything, with its threshold left blank, so the accuracy trap sits on the page next to the honest figures.
  6. Your fraud team can review 40 alerts a day, which is 360 across the nine test days. For forest, choose the threshold with the highest recall that stays within 360 alerts, and mark that row chosen=yes. Leave every other row's chosen as no. In lab/fraud-decision.md, under ## Threshold, write what this choice costs in missed fraud and in legitimate customers challenged.
  7. Adapt the adversary. Generate lab/transactions-adapted.csv for the test days with the same generator, but with fraudsters keeping every amount under 500 and pacing themselves to at most five transactions an hour. Score rules and forest on it, with forest trained as before and not retrained, and add rows rules-adapted and forest-adapted to the metrics.
  8. Under ## Adaptive adversary in the same file, explain the change in recall for each, and name the two responses the lesson gives: retraining on recent labelled data, and layered rules that are harder to probe.

Verify

cd ~/secai-lab
python - <<'EOF'
import csv
tx = {r['txn_id']: r for r in csv.DictReader(open('lab/transactions.csv'))}
train = set(open('lab/train-ids.txt').read().split())
test = [r for k, r in tx.items() if k not in train]
fraud = sum(r['is_fraud'] == '1' for r in tx.values())
print(len(tx), 'rows; fraud rate', round(fraud / len(tx), 4))
print('time split clean:', max(tx[k]['ts'] for k in train) < min(r['ts'] for r in test))
m = [r for r in csv.DictReader(open('lab/fraud-metrics.csv')) if r['chosen'] == 'yes'][0]
tp = fp = fn = 0
for s in csv.DictReader(open('lab/scores-test.csv')):
    if s['model'] != m['model']:
        continue
    flag = float(s['score']) >= float(m['threshold'])
    y = tx[s['txn_id']]['is_fraud'] == '1'
    tp += flag and y
    fp += flag and not y
    fn += y and not flag
p, r = tp / max(tp + fp, 1), tp / max(tp + fn, 1)
ok = abs(p - float(m['precision'])) < 0.01 and abs(r - float(m['recall'])) < 0.01
print(m['model'], 'alerts', tp + fp, 'precision', round(p, 3), 'recall', round(r, 3), 'PASS' if ok else 'FAIL')
EOF
awk -F, '$1 ~ /^(allow-all|rules|rules-adapted|forest-adapted)$/ {print $1, "accuracy", $3, "recall", $5}' lab/fraud-metrics.csv
grep -c '^## ' lab/fraud-decision.md

A fraud rate near 0.005, a clean time split (True), and PASS for the chosen forest row with 360 alerts or fewer. The recomputed figures must match the ones you reported. The allow-all row shows accuracy above 0.99 with a recall of 0, rules-adapted shows a far lower recall than rules, and the decision file has two sections. A FAIL means the metrics file and the scores disagree, which usually means the metrics came from the training days.

Notes

Compare forest-adapted with rules-adapted. The forest usually loses less, because it learned combinations of features rather than two fixed cut-offs, but it does lose some, and nothing in its scores announces the loss. In production the only warning is recall falling on labels that arrive weeks later. That is why the lesson treats drift monitoring and retraining as part of operating the detector rather than as an upgrade.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.