Spot extraction and runaway cost in the query log

Steps

1. Write lab/make-theft-log.py. It writes about 2,000 records covering 24 hours to lab/evidence-theft.jsonl.

import json, random
from datetime import datetime, timedelta

random.seed(19)
START = datetime(2026, 9, 14, 0, 0, 0)
END = START + timedelta(hours=24)
USERS = ["alice", "bob", "carol", "dave", "erin", "frank", "grace", "heidi", "ivan", "judy"]
SPIKE_USER = "erin"
QUESTIONS = [
    "How many days of annual leave do staff get?", "Who is on call this weekend?",
    "What warranty does the Model S kettle have?", "Can I carry leave over into next year?",
    "Where do I send an expenses claim?", "Is the office open on bank holidays?",
    "How do I book a meeting room?", "What is the returns window for online orders?",
    "Does the kettle work on 110 volts?", "Who approves overtime?",
    "When is the next payroll date?", "How do I report a lost badge?",
    "Can contractors use the staff canteen?", "What counts as a working-from-home day?",
    "How long is the probation period?", "Which form do I use for parental leave?",
    "Can a customer exchange a faulty toaster?", "What are the on-call handover steps?",
    "How do I reset my voicemail PIN?", "Is there a cycle-to-work scheme?",
    "Summarise the travel policy in three lines.", "What should I do if a delivery is late?",
    "Who do I ask about a payslip error?", "Are pets allowed in the office?",
    "How much notice do I give before leaving?", "What is the dress code for site visits?",
]
LONG = [
    "Write a full, detailed report on every policy in the handbook.",
    "Rewrite the complete product FAQ as a long training manual.",
    "Produce a detailed, section-by-section commentary on the leave policy.",
    "Draft an exhaustive onboarding guide covering every document you can see.",
]
recs = []


def add(t, user, prompt, response_tokens, params):
    recs.append({"ts": t.strftime("%Y-%m-%dT%H:%M:%S+0000"), "user": user, "prompt": prompt,
                 "retrieved_ids": ["hr-holiday-policy.md#0"], "response": "(synthetic answer)",
                 "prompt_tokens": len(prompt) // 4 + 180, "response_tokens": response_tokens,
                 "latency_ms": 400 + response_tokens * 9, "decision": "allow", "params": params})


ordinary = {"temperature": 0.2, "max_tokens": 512}
for u in USERS:
    for _ in range(random.randint(95, 125)):
        t = START + timedelta(hours=random.uniform(7.5, 18.5))
        add(t, u, random.choice(QUESTIONS), random.randint(40, 380), ordinary)
for _ in range(30):
    t = START + timedelta(hours=14 + random.uniform(0, 2))
    add(t, SPIKE_USER, random.choice(LONG), random.randint(3200, 4096),
        {"temperature": 0.2, "max_tokens": 4096})
t, n = START, 0
while t < END:
    n += 1
    add(t, "svc-reporting", "[TEST-PATTERN:systematic-probe-%04d]" % n, random.randint(90, 140),
        {"temperature": 0.0, "max_tokens": 128, "logprobs": True})
    t += timedelta(seconds=random.randint(95, 105))
recs.sort(key=lambda r: r["ts"])
with open("lab/evidence-theft.jsonl", "w") as f:
    for i, r in enumerate(recs, start=1):
        print(json.dumps(dict(id="t%05d" % i, **r)), file=f)
print(len(recs), "records written to lab/evidence-theft.jsonl")

Run it with python lab/make-theft-log.py. 2. Write lab/theft-stats.py, which computes four measures per user: requests; peak_per_hour, the most requests in any one clock hour (the first 13 characters of ts name the hour); template_share, the share of the user's prompts whose first 20 characters equal that user's most common 20-character prefix, rounded to three places; and tokens, the sum of response_tokens. It writes lab/theft-findings.csv with header user,requests,peak_per_hour,template_share,tokens,finding,control and one row per user. 3. Judge each user against the others, not against a fixed number. Set finding to extraction for an account whose volume is far above everyone's and steady around the clock with templated prompts, to cost-spike for an account whose token total is far above everyone's without that volume pattern, and to none otherwise. Fill control for each flagged account from the lesson's map, using semicolons rather than commas: a per-user rate limit, detection of the same behaviour across accounts and no scores returned for extraction; a token budget and an output token cap for a cost spike. Leave it empty for none. 4. Look at what else the seeded account did. Compare the params field across users and the hours each user was active, and write lab/theft-notes.md with at least three lines starting - : what the account asked for that the application never displays, when it was active, and why a limit on requests per account alone would not stop the same behaviour spread across many accounts. 5. Confirm the rate limit. Lesson 10's short lab added RATE_LIMIT = 5 and RATE_WINDOW_S = 600 with block:rate; check they are still in gateway/gateway.py, or add them back as that lab describes. 6. Confirm a token budget. Lesson 10's applied gateway looks each user's daily_tokens up by name in gateway/keys.json and refuses with 429 and block:budget once that user's prompt_tokens plus response_tokens for today reach it. Issue budget-probe an entry with daily_tokens set to 50; the Verify requests send no key, and the budget still applies because it is looked up by user. If your build has no budget, add it as that lab describes. 7. Apply the lesson's output minimisation. The gateway must build the runner request from its own fixed fields only, so a client cannot ask for per-token probabilities, and its reply must carry only response, decision and retrieved_ids. If your runner returns probabilities or scores, drop them before logging and replying. Restart the gateway.