Attribute every token to a caller and catch the runaway one

short · 40 min · Objective 2.5

Task

Build a cost report that attributes every token in gateway.jsonl to a caller, check that it accounts for all of them, then plant a runaway client and show that a budget check names it, before turning the finding into a ticket that has an owner, a due date and a verification.

Steps

  1. Write lab/prices.json with input_per_1k, output_per_1k and currency set to notional, and lab/budgets.json with a daily notional budget per user and a default for users not listed.
  2. Write lab/cost-report.py, which reads logs/gateway.jsonl and writes lab/cost-by-user.csv with header user,requests,prompt_tokens,response_tokens,notional_cost,share_pct. A record with no user is counted under UNATTRIBUTED rather than dropped: unattributed spend is a finding of its own.
  3. Run it on your existing traffic and save each user's mean tokens per request to lab/cost-baseline.json.
  4. Write lab/runaway.py, which sends 60 requests as batch-bot, each a long ordinary question with the product FAQ pasted in and a request for a full, detailed answer. Lesson 10's limits will refuse most of it: block:rate after five requests, and possibly block:input-tokens. Raise them for this lab only -- for the rate, a batch-bot entry under rate_limits in gateway/rules.json; for tokens, MAX_INPUT_TOKENS -- record that you did, and restore them afterwards: the point is to see what monitoring alone would catch.
  5. Write lab/budget-check.py, which exits 1 and names every user whose notional cost today exceeds their budget, or whose tokens per request exceed five times their baseline (or five times the overall baseline for a user with none), and exits 0 otherwise.
  6. Also compute the share of responses that hit the maximum output token limit, and add it to lab/cost-tickets.csv evidence if batch-bot's share is high.
  7. Create lab/cost-tickets.csv (lesson 1's lab/tickets.csv holds the support tickets, so do not reuse the name) with header id,finding,evidence,severity,owner,due,control,verification and raise one ticket for batch-bot: evidence cites the log time range, the control is a per-user token quota at the gateway, and verification is budget-check exits 0 with the quota in place.

Verify

python lab/cost-report.py
python - <<'EOF'
import csv, json
log_in = log_out = 0
for line in open("logs/gateway.jsonl"):
    r = json.loads(line)
    log_in += r.get("prompt_tokens") or 0
    log_out += r.get("response_tokens") or 0
rows = list(csv.DictReader(open("lab/cost-by-user.csv")))
rep_in = sum(int(r["prompt_tokens"]) for r in rows)
rep_out = sum(int(r["response_tokens"]) for r in rows)
top = max(rows, key=lambda r: float(r["notional_cost"]))
print("log tokens", log_in, log_out, "| report tokens", rep_in, rep_out)
print("top spender:", top["user"], top["share_pct"], "per cent")
print("PASS" if (log_in, log_out) == (rep_in, rep_out) and top["user"] == "batch-bot" else "FAIL")
EOF
python lab/budget-check.py; echo "budget check exit=$?"
awk -F, 'NR>1 && $5!="" && $6!="" {n++} END {print n+0, "ticket(s) with an owner and a due date"}' lab/cost-tickets.csv

The report's token totals equal the log's exactly, batch-bot is the top spender, the script prints PASS, the budget check exits 1 naming batch-bot, and at least one ticket has an owner and a due date. Totals that do not match mean the report is dropping records, and the dropped ones are usually the unattributed spend you most need to see.

Notes

The budget check is the monitoring; the quota in the ticket is the limit. Lesson 16's point is that the first without the second is a report that arrives after the money has gone. Re-run the check after adding the quota and close the ticket only when it exits 0 -- that run is the verification the ticket promised.

This is an independent study companion for CompTIA SecAI+ CY0-001 and is not produced by or endorsed by CompTIA.