Automate a playbook and measure what broke

applied · 90 min · Objective 1.5

Task

Take one complete triage workflow, automate it end to end except the decision, run it against real lab alerts, and measure both what it saved and what it got wrong. Objective 1.5 is efficiency and process improvement, and the measurement is the part that makes it improvement rather than change.

Steps

  1. Write the playbook by hand first, as numbered steps, for one alert type you have real examples of -- failed login bursts is a good choice.
  2. Time three manual runs. Record the time and the disposition for each.
  3. Automate every mechanical step: gather the events, count them, group by account and source, pull the account's recent history, check whether the source is a known lab host, and print a structured summary with an explicit field for anything it could not determine.
  4. Run it against ten alerts. For each, record the automated summary and your disposition.
  5. Now audit it. Hand-check three of the ten completely. Count the fields the automation got wrong, and separately the fields it reported confidently that it should have marked unknown.
  6. Compute three numbers: time saved per alert, error rate, and the fraction of errors that were silent rather than visible.

Verify

python3 /tmp/playbook.py --alert-file /tmp/alerts.json | tee /tmp/auto.txt | grep -c "^"
grep -c "UNKNOWN" /tmp/auto.txt
grep -c "confidence" /tmp/auto.txt

The second count must be non-zero. An automation that never reports UNKNOWN across ten real alerts is not certain -- it is unable to express uncertainty, which means every gap in its input is being rendered as a confident value. That is the failure mode from the short lab, now measured across a real batch.

Notes

The number to report is the third one: silent errors as a fraction of all errors. A visible error costs you a minute. A silent one costs you the disposition, and you never learn which alerts it cost you.

Keep the decision manual. You have automated the gathering, and the moment the script assigns dispositions your error rate stops being measurable, because nobody is checking the ones it closed.

This is an independent study companion for CompTIA CySA+ CS0-004 and is not produced by or endorsed by CompTIA.