Beat a do-nothing model, and see why accuracy alone misleads

short · 35 min · Objective 1.5

Task

Treat "is this customer a Business customer?" as a classification problem. Measure the accuracy of a model that always guesses the commonest answer, then of a simple rule, and compare their precision and recall. The numbers show why a single accuracy figure can flatter a useless model.

Steps

  1. Count Business and Consumer customers in data/customers.csv.
  2. The baseline model predicts Consumer for every customer. Record its accuracy as a percentage to one decimal place in lab/ml/results.txt as baseline accuracy: N.
  3. The rule model predicts Business for any customer who has placed an order of 100 units or more in data/orders.csv, and Consumer otherwise. Record its precision and recall for the Business class, as percentages to one decimal place, as rule precision: N and rule recall: N.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

grep -E '^(baseline accuracy|rule precision|rule recall): ' lab/ml/results.txt
awk -F, 'NR>1 {c[$7]++} END {for (s in c) print s": "c[s]}' data/customers.csv | sort

The baseline is 65.3% accurate while never identifying a single Business customer. The rule's precision is 100.0% -- every customer it flags is Business -- and its recall 5.8%, because only 6 of the 104 Business customers placed a bulk order. Accuracy could not have told you either of those things.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.