Find the step where the numbers first go wrong

short · 40 min · Objective 3.3

Task

Run a three-step pipeline that produces a customer count per region, record the row count after every step, and use those counts to find the step where the result first goes wrong -- without reading the code.

Steps

  1. Step 1: read data/orders.csv. Step 2: join each order to data/customers_raw.csv on lower-cased email via data/customers.csv (order customer_id to clean email, clean email to raw rows). Step 3: count distinct customer_ids per region.
  2. After each step, record the row count in lab/debug/steps.csv with header step,rows,expected, where expected is what the step should produce if nothing is wrong.
  3. Record in lab/debug/culprit.txt the first step whose rows differ from its expected count, as first bad step: N, and why.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

awk -F, 'NR>1 {print "step "$1": "$2" rows, expected "$3}' lab/debug/steps.csv
awk -F, 'NR>1 && $2!=$3 {print "first bad step: "$1; exit}' lab/debug/steps.csv
grep -E '^first bad step: ' lab/debug/culprit.txt

Step 1 has 2000 rows as expected; step 2 has 1953 where 2000 were expected, so the join is the first bad step -- blank emails drop rows and duplicate customers add them. Counting rows at every step located the fault before anyone read a line of the join.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.