Carry control totals through a pipeline and trace one record

applied · 50 min · Objective 4.3

Task

Run a four-stage pipeline -- raw, valid, joined, aggregated -- recording a row count and a control total at every stage, and follow one order from the raw file to the figure it ends up in.

Steps

  1. Stage 1 is every row of data/orders.csv; stage 2 the orders with quantity above zero; stage 3 those joined to customers and products; stage 4 the total quantity per region.
  2. Save lab/control/totals.csv with header stage,rows,quantity, where quantity is the stage's total quantity (for stage 4, rows is the number of regions).
  3. Trace order O01234 and record in lab/control/trace.txt the lines order: O01234, quantity: N, unit price: N, net value: N and region: NAME.
  4. Record in lab/control/checked.txt what was checked, by which role, and when.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

awk -F, 'NR>1 {print "stage "$1": "$2" rows, quantity "$3}' lab/control/totals.csv
grep -E '^(order|quantity|unit price|net value|region): ' lab/control/trace.txt

Stage 1 has 2000 rows and 5690 units; stage 2 has 1997 rows and 5703 -- the total goes up, because the three rows removed had negative quantities; stages 3 and 4 keep 5703, with 4 region rows at the end. O01234 is 3 units at 99.21 with no discount, 297.63, in East. A control total that changes between stages must be explained, and this one can be.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.