Write data quality tests that can stop a pipeline

applied · 50 min · Objective 5.4

Task

Write six named tests for the orders file, some that stop the pipeline and some that only warn, run them on the raw data, then on a corrected copy.

Steps

  1. Write lab/qtests/tests.py taking a folder containing orders.csv, customers.csv and products.csv, and running, in this order: quantity-positive, order-id-unique, customer-exists, product-exists, date-in-2025 (all errors) and quantity-below-100 (a warning).
  2. For each test print PASS NAME, or FAIL NAME (N rows) for a failing error test, or WARN NAME (N rows) for a failing warning. Exit 1 if any error test fails, otherwise 0.
  3. Copy the three files to lab/qtests/fixed, and remove the orders with quantity of zero or below from the copy.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

python3 lab/qtests/tests.py data; echo "exit $?"
python3 lab/qtests/tests.py lab/qtests/fixed | grep -v '^PASS'; echo "exit $?"

On the raw data: FAIL quantity-positive (3 rows), four PASS lines, WARN quantity-below-100 (6 rows), and exit 1, which stops the pipeline. On the fixed copy only the warning remains, and the exit is 0. The six bulk orders may be real; a warning asks a person, where an error would have thrown them away.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.