Flag outliers two ways and catch the error one of them misses

short · 40 min · Objective 2.2

Task

Apply the interquartile range rule to the order quantities, then a validation rule, and compare what each catches. The point is the value one method misses.

Steps

  1. Read the quantities from data/orders.csv. Calculate Q1 and Q3 with Python's statistics.quantiles(values, n=4, method='inclusive') and the IQR fences at 1.5 times the IQR.
  2. Record in lab/outliers/iqr.txt the lines lower fence: N, upper fence: N, above: N and below: N.
  3. Apply the validation rule quantity > 0, and record the order_ids that fail it in lab/outliers/invalid.csv with header order_id,quantity.
  4. Record in lab/outliers/missed.txt any invalid order the IQR rule did not flag, as missed: <order_id>.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

grep -E '^(lower fence|upper fence|above|below): ' lab/outliers/iqr.txt
awk -F, 'NR>1 {n++} END {print n" invalid order(s)"}' lab/outliers/invalid.csv
grep -c '^missed: ' lab/outliers/missed.txt

Fences of -3.5 and 8.5; 6 orders above (the genuine bulk orders) and 2 below. The validation rule finds 3 invalid orders, and 1 of them -- a quantity of -3 -- is missed by the IQR rule because it is not unusual enough. Statistical rules find the unusual; validation rules find the impossible.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.