Find the missing values and the near duplicates

short · 35 min · Objective 2.2

Task

Look for missing values in every column of the raw customer file, then look for duplicates twice: once for exact duplicate rows, and once for the same customer entered with small differences.

Steps

  1. Count blank values per column in data/customers_raw.csv and save them to lab/missing/blanks.csv with header column,blanks.
  2. Count exact duplicate rows, comparing every field, and record exact: N in lab/missing/duplicates.txt.
  3. Count near duplicates: pairs of rows whose emails match when lower-cased, ignoring blanks. Record near pairs: N.
  4. List the near-duplicate pairs in lab/missing/pairs.csv with header kept_id,duplicate_id, keeping the lower id.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

awk -F, 'NR>1 && $2>0 {print $1": "$2}' lab/missing/blanks.csv
grep -E '^(exact|near pairs): ' lab/missing/duplicates.txt
awk -F, 'NR>1 {n++} END {print n" pair(s) listed"}' lab/missing/pairs.csv

Only the email column has blanks, 20 of them. There are 0 exact duplicates and 14 near pairs, all listed: every duplicate differs from its original, so a whole-row comparison finds nothing and a comparison on the right field finds them all.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.