Generate a data dictionary, and let it catch drift

short · 40 min · Objective 5.1

Task

Write a script that builds a data dictionary from the raw customer file itself, then a second script that compares the dictionary with the file and says when the two no longer agree.

Steps

  1. Write lab/docs/dictionary.py to read data/customers_raw.csv and write lab/docs/dictionary.csv with header column,type,blank,distinct: one row per column, where blank counts empty values and distinct counts different non-empty values.
  2. Set type to integer if every non-empty value is a whole number, decimal if every one is a number with a decimal point, date if every one is YYYY-MM-DD, and text otherwise.
  3. Write lab/docs/drift.py taking a data file and a dictionary. It prints in sync: N columns when the file's header lists exactly the dictionary's columns, and otherwise changed: followed by the columns added and removed.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

awk -F, 'NR>1 {print $1": "$2", "$3" blank, "$4" distinct"}' lab/docs/dictionary.csv
python3 lab/docs/drift.py data/customers_raw.csv lab/docs/dictionary.csv

Eight columns. email has 20 blank and 294 distinct; region has 16 distinct values where four were expected; and signup_date is typed text, not date, because 30 of its values are written DD/MM/YYYY. A dictionary generated from the data describes the data's faults as well as its shape. The drift check prints in sync: 8 columns -- until someone adds a column, which is when a hand-written dictionary quietly goes wrong.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.