Version a data set by its hash

short · 35 min · Objective 5.1

Task

Register every file of the practice data by its SHA-256 hash, prove that running the register again changes nothing, and prove that regenerating the data reproduces it byte for byte.

Steps

  1. Write lab/version/register.py taking a folder. For each file in it, it calculates the SHA-256 of the file's bytes, and appends file,sha256 to lab/version/versions.csv (header file,sha256) only when the file is new or its hash differs from the last one recorded. It prints added N.
  2. Run it on data twice, and note what the second run prints.
  3. Generate the data again into lab/version/regen with the same make-datasets.py --out command, and record in lab/version/regen.txt the line regenerated identical: N of M, counting files whose hash matches the file of the same name in data.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

python3 lab/version/register.py data
awk -F, 'NR>1' lab/version/versions.csv | wc -l
cat lab/version/regen.txt

A third run prints added 0; the register holds 9 rows, one per file; and regenerated identical: 9 of 9. The generator is seeded, so the same command gives the same bytes, and a hash is then a complete description of which version an analysis used.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.