Collect files over HTTP and prove they arrived intact

applied · 45 min · Objective 1.2

Task

Serve the practice data from a local web server, download it with a script as you would from a real data portal, record the provenance of each file, and prove with checksums that what arrived is exactly what was served.

Steps

  1. In one terminal, run python3 -m http.server 8000 --bind 127.0.0.1 --directory data.
  2. Write lab/http/fetch.py, which downloads orders.csv, products.csv and survey.json from http://localhost:8000/ with Python's urllib, saves them under lab/http/downloads/, and records each file in lab/http/provenance.csv with header file,url,bytes,sha256,fetched_at.
  3. Compute the SHA-256 of the same three files in data/ and add them to lab/http/source-hashes.csv with header file,sha256.
  4. Stop the server with Ctrl+C.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

awk -F, 'NR>1 {print $1, $3}' lab/http/provenance.csv
python3 -c "import csv; a={r['file']: r['sha256'] for r in csv.DictReader(open('lab/http/provenance.csv'))}; b={r['file']: r['sha256'] for r in csv.DictReader(open('lab/http/source-hashes.csv'))}; print(sum(a[f] == b[f] for f in b), 'of', len(b), 'hashes match')"

Three files downloaded, with their sizes -- orders.csv is 66772 bytes -- and all 3 of 3 hashes match the served originals. A size check alone would miss a file altered without changing its length; a checksum would not.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.