Build the workbench and prove the data is what it claims to be

short · 30 min

Task

Install the course's tools, generate the practice data sets, make the raw data read-only, and write a first script that profiles a file the same way every time it runs. By the end you have a workbench where a mistake costs nothing and a record of what the data looked like when you started.

Steps

  1. Install Python 3 if you do not have it, and save the output of python3 --version to lab/bench/python.txt. On Windows, if python3 prints that Python was not found, run alias python3=py in Git Bash first.
  2. Download make-datasets.py from the course's data page and run python3 make-datasets.py --out data.
  3. Count the data rows (not counting the header) in data/orders.csv and data/customers_raw.csv, and record them in lab/bench/counts.csv with header file,rows, one row per file, using the file names exactly as orders.csv and customers_raw.csv.
  4. Make the data read-only with chmod a-w data/*.csv. Try to append a line to data/orders.csv with echo x >> data/orders.csv, and record the outcome in lab/bench/readonly.txt as write refused: yes or write refused: no.
  5. Write lab/bench/profile.py, which prints the number of rows and columns in data/customers_raw.csv. Run it twice, saving the output to lab/bench/run1.txt and lab/bench/run2.txt.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

grep -c 'Python 3' lab/bench/python.txt
awk -F, '$1=="orders.csv" {print "orders: "$2}' lab/bench/counts.csv
awk -F, '$1=="customers_raw.csv" {print "customers_raw: "$2}' lab/bench/counts.csv
grep -c 'write refused: yes' lab/bench/readonly.txt
cmp lab/bench/run1.txt lab/bench/run2.txt && echo "same output both runs"

The orders file has 2000 data rows and the raw customer file 314, the append was refused, and the profile script printed exactly the same thing both times. That last line is the point of the workbench: a script gives the same answer every time, which a sequence of clicks does not. Make the folder writable again with chmod u+w data/*.csv only if you need to regenerate it.

Notes

Read-only files protect against the commonest accident in data work, overwriting the original while cleaning it. The later labs all write their results under lab/, never into data/.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.