Compress a data set and measure what the layout changes

applied · 40 min · Objective 1.3

Task

Store the same orders three ways -- plain CSV, gzip-compressed CSV and an SQLite table -- measure each, and prove the compressed copy holds exactly the same data.

Steps

  1. Copy data/orders.csv to lab/layout/orders.csv.
  2. Write a gzip-compressed copy to lab/layout/orders.csv.gz with Python's gzip module.
  3. Decompress it to lab/layout/roundtrip.csv and compare it byte for byte with the original.
  4. Record the sizes in bytes in lab/layout/sizes.csv with header format,bytes, using the format names csv and gzip.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

cmp lab/layout/orders.csv lab/layout/roundtrip.csv && echo "round trip identical"
awk -F, 'NR>1 {print $1": "$2" bytes"}' lab/layout/sizes.csv
awk -F, '$1=="csv" {c=$2} $1=="gzip" {g=$2} END {printf "gzip is %.0f%% of the csv%s", 100*g/c, ORS}' lab/layout/sizes.csv

The round trip is identical, and the compressed file is about a quarter of the CSV's 66772 bytes: repetitive columns such as dates, product codes and small quantities compress well. Columnar formats go further by storing each column together, so similar values sit side by side.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.