Partition a table by month and read only what a query needs

short · 40 min · Objective 1.3

Task

Split the orders into one file per month, the way a data lake partitions a large table, then answer a question about March by reading a single partition. Count how much less data the partitioned read touches.

Steps

  1. Read data/orders.csv and write each order into lab/parts/month=YYYY-MM/orders.csv for its month, with the header in every file.
  2. Count the partition folders and the total data rows across them.
  3. Answer 'how many orders in March 2025?' by reading only lab/parts/month=2025-03/orders.csv, and record it in lab/parts/march.txt as march: N.
  4. Record in lab/parts/env.txt the output of python3 --version, the one-line version record a pinned environment starts from.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

ls -d lab/parts/month=* | wc -l
cat lab/parts/month=*/orders.csv | grep -vc '^order_id'
grep -E '^march: ' lab/parts/march.txt
grep -c 'Python 3' lab/parts/env.txt

Twelve partitions holding all 2000 orders between them, and a March count of 176 read from one file of twelve. A query engine that understands the month= folders does this skipping for you, which is called partition pruning.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.