Record lineage and run an impact analysis

applied · 50 min · Objective 5.1

Task

Write down the lineage of the one-page report as a list of edges, then write the script that answers the two questions lineage exists for: what breaks if this changes, and where did this number come from?

Steps

  1. Save lab/lineage/edges.csv with header input,output and these edges: customers_raw.csv to customers.csv; orders.csv to valid_orders; valid_orders to joined; customers.csv to joined; products.csv to joined; joined to by_region; by_region to report.html; joined to by_category; by_category to category-chart.svg.
  2. Write lab/lineage/impact.py NAME, printing everything downstream of NAME, directly or indirectly, as N: followed by the names sorted and separated by a comma and a space. With --up before the name, it prints everything upstream instead.
  3. Save lab/lineage/owners.csv with header node,owner, naming an owner for each of the four source files.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

python3 lab/lineage/impact.py customers_raw.csv
python3 lab/lineage/impact.py by_region
python3 lab/lineage/impact.py --up report.html
awk -F, 'NR>1 && $2!=""' lab/lineage/owners.csv | wc -l

A change to customers_raw.csv reaches 6 things: by_category, by_region, category-chart.svg, customers.csv, joined and report.html -- including a chart that never mentions customers. by_region reaches only report.html. The report has 7 upstream nodes, the four source files among them, and each source has an owner. The same edge list answers both directions.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.