Versioning and data lineage

Why this matters

Two questions come up about every important number sooner or later: where did this come from? and why is it different from last time? Lineage answers the first by tracing a figure back through every transformation to its source. Versioning answers the second by recording what changed, and when. Without them, both questions become investigations; with them, they become lookups.

The exam's data management objective names documentation, versioning and data lineage together. This lesson covers versioning data and code, lineage from source to report, impact analysis, master data and the single source of truth, and the roles that own data.

The lesson

Versioning data sets and the code that produced them

Versioning keeps a history of changes so earlier states can be identified, compared and recovered.

For code, such as queries, cleaning scripts and pipeline definitions, version control systems such as Git are the standard. Every change is a commit with a message, an author and a timestamp, and any past version can be compared with the current one.

For data, versioning is harder, because data sets are large and change continuously. Common approaches include:

• Snapshots: saving a copy of a data set at a point in time, such as each month end, so reports can be reproduced as they were.
• Dated file names or partitions: orders_2025-09-30, so the version is part of the name.
• Slowly changing dimensions in a warehouse, which keep history within a table: when a customer moves region, a new row records the new region with its valid-from date, rather than overwriting the old one. This lets reports show a customer's region as it was at the time of each order.
• Data versioning tools that track changes to data sets the way Git tracks code.

The key principle is that a published result should be reproducible: given the version of the code and the version of the data used, it should be possible to produce the same figure again. Recording both, together, with each important output makes that possible.

Data lineage: from source to report

Data lineage is the record of a data element's journey: where it originated, every system it passed through, every transformation applied to it, and every report or model that uses it.

Lineage can be described at different levels:

• Table level: this dashboard reads this warehouse table, which is loaded from these two source tables.
• Column level: the "net revenue" column is calculated from quantity, unit price and discount, which come from these source fields.

Lineage serves several purposes. It supports trust, because users can see that a figure comes from an approved source. It speeds troubleshooting, because when a number is wrong, the lineage shows every step where it might have gone wrong. It supports compliance, because regulations may require showing where personal data flows and how reported figures are derived.

Lineage can be documented by hand, in diagrams or catalogues, or captured automatically by tools that read pipeline and query definitions. Automatic lineage stays accurate as pipelines change; hand-drawn diagrams tend to drift.

The simplest lineage an analyst can keep is the one this course has encouraged throughout: scripts that read from named raw sources and write to named outputs, kept in version control, which are a readable map of where every number came from.

Impact analysis: what breaks if this changes

Lineage also works forwards. Impact analysis asks: if this source, table or field changes, what depends on it?

Before renaming a column, changing a definition, retiring a table or altering a pipeline, impact analysis identifies every downstream report, dashboard, model and extract that would be affected. Without it, changes break things silently: a renamed field that one report uses, discovered weeks later when someone notices its numbers froze.

With lineage available, impact analysis is a query: list everything downstream of this column. The affected owners can then be told in advance, the dependent items updated in the same change, and the change tested against them. This is one of the strongest practical arguments for maintaining lineage at all.

Master data and a single source of truth

Master data is the core reference data about the key entities an organisation works with: customers, products, suppliers, employees, locations. It is used across many systems, and problems arise when each system keeps its own version: a customer with one address in the billing system and another in the delivery system, or a product with different names in sales and inventory.

Master data management (MDM) establishes a single, authoritative version of each entity, often called the golden record, reconciling duplicates and conflicts across systems and distributing the agreed version back to them. The duplicate customers in the course's raw customer file are exactly the problem MDM exists to prevent.

The broader idea is a single source of truth: for each important piece of information, one agreed, authoritative source that everyone uses. When two reports disagree, the one built on the source of truth is the reference. In practice, organisations often maintain several governed sources, each authoritative for its own domain, such as finance for revenue and the customer system for customer details, and the governance task is making sure everyone knows which is which.

Data stewards, owners and custodians

Governance needs people with clear responsibilities, and three roles recur.

A data owner is accountable for a data domain, such as customer data or financial data, usually a senior business person. The owner decides who may access the data, approves definitions and classifications, and is accountable for its quality and appropriate use.

A data steward manages the data day to day on the owner's behalf: maintaining definitions and documentation, monitoring quality, resolving issues, and acting as the expert people ask about the data. Stewards are often in the business rather than in IT.

A data custodian looks after the technical environment where data is stored and processed: running the databases, applying security controls, taking backups and implementing access decisions. Custodians are usually in IT or a data platform team.

Analysts are data users: responsible for using data within the rules, raising quality issues with stewards, and asking owners before using data for a new purpose. Knowing who holds each role for the data you use is what makes the earlier advice, ask the data owner, possible.