Documentation: data dictionaries and metadata

Why this matters

Data without documentation is a puzzle that each new user has to solve again. What does this column mean? Is this amount before or after tax? Which of these three customer tables is the real one? Every analyst who guesses may guess differently, and their reports will disagree. Documentation is how an organisation's knowledge about its data outlives the people who hold it.

Data governance is the set of policies, roles and practices that make data trustworthy and properly used, and documentation is its foundation. This lesson covers data dictionaries and business glossaries, the kinds of metadata, data catalogues, documenting data as it is created, and keeping documentation true.

The lesson

Data dictionaries and business glossaries

A data dictionary describes the fields in a data set or database. For each column, it typically records:

• the name and a plain-language description;
• the data type and format, such as date in ISO format or decimal to two places;
• allowed values or ranges, such as the four regions, or quantity greater than zero;
• whether it can be empty, and what an empty value means;
• the source of the value and any transformation applied;
• keys and relationships to other tables.

A data dictionary answers "what is in this column?". A business glossary answers a different question: "what do we mean by this term?". It defines business concepts, such as "active customer", "net revenue" or "churn", in language agreed across the organisation, independent of any one table. The two link together: the glossary defines "net revenue", and the dictionary shows which columns in which tables implement it.

Both prevent the most common cause of conflicting reports: two people using the same word for different things.

Technical, business and operational metadata

Metadata is data about data. It is usually grouped into three kinds.

Technical metadata describes structure: table and column names, data types, keys, indexes, file formats, sizes and locations. Much of it can be collected automatically from databases and files.

Business metadata describes meaning: definitions, business rules, owners, sensitivity classifications, and how the data should and should not be used. It has to be written by people who understand the business.

Operational metadata describes processing: when data was loaded or last refreshed, how many rows arrived, which pipeline produced it, and any errors or quality results. It is generated by the systems that move and transform data.

Together, they answer the questions every analyst asks of an unfamiliar data set: what is it, what does it mean, who owns it, where did it come from, and how current is it?

Data catalogues and finding what already exists

In a large organisation, the problem is often not missing data but data nobody can find. Analysts rebuild data sets that already exist, or use an outdated copy because they did not know about the current one.

A data catalogue is an organised, searchable inventory of an organisation's data assets: databases, tables, files, reports and dashboards, each with its metadata. Users can search for "customer", see which data sets exist, read their descriptions, check their owners, quality and freshness, and see which are certified as authoritative.

Catalogues typically combine automatic collection of technical metadata with business metadata added by data owners and stewards, and many also show lineage and usage, such as which reports use a table. Their value depends on being maintained; a catalogue full of stale or missing descriptions is soon ignored.

The analyst's habit is to search before building: check the catalogue, or ask the data team, before creating a new data set, in case a governed one already answers the question.

Documenting a data set as it is created

Documentation written later is rarely written at all, and what is written is reconstructed from memory. The most reliable time to document a data set is when it is created, when the reasons for each decision are still fresh.

For a new data set, table or report, record at least:

• its purpose and intended users;
• its sources, including queries or file names and extraction dates;
• its grain: what one row represents;
• each field's definition, especially derived fields and their formulas;
• filters and exclusions, and why they were applied;
• known limitations and quality issues;
• its owner and how often it is refreshed.

Much of this can live with the work itself: comments in code, a README file beside a data set, descriptions stored in the database or BI tool. Documentation kept next to what it describes is more likely to be found and to be updated.

Keeping documentation true as the data changes

Documentation that is wrong is worse than none, because it is trusted. Data changes constantly: new fields are added, definitions are revised, sources are replaced. Each change can leave the documentation describing something that no longer exists.

Keeping documentation accurate needs process rather than good intentions:

• Make documentation part of the change: a change to a table or measure is not complete until its documentation is updated, ideally reviewed together.
• Assign owners responsible for each data set's documentation.
• Generate what can be generated: technical and operational metadata from the systems themselves, so it cannot fall out of date.
• Review periodically, prompted by the catalogue, to confirm descriptions are still correct.
• Date the documentation, so readers can judge how current it is.

Version control helps here too: documentation stored alongside code in the same repository changes in the same commits, and its history shows when each definition changed.