Retention and audits

Why this matters

Organisations are required both to keep data and to get rid of it. Financial records must be kept for years; personal data must not be kept longer than needed. Data under investigation must not be deleted, however old. And at any time an auditor may ask who accessed a data set, who changed a figure, and how the organisation knows. Analysts create copies, extracts and reports that fall under all of these rules.

The exam's compliance objective names retention and audits. This lesson covers retention periods and deletion, legal holds, audit trails, preparing for an audit, and disposal with proof that it happened.

The lesson

Retention periods and why data must also be deleted

A retention period is how long a type of record must be kept. It is set in a retention schedule, a policy that lists record types with their periods and the reason for each.

Periods come from several sources: laws and regulations, such as tax and accounting rules that require financial records to be kept for a number of years; industry rules; contracts; and business need. They vary widely between record types and jurisdictions, which is why the schedule, not memory, is the reference.

Retention cuts both ways. Keeping records too short breaks legal obligations and loses evidence. Keeping them too long creates risk: privacy laws such as GDPR require personal data to be kept no longer than necessary for its purpose, old data can be breached, and everything retained can be requested in legal proceedings. Storage is cheap, but the liability of unnecessary data is not.

For analysts, retention applies to copies as well as systems of record. An extract saved on a laptop, a data set in a personal folder, or a report attachment in an email is still data subject to the schedule, and copies are exactly what tends to be forgotten when the original is deleted on time.

Legal holds

A legal hold (or litigation hold) suspends normal deletion for data relevant to actual or anticipated legal proceedings, investigations or audits. Once a hold is in place, the relevant data must be preserved, even if its retention period has expired and even if it would otherwise be routinely deleted.

A legal hold overrides the retention schedule. Deleting held data, even through an automatic process, can have serious legal consequences, and is treated as destroying evidence.

Holds are issued by the legal team and specify what is covered, such as data about a particular customer, product or period. Everyone who holds relevant data, including analysts with extracts or reports, must be told and must preserve it. When the matter ends, the hold is released and normal retention resumes.

Automated deletion processes therefore need a way to exempt held data, and a record showing that they did.

Audit trails: who accessed and changed what

An audit trail (or audit log) records activity on data and systems: who accessed which data, when, and what they did, including viewing, creating, changing, exporting and deleting.

Useful audit trails record:

• the user or service account;
• the action and the object it affected;
• the time;
• for changes, the before and after values, or enough to reconstruct them;
• where relevant, the source, such as the machine or network location.

Audit trails serve several purposes: detecting inappropriate access, investigating incidents, proving compliance, and answering the question of why a figure changed. To be trusted, they must be protected from tampering, so that the people whose actions are logged cannot edit or delete the log, and they must themselves be retained for an appropriate period.

For analysts, the practical points are that access to sensitive data is often logged and reviewed, and that analytical work should itself leave a trail: version-controlled code, dated extracts and recorded validation.

Preparing for an audit, and evidence an auditor accepts

An audit is an independent examination of whether an organisation's data practices comply with laws, regulations, standards or its own policies. It may be internal or external, scheduled or triggered by an incident.

Auditors ask for evidence, not assurances. Evidence they typically accept includes:

• policies and procedures, such as the retention schedule and access policy;
• records that the procedures were followed, such as access reviews, approval records, validation records and deletion logs;
• system configuration, such as access control settings and encryption status;
• audit logs showing who did what;
• samples: auditors often select specific records or transactions and ask to see how each was handled.

Preparation is mostly continuous rather than last-minute: documenting processes, keeping records as work is done, and storing them where they can be found. Findings from an audit are recorded with agreed actions and deadlines, and following up on them is part of the next audit.

Disposal and proving that it happened

When data reaches the end of its retention period, and no legal hold applies, it should be disposed of securely and completely.

Secure disposal means the data cannot be recovered. For digital data, it includes deleting it from primary systems, from replicas and copies, from reports and extracts, and, according to policy, from backups, which are often the hardest part. For storage media being retired, disposal may require secure wiping or physical destruction. Anonymising data so it no longer relates to identifiable people is sometimes an alternative to deletion for data still useful in aggregate, provided the anonymisation is genuine.

Disposal should be recorded: what was disposed of, when, by what method, and by whom, with any certificate of destruction from a disposal provider. This record is the evidence an auditor will ask for, and it proves the organisation met its obligation to delete, which is as much a compliance requirement as the obligation to keep.