Logging: what to collect, and what you will regret not collecting

Why this matters

Every investigation is limited by what was recorded before anyone knew there was a problem. You cannot go back and collect last month's process launches. This is the one area of security operations where decisions made quietly, on a budget call, determine whether a future incident is answerable at all.

CS0-004 puts logging in the architecture objective for that reason: it is infrastructure, not housekeeping. The exam expects you to know which sources answer which questions, and to recognise a scenario where the honest answer is "that data does not exist".

The lesson

Log sources by tier: network, host, application, cloud, identity

Think in tiers, because each answers a different question.

Network — firewall, proxy, DNS, flow records, IDS. Answers what talked to what. Broad coverage, shallow detail, and increasingly limited by encryption.

Host — process creation, authentication, service and scheduled task changes, file and registry activity, EDR telemetry. Answers what actually ran. Deep detail, and the only place to see activity that never crosses the network.

Application — web server logs, database audit trails, line-of-business application logs. Answers what was done to the data. Rich and specific, but uneven: every application logs differently, and many log almost nothing useful.

Cloud — control-plane audit logs, storage access logs, flow logs. Answers what was changed in the infrastructure, which is where cloud intrusions actually happen. An attacker who steals a cloud credential may never touch a host.

Identity — authentication, token issuance, MFA, consent, directory changes. Answers who was granted what. Covered in detail in the previous lesson.

The tiers are complementary, not substitutable. A proxy log tells you a host requested a domain; only host telemetry tells you which process did. An investigation with only one tier keeps arriving at questions it cannot answer.

For the exam, a useful ranking by value per gigabyte:

1. Process creation with command line — the single most useful host field.
2. DNS queries — the closest thing to a complete index of intent.
3. Proxy or web gateway logs with user attribution.
4. Authentication and identity events.
5. Cloud control-plane audit logs.

Time synchronisation and why skew ruins a timeline

Every log line is a claim about when, and an investigation is a claim about order. If clocks disagree, the order is wrong, and a wrong order produces wrong causation.

The practical requirements:

• Synchronise everything to the same time source, and make that source authoritative and monitored.
• Store in UTC, and render in local time only for humans. Timelines that mix local times across regions are how "the attacker was in the system before the phishing email arrived" gets concluded.
• Watch for daylight saving transitions, which can duplicate or skip an hour in local-time logs.
• Record and check skew. A device drifting by minutes is common and is enough to invert the order of two related events.
• Know which timestamp you are reading. Event time, ingest time and index time differ, sometimes by a lot if a collector was backed up. Correlating on ingest time when you meant event time produces confident nonsense.

In evidence terms this matters even more: a timeline that cannot be defended because the clocks were wrong is a timeline that cannot be used.

Normalisation, parsing and field mapping

Raw logs arrive in dozens of formats. Parsing extracts fields from a line; normalisation maps those fields to a common schema so that a single query can span sources.

Without it, "show me everything involving 10.4.2.19" becomes twelve different queries against twelve field names for the same concept — src_ip, source.ip, ClientIP, c-ip, and so on.

What goes wrong, and what to watch for:

• A parser breaks silently after a vendor update. The logs still arrive and still index; the fields are now empty. Every detection depending on those fields quietly stops firing, and nothing alerts, because absence of alerts is the normal state.
• Fields are mapped to the wrong concept. A source and destination transposed in a parser inverts the direction of every conversation.
• Truncation. Long command lines are exactly the interesting ones, and are exactly what gets cut at a field limit.
• Timezone assumptions in the parser reintroduce the skew problem.

The defensive habit: monitor volume and field population per source, not just that data is arriving. A source whose event count is normal but whose key field is suddenly 90% empty is a broken parser, and you want to know within the hour rather than during an incident.

Retention, cost and the 90-day conversation

Retention is where security requirements meet a budget, and the conversation is worth understanding because you will be in it.

The tension is simple. Storage costs money; intrusions are frequently discovered long after they begin. Published dwell times vary, but a meaningful share of intrusions are found more than three months after the initial compromise — often because a third party told the victim. If you keep 30 days of logs, those investigations begin with the evidence already deleted.

How mature teams resolve it:

• Tier the retention. Keep high-value, low-volume sources for a long time (identity, DNS, proxy, cloud audit) and high-volume, low-value sources briefly. This is far cheaper than one uniform policy and answers more questions.
• Hot and cold storage. Recent data searchable instantly; older data archived cheaply and restorable when needed. Slow access to old logs beats no access.
• Summarise before discarding. Aggregates — connection counts per host per day — survive at a fraction of the size and still support "when did this start?".
• Know your regulatory floors. Some frameworks mandate minimums, and those are floors rather than targets.

A useful way to frame it to a budget holder: retention is not a storage decision, it is a decision about the maximum age of an intrusion you are able to investigate.

Coverage gaps and how to find them before an incident does

The dangerous property of a logging gap is that it is invisible from inside the tool. A SIEM shows what it has. It cannot show what it never received.

Ways gaps appear:

• A host is built without the agent or the forwarder.
• An agent crashes, or is stopped by an attacker, and nobody notices.
• A network change routes traffic around a sensor.
• A licence limit silently drops events above a volume threshold.
• A new cloud subscription is created outside the logging configuration.
• A log source is switched off during troubleshooting and not switched back.

How to find them deliberately:

• Compare against an independent inventory. The tool's own list is circular; a separate asset source is the only way to see absence.
• Alert on log-source silence. Every source should have an expected heartbeat, and its absence should page someone. This is one of the highest value detections in any SOC and one of the least implemented.
• Track per-source volume trends. A 40% drop overnight is a finding.
• Map coverage against a technique framework to find the categories you cannot detect at all — covered in the attack frameworks lesson.
• Test with atomic activity. Run a known benign technique in the lab you built earlier and confirm it appears. If it does not, you have found the gap the cheap way.

The last one is the theme of this whole course: a detection nobody has ever seen fire is a hypothesis, not a control. Silence is the same shape whether you are safe or blind, and only a test distinguishes them.

Topics this lesson owns

• [x] Log sources by tier: network, host, application, cloud, identity
• [x] Time synchronisation and why skew ruins a timeline
• [x] Normalisation, parsing and field mapping
• [x] Retention, cost and the 90-day conversation
• [x] Coverage gaps and how to find them before an incident does