Metrics, logs and traces: the three signals

Why this matters

Observability is what makes every other operational objective possible. You cannot right-size without measurement, cannot autoscale on a signal you do not collect, cannot troubleshoot a distributed system without traces, and cannot prove compliance without logs.

It is also where the cloud differs most sharply from traditional monitoring. Instances are ephemeral, so anything stored on them disappears with them. There are far more moving parts. And the provider shows you some things and not others — the gap between what the platform can see and what only the guest can see is the source of a lot of confusion, and of at least one common exam question.

The lesson

What each signal is good at, and the question you cannot answer with metrics alone

Three signals, three jobs.

Metrics. Numbers over time — CPU percentage, request count, queue depth, error rate. Cheap to store, cheap to query, ideal for dashboards, thresholds and trends. Their limitation is that they are aggregates: a metric tells you the error rate rose to 4%, and cannot tell you which requests failed or why. Metrics answer "is something wrong, and roughly where?"

Logs. Discrete events with detail and context — a request, an error, an authentication, a configuration change. They answer "what exactly happened?" They are far more expensive to store and search, and they are the signal that compliance and forensics depend on.

Traces. The path of a single request across every service it touched, with timing at each hop. They answer "where did the time go, and which component caused this?" — a question that is nearly unanswerable with metrics and logs alone once a request crosses more than two or three services.

The question in the heading: why is this request slow? Metrics show elevated latency somewhere. Logs from each service show requests that each look reasonable. Only a trace shows that one call took 800ms of the 900ms total, in a service nobody suspected. That is the specific gap traces exist to close.

A fourth signal, events, is worth naming: platform-level records of what changed — a resource created, a permission modified, a configuration updated. These are the audit trail, and they answer the first question of every incident, "what changed?"

Provider metrics against in-guest metrics, and the memory figure that needs an agent

This distinction is examined specifically and catches people out.

The provider's monitoring sees the instance from outside, at the hypervisor level: CPU utilisation, network in and out, disk operations, status checks. It is available with no configuration and no agent.

It cannot see inside the guest. Specifically:

• Memory utilisation. The hypervisor knows how much memory is allocated to the instance, not how much the operating system is using. Memory requires an agent, and this is the single most commonly missed metric in cloud monitoring — including during the right-sizing exercise in objective 1.7, where its absence causes real mistakes.
• Disk space used within a filesystem, as opposed to volume-level I/O. Also needs an agent, and a full filesystem is one of the most common causes of an instance that is running and not working.
• Process-level detail — which process is consuming what.
• Application logs and metrics, which are inside by definition.

So: install the agent. The provider's agent collects memory, disk space, process information and log files, and ships them to the same place as the platform metrics. Bake it into the image (objective 2.4) so every instance has it without anyone remembering.

Note the complementary case: steal time (objective 1.2) is visible inside the guest and describes something outside it. So the two views are not simply "outside is less detailed" — they see different things, and diagnosis often needs both.

Centralised logging, retention, and the cost of keeping everything forever

Logs must leave the instance, immediately and continuously.

The reasons are absolute in a cloud: instances are terminated by scaling, rolling updates and failures, and when one goes it takes its local logs with it. The most important logs in an organisation's history are frequently the last ones written by a machine that then died — which is precisely the case local storage cannot serve.

Centralisation also gives you searching across the fleet, correlation between services, retention independent of infrastructure, and access control so that people who should read logs can, and people who should not cannot. And it makes logs tamper-resistant: an attacker who compromises an instance can delete its local logs but not the copies already shipped elsewhere. That is a security control in its own right, and objective 6.3 depends on it.

The counterweight is cost, and log ingestion and retention is a genuinely large line item in mature estates. Manage it deliberately:

• Tier retention by value. Security and audit logs are kept for years because regulation requires it. Debug logs are kept for days. Applying one retention to everything means either paying too much or losing what you needed.
• Move older logs to cheaper storage. Recent logs in an indexed, searchable system; older logs in object storage where they are cheap and still queryable, if more slowly.
• Sample high-volume, low-value logs — but never sample audit or security logs.
• Control verbosity. A debug log level left on in production is a very common and very expensive mistake, and it is exactly what the cost anomaly detection in objective 1.8 catches.
• Structure the logs. Structured (JSON) logs are queryable without fragile text parsing, and the cost of retrofitting structure later is high.

One thing to standardise early: a correlation ID carried through every service and written into every log line. Without it, correlating a user's journey across services is guesswork. With it, it is a single query — and it is also what ties logs to traces.

Distributed tracing across services, and the request you can finally follow end to end

Distributed tracing assigns each incoming request an identifier, propagates it through every downstream call, and records a span for each operation with its timing, parent and metadata. Assembled, the spans form a trace: a timeline of the whole request.

What it gives you that nothing else does:

• Where the time actually went, hop by hop.
• The dependency graph as it really is, derived from observed calls rather than from documentation — which is the same problem as the dependency mapping in objective 2.1, solved continuously.
• The failing hop, identified directly rather than inferred.
• The N+1 pattern — one request making hundreds of small downstream calls — which is invisible in metrics and obvious in a trace. This is the chatty application from objective 2.1, caught in the act.

Two practicalities:

• Propagation must be complete. A service that does not forward the trace context breaks the chain, and everything beyond it is invisible. Adopting a standard propagation format across all services is what makes this work.
• Sampling is normal. Tracing every request is expensive, so a percentage is captured — ideally with rules that always keep errors and slow requests, since those are the ones worth having.

Tracing matters in proportion to how distributed a system is. For a monolith it adds little; for microservices, serverless and event-driven designs (objective 5.6) it is close to essential, because no single log tells the story.

Baselines, so 'high CPU' means something measured rather than something felt

The final idea, and it connects observability back to every other domain.

A baseline is a recorded picture of normal: normal CPU across the day and the week, normal request rate, normal latency distribution, normal error rate, normal costs. Without it, every judgement is a guess and every threshold is arbitrary.

What baselines make possible:

• Alerts that mean something (next lesson). "CPU above 80%" is arbitrary; "CPU above the 99th percentile for this hour of the week" is informed.
• Capacity planning, because trends are visible.
• Change verification. Comparing after a deployment against before is how you know whether it helped — and it is the comparison the migration in objective 2.3 needed.
• Anomaly detection, which is baselining performed automatically.

Practical points: record percentiles rather than averages, because a mean latency hides the tail that users actually experience, and p50/p95/p99 tell three different and useful stories. Re-baseline after significant change, or you are comparing against a system that no longer exists. And baseline the business metrics too — orders per hour, sign-ins per minute — because those detect problems that infrastructure metrics miss entirely. A deployment that breaks checkout while every server looks perfectly healthy is caught by the business metric and by nothing else.

What to take into the exam

• Metrics = is something wrong. Logs = what exactly happened. Traces = where did the time go. Platform events answer "what changed?"
• Memory and filesystem usage need an in-guest agent. The hypervisor cannot see inside; steal time is the reverse case.
• Ship logs off the instance immediately — ephemeral instances take local logs with them, and centralised logs resist tampering.
• Tier log retention by value; never sample audit or security logs; a debug level left on in production is a classic cost incident.
• Correlation IDs are what make cross-service investigation possible.
• Tracing needs complete context propagation; one service that does not forward it blinds everything downstream.
• Baseline with percentiles, re-baseline after change, and baseline business metrics as well as infrastructure ones.