Monitoring and logging

Why this matters

The previous lesson's method starts with "identify the problem", and everything in it depends on having information. Monitoring is how you have it before a user tells you, and logging is how you find out what happened when you were not watching.

The failure this prevents is specific and common: a disk that had been filling for six weeks, on a machine nobody was watching, taking down an application at 04:00 on a Sunday. Nothing about that was sudden. It was invisible.

Domain 5 is 22% of the exam and this objective is where the vocabulary lives — SLA, SLO, SLI, SNMP, MIB, traps, webhooks. It is unusually terminology-heavy, so the definitions matter as much as the tools.

The lesson

Service monitoring

Service monitoring is the continuous checking of whether something is working, and it comes in layers that answer different questions.

Is the process alive? The cheapest check, and the least informative:

systemctl is-active nginx
systemctl list-units --state=failed
ss -tlnp | grep :443             # is it actually LISTENING?

A process can be running and useless — deadlocked, out of file descriptors, unable to reach its database. "The service is running" is not the same as "the service is working", and monitoring that only checks the former is how an outage goes unnoticed for an hour.

Does it respond correctly? A functional check, which is what you actually want:

curl -sf -o /dev/null -w '%{http_code} %{time_total}\n' https://app.example.com/healthz

Is it giving the right answer, quickly enough? Latency and error rate, which is where the objectives below come in.

The layer above is synthetic monitoring — a scripted transaction from outside your network, doing what a user does. It is the only kind that catches "the site works from inside the datacentre and DNS is broken for everyone else".

SLA, SLO and SLI

Three terms that are constantly confused and are on the objectives explicitly. The clean way to hold them is measurement, target, contract:

Service-level indicator (SLI) — the measurement. A number you actually collect. "99.95% of requests returned a non-5xx status in the last 30 days." "95th percentile latency is 240 ms."

Service-level objective (SLO) — the internal target for that indicator. "99.9% availability over 30 days." "95th percentile under 300 ms." This is what you engineer toward and what your alerts are derived from.

Service-level agreement (SLA) — the contract with the customer, including the consequence of missing it: service credits, penalties, termination rights.

The SLA is always looser than the SLO, deliberately. If you promise customers 99.9% and target 99.9% internally, you breach the contract the first time you miss. Target 99.95% internally and promise 99.9%, and you have room to fail before it costs money.

That gap has a name worth knowing: the error budget. An SLO of 99.9% over 30 days permits about 43 minutes of downtime. That is not a failure allowance to be minimised to zero — it is a budget to be spent on deploying changes. Unspent budget means you are shipping too slowly; exhausted budget means you stop deploying and stabilise.

Two practical points the exam likes. Availability percentages translate into surprisingly little time: 99.9% is 43 minutes a month, 99.99% is 4.3 minutes — which is less than one person can notice and respond to, so it implies automatic failover rather than an on-call rota. And an SLI must measure what the user experiences: CPU utilisation is not an SLI, because no customer has ever cared about it.

Configurations and thresholds

A threshold is the value at which a check changes state, and choosing them badly is the most common way a monitoring system becomes useless.

# the shape, in any tool
checks:
  - name: disk-root
    metric: disk_used_percent
    path: /
    warning: 80
    critical: 90
    for: 10m                  # must persist -- do not alert on a spike

Two levels — warning and critical — is the standard configuration: warning is "look at this during working hours", critical is "wake someone". Collapsing them into one loses that distinction and turns everything into an emergency.

The mistakes to avoid, all of which produce the same end state:

Thresholds too tight produce constant noise, and a team that ignores alerts. This is by far the most damaging failure, because it disables the system without disabling it — the alerts still fire, nobody reads them, and a real one arrives among two hundred false ones.

Thresholds too loose mean the alert arrives after the user's complaint.

No duration requirement means a momentary CPU spike pages someone at 3 a.m. Requiring a condition to persist for five or ten minutes removes most false alarms at almost no cost in detection time.

Static thresholds on variable workloads — 70% CPU is normal at midday and alarming at midnight. Either set them per-service or use a baseline.

Two thresholds that deserve setting on every machine, because their failure modes are silent: disk space at 80/90, and inodes, which fill independently of space and produce "No space left on device" on a disk showing 40% free. And certificate expiry, at 30 and 7 days, which prevents an entirely predictable outage.

Data acquisition methods

How the monitoring system actually gets its numbers. The first distinction is direction:

Pull (polling) — the monitoring server asks each target on a schedule. Prometheus works this way. The server controls the rate, targets need no knowledge of where to send data, and a target that stops answering is itself the signal.

Push — the target sends data to a collector. Better for short-lived jobs that may not exist when a poll arrives, and for hosts behind NAT that cannot be reached inbound. The cost is that silence is ambiguous: nothing arriving might mean healthy-and-idle or dead.

Agent versus agentless

The same agent/agentless trade-off as configuration management, applied to data collection:

Agents (node_exporter, Telegraf, the Datadog or Zabbix agent) give far richer data and are the norm for servers you own. Agentless — SNMP, SSH, or an HTTP endpoint — is what you use for appliances, switches, printers and anything you cannot install software on.

SNMP, MIBs and traps

Simple Network Management Protocol is the old agentless standard, and it is on the objectives because network hardware still speaks nothing else.

It is a request/response protocol over UDP 161. Each value has a numeric address — an OID — and the management information base is the dictionary that maps those numbers to names and meanings. 1.3.6.1.2.1.1.3.0 means nothing until a MIB tells you it is sysUpTime; loading the vendor's MIB is what makes a switch's proprietary counters legible.

snmpwalk -v2c -c public switch01 1.3.6.1.2.1.1     # walk the system tree
snmpget -v2c -c public switch01 sysUpTime.0
snmpget -v3 -l authPriv -u monitor -a SHA -A "$AUTHPASS" \
        -x AES -X "$PRIVPASS" switch01 sysUpTime.0

Traps invert the direction. Instead of the manager polling, the device sends an unsolicited message to the manager (UDP 162) when something happens — an interface goes down, a power supply fails. Traps are the push half of SNMP, and they matter because polling every five minutes cannot catch a link that flapped for ten seconds.

An inform is a trap that expects an acknowledgement, and is therefore retransmitted if lost — worth knowing because a plain trap is fire and forget over UDP, so a dropped one is simply never seen.

The security point the exam asks about: SNMP v1 and v2c authenticate with a community string sent in clear text, and the default is public. That is a read-only password on the wire for anyone watching, and community strings routinely survive on production equipment for years. v3 adds real authentication and encryption and is the only version to use on anything that matters.

Webhooks

A webhook is the modern push mechanism: an HTTP POST to a URL you supply, carrying a JSON payload, sent when an event occurs.

curl -X POST "$ALERT_WEBHOOK_URL" \
  -H 'Content-Type: application/json' \
  -d '{"text":"disk /var at 92% on web01","severity":"warning"}'

Webhooks are how monitoring reaches the places people actually look — chat, paging services, ticketing — and how systems chain together: an alert fires a webhook, which triggers a pipeline, which runs a remediation playbook.

Three things to know. The URL is a credential — anyone holding it can post as you, so it belongs in a secret store, not in a repository. Incoming webhooks should be verified, usually by an HMAC signature header, or anyone can forge events into your system. And delivery is not guaranteed: the sender may retry, which means your handler must be idempotent, or it may not, which means a missed event is missed.

Health checks

A health check is an endpoint or command whose whole job is to answer "are you working?". They are what orchestrators, load balancers and monitoring all consult.

# in a container
healthcheck:
  test: ["CMD-SHELL", "curl -sf http://localhost:8080/healthz || exit 1"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 60s

# systemd's version of the same idea
[Service]
Restart=on-failure
RestartSec=5s
WatchdogSec=30s

The distinction worth carrying: a liveness check asks "should I restart this?" and a readiness check asks "should I send it traffic?". They are different questions — an application still loading its cache is alive but not ready, and restarting it is exactly the wrong response.

Two design rules. A health check should test the service's own function, not its dependencies. If your check fails because the database is down, then a database blip restarts every application instance simultaneously and turns a small problem into a large one. And start_period matters: without it, a slow-starting application is killed by its own health check before it ever becomes healthy, in a loop.

Logging and log aggregation

Logging on a modern Linux system means the journal, plus the traditional files:

journalctl -u nginx                     # one unit
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b                    # errors this boot
journalctl -f                           # follow, like tail -f
journalctl -k                           # kernel messages
journalctl --disk-usage
journalctl --vacuum-time=30d

The journal is binary, indexed and structured, which is why filtering by unit, priority, boot and time is instant. rsyslog still writes plain text to /var/log/ for tools that expect it and for forwarding.

Priorities run 0 (emerg) to 7 (debug), and -p err shows 3 and above.

Log aggregation ships logs from every host to a central system. The reasons are not administrative tidiness:

• A compromised host's local logs cannot be trusted — an attacker's first act is to edit them. Logs already shipped elsewhere are evidence.
• A host that is down cannot be logged into, and its logs are exactly what you need.
• Correlation across hosts is impossible locally. A request touching a load balancer, three services and a database leaves five logs on five machines.
• Retention beyond what a local disk holds.

# rsyslog: forward everything, over TCP
*.* @@logserver.example.com:514

# systemd-journal-remote, or an agent
journalctl -o json | promtail ...

The stacks are ELK/OpenSearch (Elasticsearch, Logstash, Kibana), Loki with Grafana, or a hosted service. What matters more than the choice: use a structured format, ideally JSON, because parsing free text with regexes breaks every time someone changes a message. And synchronise clocks with NTP — correlating logs across hosts is impossible if their timestamps disagree, which is the same point the installation lesson made about doing time first.

Do not forget rotation. Log aggregation does not stop the local disk filling; logrotate and journalctl --vacuum-size still matter, and an unrotated log is one of the most common causes of a full /var.

Alerts, events and notifications

Three words used interchangeably in conversation and distinguished on the objectives:

Events are things that happened. A service restarted, a disk crossed 80%, a deployment completed, a login failed. Most are informational and most are never looked at individually — they are the raw material.

Alerts are events that meet a condition somebody decided is worth acting on. The threshold turns an event into an alert.

Notifications are the delivery of an alert to a human: email, chat, SMS, a page. One alert may produce several notifications, or none if it is only recorded.

event  ->  (threshold / rule)  ->  alert  ->  (routing)  ->  notification

The reason to keep them separate is that each stage is tuned differently. Collect events generously — storage is cheap and you cannot investigate what you did not record. Alert selectively. Notify very selectively.

Every alert must be actionable. The test is simple: what do I do when this fires? If the answer is "look at it and decide it is fine", it is not an alert; it is a dashboard. An on-call rota receiving twenty pages a night stops reading them, and then misses the one that mattered — the most common way a monitoring system fails while appearing to work.

Routing matters as much as thresholds: critical to a pager, warning to a chat channel, everything to the dashboard. And alerts need grouping and inhibition — a failed switch should page once about the switch, not sixty times about every host behind it.

On the exam

• "Running" is not "working". Prefer a functional check to a process check.
• SLI is the measurement, SLO is the internal target, SLA is the contract. The SLA is deliberately looser than the SLO; the gap is the error budget. 99.9% is about 43 minutes a month.
• Thresholds need a warning and a critical level and a duration, or you get noise. Watch inodes as well as disk space, and certificate expiry.
• Pull means silence is itself a signal; push suits short-lived jobs and hosts you cannot reach.
• Agent gives detail and needs installing; agentless suits appliances.
• SNMP: OIDs are numeric addresses, MIBs translate them, traps are unsolicited messages from device to manager on UDP 162. v1/v2c community strings are clear text — use v3. An inform is an acknowledged trap.
• Webhooks are HTTP POSTs on an event; the URL is a credential, and incoming ones should be signature-verified.
• Health checks: liveness asks "restart?", readiness asks "send traffic?". Do not fail your health check because a dependency is down.
• Log aggregation exists because a compromised host's logs cannot be trusted, a dead host cannot be logged into, and correlation needs one place. Synchronise clocks; keep rotating locally anyway.
• Events happen, alerts are events worth acting on, notifications deliver them to a person. An alert that is not actionable trains people to ignore alerts.