SNMP, flow data and logging

Why this matters

The previous lesson covered the instruments. This one covers what you do with what they produce: establish what normal looks like, alert when reality departs from it, and collect logs somewhere they can be correlated.

The single most useful idea here is the baseline. Almost every alerting mistake — thresholds that never fire, thresholds that fire constantly — comes from not having one.

The lesson

Baseline metrics

A baseline is a record of normal performance, gathered over enough time to include the normal variation. Without one, "the CPU is at 60%" is not information: it might be this device's idle state or twice its usual peak.

A useful baseline covers:

• Interface utilisation and error counters
• CPU and memory on network devices
• Latency and jitter across key paths
• Throughput for important services
• Typical traffic mix, from flow data

And it has to span a representative period — at minimum a full week, so that weekday and weekend differ, and ideally a month, so that period-end peaks are included. A baseline taken during a quiet week makes every busy Monday an incident.

Baselines are what turn monitoring into two useful capabilities:

Capacity planning. A link at 40% growing 3% a month has about eighteen months left. That is a budget conversation with a date on it rather than an emergency.

Anomaly detection. Departure from the baseline is the signal — a server that normally sends 2 GB a day sending 40 GB is worth investigating even though no threshold was crossed.

Re-baseline after any significant change, or the baseline describes a network that no longer exists.

Anomaly alerting and notification

Alerting is the mechanism that tells a human something needs attention. Getting it right is mostly about restraint.

Two approaches, and good systems use both:

Threshold alerting fires when a metric crosses a fixed value — disk above 90%, interface above 80% for ten minutes. Simple and predictable; it needs sensible values, which come from the baseline.

Anomaly alerting fires on departure from the learned normal pattern, so it can catch what no threshold would: traffic at 3 a.m. on a link that is always idle then, even though the absolute number is low.

The failure that matters most is alert fatigue. A system that sends hundreds of notifications trains its recipients to ignore them, and then the one that mattered is ignored too. The disciplines that prevent it:

• Alert on symptoms users feel, not on every metric you can collect.
• Use a duration — "above 80% for ten minutes" rather than an instantaneous spike.
• Suppress dependants. If a core switch fails, alert on the switch, not on the two hundred devices behind it. Good platforms understand topology well enough to do this.
• Route by severity. Page for outages; email for trends; ticket for maintenance.

Every alert should be actionable. If nobody does anything when it arrives, it should not exist.

Log aggregation and syslog collectors

Every device keeps logs locally, and local logs have three problems: they are small and roll over, they are invisible until someone opens that device, and an intruder can delete them.

Log aggregation sends logs to a central collector. A syslog collector receives syslog messages — the standard on UDP 514 — from network devices, servers and appliances.

Centralising gives you:

• Correlation. One timeline across devices, which is the only way to reconstruct a sequence. This is why the NTP lesson matters here — correlation is impossible if clocks disagree.
• Retention beyond what a device can hold.
• Tamper resistance. An attacker who clears a device's logs cannot easily reach what was already shipped elsewhere.
• Search across everything at once.

Syslog messages carry a facility (what produced it) and a severity, where 0 is Emergency and 7 is Debug — lower is worse, which is the opposite of most people's intuition and is therefore examinable. Configuring a device to send severity 4 and below sends warnings and worse, and leaves out the noise.

Note that plain syslog over UDP is unauthenticated and unencrypted, and can be lost silently. Syslog over TLS or relayed over TCP addresses that where it matters.

SIEM

A SIEM (Security Information and Event Management) platform is log aggregation plus security analysis. It ingests logs from everywhere — firewalls, servers, endpoints, applications, identity systems — normalises them into a common format, and then correlates.

Correlation is the value. Individually, a failed login is nothing, a firewall deny is nothing, and a new administrative account is nothing. A SIEM notices that the same source produced forty failed logins, then one success, then created an account, within four minutes — and raises that as one incident.

SIEMs also provide long-term retention for compliance, dashboards, and investigation tooling. The costs are real: they are expensive, they need tuning to avoid drowning analysts in false positives, and they need someone to read them. An untuned SIEM is an expensive way to generate noise.

API integration

Modern monitoring is not a closed box. API (Application Programming Interface) integration lets platforms exchange data and actions with other systems — which is what makes automated response possible.

Typical uses: create a ticket automatically when an alert fires; post to a chat channel; pull device inventory from the CMDB so the monitoring system's list stays correct; trigger an automation playbook to collect diagnostics or restart a service; feed metrics into a capacity dashboard.

The exam framing is usually workflow: an alert that raises a ticket and notifies a channel without anyone retyping anything, so the response starts immediately and nothing is lost in a handover.

Port mirroring

In a switched network, a switch forwards a frame only to the port the destination is on. So a capture device plugged into a spare port sees almost nothing — broadcasts and its own traffic.

Port mirroring — SPAN on Cisco equipment — solves it by copying traffic from one or more source ports (or a whole VLAN) to a destination port where the analyser sits.

Two limits worth knowing, because both produce misleading captures:

• The destination port must have enough capacity. Mirroring two saturated 1 Gb ports to one 1 Gb port means the switch drops some of the copy, and your capture is silently incomplete — which can look exactly like packet loss on the network.
• Mirroring costs the switch some resources, and some platforms limit how many sessions can run at once.

The alternative is a network tap, a passive device inserted into the link itself. It cannot drop frames, does not load the switch, and typically keeps working if it loses power — but it must be physically installed in the path, which means an outage to fit it.

Mirroring can be ad hoc — configured for a specific investigation and removed afterwards, which is the usual case — or scheduled, running at set times to sample traffic regularly, for example capturing a window each night for compliance or baselining.

What you actually monitor

Four categories, and a complete picture needs all four:

Availability monitoring asks whether it is up. Ping, port checks, service checks. Binary, cheap, and the first thing anyone notices when it is wrong.

Performance monitoring asks whether it is fast enough — latency, throughput, response time, utilisation. Something can be entirely available and unusable, and this is the category that sees it.

Traffic analysis asks what is actually flowing, using flow data and capture. It answers "why is the link full" and "what is this host doing".

Configuration monitoring watches for changes to device configuration. It detects the drift described in the IaC lesson, catches unauthorised changes, and gives you a diff when something breaks — because the honest answer to "what changed?" is usually "something", and a configuration monitor is what turns that into a specific line.