Centralise logs and find one event across three hosts

applied · 50 min · Objective 3.2

Task

Send logs from all three lab VMs to one collector, then trace a single event across them. Centralised logging is a control you appreciate only when you need to correlate, and the correlation is the exercise.

Steps

  1. On Host B, enable the syslog receiver in /etc/rsyslog.conf by uncommenting the imudp module and input(type="imudp" port="514"). Restart rsyslog and confirm it is listening.
  2. On Host A and the router, add a forwarding rule — *.* @192.168.20.10:514 for UDP — and restart. The single @ is UDP; @@ is TCP, and the difference is whether a lost log line matters to you.
  3. Generate a distinctive event on each client: logger -p local0.warning "canary-$(hostname)-$(date +%s)".
  4. On the collector, confirm all three arrived and note what distinguishes them: the hostname field, the facility and the severity. Those three fields are what make a central log searchable rather than merely large.
  5. Now do the correlation exercise. From Host A, ssh to the router with a wrong password twice, then correctly. Find every log line that event produced, on both hosts, and put them in time order.

Verify

sudo ss -ulnp | grep 514
logger -p local0.warning "canary-test-event"
sudo grep -h "canary" /var/log/syslog | tail -5
sudo grep -hE "sshd.*(Failed|Accepted)" /var/log/syslog | tail -6

The collector must be listening on UDP 514, and the canary lines from all three hosts must appear in one file with three different hostnames. The ssh sequence must show two failures followed by an acceptance, in order.

Notes

The correlation step exposes the problem central logging exists to solve and the one it creates. It solves "which host did this happen on" — you no longer need to guess before you look. It creates "whose clock do I trust", because three hosts with drifting clocks produce a timeline that is simply wrong.

That is why NTP and logging belong in the same lesson. Set up chrony on all three VMs pointing at the same source and the timestamps become comparable. Without it, correlation across hosts is guesswork dressed as evidence.

The severity levels are examinable and worth reading off a real log: emergency, alert, critical, error, warning, notice, informational, debug. The practical advice is to alert on error and above, review warnings daily, and keep the rest for when you need them — because a log that alerts on everything trains people to ignore it.