Build an alert that is actionable, not noise

short · 30 min · Objective 5.1

Task

Configure a threshold with a warning and a critical level and a duration requirement, prove it does not fire on a momentary spike, and prove it does fire on a sustained problem. Then centralise a log and confirm it survives local tampering.

Steps

  1. Write a disk-usage check for /mnt/mon with two thresholds -- warning at 80, critical at 90 -- and a requirement that the condition persist for a set number of samples before it alerts. Use the numeric operators, not string ones. Save it as ~/lab-mon/check-disk.sh, printing ALERT warning or ALERT critical when it alerts and ok otherwise. So the Verify can replay steps 2 and 3, give it a scenario argument: transient fills /mnt/mon to 85% for one sample and frees it, sustained fills it to 92% and holds; either way it then takes its samples and reports.
  2. Fill a loop filesystem to 85% briefly and then free it, and confirm the check does NOT alert on the transient -- the duration requirement absorbs it.
  3. Fill it to 92% and keep it there, and confirm the check DOES alert, at critical level.
  4. Check inodes as well as space with df -i, and demonstrate the alert firing for inode exhaustion on a filesystem that still shows free space.
  5. Forward logs to a second host (or a local collector) with rsyslog over TCP, writing the origin's messages to /var/log/from-origin.log on the collector as in the authentication lab (if $fromhost == 'origin' then { action(type="omfile" file="/var/log/from-origin.log") stop }). Generate an auth event and confirm it arrives.
  6. Play the attacker: truncate the local log as root and confirm the forwarded copy is unaffected. State what centralising buys and what it does not.
  7. Confirm local rotation still matters: show that forwarding does not stop the local disk filling, and that logrotate is still required.

Verify

cd ~/lab-mon
# transient must not alert, sustained must:
./check-disk.sh transient | grep -q '^ALERT' && echo "FALSE ALARM on spike -- fail" || echo "spike absorbed"
./check-disk.sh sustained | grep -q '^ALERT critical' && echo "sustained problem alerts"
df -i / | awk 'NR==2 {print "inode use:", $5}'
# forwarded log survives local truncation:
: > /var/log/secure                        # on the origin
grep -c . /var/log/from-origin.log         # on the collector: still non-zero

The spike being absorbed and the sustained condition alerting is the whole point: a threshold with no duration requirement pages someone at 3am for a momentary blip, and a team that is paged for blips stops reading pages.

Notes

The test of an alert is: what do I do when this fires? If the answer is "look and decide it's fine", it is a dashboard, not an alert. Every threshold here has a warning level for working hours and a critical level for waking someone, because collapsing them turns everything into an emergency.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.