Build an alert that is actionable, not noise

short · 30 min · Objective 5.1

Task

Configure a threshold with a warning and a critical level and a duration requirement, prove it does not fire on a momentary spike, and prove it does fire on a sustained problem. Then centralise a log and confirm it survives local tampering.

Steps

  1. Write a disk-usage check with two thresholds -- warning at 80, critical at 90 -- and a requirement that the condition persist for a set number of samples before it alerts. Use the numeric operators, not string ones.
  2. Fill a loop filesystem to 85% briefly and then free it, and confirm the check does NOT alert on the transient -- the duration requirement absorbs it.
  3. Fill it to 92% and keep it there, and confirm the check DOES alert, at critical level.
  4. Check inodes as well as space with df -i, and demonstrate the alert firing for inode exhaustion on a filesystem that still shows free space.
  5. Forward logs to a second host (or a local collector) with rsyslog over TCP. Generate an auth event and confirm it arrives.
  6. Play the attacker: truncate the local log as root and confirm the forwarded copy is unaffected. State what centralising buys and what it does not.
  7. Confirm local rotation still matters: show that forwarding does not stop the local disk filling, and that logrotate is still required.

Verify

cd ~/lab-mon
# transient must not alert, sustained must:
./check-disk.sh transient | grep -qi alert && echo "FALSE ALARM on spike -- fail" || echo "spike absorbed"
./check-disk.sh sustained | grep -qi 'critical' && echo "sustained problem alerts"
df -i / | awk 'NR==2 {print "inode use:", $5}'
# forwarded log survives local truncation:
: > /var/log/secure; grep -c . /var/log/from-origin.log   # non-zero on the collector

The spike being absorbed and the sustained condition alerting is the whole point: a threshold with no duration requirement pages someone at 3am for a momentary blip, and a team that is paged for blips stops reading pages.

Notes

The test of an alert is: what do I do when this fires? If the answer is "look and decide it's fine", it is a dashboard, not an alert. Every threshold here has a warning level for working hours and a critical level for waking someone, because collapsing them turns everything into an emergency.