Make a degraded array raise an alert instead of waiting to be found

short · 40 min · Objective 4.1

Task

Configure mdadm's monitor to run a program on array events, fail a member, and confirm an alert was raised within a minute -- then rebuild and confirm that was reported too. A degraded array nobody knows about is one failure from total loss; the alert is the difference.

Steps

  1. Write /usr/local/bin/raid-alert.sh, which appends the date and its arguments (event, device, member) to /var/log/raid-alerts.log. Make it executable and copy it to lab/raidalert/raid-alert.sh.
  2. Set PROGRAM /usr/local/bin/raid-alert.sh in mdadm.conf and start the monitor with mdadm --monitor --scan --daemonise --delay=10.
  3. Fail one member with mdadm --fail, note the time in lab/raidalert/failed-at.txt, and wait one minute.
  4. Remove the failed member, add a fresh loop device, and wait for the rebuild to finish.
  5. Copy /var/log/raid-alerts.log to lab/raidalert/alerts.log and record in lab/raidalert/response.txt the three things you would do on receiving the Fail alert on a real server, in order.

Verify

These checks run in a POSIX shell: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. A stock Windows PowerShell or Command Prompt has no awk or grep, so there the first line fails.

grep -c 'PROGRAM' /etc/mdadm/mdadm.conf /etc/mdadm.conf 2>/dev/null
grep -Ec 'Fail' lab/raidalert/alerts.log
grep -Ec 'RebuildFinished|RebuildStarted|SpareActive' lab/raidalert/alerts.log
grep -Eic 'backup|replace|identify|slot' lab/raidalert/response.txt

The log records the Fail event and at least one rebuild event. The response should start with confirming backups are current and identifying the exact failed slot, before replacing anything -- the array has no redundancy left until the rebuild finishes.

Notes

In production the MAILADDR line sends the same events by email, and most monitoring agents read /proc/mdstat directly. Hardware RAID controllers raise equivalent alerts through their management software and the server's management controller.

This is an independent study companion for CompTIA Server+ SK0-005 and is not produced by or endorsed by CompTIA.