Degrade an array and rebuild it

short · 35 min · Objective 1.3

Before you start — what this lab needs

A disposable virtual machine. Loop files keep this off real storage, but mdadm is destructive when pointed at the wrong device:

for n in 1 2 3; do
  dd if=/dev/zero of=/tmp/md$n.img bs=1M count=128
  losetup -f --show /tmp/md$n.img
done
losetup -a

Task

Build a RAID 1 mirror from loop devices, fail a member deliberately, observe the degraded state, and rebuild. Then demonstrate that RAID protects against a disk and not against you.

Steps

Create a mirror across two loop devices: mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/loop0 /dev/loop1.
Read cat /proc/mdstat and identify the [UU] status. Format and mount it, and write a file you can recognise later.
Fail one member: mdadm --manage /dev/md0 --fail /dev/loop1. Read /proc/mdstat again and find [U_].
Confirm the filesystem is still readable and writable while degraded -- that is the point of the mirror.
Remove the failed member, add the spare loop device, and watch the rebuild progress in /proc/mdstat. Note the speed and recovery lines.
Once [UU] returns, demonstrate the limit of RAID: delete your file with rm. Confirm it is gone from both members, because the mirror faithfully replicated the deletion.
Record the array in /etc/mdadm.conf with mdadm --detail --scan, and explain what happens on the next boot without it.

Verify

grep -A2 '^md0' /proc/mdstat | grep -o '\[U*_*\]'    # [UU] healthy, [U_] degraded
mdadm --detail /dev/md0 | grep -E 'State :|Active Devices|Failed Devices'
mount | grep -q /mnt/md && echo "mounted throughout"
test -f /mnt/md/important.txt || echo "rm removed it from BOTH members"

The last line is the lesson people skip. A mirror is two copies of whatever you did, including the mistake.

Notes

Tear down with umount, mdadm --stop /dev/md0, mdadm --zero-superblock on each loop device, then losetup -d. Skipping the zero-superblock step leaves metadata that makes the array reassemble itself unexpectedly at the next boot.