Recover a machine that will not boot

applied · 45 min · Objective 5.2

Task

Break a machine's boot in the two ways that happen most -- a bad fstab entry and a broken bootloader configuration -- and recover each from the console and from rescue media. Boot recovery is learned under pressure or learned here; here is cheaper.

Steps

  1. fstab fault. Add an fstab entry with a wrong UUID and NO nofail. Reboot and watch the machine drop to emergency mode.
  2. Recover: log in at the console, note the root filesystem is read-only, mount -o remount,rw /, fix the UUID (or add nofail), verify with mount -a, and reboot.
  3. Establish the habit that prevents it: after any fstab edit, mount -a while you still have a shell, and findmnt --verify to validate without mounting.
  4. Bootloader fault. Simulate a broken GRUB configuration -- a wrong root or a missing entry -- and reboot to a grub> prompt.
  5. Recover interactively at the prompt: ls to find the partition, set root, linux and initrd with the right paths, and boot.
  6. Repair properly from rescue media: mount the root and boot filesystems, bind-mount /dev /proc /sys, chroot, reinstall GRUB and regenerate the config with grub2-mkconfig. Explain why editing grub.cfg directly would not have lasted.
  7. Confirm a normal reboot succeeds.

Verify

findmnt --verify --verbose | tail -3          # fstab is valid
grep -q nofail /etc/fstab && echo "non-essential mount is nofail"
# after GRUB repair, confirm the generated config names the right root:
grep -o 'root=[^ ]*' /boot/grub2/grub.cfg | head -1
findmnt -n -o SOURCE /                         # must match the above
# a normal boot works:
systemctl is-system-running --wait

The root named in the regenerated grub.cfg matching what findmnt reports for / is the proof the bootloader repair is correct. The habit from step 3 -- mount -a before rebooting -- is what turns the fstab fault from a console trip into a caught typo.

Notes

Editing grub.cfg by hand is the mistake to unlearn: it is generated, and the next grub2-mkconfig or kernel update discards the edit, producing a fault weeks later with no apparent cause. Change /etc/default/grub and regenerate. The commonest real cause of both faults is a disk renamed or replaced so a UUID no longer resolves.