Recover a machine that will not boot

applied · 45 min · Objective 5.2

Task

Break a machine's boot in the two ways that happen most -- a bad fstab entry and a broken bootloader configuration -- and recover each from the console and from rescue media. Boot recovery is learned under pressure or learned here; here is cheaper.

Steps

  1. fstab fault. Add an fstab entry with a wrong UUID and NO nofail. Reboot and watch the machine drop to emergency mode.
  2. Recover: log in at the console, note the root filesystem is read-only, mount -o remount,rw /, fix the UUID (or add nofail), verify with mount -a, and reboot.
  3. Establish the habit that prevents it: after any fstab edit, mount -a while you still have a shell, and findmnt --verify to validate without mounting.
  4. Bootloader fault. Simulate a broken GRUB configuration -- a wrong root or a missing entry -- and reboot to a grub> prompt.
  5. Recover interactively at the prompt: ls to find the partition, set root, linux and initrd with the right paths, and boot.
  6. Repair properly from rescue media: mount the root and boot filesystems, bind-mount /dev /proc /sys, chroot, reinstall GRUB and regenerate the config with grub2-mkconfig -o /boot/grub2/grub.cfg. Reinstalling GRUB differs by firmware: on BIOS, grub2-install /dev/<disk>; on UEFI, do NOT run grub2-install on RHEL 9 -- reinstall the packages instead (dnf reinstall grub2-efi-x64 shim-x64), which keeps Secure Boot's signed chain intact. Explain why editing grub.cfg directly would not have lasted.
  7. Confirm a normal reboot succeeds.

Verify

findmnt --verify --verbose | tail -3          # fstab is valid
grep -q nofail /etc/fstab && echo "non-essential mount is nofail"
# after GRUB repair, confirm the default boot entry names the right root:
grubby --info=DEFAULT | grep -o 'root=[^ "]*'
findmnt -n -o SOURCE /                         # must match the above
# a normal boot works:
systemctl is-system-running --wait

The root named in the default boot entry matching what findmnt reports for / is the proof the bootloader repair is correct. Ask grubby rather than grepping grub.cfg: RHEL 8 and 9 keep each kernel's arguments in Boot Loader Spec entries under /boot/loader/entries/, and the generated grub.cfg carries no per-kernel root= for a grep to find. The habit from step 3 -- mount -a before rebooting -- is what turns the fstab fault from a console trip into a caught typo.

Notes

Editing grub.cfg by hand is the mistake to unlearn: it is generated, and the next grub2-mkconfig or kernel update discards the edit, producing a fault weeks later with no apparent cause. Change /etc/default/grub and regenerate. The commonest real cause of both faults is a disk renamed or replaced so a UUID no longer resolves.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.