Diagnose the storage failures by their signatures

short · 35 min · Objective 5.2

Task

Reproduce the storage failures that have distinctive signatures -- inode exhaustion, the deleted-but-open file, the OOM-killed process, the unkillable D-state process -- and identify each from its first symptom. Recognising the signature is the skill; the fix follows from it.

Steps

  1. Inode exhaustion. mkfs.ext4 -N 256 /tmp/small.img, mount it, and create empty files until touch fails with "No space left on device". Confirm df -h shows free space and df -i shows 100% -- the signature.
  2. Deleted-but-open. Start a process holding a log open, delete the log, and confirm du finds nothing while the space stays used. Find it with lsof +L1 and reclaim it by restarting the process.
  3. OOM kill. Run a process that allocates until the kernel kills it (tail /dev/zero in a memory-limited cgroup or container). Confirm exit 137 and the "Out of memory: Killed process" line in dmesg.
  4. D-state. Reason about the uninterruptible-sleep case: if you have a spare NFS mount, make the server unreachable and observe a process stuck in D that kill -9 cannot touch, with load climbing while the CPU idles.
  5. systemd unit failure. Point a unit's ExecStart at a missing binary and read status=203/EXEC from systemctl status.
  6. For each, write the one-line signature: the first observation that identifies it before any deeper investigation.

Verify

# inode exhaustion: space free, inodes full
df -h /mnt/small | awk 'NR==2{print "space", $5}'; df -i /mnt/small | awk 'NR==2{print "inodes", $5}'
# deleted-but-open:
lsof +L1 2>/dev/null | grep -q deleted && echo "deleted-but-open found"
# OOM:
dmesg | grep -qi 'killed process' && echo "OOM signature present"

The pairing of df -h showing free space and df -i showing 100% is the signature to lock in: "No space left on device" on a disk that is mostly empty is inodes, every time, and df -i is the one command that reveals it.

Notes

Each failure here has a first symptom that identifies it without deeper digging: free space with a full disk is inodes; du and df disagreeing is a deleted-but-open file; exit 137 is the OOM killer; a process immune to kill -9 is in D state waiting on I/O. Learn the signatures and the diagnosis is immediate.