Exhaust inodes on a disk that is nearly empty

short · 30 min · Objective 1.3

Before you start — what this lab needs

A disposable virtual machine. You will create and fill filesystems on loop files, which keeps it away from real storage, but the mkfs commands are destructive if aimed wrongly:

dd if=/dev/zero of=/tmp/small.img bs=1M count=64

Task

Produce the signature failure -- "No space left on device" on a filesystem with plenty of free space -- so that you recognise it instantly when it happens for real. Then confirm that xfs does not have the same limit.

Steps

Create an ext4 filesystem with deliberately few inodes: mkfs.ext4 -N 512 /tmp/small.img. Mount it at /mnt/inodes.
Confirm the starting position with both df -h /mnt/inodes and df -i /mnt/inodes. Note how much space is free and how many inodes exist.
Create empty files until it fails: for i in $(seq 1 1000); do touch /mnt/inodes/f$i || break; done. Read the error.
Run df -h and df -i again. The first shows a nearly empty filesystem; the second shows 100% inode use. This is the whole lesson.
Delete some files and confirm writing works again -- and note how slow the deletion is relative to the space it frees.
Now do the same with xfs: mkfs.xfs -f /tmp/small.img, remount, and create the same number of files. Confirm that inode use rises but does not hit a fixed ceiling, because xfs allocates inodes dynamically.
Find which directory holds the most files on a real system: find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -rn | head.

Verify

df -h /mnt/inodes | awk 'NR==2 {print "space used:", $5}'
df -i /mnt/inodes | awk 'NR==2 {print "inodes used:", $5}'
touch /mnt/inodes/one-more 2>&1 | grep -qi 'no space' && echo "reproduced"

The two percentages must be wildly different -- space in single figures, inodes at 100%. If both are high you filled it with data rather than with empty files; use touch, not dd.

Notes

The real-world versions are a mail queue, a session directory, a cache that never expires, and an application logging one file per request. All four use almost no space and all four will stop the machine writing. df -i belongs next to df -h in your monitoring, because nothing else reports it.