Exhaust disk space and inodes, and find what filled them

applied · 50 min · Objective 4.2

Task

Give a service a small volume for its logs, then exhaust it twice -- once by filling the space, once by exhausting its inodes with many tiny files while space remains -- and diagnose each from the application's failure back to the cause. The second case is the one that fools people: df -h says there is room.

Steps

  1. Create and mount the volume, and write a small script /usr/local/bin/applog.sh that appends a timestamped line to /srv/applogs/app.log every second and logs to the journal if the write fails. Run it as a systemd service.
  2. Fill the space: write one large file with fallocate until the volume is full. Save df -h /srv/applogs, du -sh /srv/applogs/* and the service's journal errors to lab/exhaust/space.txt. Delete the file.
  3. Exhaust the inodes: create empty files in a loop until creation fails. Save df -h /srv/applogs, df -i /srv/applogs and the service's errors to lab/exhaust/inodes.txt.
  4. Find the directory holding most files with find /srv/applogs -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n | tail, save it to lab/exhaust/culprit.txt, and clean it up.
  5. Record in lab/exhaust/alerts.txt the two monitoring thresholds that would have warned before either failure.

Verify

These checks run in a POSIX shell: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. A stock Windows PowerShell or Command Prompt has no awk or grep, so there the first line fails.

grep -Eic 'No space left on device' lab/exhaust/space.txt lab/exhaust/inodes.txt
grep -E '100%' lab/exhaust/inodes.txt | head -2
awk '/IUse%/ {getline; print "inode use: "$5}' lab/exhaust/inodes.txt
grep -Eic 'inode' lab/exhaust/alerts.txt

Both failures report no space left on device, but the inode capture shows space still free with inode use at 100 per cent. A monitoring system watching only disk space would have stayed silent through the second failure, which is why the alerts file must include an inode threshold.

Notes

Memory leaks are the third common exhaustion: a process whose memory grows without levelling off until the kernel's out-of-memory killer ends something, recorded as an Out of memory: Killed process line in journalctl -k.

This is an independent study companion for CompTIA Server+ SK0-005 and is not produced by or endorsed by CompTIA.