Reclaim a filesystem that df and du disagree about

applied · 35 min · Objective 2.1

Task

Reproduce the deleted-but-open-file situation deliberately, diagnose it with the tools you would use in an incident, and reclaim the space. Then do the same thing the right way, so the situation never arises.

This is the single most common "the disk is full and I cannot find why" call, and reproducing it once is worth more than reading about it three times.

Steps

  1. Create a directory and start a process that holds a log open and keeps writing to it. In one shell: mkdir -p /var/log/demo && (while true; do date >> /var/log/demo/app.log; sleep 1; done) &
  2. Let it run for a few seconds, then confirm the file is growing with ls -l /var/log/demo/app.log.
  3. Delete the file while the writer still has it open: rm /var/log/demo/app.log.
  4. Observe the disagreement. du -sh /var/log/demo now reports almost nothing, while the space is still allocated. On a real system df would still show it used.
  5. Find the culprit with lsof +L1, which lists open files whose link count is zero. Identify the PID.
  6. Reclaim the space by stopping that process, and confirm the change.
  7. Now do it correctly: recreate the writer, and instead of deleting the file, truncate it in place with : > /var/log/demo/app.log. Confirm the writer keeps working and the space is released immediately.

Verify

lsof +L1 2>/dev/null | head          # after step 3: names the deleted file
lsof +L1 2>/dev/null | wc -l         # after step 6: no deleted-but-open files
ls -l /var/log/demo/app.log          # after step 7: exists and is small

The key observation is between steps 4 and 5: du finds nothing and lsof +L1 finds the file. If lsof +L1 is empty after step 3, your background writer exited -- check with jobs and restart it.

Notes

Truncating with : > or > file keeps the inode, so the writer's file descriptor stays valid and it carries on appending from offset zero. Deleting breaks that, which is why logrotate has both a copytruncate option and a postrotate signal -- two different solutions to exactly this problem.