df says the disk is full, du cannot find the space, and neither is lying

Paul Wellner Bou

Well-known member
Registered
Joined
Apr 20, 2016
Messages
107
Points
16
Hit 100 percent on a VPS and spent an hour with du adding up to maybe thirty gigabytes less than df was reporting. The gap was real, not a rounding problem.

It was a log file that had been deleted while the service still had it open. The directory entry is gone so du cannot see it, but the inode stays allocated until the process closes the handle, so df still counts it. lsof +L1 lists exactly these. Restarting the service released it immediately, no reboot needed.

Since then I check two more things before digging. df -i in case it is inodes rather than bytes, which looks identical from the outside and is usually a mail queue or a session directory. And files hidden under a mount point, where something wrote to a directory before the filesystem was mounted over it. Bind mounting the root somewhere else and running du there shows those.

One habit I changed: truncating a log with > file instead of deleting it, so space comes back without a restart. That works for a process appending, less so for one holding an offset, and I have not tested enough cases to say where the line is.
 

Latest Hosting OffersNew Reviews

Sponsors

Tag Cloud

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top