Question

relevant software:
Red Hat Enterprise Linux Server release 6.3 (Santiago)
cpanel installed 11.34.0 (build 7)

background and problem:
I was getting a disk usage warning (via cpanel) because /var seemed to be filling up on my server. The assumption would be that there was a log file growing too large and filling up the partition. I recently removed a large log file and changed my syslog config to rotate the log files more regularly. I removed something like /var/log/somefile and edited /etc/rsyslog.conf. This is the reason I was suspicious of the disk usage report warning issued by cpanel that I was getting because it didn’t seem right.

This is what df was reporting for the partitions:

$ [/var]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             9.9G  511M  8.9G   6% /
tmpfs                 5.9G     0  5.9G   0% /dev/shm
/dev/sda1              99M   53M   42M  56% /boot
/dev/sda8             883G  384G  455G  46% /home
/dev/sdb1             9.9G  151M  9.3G   2% /tmp
/dev/sda3             9.9G  7.8G  1.6G  84% /usr
/dev/sda5             9.9G  9.3G  108M  99% /var

This is what du was reporting for /var mount point:

$ [/var]# du -sh
528M    .

clearly something funky was going on. I had a similar kind of reporting inconsistency in the past and I restarted the server and df reporting seemed to be correct after that. I decided to reboot the server to see if the same thing would happpen.

This is what df reports now:

$ [~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             9.9G  511M  8.9G   6% /
tmpfs                 5.9G     0  5.9G   0% /dev/shm
/dev/sda1              99M   53M   42M  56% /boot
/dev/sda8             883G  384G  455G  46% /home
/dev/sdb1             9.9G  151M  9.3G   2% /tmp
/dev/sda3             9.9G  7.8G  1.6G  84% /usr
/dev/sda5             9.9G  697M  8.7G   8% /var

This looks more like what I’d expect to get.

For consistency this is what du reports for /var:

$ [/var]# du -sh
638M    .

question:
This is a nuisance. I’m not sure where the disk usage reports issued by cpanel get their info but it clearly isn’t correct. How can I avoid this inaccurate reporting in the future? It seems like df reporting wrong disk usage is a strong indicator of the source problem but I’m not sure. Is there a way to ‘refresh’ the filesystem somehow so that the df report is accurate without restarting the server? Any other ideas for resolving this issue?

Answer 1

If you remove a file, but it is still open by some process, the disk space is not recovered — the process continues to access that file. This is a common problem with log files, because syslogd keeps them all open.

The disk space reported by du doesn’t include this file because it works by walking down the directory hierarchy adding up the sizes of all the files it finds. But this file can’t be found in any directory, so it’s not counted. df reports the actual space used in the filesystem.

The logfile rotation script sends a signal to syslogd telling it to close an reopen all its log files. You can accomplish this with:

killall -HUP syslogd

You also need to do this to get syslogd to use your modified syslog.conf.