Came across an issue today where a client was receiving emails about /tmp on his CPanel server being full. Websites were not responding either. I logged in, and df-h indeed showed 100% usage, but looking inside the directory there were no large files:
root@server1 [/tmp]# df -h /tmp Filesystem Size Used Avail Use% Mounted on /usr/tmpDSK 485M 460M 0 100% /tmp root@server1 [/tmp]# ls -lah /tmp/ total 23K drwxrwxrwt 3 root root 1.0K May 6 15:12 ./ drwxr-xr-x 28 root root 4.0K Apr 22 01:37 ../ -rwxr--r-- 1 root root 6.0K Jan 26 09:28 aquota.user* drwx------ 2 root root 12K Jan 20 18:25 lost+found/ lrwxrwxrwx 1 root root 30 May 6 15:12 mysql.sock -> ../../var/lib/mysql/mysql.sock= root@server1 [/tmp]#
There were no files in lost+found either. After restarting Mysql, disk space usage on /tmp returned to normal:
root@server1 [/tmp]# /etc/init.d/mysql restart Shutting down MySQL........................................[ OK ] Starting MySQL. [ OK ] root@server1 [/tmp]# df -h /tmp Filesystem Size Used Avail Use% Mounted on /usr/tmpDSK 485M 11M 449M 3% /tmp root@server1 [/tmp]#
Basically what happened here is that at some point the files taking up all the space in /tmp were deleted from the system, but weren’t actually deleted because some process had a lock on them. The deleted files will continue to take up disk space until the process that’s using them exits. You can find open deleted files with lsof:
root@server1 [~]# lsof | grep tmp queueproc 5133 root 0r REG 7,0 16817 6103 /tmp/sh-thd-1300762559 (deleted) cpdavd 5170 root 0r REG 7,0 16817 6103 /tmp/sh-thd-1300762559 (deleted) mysqld 28569 mysql 4u REG 7,0 0 12 /tmp/ib5AISFk (deleted) mysqld 28569 mysql 5u REG 7,0 0 13 /tmp/ibSyq2LH (deleted) mysqld 28569 mysql 6u REG 7,0 0 14 /tmp/ib3ZkcS4 (deleted) mysqld 28569 mysql 7u REG 7,0 0 15 /tmp/ibIOXl0r (deleted) mysqld 28569 mysql 11u REG 7,0 0 16 /tmp/ibKMMLdP (deleted) root@server1 [~]#
Restarting the service that’s holding onto the deleted files frees up the disk space. I would also recommend creating a larger /tmp directory on CPanel machines as it’s quite small, but I’ll save that for another post.