| tags:admin cpanel disk linux mysql sysadmin tmp categories:Tech

CPanel /tmp 100% Full

Came across an issue today where a client was receiving emails about /tmp on his CPanel server being full. Websites were not responding either. I logged in, and df-h indeed showed 100% usage, but looking inside the directory there were no large files:

root@server1 [/tmp]# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
/usr/tmpDSK           485M  460M     0 100% /tmp
root@server1 [/tmp]# ls -lah /tmp/
total 23K
drwxrwxrwt  3 root root 1.0K May  6 15:12 ./
drwxr-xr-x 28 root root 4.0K Apr 22 01:37 ../
-rwxr--r--  1 root root 6.0K Jan 26 09:28 aquota.user*
drwx------  2 root root  12K Jan 20 18:25 lost+found/
lrwxrwxrwx  1 root root   30 May  6 15:12 mysql.sock -> ../../var/lib/mysql/mysql.sock=
root@server1 [/tmp]#

There were no files in lost+found either. After restarting Mysql, disk space usage on /tmp returned to normal:

root@server1 [/tmp]# /etc/init.d/mysql restart
Shutting down MySQL........................................[  OK  ]
Starting MySQL.                                            [  OK  ]
root@server1 [/tmp]# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
/usr/tmpDSK           485M   11M  449M   3% /tmp
root@server1 [/tmp]#

Basically what happened here is that at some point the files taking up all the space in /tmp were deleted from the system, but weren’t actually deleted because some process had a lock on them. The deleted files will continue to take up disk space until the process that’s using them exits. You can find open deleted files with lsof:

root@server1 [~]# lsof | grep tmp    
queueproc  5133      root    0r      REG                7,0    16817       6103 /tmp/sh-thd-1300762559 (deleted)
cpdavd     5170      root    0r      REG                7,0    16817       6103 /tmp/sh-thd-1300762559 (deleted)
mysqld    28569     mysql    4u      REG                7,0        0         12 /tmp/ib5AISFk (deleted)
mysqld    28569     mysql    5u      REG                7,0        0         13 /tmp/ibSyq2LH (deleted)
mysqld    28569     mysql    6u      REG                7,0        0         14 /tmp/ib3ZkcS4 (deleted)
mysqld    28569     mysql    7u      REG                7,0        0         15 /tmp/ibIOXl0r (deleted)
mysqld    28569     mysql   11u      REG                7,0        0         16 /tmp/ibKMMLdP (deleted)
root@server1 [~]#

Restarting the service that’s holding onto the deleted files frees up the disk space. I would also recommend creating a larger /tmp directory on CPanel machines as it’s quite small, but I’ll save that for another post.