4,461
edits
Web services: Difference between revisions
add "No space left on device" error on a LAMP web server
change headers |
add "No space left on device" error on a LAMP web server |
||
| Line 154: | Line 154: | ||
Donating to EFF: https://eff.org/donate-le | Donating to EFF: https://eff.org/donate-le | ||
</source> | </source> | ||
= Miscellaneous = | |||
== "No space left on device" error on a LAMP web server == | |||
*Tested on: Ubuntu 12.04 Precise | |||
*Difficulty: 2/10 | |||
*Time: >10 minutes + number of files to delete + your WPM | |||
If you try to '''create a blank file''', | |||
<source lang="bash"> | |||
$ touch forcefsck | |||
touch: cannot touch 'forcefsck': No space left on device | |||
</source> | |||
you get a report back saying there is no space left on device. However, when you '''check the disk space''': | |||
<source lang="bash"> | |||
$ df -h | |||
Filesystem Size Used Avail Use% Mounted on | |||
/dev/sda1 52G 28G 22G 58% / | |||
udev 7.9G 4.0K 7.9G 1% /dev | |||
tmpfs 3.2G 524K 3.2G 1% /run | |||
none 5.0M 0 5.0M 0% /run/lock | |||
none 7.9G 140K 7.9G 1% /run/shm | |||
</source> | |||
There is still 58% of disk space left, so something else is wrong. After googling about this, it turns out that my inode was running out. To '''check the number of inodes''': | |||
<source lang="bash"> | |||
$ df -i | |||
Filesystem Inodes IUsed IFree IUse% Mounted on | |||
/dev/sda1 3393040 3393020 20 100% / | |||
udev 2050686 489 2050197 1% /dev | |||
tmpfs 2052885 384 2052501 1% /run | |||
none 2052885 2 2052883 1% /run/lock | |||
none 2052885 47 2052838 1% /run/shm | |||
</source> | |||
''inode'' stands for index node, which is an index for a file/folder/device/etc. in the Unix file system scheme. | |||
To '''find out which folder is causing this massive hemorrhage of inodes''': | |||
<source lang="bash"> | |||
$ sudo -s | |||
$ cd / | |||
$ for i in /*; do echo $i; find $i -type f | wc -l; done | |||
(...) | |||
/home | |||
34293 | |||
/initrd.img | |||
0 | |||
/initrd.img.old | |||
0 | |||
/lib | |||
14655 | |||
/lib64 | |||
0 | |||
/lost+found | |||
0 | |||
/media | |||
0 | |||
/mnt | |||
0 | |||
/opt | |||
1402 | |||
(...) | |||
</source> | |||
It looks like there is a lot of inodes in /var for some reason, now we need to narrow down to a specific directory: | |||
<source lang="bash"> | |||
$ for i in ./* ; do echo $i; find $i -type f | wc -l; done | |||
(...) | |||
./crash | |||
1 | |||
./lib | |||
3186175 | |||
./local | |||
0 | |||
./lock | |||
0 | |||
(...) | |||
$ cd lib | |||
$ for i in ./* ; do echo $i; find $i -type f | wc -l; done | |||
(...) | |||
./pam | |||
6 | |||
./php5 | |||
3012602 | |||
./plymouth | |||
1 | |||
(...) | |||
</source> | |||
You can check the number of files in any directory by issuing '''ls -l | wc -l''' but I couldn't even do this because there were millions of files that have accumulated over a year. These files had accumulated because PHP isn't doing the garbage collection. Your session.gc_probability may be set to 0. Change it to 1. | |||
<source lang="bash"> | |||
$ /usr/lib/php5/maxlifetime | |||
24 | |||
</source> | |||
It's 24 minutes. Now, here is the command to delete all of the older files. | |||
<source lang="bash"> | |||
$ find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec rm {} \; | |||
</source> | |||
This isn't necessary if you have the garbage collection enabled from the PHP configuration, but here is a cron job to run every hour as a root if this isn't caused by PHP. | |||
<source lang="bash"> | |||
$ crontab -e | |||
0 /usr/bin/find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec /bin/rm {} \; | |||
</source> | |||
===References=== | |||
http://pim.famnit.upr.si/blog/index.php?/archives/172-Running-out-of-inodes,-no-space-left-on-device,-php-not-cleaning-sessions.html (accessed on July 30, 2012) | |||