Bdtyktl

I have a problem with a server running Ubuntu 16.04.05. After running a python script who created more than 65 million of files I have an error message saying that I have no space left:

12 zip -r 31.zip 31

zip I/O error: No space left on device

zip error: Could not create output file (31.zip)

I found some similar problems online: No space left on the device though there is space and No space left on device even though there is, but I think I have a different one. The solutions posted in these two answers are not working for me.

The following is the structure of the storage:

➜  12 df -k

Filesystem      1K-blocks      Used Available Use% Mounted on

udev             32963768         0  32963768   0% /dev

tmpfs             6597356      9244   6588112   1% /run

/dev/sda1        32895856  26382068   4819736  85% /

tmpfs            32935580        96  32935484   1% /dev/shm

tmpfs                5120         0      5120   0% /run/lock

tmpfs            32935580         0  32935580   0% /sys/fs/cgroup

/dev/sdb       1031992064 664288464 322578204  68% /media/hdd1

tmpfs             6587116        20   6587096   1% /run/user/118

tmpfs             6587116         0   6587116   0% /run/user/1001

I am working and trying to zip a folder in /media/hdd1 which should have plenty of space left. The dimension of the folder I am trying to zip is:

➜  12 du -sh 31

49M     31

EDIT:

➜  12 df -i

Filesystem       Inodes    IUsed   IFree IUse% Mounted on

udev            8240942      478 8240464    1% /dev

tmpfs           8246695      617 8246078    1% /run

/dev/sda1       2097152   500872 1596280   24% /

tmpfs           8233895        5 8233890    1% /dev/shm

tmpfs           8233895        6 8233889    1% /run/lock

tmpfs           8233895       16 8233879    1% /sys/fs/cgroup

/dev/sdb       65536000 65536000       0  100% /media/hdd1

tmpfs           8246695       15 8246680    1% /run/user/118

tmpfs           8246695        4 8246691    1% /run/user/1001

As ByteCommander explained, you have a problem with inodes.

Another way to takle this problem is to look for the millions of little files that are consuming the inodes.

sudo du -a -x -d 1 --inodes /media/hdd1 | sort -nr | head -20

will show you the directories consuming the most inodes on /media/hdd1 - you can then travel down through the directories until you find one filled with a bunch of files.

"After running a python script who created more than 65 million of files" here is your problem.

The ext4 file system Ubuntu uses has only a limited number of so-called "inodes", which is determined when you format a partition. You need at least one inode per file or directory on the file system.

Now if you create huge numbers of small files, you'll exhaust the available inodes faster than the available storage space. Both equally result in the same symptoms of a seemingly full disk though.

You will need to get rid of those tons of tiny files and e.g. put them on a different drive, in a zip file, or delete them altogether.

Unfortunately the number of inodes can not be changed dynamically, as far as I know. If you really need more on your current partition, you'll probably need to back up your data and reformat everything.

Related:

• Running out of inodes


• How can I create an EXT4 partition with an extra large number of inodes?


• Drawbacks of increasing number of inodes in EXT4