Help, var, etc out of space on Ubuntu Server with plenty of space, Can't upgrade any packages, var and etc out of space Options

[DrewG]

Hi, I have an Ubuntu server 14.04 set up on an old laptop. It has been working well for a year or two.

It is configured to boot off of a flash drive. I have two external USB hard drives set up for SMB shares. It does not use the internal hard drive on the laptop. I set it up originally by following a guide, but don't remember the details of how it works.

I recently tried to update some services through Webmin (as I had been doing periodically) and was not able to. The message was:

Failed to fetch package list : Failed to open /etc/webmin/package-updates/current.cache for writing : No space left on device

The flash drive and the USB hard drives have plenty of space. I have tried restarting the machine and clearing tmpfs and it does not help.

I would appreciate any help.

df -h reports:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 14G 11G 2.2G 84% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 487M 4.0K 487M 1% /dev
tmpfs 100M 3.0M 97M 3% /run
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdc1 2.8T 2.0T 811G 71% /mnt/Seagate3TB
/dev/sdb1 1.9T 1.6T 309G 84% /mnt/External2TB


Operating system: Ubuntu Linux 14.04
Webmin version: 1.690
Kernel and CPU Linux 3.13.0-65-generic on i686
Processor information Genuine Intel® CPU U1400 @ 1.20GHz, 1 cores
System uptime 26 days, 20 hours, 38 minutes
Running processes 88
CPU load averages 0.00 (1 min) 0.01 (5 mins) 0.05 (15 mins)
CPU usage 3% user, 0% kernel, 0% IO, 97% idle
Real memory 312.14 MB used, 992.35 MB total
Virtual memory 32.61 MB used, 1012 MB total
Local disk space 3.42 TB used, 4.56 TB total

Package updates 150 package updates are available

Update packages through webmin:
Failed to fetch package list : Failed to open /etc/webmin/package-updates/current.cache for writing : No space left on device

sudo dpkg --configure -a
dpkg: error: unable to create new file '/var/lib/dpkg/status-new': No space left on device

[michaelk]

I doubt 150 updates would require 1.5GB ( less 5% for reserved space). If you have lots of files on your / partition then you might of run out of inodes. Post the output of the command

df -i

Check your /boot. If you have lots of old kernels that is one way to easily gain some space.

[DrewG]

OK, I'm not sure what inodes are but will read up on them.

Here is the output of df -i:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 912128 912128 0 100% /
none 127020 2 127018 1% /sys/fs/cgroup
udev 124603 510 124093 1% /dev
tmpfs 127020 507 126513 1% /run
none 127020 1 127019 1% /run/lock
none 127020 1 127019 1% /run/shm
none 127020 2 127018 1% /run/user
/dev/sdc1 212556759 624 212556135 1% /mnt/Seagate3TB
/dev/sdb1 324079820 146252 323933568 1% /mnt/External2TB


This is what is in /boot:

abi-3.13.0-24-generic initrd.img-3.13.0-49-generic
abi-3.13.0-29-generic initrd.img-3.13.0-51-generic
abi-3.13.0-30-generic initrd.img-3.13.0-52-generic
abi-3.13.0-32-generic initrd.img-3.13.0-53-generic
abi-3.13.0-33-generic initrd.img-3.13.0-54-generic
abi-3.13.0-34-generic initrd.img-3.13.0-55-generic
abi-3.13.0-35-generic initrd.img-3.13.0-57-generic
abi-3.13.0-36-generic initrd.img-3.13.0-58-generic
abi-3.13.0-37-generic initrd.img-3.13.0-59-generic
abi-3.13.0-39-generic initrd.img-3.13.0-61-generic
abi-3.13.0-40-generic initrd.img-3.13.0-62-generic
abi-3.13.0-43-generic initrd.img-3.13.0-63-generic
abi-3.13.0-44-generic initrd.img-3.13.0-65-generic
abi-3.13.0-46-generic memtest86+.bin
abi-3.13.0-48-generic memtest86+.elf
abi-3.13.0-49-generic memtest86+_multiboot.bin
abi-3.13.0-51-generic System.map-3.13.0-24-generic
abi-3.13.0-52-generic System.map-3.13.0-29-generic
abi-3.13.0-53-generic System.map-3.13.0-30-generic
abi-3.13.0-54-generic System.map-3.13.0-32-generic
abi-3.13.0-55-generic System.map-3.13.0-33-generic
abi-3.13.0-57-generic System.map-3.13.0-34-generic
abi-3.13.0-58-generic System.map-3.13.0-35-generic
abi-3.13.0-59-generic System.map-3.13.0-36-generic
abi-3.13.0-61-generic System.map-3.13.0-37-generic
abi-3.13.0-62-generic System.map-3.13.0-39-generic
abi-3.13.0-63-generic System.map-3.13.0-40-generic
abi-3.13.0-65-generic System.map-3.13.0-43-generic
abi-3.13.0-66-generic System.map-3.13.0-44-generic
abi-3.13.0-67-generic System.map-3.13.0-46-generic
config-3.13.0-24-generic System.map-3.13.0-48-generic
config-3.13.0-29-generic System.map-3.13.0-49-generic
config-3.13.0-30-generic System.map-3.13.0-51-generic
config-3.13.0-32-generic System.map-3.13.0-52-generic
config-3.13.0-33-generic System.map-3.13.0-53-generic
config-3.13.0-34-generic System.map-3.13.0-54-generic
config-3.13.0-35-generic System.map-3.13.0-55-generic
config-3.13.0-36-generic System.map-3.13.0-57-generic
config-3.13.0-37-generic System.map-3.13.0-58-generic
config-3.13.0-39-generic System.map-3.13.0-59-generic
config-3.13.0-40-generic System.map-3.13.0-61-generic
config-3.13.0-43-generic System.map-3.13.0-62-generic
config-3.13.0-44-generic System.map-3.13.0-63-generic
config-3.13.0-46-generic System.map-3.13.0-65-generic
config-3.13.0-48-generic System.map-3.13.0-66-generic
config-3.13.0-49-generic System.map-3.13.0-67-generic
config-3.13.0-51-generic vmlinuz-3.13.0-24-generic
config-3.13.0-52-generic vmlinuz-3.13.0-29-generic
config-3.13.0-53-generic vmlinuz-3.13.0-30-generic
config-3.13.0-54-generic vmlinuz-3.13.0-32-generic
config-3.13.0-55-generic vmlinuz-3.13.0-33-generic
config-3.13.0-57-generic vmlinuz-3.13.0-34-generic
config-3.13.0-58-generic vmlinuz-3.13.0-35-generic
config-3.13.0-59-generic vmlinuz-3.13.0-36-generic
config-3.13.0-61-generic vmlinuz-3.13.0-37-generic
config-3.13.0-62-generic vmlinuz-3.13.0-39-generic

... trimmed because post was too long

[michaelk]

/dev/sda1 912128 912128 0 100% /

As expected you have run out of inodes. In a nutshell it is a data structure that contains the files metadata. inodes are created when the partition is formatted and the number can not be easily changed on a running system. At the moment the only way to fix your problem is to delete files in sda1. Old kernels is a start but without knowing how much other stuff you have I do not know what else to suggest. Take a look at the files in your home directories.

tuxtweaks.com/2009/12/remove-old-kernels-in-ubuntu/
help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernelsA

[DrewG]

Thank you. I tried to figure this out for a while. I'm running a command that will tell me what directories have the most files, hopefully will give me a place to start deleting.

[DrewG]

The problem was in /usr/src

It had a lot of old versions of the kernel in it. I have been removing them one by one with the following command:

sudo apt-get remove --purge linux-image-3.13.0-34-generic linux-headers-3.13.0-34 linux-headers-3.13.0-34-generic

and have already freed up about 30% of my inodes.

Is there a configuration option that will keep this from happening in the future? Maybe limit the old saved kernels to the last X versions?

[michaelk]

Ubuntu does not remove old kernels for some unknown reason. I have not played with it in awhile so I don't know.

[DrewG]

Now I'm stuck again. After a restart, eth0 is broken and I can't get it on the network.

ifconfig only shows "lo".
lspci shows the Ethernet hardware in the laptop.
/etc/network/interfaces has a listing for
iface eth0 inet dhcp
I added
auto eth0

I have rebooted the networking gear it is attached through.

All commands that reference eth0 come back with "device not found".

Any ideas?

[DrewG]

OK, I figured out that I could use an older kernel in the grub boot menu to get networking to work. I went back to 3.13.0-65 (from 0-67) and it started to work. I then updated all the packages and rebooted into 3.13.0-79 and it works. My failed upgrades when I was out of inodes must have done something bad to networking, or else there was a problem in those kernels that was fixed later.

Thanks for your help.

[michaelk]

Great glad you got it working

[jazz123]

That sounds me a lot!