ubuntu 18.04 server

The well kwnown df(disk free) and du(disk usage) are used to display disk usage. But they don’t always tell the same story.

Before going further, there is a ncurses based version of du : ncdu.

Something has eaten my disk space and I can’t find what!

I somehow manage a server for a friend who has a small business, well somehow, because I used to manage it, but since there is nothing to manage, I almost forget about it.

This server runs ubuntu 10.04.3 LTS (yes a 9 years old release, unsupported since 2015), runs a plethora of services (directory/file/printing/telephony/fax/snmp/imap/virtual machines/VPN/…) flawlessly. As we say in engineering, If it works, don’t touch it!.

But this week, my friend called me twice saying system partition was full. This is the biggest flaw of this server, the system partition is a little bit small and is periodically filled (mainly) by the log files. logrotate may not be configured as well as it should.

Ok, the disk is full, now what?

Diagnostic

First of all : df -h /

user@server:~$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.6G  4.6G  0.0G  100% /
user@server:~$

Then : sudo du -sxh /

user@server:~$ sudo du -xsh /
3.3G	/
user@server:~$

So df says “4.6G used” and du tells “3.3G used”, why? Because some files may be deleted but still in use by other programs (see that excellent answer on unix.stackexchange).

So let’s try sudo lsof -s -nP +L1:

...
imap     3862      XXX   14u   REG    9,0        592  0    132629 /var/mail/indexes/XXX/.Deleted Messages/dovecot.index (deleted)
imap    11242    XXXXX    7u   REG    9,0    1073712  0    137771 /var/mail/indexes/XXXXX/.Sent/dovecot.index (deleted)
openvpn 23678 nobody      1w   REG    9,0 1369666431  0    131769 /var/log/openvpn (deleted)
apache2 13815 www-data    9w   REG   0,17          0  0 369358297 /var/run/apache2/ssl_mutex (deleted)
apache2 13833 www-data    9w   REG   0,17          0  0 369358297 /var/run/apache2/ssl_mutex (deleted)
imap    25994    XXXXX    7u   REG    9,0       1296  0    131647 /var/mail/indexes/XXXXX/.INBOX/dovecot.index (deleted)
...

So /var/log/openvpn has been deleted, but it’s still used by openvpn and it still consumes 1369666431 bytes (1.27 GiB) on the disk!. It may be possible to truncate the file while the openvpn server use it, but I just restarted the openvpn server and now df and du now agree.

user@server:~$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.6G  3.3G  1.3G  71% /
user@server:~$ sudo du -sxh /
3.3G	/
user@server:~$

Some space has been freed, I have reconfigured openvpn to be less verbose, logrotate to rotate openvpn’s log file, and now I can once more forget this server, Great success !

ncdu - a tool for finding your big files

ncdu is the command line equivalent of gnome’s disk analyser (baobab), or not to name it, WinDirStat.

Sample output of ncdu -x /

ncdu 1.11 ~ Use the arrow keys to navigate, press ? for help
--- / -----------------------------------------------------------------
   18.9 GiB [##########] /var
   16.0 GiB [########  ]  swapfile
    6.2 GiB [###       ] /usr
    4.1 GiB [##        ] /root
    1.2 GiB [          ] /lib
  252.3 MiB [          ] /boot
   19.6 MiB [          ] /sbin
   19.0 MiB [          ] /etc
   16.0 MiB [          ] /bin
    6.6 MiB [          ] /tmp
   16.0 KiB [          ] /opt
    8.0 KiB [          ] /mnt
    8.0 KiB [          ] /media
    8.0 KiB [          ] /.config
    8.0 KiB [          ] /snap
    4.0 KiB [          ] /lib64
    4.0 KiB [          ] /srv
@   0.0   B [          ]  initrd.img.old
@   0.0   B [          ]  initrd.img
@   0.0   B [          ]  vmlinuz.old
@   0.0   B [          ]  vmlinuz
>   0.0   B [          ] /sys
>   0.0   B [          ] /run
>   0.0   B [          ] /proc
>   0.0   B [          ] /home
>   0.0   B [          ] /dev
>   0.0   B [          ] /data

 Total disk usage:  46.8 GiB  Apparent size:  46.8 GiB  Items: 757332

~~~

Question, remark, bug? Don't hesitate to contact me or report a bug.