Andrew Que Sites list Photos
Projects Contact
Main

All of my computers run Linux and need networking to share files. I have Network File System (NFS) for a couple of machines, but often use SSH File System (SSHFS) instead. The layer of encryption makes SSHFS slower, but because the connection is based on SSH there are user permissions and authentication. NFS pretty much has no restrictions except for limited what IP addresses can make a connection. User permissions are enforced, but there is nothing stopping a malicious client from simply taking a static IP address and using the root account. I guess when it was created, only administrators could setup shares and clients using those shares. These days it falls rather short of being a good general purpose file sharing protocol. If everything on the network is trusted NFS would be a good option. For the computers of DrQue.net, which are on a house network, SSHFS is the preferred method.

November 20, 2021

Web Server with both PHP 5.6 and 7.4

With my experiments on the container and then the Sun-Pi fully functional it was time to get both versions of PHP running on the primary web server, the Web-Pi. To do this, I had the Sun-Pi take over hosting DrQue.net for about an hour. This allowed me to get everything working on the Web-Pi without taking down the site.

When compiling on the Web-Pi using a parallel build I ran out of memory. I had to shut down several services to get the compile to complete. I could have just run without building in parallel, but I didn’t want to wait. After PHP 5.6 with FPM was built and installed, I went about installing PHP 7.4.

The Web-Pi was not running the latest release of Raspbian (now called Raspberry Pi OS… whatever). So when I installed php-fpm I actually got PHP 7.3. The distribution update on the Pi isn’t as straight forward as I had thought. I had to edit the /etc/apt/sources.list in order to get the distribution update to register as available. Afterward I had PHP 7.4 with FPM.

Since this is the third time I’ve run this setup it went fairly quick. PHP 5 was set as the default which allows all the existing sites on DrQue.net to run. I wanted to setup PHP 7 for the sandbox area so I can start porting my active sites. One problem I was having is that when PHP is installed as an Apache module, I am able to specify parameters in a virtual host. I use this to turn on displayed errors and turn the error reporting all the way up. It is generally good practice to have all the errors/warnings/notifications displayed when developing to point out potential problems. Getting this to happen, however, was turning out to be a real pain.

My first solution was to create two FPM sockets, one for production, and one for development. The configuration files were the only difference. Seemed a bit of overkill, but it did work. Then I read about the .user-ini file. Turns out that only works when PHP is an Apache module. However, using the .htaccess file was supposed to work for FPM. The part that wasn’t clear was that the format of how to set PHP parameters.

SetEnv PHP_VALUE "error_reporting=E_ALL & ~E_NOTICE"
SetEnv PHP_VALUE "display_startup_errors=on"
SetEnv PHP_VALUE "log_errors=off"
SetEnv PHP_VALUE "display_errors=on"

This worked. I wondered if I could do the same in the virtual host itself. Turns out that also works just fine.

<VirtualHost *:80>
  ServerNameDocumentRoot "…"

  Include conf-available/php7.conf

  SetEnv PHP_VALUE "error_reporting=E_ALL"
  SetEnv PHP_VALUE "display_startup_errors=on"
  SetEnv PHP_VALUE "log_errors=off"
  SetEnv PHP_VALUE "display_errors=on"

</VirtualHost>

Took me awhile to find that solution, but it was what I needed. So the main site is operating on PHP 5.6 as usual, although it now using FPM rather than an Apache module. My sandbox area runs on PHP 7.4 with all errors displayed. I can finally start updating my PHP pages.

   Yesterday my 512 GiB microSD card arrived, allowing the Sun-Pi to mirror the Web-Pi.  After getting the base OS installed I started the synchronization process last night and it was finished by morning.  Today it was time to run the process of installing PHP 5 and 7.  The process went pretty smoothly with only minor hiccups.  I setup a couple virtual hosts to test some of my PHP sites to see that PHP 5 was fully functional.  With that verified, I wanted to see if PHP 7 worked for any of my sites.
   PHP 7 insisted on having a timezone.  The photoblog scripts I updated in 2015, and were mostly functional.  The photo gallery is an old script I have tried to replace a couple of times but never liked the replacement.  It doesn't work in PHP 7.  I found a couple other items I needed as well.  mysqli, bcmath, gd are not included with PHP 7 by default and have to be added.
   The last discovery I made was messing up when I made the mount for synchronization.  I used an SSHFS mount, but forced user permissions.  While I had the file data, the user permissions were not preserved.  Tried again using an NFS mount and that took care of those issues.
   One other item I wanted to add was enabling error and warning messages for some websites.  I have a sandbox area used for testing.  In the past I used phpFlag to override PHP settings.  This no longer works with PHP in FPM mode.  I had to setup two FPM sockets using different configuration files, one for production sites, and the other for the sandbox.  Can't say I like that setup, but it does work.
   Pictured is Zach at breakfast.  This morning was my first cold ride of the season.  Temperature was 20 °F (-6.7 °C) and that is a chilly ride.  The previous evening I prepared my bike and got out all my winter riding gear.  Usually below 25 °F (-4 °C) I wear a neoprene face mask and ski goggles.  The downside is that the mask becomes wet from breath and putting it back on for the second portion of the ride is uncomfortable.  Since I would have a tailwind on the ride to breakfast, I opted to leave off the mask.  It was cold, especially downhill, but not too bad.  The rest of my outfit was pretty standard.  Heavy shirt (for anything below 25 °F), fleece fitted pants, medium socks, ear muffs and riding gloves.  I put on the bicycle handlebar mitts and I don't really need gloves with those. 
   For the return ride, I put on the face mask.  Despite the headwind my face was comfortable.  My hands were not and I had to stop to put on gloves.  The problem is that the bike sitting in the cold for an hour cools off the hands quickly.  I forgot my chemical hand warmers which would have done the trick.  Instead I brought my gloves with battery warmers.  They had my hands back up to temperature in no time, but they are a little difficult to use with the bike mitts.

November 18, 2021

Compiling PHP 5 with FPM

This is the last part of my series on getting Apache 2 server that runs both PHP 7 and 5 with a custom compiling of PHP 5.

The last step to getting PHP 5 and 7 running is to custom compile PHP 5. For this we will need to setup a build environment.

apt install build-essential gcc make

Then we will need some libraries.

apt install apache2-dev build-essential gcc make \
    libxml2-dev  libsqlite3-dev  zlib1g-dev  libbz2-dev  libcurl4-openssl-dev \
    libgmp-dev libonig-dev libjpeg-dev libgd-dev libmcrypt-dev

If compiling on x86, there are two problems that will be encountered. The libraries curl and gmp both install to x86-specific directories. We need links from the main include directory.

ln -s /usr/include/x86_64-linux-gnu/curl /usr/local/include/curl
ln -s /usr/include/x86_64-linux-gnu/gmp.h /usr/local/include/gmp.h

On ARM the paths look like this.

ln -s /usr/include/arm-linux-gnueabihf/curl /usr/include/curl
ln -s /usr/include/arm-linux-gnueabihf/gmp.h /usr/include/gmp.h

Now we need the source code for PHP 5. They can be found on the download releases page. I am interested in the last PHP 5 release which is 5.6.40.

wget https://www.php.net/distributions/php-5.6.40.tar.gz
tar xvfz php-5.6.40.tar.gz

The configuration is one I use, but if you are compiling your own version of PHP you will likely have some idea of what other PHP packages you want included.

./configure \
    --prefix=/usr/local/php5 \
    --with-config-file-path=/etc/php/5.6 \
    --enable-fpm \
    --with-fpm-user=www-data \
    --with-fpm-group=www-data \
    --with-apxs2 \
    --enable-sockets \
    --with-mysql \
    --with-mysqli \
    --with-pdo-mysql \
    --enable-ftp \
    --enable-sysvsem \
    --enable-sysvshm \
    --enable-sockets \
    --enable-wddx \
    --with-mhash \
    --with-gd \
    --with-zlib \
    --enable-exif \
    --with-mcrypt \
    --with-gmp \
    --with-curl \
    --enable-bcmath \
    --enable-mbstring \
    --with-bz2 \
    --enable-shmop \
    --with-mcrypt \
    --enable-maintainer-zts

Highlighted in yellow are lines to tell the configuration that we want to install PHP 5 in /usr/local/php5, and we want the configuration files located in /etc/php/5.4. Feel free to modify these paths, but take note of them as they are used latter in the setup. In green are lines important to getting FPM.

Once setup, simply make and install.

make -j 16
make install

My system has a 16-thread CPU and the build only takes a minute. Adjust the value according to your CPU thread count as PHP builds just fine in parallel.

Next we need to configure PHP 5.

mkdir /etc/php/5.6
cp php.ini-production /etc/php/5.6/php.ini
cp sapi/fpm/php-fpm.conf /etc/php/5.6/php-fpm.conf

This will copy the default configuration files to the PHP 5 configuration directory. There are a couple of small changes needed in the FPM configuration.

nano /etc/php/5.6/php-fpm.conf

First, the PID line needs un-commented and changed like this:

;pid = run/php-fpm.pid → pid = /run/php5-fpm.pid 

The listening socket needs to change from a IP socket to a Unix socket:

listen = 127.0.0.1:9000 → listen = /run/php5-fpm.sock

Then the owner lines need to be enabled.

;listen.owner = www-data → listen.owner = www-data
;listen.group = www-data → listen.group = www-data
;listen.mode = 0660 → listen.mode = 0660

Now we need to create a service to run FPM for PHP 5 and start it.

cat << EOT >> /lib/systemd/system/php5-fpm.service
[Unit]
Description=The PHP 5 FastCGI Process Manager
After=network.target

[Service]
Type=simple
PIDFile=/run/php5-fpm.pid
ExecStart=/usr/local/php5/sbin/php-fpm --nodaemonize --fpm-config /etc/php/5.6/php-fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target
EOT

systemctl enable php5-fpm.service
systemctl daemon-reload
systemctl start php5-fpm.service

Verify the service is running:

root@php75:~# systemctl status php5-fpm.service

 php5-fpm.service - The PHP 5 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php5-fpm.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Tue 2021-11-16 00:14:32 UTC; 2s ago
   Main PID: 220157 (php-fpm)
      Tasks: 3 (limit: 38336)
     Memory: 7.0M
     CGroup: /system.slice/php5-fpm.service
             ├─220157 php-fpm: master process (/etc/php/5.6/php-fpm.conf)
             ├─220158 php-fpm: pool www
             └─220159 php-fpm: pool www

If the service started we can now use it the same was as the distribution PHP-FPM. Let’s create another virtual host for this:

mkdir /var/www/html/php5

cat <<EOT >> /var/www/html/php5/index.php
<?php phpinfo(); ?>
EOT

cat <<EOT >> /etc/apache2/sites-available/php5.conf
<VirtualHost *:80>
      ServerName php5.php7_5
      DocumentRoot "/var/www/html/php5"

      <FilesMatch \.php$>
          SetHandler "proxy:unix:/run/php5-fpm.sock|fcgi://localhost"
      </FilesMatch>
</VirtualHost>
EOT

a2ensite php5
systemctl reload apache2

Again, modify /etc/hosts on the host machine to have php5.php7_5 point to the container IP address. Now if you go to http://php5.php7_5 you should see the PHP information page for PHP 5.6 rather than PHP 7. Both http://php5.php7_5 and http://php7.php7_5 should work.

The take away lessons I learned from this:

  • Build the other version(s) of PHP with configuration parameters to install them in separate locations. This will allow many versions of PHP to be available.
  • Copy over the default PHP-FPM configuration and make changes to reflect how you want it to run.
  • Use a Unix socket instead of a TCP/IP socket for FastCGI.
  • Each version of PHP you want to add to Apache must be running as a FastCGI service.
  • A default PHP handler can be added to the top level of the Apache configuration, and virtual hosts can override this version with their own.

It took me about 2 full days of experimenting to get this setup working, but maybe this information will prove useful to someone else. Also, I’m sure I’ll forget what I did and I might need this in the future.

After getting the container working I tested this setup on a Raspberry Pi and had it working there as well. I have a large micro SD card on order to build a backup server for DrQue.net, and when it arrives I should be able to fully test this setup for DrQue.net itself. With PHP 8 now released, I might add a third PHP handler so I can get my PHP code into this decade. For now, however, I will live with just being able to run PHP 5 and 7 together.

November 17, 2021

Apache with PHP 7 MPM

In my quest to setup an Apache 2 server that runs both PHP 7 and 5, I created a Linux container to experiment with. Now that there is a container, it is time to get the container configured.

First, let’s grab a couple of tools we will need:

sudo apt nano wget

There is no reason to compile our own Apache server so we can just install it normally:

sudo apt install apache2

That will install Apache and it will start running. A container gets its own IP address. By default it is on a virtual subnet local to the host computer. There are other options but this works fine for the tests we’re about to run. When Apache installs, it also starts serving a default website. To see this page we just need to know the IP address of the container. Debian has ifconfig installed.

root@php75:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.3.246  netmask 255.255.255.0  broadcast 10.0.3.255
        inet6 fe80::216:3eff:fe44:a4ac  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:44:a4:ac  txqueuelen 1000  (Ethernet)
        RX packets 47  bytes 7637 (7.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 2060 (2.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Here we can see the IP address assigned to this container is 10.0.3.246. Putting that into a browser on the host machine will bring up the default Apache 2 page showing the server is installed and running.

Something that we will need are a couple of virtual hosts. Since our container is called php7_5 we’ll make the virtual hosts a.php7_5 and b.php7_5. Let’s start by creating two sites:

mkdir /var/www/html/a /var/www/html/b

cat <<EOT >> /var/www/html/a/index.html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Test A</title>
  </head>
  <body>
    <h1>Test A</h1>
    <p>Hello world!</p>
  </body>
</html>
EOT

cat <<EOT >> /var/www/html/b/index.html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Test B</title>
  </head>
  <body>
    <h1>Test B</h1>
    <p>Hello world!</p>
  </body>
</html>
EOT

cat <<EOT >> /etc/apache2/sites-available/a.conf
<VirtualHost *:80>
        ServerName a.php7_5
        DocumentRoot "/var/www/html/a"
</VirtualHost>
EOT

cat <<EOT >> /etc/apache2/sites-available/b.conf
<VirtualHost *:80>
        ServerName b.php7_5
        DocumentRoot "/var/www/html/b"
</VirtualHost>
EOT

a2ensite a b
systemctl reload apache2

In order to view these virtual hosts we need to bring up a web browser on the host machine and add the sites to the /etc/hosts. So on the host machine:

echo 10.0.3.246       php7_5 a.php7_5 b.php7_5 >> /etc/hosts

Now in the web browser you should be able to go to http://a.php7_5 or http://b.php7_5 and see either “Test A” or “TestB.” This shows that the virtual hosts are working.

Next we will install the distribution version of PHP with FPM.

apt install php-fpm

This does install PHP as part of the process, but PHP-FPM will be setup as a service. You can check to make sure the service is running. Based on what distribution, you might need to change the 7.4 to whatever version of PHP installed.

root@php75:~# systemctl status php7.4-fpm
 php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Mon 2021-11-15 22:40:43 UTC; 21s ago
       Docs: man:php-fpm7.4(8)
    Process: 10250 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php7.4-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
   Main PID: 10247 (php-fpm7.4)
     Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec"
      Tasks: 3 (limit: 38336)
     Memory: 8.3M
     CGroup: /system.slice/php7.4-fpm.service
             ├─10247 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             ├─10248 php-fpm: pool www
             └─10249 php-fpm: pool www

Highlighted is the important piece of information—the FPM socket. Remember that for latter. Now we need to install FastCGI for Apache and enable all the modules needed to make it work:

apt install libapache2-mod-fcgid
a2enmod actions alias proxy_fcgi fcgid

Now let’s make a virtual host to use it.

mkdir /var/www/html/php7

cat <<EOT >> /var/www/html/php7/index.php
<?php phpinfo(); ?>
EOT

cat <<EOT >> /etc/apache2/sites-available/php7.conf
<VirtualHost *:80>
      ServerName php7.php7_5
      DocumentRoot "/var/www/html/php7"

      <FilesMatch \.php$>
          SetHandler "proxy:unix:/run/php/php7.4-fpm.sock|fcgi://localhost"
      </FilesMatch>
</VirtualHost>
EOT

a2ensite php7
systemctl reload apache2

Here it is important to change the socket to whatever was reveled by systemctrl status. Again, add php7.php7_5 to the host machine’s /etc/hosts file. Now when a browser is used to navigate to http://php7.php7_5 it should print the PHP information page. In the table under Server API it should read FPM/FastCGI.

Note that we could define the FileMatch section of the virtual host in the main Apache configuration file. That would enable PHP 7 in FPM for all virtual hosts by default. This will be useful in the future.

The important take away lessons are this:

  • PHP FastCGI needs to run as a system service. The package php-fpm takes care of that for the distribution maintained version of PHP.
  • Apache can PHP setup as FastCGI if it is setup for each virtual host.
  • If FastCGI isn’t setup correctly, one of two things happened. Either the PHP isn’t being parsed, which results in the raw PHP being received, or an error is encountered. In the event of an error, check the /var/logs/apache2/error.log. The error I found most often was not finding the socket because I got the path wrong.

This article of the series on running PHP 5 and 7 in Apache will cover getting a test environment setup to build PHP with FastCGI Process Manager (FPM).

The first thing I wanted to do was setup a clean environment to do the build. Initially I was thinking I should get a virtual machine running. However, I’ve lately taking a liking to Linux containers (LXC). I set one up for a work project that needed to CentOS build environment. The build does better with a lot of processing power and memory, and a container is better than a virtual machine for this. So I followed a similar recipe to get my build environment setup for the PHP 7 and 5 test. Some people may choose to use Docker for container management. LXC is so simple and this doesn’t need to be deployed, so I don’t see a reason to add Docker.

I run Linux Mint on the Snow Dragon and LXC is installed. A coworker runs a much more lien Liunx OS and had to install it. Since I didn’t have to do this I will assume LXC is installed. For the test environment I am just going to use the latest Debian. So the container creation command looks like this:

sudo lxc-create -t /usr/share/lxc/templates/lxc-download -n php7_5 -- -d debian -r bullseye -a amd64 

With the container created, I don’t need to do anything else to get it configured. Sometimes directories between the container and the host need to be shared. In that case, one could edit the configuration file for the container.

sudo nano /var/lib/lxc/php7_5/config

And in that file the add the line:

lxc.mount.entry = <host_path> <client_path> none rw,bind,create=dir 0 0

Where host_path is the location to share on the host machine, and client_path is where it shows up in the running container. Note that one can change the rw part of the mount to ro to create a read-only location. Nice if you want to access files, but don’t want the container to be able to make changes.

One other item in the config

Now that the container is created, need to start it:

sudo lxc-start php7_5

Once it is running we can login:

sudo lxc-attach php7_5

Logging out of the container is as simple as typing “exit”.

To shutdown the container the command:

sudo lxc-stop php7_5

And when finished testing we can remove the container with everything in it:

sudo lxc-destroy php7_5

Just like a virtual machine, anything done in the container only effects the container. However, memory and CPU resources are shared with the host. This is perfect for build environments as the container will run at native speeds with full memory access, but without any risk of messing up the host machine’s build environment.

November 15, 2021

Running Apache with PHP 5 and 7

For some time I have wanted to be able to have DrQue.net run both PHP 5 and PHP 7. All of the sites on DrQue.net were developed in PHP 4 or 5, and most will no longer work in PHP 7. However, PHP 5 is end-of-life and for the maintained areas of the site I’d like to switch over to using PHP 7. I had looked into running two versions of PHP before but never got it working. The examples that existed used repositories that had various compiled versions of PHP. I would have been fine using that, but the repositories did not have ARM—just x86. So when I migrated to the Web-Pi I simply compiled PHP 5 from the source and used that. Functional, but I have no way to port to PHP 7 because the server currently doesn’t run it. Time to change that.

I’ve read the way to do this is to use FastCGI. Then each virtual host can specify which PHP version it wishes to use. All I needed to do is figure out how this works and compile my own. In the articles of this series I will outline what I did for a test environment so that I could assemble the pieces to roll my own Apache 2 server running both PHP 5 and 7 built form source code.

New Shear Pins

New Shear Pins

   David Blowie, our snow blower, got an oil change and fired up for a few minutes in preparation for the coming winter.  Last year I managed to break every shear pin on the auger.  They were so stuck I had to take Mr. Blowie to the repair shop to have a professional remove them.  We hadn't run the snow blower since and I had not replaced the shear pins.  Took care of that today.  Alright Wisconsin weather, it is your turn.  Let's have some snow!

November 12, 2021

Linux Missing Disk Space due to Removed Open Files

Started getting warning e-mails from the Emerald Dragon about low disk space. It is setup to do this when the disk space is below 75%. I did some basic cleanup and get the usage under 75% by removing old log files, cache, etc. However, the system has a 16 GB eMCC and I wasn’t seeing anything using this space.

I started by checking the free space using disk free (df -h).

root@EmeraldDragon:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            716M     0  716M   0% /dev
tmpfs           172M   23M  150M  13% /run
/dev/mmcblk0p2   15G   11G  4.0G  72% /
tmpfs           859M     0  859M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           859M     0  859M   0% /sys/fs/cgroup
/dev/mmcblk0p1  128M   19M  109M  15% /media/boot
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           172M     0  172M   0% /run/user/0
tmpfs           172M     0  172M   0% /run/user/1000

Then I did a disk usage command (du -h) on root:

root@EmeraldDragon:/# du -h /
...
2.7G	/

This tells me that all the files on the system consume only 2.7 GB of space, but the disk free command says that 11 GB are used. How can the sum of all the files on disk be less than the used space on disk?

In Linux, this can happen because when you delete a file used by a running process, the file doesn’t go away until the running process stops. Typically the discrepancy is fairly small. However, the Emerald Dragon has been running over 1,570 days.

Now the trick was to find the process holding open a large collection of deleted files. The command list open files (lsof) is helpful for this. Combine with grep I could get a list of open but deleted files. In this list I found the following:

root@EmeraldDragon:/# lsof | grep deleted
… 
rsyslogd    675           syslog    6w      REG              179,2 1420744573       1656 /var/log/syslog (deleted)
rsyslogd    675           syslog    8w      REG              179,2 5424049544       1667 /var/log/auth.log (deleted)
rsyslogd    675           syslog    9w      REG              179,2 1033132864      11460 /var/log/mail.log (deleted)
in:imuxso   675   701     syslog    6w      REG              179,2 1420744573       1656 /var/log/syslog (deleted)
in:imuxso   675   701     syslog    8w      REG              179,2 5424049544       1667 /var/log/auth.log (deleted)
in:imuxso   675   701     syslog    9w      REG              179,2 1033132864      11460 /var/log/mail.log (deleted)
in:imklog   675   702     syslog    6w      REG              179,2 1420744573       1656 /var/log/syslog (deleted)
in:imklog   675   702     syslog    8w      REG              179,2 5424049544       1667 /var/log/auth.log (deleted)
in:imklog   675   702     syslog    9w      REG              179,2 1033132864      11460 /var/log/mail.log (deleted)
rs:main     675   703     syslog    6w      REG              179,2 1420744573       1656 /var/log/syslog (deleted)
rs:main     675   703     syslog    8w      REG              179,2 5424049544       1667 /var/log/auth.log (deleted)
rs:main     675   703     syslog    9w      REG              179,2 1033132864      11460 /var/log/mail.log (deleted)

It looks like the system log daemon has several removed files open. So I gave it a restart and checked the disk usage.

root@EmeraldDragon:/# service rsyslog restart
root@EmeraldDragon:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            716M     0  716M   0% /dev
tmpfs           172M   23M  150M  13% /run
/dev/mmcblk0p2   15G  2.8G   12G  20% /
tmpfs           859M     0  859M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           859M     0  859M   0% /sys/fs/cgroup
/dev/mmcblk0p1  128M   19M  109M  15% /media/boot
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           172M     0  172M   0% /run/user/0
tmpfs           172M     0  172M   0% /run/user/1000

That was it.

Good lesson learned for a server that has run continuously for 4 ¼ years. Most people would never run into this problem because of both system restarts and large file systems. The Emerald Dragon is rather special in that regard.