So a few days ago I wrote about using a Network Block Device (NBD) for doing completely secure off-site backups. This article will expand on that idea and detail setting up a complete system. Let’s begins with an overview of the setup.
The graphic above shows the setup that will be outline in this article. A Raspberry Pi 4 connected to an external USB drive makes up the backup server, but the technique would work with any Internet ready computer with a mass storage device.
The setup works by running a Linux-based computer with a NBD server sharing an attached hard drive over an SSH tunnel. This allows a client computer to open an SSH tunnel to the device and connect to the hard drive as a block device. Linux Unified Key Setup (LUKS) is used to encrypt the drive at the block device level.
The encryption/decryption of the data stored on the disk happens not on the remote backup system that has the disk connected, but the client to backup. At no time does the backup system have any sensitive information. This system allows the backup system to safely operate in an insecure locations. Should the backup system be stolen or hacked the data is still secure. Even if the attacker could control the backup system they would have no advantage getting access to the plaintext data saved on the disk.
There are several steps in setting up such a backup system. Most of the steps will be outlined in this article. Items that will not be covered:
- How the IP address of the remote machine is maintained. It is assumed that someone implementing this can handle getting the address to the remote machine.
- Router setup to map the SSH port from the Internet to the backup system.
In addition, familiarity with setting up a Linux-based system is assumed, as well as basic knowledge of the Raspberry Pi.
Commands will need to run on both the backup system and the client to backup. These commands are distinguished by the two prompts:
Commands running on the backup system will start like this:
And commands on the client like this:
Step 1: Make an operating system SD card
For the Raspberry Pi we will use Raspberry Pi OS Lite. No monitor is used and thus no graphics are required for the backup server. We will skip installing the operating system but assume it installs with SSH enabled. See this article for how that is done. Nothing precludes using a Raspberry Pi with full desktop installed, just make sure to install an SSH server.
Step 2 Repurpose backup user (optional)
Debian-based systems (including the Raspberry Pi) have a default user called backup. We can repurpose this user to actually perform backups. By default, the backup user cannot login in any manner. We want to allow this user to be able to establish an SSH tunnel.
The backup user needs a home directory so to hold SSH configuration. We could use the standard /home directory, but since this user can’t actually login and only needs an SSH key, we can put the home directory elsewhere. The default home of backup is /var/backup. This directory contains some backup files and probably best to use another path. The directory /var is typically for variable data, and this user’s home directory will never change. I will use /etc/backup as /etc is for configurations files and our user data is more of a configuration. It doesn’t actually matter what path you choose.
backup@backup-server:~ $ sudo mkdir -p /etc/backup/.ssh
backup@backup-server:~ $ sudo chown -R backup:backup /etc/backup
backup@backup-server:~ $ sudo chmod 700 /etc/backup/.ssh
Modify the backup user to use this directory:
backup@backup-server:~ $ sudo usermod -d /etc/backup backup
In order for the backup user to create an SSH tunnel we need to give it an authorized SSH key. This allows the client computer to establish the tunnel as the backup user. The key should be for the user doing the backups on the client. If backups are a cron job, the root used can be used. The public key is stored in ~/.ssh/ and typically have the name id_*.pub with the actual name depending on the key type. We’ll assume an ed25519 key type.
user@client:~ $ cat ~/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBNh7YrOdzl30lFHDNFUkSt2+cICYmEp6eKsGR/rQ0w7 root@computer
Copy this key and add it to the authorized keys on the backup server.
backup@backup-server:~ $ sudo echo \
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBNh7YrOdzl30lFHDNFUkSt2+cICYmEp6eKsGR/rQ0w7 root@computer" \
backup@backup-server:~ $ sudo chmod 600 /etc/backup/.ssh/authorized_keys
We should now be able to connect to the backup server using the backup user, but not get a prompt.
user@client:~ $ ssh backup@backup-server
If successful you should get the banner followed by the text
This account is currently not available.
Connection to backup-server closed.
This is exactly what we want. The backup user will allow an SSH tunnel to be established, but have no other abilities.
For added security we could limit the client computer so it can only forward the NBD port by adding the prefix permitopen="localhost:10809" to the line in the authorized_keys file. The file would look something like this:
permitopen="127.0.0.1:10809" ssh-ed25519 AAAA… root@computer
Without this the network on the remote computer is exposed to the backup client computer via tunneling.
Step 3: Disable non-SSH logins (optional)
Since we will be using this device over an SSH tunnel there is no need to ever allow normal logins. If the backup user from step 2 was repurpose, the next line can be skipped. Otherwise we first have to get the client’s SSH key to the Pi.
user@client:~ $ ssh-copy-id <backup server>
This will make it so the client computer (i.e. the computer that runs the backups) is recognized by the Pi. Verify this works by connecting via SSH. If there is no prompt for a password, it worked.
Now delete the passwords for both root and the backup user.
backup@backup-server:~ $ sudo passwd -d root
backup@backup-server:~ $ sudo passwd -d <backup user>
This will make it so you can no longer login using a terminal—you must use SSH from the client computer.
Step 4: Setup the NBD server
Install the Network Block Device server:
backup@backup-server:~ $ sudo apt install nbd-server
Configure the server:
backup@backup-server:~ $ sudo cat <<EOT >> /etc/nbd-server/config
listenaddr = 127.0.0.1
exportname = /dev/sda
backup@backup-server:~ $ sudo service nbd-server restart
We assume the attached disk being used for backups is /dev/sda which is a safe assumption for single drive connected to a Raspberry Pi. If the backup server has more drives to share, then they can be added the /etc/nbd-server/config file. In addition, since NBD shares block devices, mdadm RAID-arrays could be shared.
Step 5: Connect to NBD
Install the Network Block Device client on client computer.
user@client:~ $ sudo apt install nbd-client
user@client:~ $ sudo modprobe nbd
Start an SSH tunnel:
user@client:~ $ ssh -f -N -L 10000:127.0.0.1:10809 <backup server>
Attach to block device:
user@client:~ $ sudo nbd-client 127.0.0.1 10000 /dev/nbd0 -N usb
If this worked, the device /dev/nbd0 will now be ready for use. If it is already a formatted drive it can be mounted. For the next section we will assume the drive is empty and not yet setup. To have encrypted backups, LUKS needs to be setup on the disk. Otherwise, there isn’t much of a point to doing backups in this way.
Step 6: Partition Disk
If the disk is already setup with LUKS, you can skip this setup. Otherwise, we need to setup the disk. This article assumes we start from step 5 with the backup server now tunneled to the client and the NBD connected. These steps would also work if the drive were connected directly to the client computer, albeit using different block device paths, which is useful for setup and initial synchronization.
First we need to create a key. You can use a password if you like, but a key file is better. Backups are typically automated and a key file on the client can be used to mount the remote drive as part of the backup script. We will assume the client computer is secure, but the backup server is not. Thus we just store the key in plain text on the client computer.
user@client:~ $ dd if=/dev/random bs=32 count=1 > backupKey.bin
This will generate a random key file called backupKey.bin. After creating this file, keep it safe. If you lose this file, there is absolutely no way of recovering your backup data. Placing it on a USB disk and store it in a fireproof box or safe deposit box. You could encrypt it and place it on the backup computer, cloud storage, or even e-mail it to yourself (assuming you have web-based e-mail).
In this instance, the key file is 32 bytes, or 256 bits. There is no reason it needs to be this size and could be larger. However, 256 bits is a typical block cipher key size and should be sufficient.
Now use this key to setup LUKS on the remote drive:
user@client:~ $ sudo cryptsetup -v luksFormat --key-file=backupKey.bin /dev/nbd0
This will setup the encrypted disk. Now we need to open it and create a file system.
user@client:~ $ sudo cryptsetup open /dev/nbd0 backupDrive –key-file=backupKey.bin
user@client:~ $ sudo mkfs.ext4 /dev/mapper/backupDrive
In order to use this disk we need a mount point. We will make one in /mnt/backupDrive.
user@client:~ $ sudo mkdir /mnt/backupDrive
user@client:~ $ sudo mount /dev/mapper/backupDrive /mnt/backupDrive
The only thing that might need to be done is changing the owner of the backup drive. Depends on if you are doing backups as root, yourself, or a backup user. In this case, the backup drive mount will be owned by the user backup.
user@client:~ $ sudo chown backup:backup /mnt/backupDrive
That’s it. The disk is now ready for backups.
Step 7:Initial Backup
For backups I typically use rsync. The first pass could take awhile, but subsequent runs will only update files that have changed. The command looks like this:
user@client:~ $ rsync -av --delete /<source path>/ /mnt/backupDrive
Best to do this and leave it for awhile such as letting it run overnight.
As this point the backups to the backup system are possible. In the next article we will cover automating the system.