Andrew Que Sites list Photos
Projects Contact
Life Returns

Life Returns

   Enabled nightly backups of the Data-Dragon.  This is done through rsync to a Raspberry Pi 4 with 4x 10 TB USB hard drives in a RAID-5 configuration.  Initially I thought I would use wake-on-lan, but the Pis do not support it.  However, the drives spin down after a few minutes of being idle.  This is pretty much perfect as a an idle Raspberry Pi barely consumes any power.  So the drives spend most of the day powered down and wake in the evening for backups.  The Data-Dragon does this using a cron job.  I had debated on having the Pi run the cron job, but the fact it, if the Data-Dragon isn't online the cron job won't run.  I've had issues in the past with rsync removing all files because a mount failed.  This avoids that ever being a possibility.
   I noticed in the kernel logs the following message:
[   16.908469] EDAC amd64: Node 0: DRAM ECC disabled.
[   16.908473] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
                Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
                (Note that use of the override may cause unknown side effects.)
   This means the error-correcting RAM I have isn't actually being used.  My buddy Pluvius helped me look into this issue and we believe the ECC fetcher was removed in a recent BIOS update.  They had some clever but misleading verbiage.
Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules (operate in non-ECC mode)
   So you can buy ECC RAM and it will work—it just won't have any error correction.  This is more than irritating.  Pretty much all forms talking about running a ZFS file system strongly recommend using ECC RAM—which I did.  However it isn't actually working.
Horse has a Look at the Cyclist

Horse has a Look at the Cyclist

The Data-Dragon has undergone 14 drive tests, and most have tested all 13 of the 4 TB disks. A total of 648 TB (648,127,498,862,592 bytes actually) of data was written and verified as a result. Despite some setbacks caused by controllers, I have fairly high confidence these drives are functional.

So what caused the problems with the original Data-Dragon setup? Unknown. For now we will have to live with this answer. The Data-Dragon is setup with ZFS in RAIDz2, meaning it can survive a two drive failure. There is also a backup server whose drives are normally spun down. Thus there should be little chance of such a failure occurring again.

Playground Closed

Playground Closed

   I ordered another 16 GB of RAM for the Data-Dragon putting the total to 40 GB.  While 24 GB had seemed to be doing alright, I noticed the RAM usage was around 75% and I still don't have all the virtual machines running.  This should take care of that problem.
   Pictured is Parisi Park in Middleton with a sign stating the playground is closed.  All the parks I bike past have similar signs.
   Youtube at some point started displaying more irrelevant videos suggestions.  I didn't mind the suggestions when they were related to the topic I was searching for, but now it just seems the suggestions are purely ad driven.  To help filter some of the chaff I've added two filters that block youtube's "Recommend for you" and "Free with Ads". 
# Block youtube's "Recommended for you" video listings. span:has-text(/Recommended for you/)

# Block youtube's "Free with Ads" video listings. span:has-text(/Free with Ads/)
Neither of these categories have ever suggested anything I need to see and are rarely related to my search.  These filters work in uBlock Origin.
   After I finished the restore from the Backup-Dragon I needed to move the system to a new location.  Once it was moves I proceeded to destroy the LUKS header with a bad pipe command.  Now I had an inaccessable 30 TB RAID-5 array.  I have since learned how to backup the LUKS header, but that means my backup is, for all particle purposes, gone.
   I don't feel too bad about this.  The data has been restored so I won't be missing anything.  I just need to repeat the backup process.  That will take awhile, but isn't a problem.  It has been running for 4 days now and transferred 9.0 of 12.1 TB.  It is going slightly faster than the first round.  When Zach built the server, he used a USB hub.  The Raspberry Pi 4 has 4x USB ports, and this system has 4x 10 TB drives.  However, when he set things up he wanted to have a keyboard and used a USB hub so he could plug one in.  I don't need a keyboard and got rid of the hub.  Seems to have improved speeds some.
   My original plan had been to setup the backup server to actually do periodic backups.  Those scripts will have to wait a few more days.
   I hadn't noticed until I started looking at CPU flags using cat /proc/cpuinfo.  I found the Data-Dragon didn't have virtualization instructions.  This system is based on an AMD Ryzen and the entire Ryzen series has support for virtualization.  That means virtualization was disabled by the BIOS.  Finding it was a bit of a chore as the BIOS manufacturer placed it at the third level of options with names that didn't give a clue as to what they were for.  I found it in: MIT tab -> Advanced Frequency Settings (?!) -> Advanced Core settings -> Enable SVM.  Completely obvious, right?
   Today the additional 16 GB of RAM for the Data-Dragon arrived.  This places the system at 24 GB and should improve operations.
   This is the first time I've power cycled the Data-Dragon since it was restored, and as I expected, the reboot didn't go as smoothly as I had hoped.  Proxmox uses ZFS as a storage location for virtual machines/containers.  The problem is that the ZFS file system is not automatically mounted when the system starts.  That meant my VMs running from the ZFS pool were not available.  That I was expecting.  What I didn't expect was that after I mounted the file system, these VMs were still not available.
   Turns out the Proxmox adds a new pool when you add a VM using ZFS for storage, it makes a new mount point.  That's not how I expected that to work, but once I figured this out I was quickly able to restart the VMs using this ZFS storage.
   I still have much to learn about Proxmox, but I've actually been using it since 2017.  We have a build server at work I built up with Proxmox.  Although I don't use it much I learned a good deal from having to run it.
   Yesterday, well ahead of schedule, the transfers from the Backup-Dragon to the Data-Dragon completed.  Clearly reads go faster than writes.  This completion means the Data-Dragon can get back to being the data server.  Initially the Data-Dragon was only supposed to host the file system, but with the departure of Zach we lost our virtual machine host.  In addition, I was going to use a Linux software RAID-5 for hosting storage.  After some research I decided that ZFS was a much better solution for all the things I wanted to do.  In addition, the Data-Dragon must now be a virtual machine host.  The problem: I speced out the system to only use 8 GB of RAM.  That is on order, but with the virus induced lockdown it's delivery has been delayed.
   Despite not have much RAM, I was still able to setup the first VM needed by the system: drive sharing.  Most of the house uses some form of the Windows operating system.  The Data-Dragon runs Proxmox, a Linux-based hypervisor, and all of my computers simply connect using NFS.  I found that Proxmox has several container TurnKey templates, one of which is expressly designed to be a file server.  I generally like containers because they are smaller and take less resources to run.  This container was exactly what I needed to get the public areas server by the Data-Dragon accessible from the rest of the house.  Even with the limited RAM running this isn't a problem.

April 20, 2020

Happy TCD!

Emerald-Dragon 1000 days of runtime

The Emerald Dragon achieves 1,000 days of runtime today without a reboot. This machine has faithfully run DNS and e-mail for since April of 2017. It hasn’t had a reboot since July of 2017.