Andrew Que Sites list Photos
Projects Contact
Main

April 10, 2022

RAM upgrade

64 GiB RAM

64 GiB RAM

When I built the Snow Dragon I thought 32 GiB would be plenty for RAM for most of my day to day operations. However, applications keep growing larger and I need to run larger virtual machines.

Programs like Atom are Electron-based which means they are basically a fully contained web browser and everything that entails. While functional they are hardly efficient. For work I need to run a Windows 10 virtual machine for a project in order to use the client's VPN. The client prepared the VM which includes their virus protection and everything else their IT department wants. I had initially setup this machine to use 4 GiB of RAM, but really it wants more. And my parallel Python scripts are not easy on RAM either.

Finding more ECC RAM during the chip shortage wasn't an easy task. Pluvius found me some and I ordered another 32 GiB. Now the Snow Dragon runs with 64 GiB. That should improve my working situation.

April 09, 2022

SQL Optional Relation

While I have used MySQL/MariaDB since 2005, I am by no means an SQL expert. At some point in the new gallery scripts I added a relation to the gallery table. Each gallery can have a thumbnail. Internally the way I set this up was to use the hash of the thumbnail in the thumbnail column. That information allows the thumbnail to be displayed without any additional information. However, I wanted to know the thumbnail size so the space could be reserved before loading the image. My current gallery keeps changing the page length as images are loaded, and it is annoying when searching by date, which are listed at the bottom, to have the page constantly moving until all images are loaded.

So I added a relation so the thumbnail could be load the thumbnail image details and add that into the columns returned.

SELECT
  *,
  GalleryTable.id AS id
FROM
  GalleryTable,
  PictureTable
WHERE
  GalleryTable.thumbnail = PictureTable.hash

However, this failed to return rows where there isn't a thumbnail. That's because there is no relation in that case. My workaround might not be the best, but is functional.

SELECT
  GalleryTable.*,
  PictureTable.width,
  PictureTable.height
FROM
  GalleryTable,
  PictureTable
WHERE
  GalleryTable.thumbnail = PictureTable.hash
UNION ALL
SELECT
  *,
  NULL AS width,
  NULL AS height
FROM
  GalleryTable
WHERE
  thumbnail IS NULL

In this setup we are doing two different selects. The first is the relational select, which will cover all the galleries that have thumbnails. The second is a select that only selects records that do not have thumbnails. A union requires that both queries have the same columns, so for the second query I just add columns for width and height that have null for the value.

Now I should have switch to using the picture id rather than the picture hash in the gallery table. The result is that the query for the galleries is quite slow. To speed it up, I turned the thumbnail column into an index. The query went from 4 seconds to tens of milliseconds. That makes sense, but I was surprised at just how much of an improvement this made.

I should switch from using the hash to using the picture id for the thumbnail. However, I don't even want to use an SQL database. I'd like to switch to something like MongoDB. Then if I make that switch, I should get ride of PHP and make the gallery run in NodeJS. I'm undecided about doing a complete rewrite of the server-side.

Early Spring Snow

Early Spring Snow

   With the work completed yesterday on the new gallery script, I decided the next step was to create an all new gallery reference set.  This includes all the thumbnails and full-size files.  After I had it written I left the script to run.  315 minutes (5 hours, 15 minutes) is was complete.  171,328 files spanning 148.5 GiB of space.
   In the evening I worked on importing all this data into the new gallery scripts.  There was a fair bit of work to do, but I ended a very late evening with a mostly ported gallery.  That should be a win, however...
   I'm not happy about the performance of the new gallery.  Loading the gallery is slow, and the incremental loading of large image sets creates a UX I don't like.  It is not quite as functional as my old gallery, and I will have to work on that.
   First breakfast with people for several weeks, and the first time I've biked to breakfast this year.  It has been raining a lot lately—it is April after all—and I wasn't sure if I would be able to bike in.  However when I looked outside after I got up I saw that there were actually snow furies.  I don't like biking in cold rain, but I can bike when it is snowing just fine.  We had a little snow accumulation, but it didn't last the day.  Weather will warm up some over the next couple of days.

April 07, 2022

Getting Complete List of Gallery Photos

The last time I stopped working on the new photo gallery scripts was because the difficulty of importing my old gallery. There are over 30,000 pictures from 3,000+ different dates and grouped into 100+ galleries. There are a few problems. My original gallery script worked with a mixture of thumbnail sizes. In the beginning I had a mixture of mainly two aspect ratios: 4:3 for my older images and 3:2 for my DSLR photography. My older images were captured for standard screen sizes (320x240, 640x480, 1024x768, ext). When I did thumbnails and available download sizes, I would use this 4:3 scale. For my DSLR photography I started using the base size of 300x300, meaning the image could not exceed 300 pixels in width or height. This typically meant thumbnails of 300x200 for landscape and 200x300 for portrait oriented shots. I started with the bounding pixels sizes of 75, 150, 300, 600 for thumbnail sizes, latter adding 900, 1800 and then 3600 and full size.

The way the thumbnails are stored is by the bounding size limits. For example, 300x300 for images where the maximum width or height must be 300 or below. The older images were often landscape only, so I also had directories like 320x240. To represent full-size images I used 9999x9999. Sometimes there would be mixes, like: 75x75, 150x150, 320x240, 640x480.

The photo gallery was written about a year after I started shooting DSLR and I didn't yet have many pictures. Thus the work to incorporate those older images not in 4:3. Now some 17 years latter, 4:3 is the overwhelming majority of my pictures. There is no reason to keep a mix of aspect ratios since we have the full-size version. A 320x240 picture can still have a 300x300 limited thumbnail, and space is no longer a concern. The new gallery will keep a smaller set of thumbnails and be able to resize the full-size image dynamically if desired.

The main problem is tracking down all the full-size versions of pictures. The first complication is that when I first started doing digital photography, I typically would save in the PaintShop Pro format (PSP). Nothing but PaintShop Pro can read this format. By 2005 I stopped using PSP and switched to saving in lossless JPEG, or if I used layers, I would save in both formats. At some point more recently I wrote a script to convert all the PSP images to lossless JPEG, so I do have a common format for all the images.

Another problem is that I didn't start including full-size images until around 2011. This isn't too much an issues since most of the images come from my photography directory and there is a 1:1 correlation between the web directory and the source. However, that isn't always the case. Sometimes I have cellphone camera pictures, or pictures taken from friend's cameras.

To solve this problem, I started by exporting the entire database of images to a JSON file. Then I could parse this file and find sources for each of the images. If a source wasn't found, I could list the problem image and address it. There were several dozen, but it wasn't actually that bad. In my first round, I just had to create links to full-size version for the old aspect ratio images. There were also missing images. Many of these were renames or images that ended up in the gallery by accident. It took a bit of sorting to straighten that all out, but I now have an image gallery that has a 1:1 translation to full-size source images. That means I can use this information to index the new gallery.

The new image gallery is very much vaporware at this point, but I keep edging closer to making it a real thing.

April 06, 2022

New ISP

Tea

Tea

   DrQue.net had a short period of downtime this afternoon as Elmwood Park switched ISPs.  The DNS change happened pretty quick as within 30 minutes a had a coworker able to grab files from my site.  We are now (back) on a fiber with speeds of 1 Gbit/sec in both download and upload.  That's a significant improvement over the coaxial cable connection we had before.  Speed tests showed us holding over 870 Mbit/sec in both directions, although how accurate such tests remains in question.  ISPs cheat by using traffic shaping to give priority to speed tests sites and throttle speed for everything else.  What we actually achieve remains to be seen.  Still, happy to be back on fiber.
   Occasionally nemo lags for a long time when opening a new window. Some searching found this post. The workaround of running killall gvfsd-trash does seem to fix the issue.  The more I run Linux, the more I find little quirks like this.  All large dynamic software has such bugs, and finding workarounds until the issue is fixed is nice.
   Dinner with Pokie this evening.  We tried a new Russian restaurant that was wonderful.  Pictured is my tea.
Andrew Que, April of 2022

Andrew Que, April of 2022

   Work wanted a picture that wasn't the back of my head.  Canon's EF 85 mm f/1.2L is considered thee portrait lens.  I don't have one, but it does seem that 85 mm is the desired focal length for portraits and my EF 28-300 mm f/3.5-5.6L will have no problem shooting at 85 mm.  My lens won't have the super shallow depth of field but I was pretty sure the pictures would be nice nonetheless.
   For the setup I tried out three basic LED lighting fixtures I picked up some time ago.  They have the ability to tune the color temperature and were really meant for webcams.  Still, I was pretty sure they would work for this setup as well.  I used the area I have my bike trainer setup.  My camera plugs into an HDMI TV and I have one for watching Star Trek while I ride.  This would allow me to see myself as I setup the shot.  I hung a mostly black blanket for a backdrop, turned off all other lighting, and took several shots.  I think the results are an accurate depiction of what I look like.  The lighting isn't perfect and I have more shadow on one of my eyes than the other.  Still, for a quick portrait, I think it turned out well.
 
SLS on pad

SLS on pad

   Not a great picture because it was taken from a moving bus, but this is the Space Launch System (SLS) on the pad 39B.  It was scheduled to undergo a wet-dress rehearsal today, but that was scrubbed after problems with the ventilation fans on the Mobile Launcher Platform (MLP).  This should only delay the test by a day, but SLS is behind schedule and severely over budget.  A single launch is estimated to cost $2 billion.  We have yet to see if it is able to deliver.
AC Fog

AC Fog

   Return trip from Florida turned into a something longer after the only two storms in the country were over northern Florida and Chicago.  We spent a couple hours sitting on a plane parked on a taxiway.  Observed an interesting phenomenon.  The plane, and Airbus A320, had a powerful air conditioning unit that kept the plane so cool I had to get out my sweatshirt.  I overheard a flight attendant say the temperature at the back of the plane was 63°F/17°C because something wasn't set correctly.  Condensation was forming at the vents and at one point was so heavy the plane looked foggy.  I've since read that the way aircraft cooling systems work is by compressing air, cooling it, then decompressing it.  The drop in pressure creates cold air.  This cold air is then mixed with outside air to obtain a desired temperature.  The decompressed air is likely to have lost its moisture in the expansion process, but mixing back in the humid outside air is what likely caused the condensation.  The outside air had high humidity, but when mixed with the cold air was unable to hold that humidity, causing the water vapor to condense into clouds of fine water droplets.  Neat effect.
The Orbiter Atlantis

The Orbiter Atlantis

   Fairly busy day.  Visited the Kennedy Space Center in Cape Canaveral.  Pictured is the space shuttle orbiter Atlantis.  I've seen Discovery, Independence and now Atlantis.  I like this display of the shuttle the best.  Both Discovery and Independence are really only visible from underneath.  At Kennedy, they have hung the orbiter at an angle with the bay doors open, and there is a multi-tiered balcony for viewing.  This allows a much better view of one of the most complex machines every built.