Andrew Que Sites list Photos
Projects Contact
Main

July 09, 2014

Using Polynomial Regression for Interpolation

One of the reasons I started researching polynomial regression was its use for interpolation. The applications for this are limited as larger degree polynomials can easily oscillate wildly, but there are still times it can be useful. Today I found a use I want to share.

For the past several days our computer ππ has been logging the amount of light (in lux) that falls on our house's roof top. To my dismay it often crashes and I haven't figured out the cause. So gaps exist in my data. Today I got the most complete graph I've yet had, but there was a drop out just after 7:00 pm that lasted until 8:40 pm. The sun was setting during this time and all the days useful data had pretty much been logged. I am still left with a gap in my data.

To fill in the gap we could just do a linear approximation between the two points on either end of the data. When we do, our graph of the data right around the gap looks like this:

Here you can see the interpolation line in red does connect the two points closest to the gap, but doesn't follow the curve. The approximation is alright would probably be functional for what I am trying to do. However, I know I can get a better curve. This is where polynomial regression can be used. Using a polynomial function to approximate the existing data will give us a continuous function that will also fill in the missing data. Since a line won't do a good job, there is no point trying linear regression. So let us next start with quadratic regression (a 3 coefficient polynomial).

This isn't a great representation. Now cubic regression (4 coefficient polynomial).

This looks much better. The curve follows along most of the existing data points and makes a nice transition. We could stop here and calculate all the missing data points using this polynomial, but will a higher degree polynomial with more coefficients make the curve fit even better? Let us try quartic regression (5 coefficients).

This curve now runs through almost all the existing data points and is what we are looking for. At this point, the curve fits the data better than the data itself as the true data has noise. This is the curve I used to interpolate the data.

For the sake of inquiry we can continue to increase the degree of the polynomial. Octic regression (9 coefficients) produces this:

Here we start to see an oscillation being introduced into the data. While it fits and may be what the happened during the missing data period, it isn't any better than the curve produced with quartic regression.

Some strange things happen once more than 9 coefficients are used. At 10 coefficients we get this:

Simply knowing the physical phenomenon taking place, this graph isn't possible. However, it does actually have a better coefficient of determination than the lower degree polynomial. So it does make sense to select which curve to use based on it's visual attributes. The algorithm is designed simply to minimize the residual error with a given degree polynomial, and knows nothing of the restraints imposed by the data source.

This work was all done using my online polynomial regression calculator and only took a couple of minutes. A slightly modified version produced the graphs for this article as I removed the (meaningless) units and allowed larger degree polynomials.

July 08, 2014

Repairs and the Raspberry Pi Watchdog

   This is a project I started for a friend.  She had this dresser that was build by her father when she was a child.  It had fallen apart during a move and she asked if Xiphos and I might be able to put it back together.  Having looked at some pictures I could see it had been glued together, and that all of the glue joins had simply given out.  Re-glueing isn't hard so I asked her to bring it over.  I started by sanding off the old glue and piecing the dresser back together.  Happy with how things were fitting, I glued the first piece back together.  It wasn't fully disconnected and I lifted the unglued end, and poured a generous helping of glue into the gap.  After that I clamped it down and used a wet rag to remove the glue that squirted out.  Here is the skeleton of the dresser under my largest clamp as the glue dries.
   While it was a sunny day today Operation Lux crashed in the early morning hours and wasn't reset again until the middle of the day.  So most of the day's data was lost.  Not pleased looked into enabling the Raspberry Pi's build-in watchdog timer.  For those not familiar with this concept, a watchdog timer is a device that has to periodically have some condition satisfied.  If not, the watchdog will reset the device.  Under normal conditions the software is setup to regularly preform the task of updating the watchdog.  If the software should crash or stop responding, the watchdog doesn't get its update.  In this way there is a hardware backup in case software stops functioning.  Watchdogs are very common in the embedded world and I've used them many times.  However, getting the watchdog to work on the Raspberry Pi turned out to be fruitless.  After enabling the driver the system just continuously reset.  I had about 4 seconds to log in and try a command before the next reset took place.  In the end I had to pull the flash device out of the Pi and edit the config file that loads the watchdog in order to stop the resets.  I am not happy about this.  The Pi is suppose to have exactly the kind of protection I was looking for.  The watchdog interface allows a ping to be sent to an IP address, and if it is unable to ping that address will let the watchdog reset the system.

July 07, 2014

Cloudy day for data

Kelly

Kelly

   A cloudy day today, and ππ measured an average just under 7000 Lux for the 24 hour period.  That is significantly worse than I thought it would be.  To make matters worse, ππ has been locking up, producing gaps in my data.  I'm not sure where the problem comes from.  Could be power supply, the SBC, software, or something else.  However, we data.

July 06, 2014

Tracking The Cult

Andel has fallen

Andel has fallen

   Our adventure today started with some basics.  We needed a thieves guild license.  Not that anyone in our group is a thieve, but the way the town works is that thievery is so high that they made a guild such that when you buy a license you pay the wages of "honest" thieves.  A thieve who steals from you will return your money if they find your are a member of the guild.  It's kind of like being required to buy medical insurance.
   We wanted to find out more about happenings on the road and see if anyone had information about zombies and missing caravans.  At the Drunk'n Eagle we talked to a fat intoxicated merchant who just arrived in a trading caravan from Twilight City.  He talked of centaurs and wild elves, and a group of goblins who were being attacked by undead.  Returning back to Dragon's Glory Inn we received two letters.  One was from Ankus Lightfoot telling us the address of the merchant's wife we were looking for.  The other was from Olf, the cook at Marv and Elenoria's bar in Middleton giving us an uninteresting update. 
   Putting on our best clothing the group walked to the nice part of town where the merchants lived.  There we found the house and knocked on the door.  The maid said the lady of the house was currently busy and we said we would wait.  After awhile an Ork gardener who had clear dressed in a hurry walked out, followed by Freya.  We passed on the news of Freya's husband and returned what things of his we were carrying.
   The group noticed we were being followed by the city watch.  So on returning to the bar, we invited them in for a drink.  They wouldn't tell us why we were being followed but did talk about strange attacks on group in the town.  This included attacks by the undead.  They suddenly stood at attention and the constable entered.  He had a job for us.  There were a group of religious fanatics somewhere in the slums and they might be doing something related to necromancing.  We were to investigate and find out.
   Ellenoria decided to get into the slums by dressing like a Lady of Negotiable Affection.  This would allow Ellenoria, Annalis, and Andel to look like they were on business in the slums with Marv escorting them.  The group was stopped by some ruffians consisting of a human, half-ork and half-oger.  Ellenoria explained she was off duty but were looking for some cloaked figures who owed us money.  They seemed creeped out by these guys, told us were we could find the house and let us on our way.
   We found the black house and staked it out for a night from the rubble of a near by building.  People were coming and going in groups of 4.  All were dressed in black cloaks and all looked the same.  The groups split up when they left, and merged again as they returned.
   The following day Ellenoria, Marv, Andal and Dubbie went in search of a path back to town that avoided people.  They got Jam and Sparky, then returned to the house.  That day Marv, Andel, and Annilis each followed a cloaked figure to see if they where they went.  Annilis, with her excellent hiding skills, found the person went to a store and came out with a bag of flour.  Andel used her hawk Dubbie to follow her person from a good distance and  Marv's target knew they were being followed.  Ducking back, Marv tried to hide and try again, but lost his target.  Andel found her target went into a manhole, and followed.  After searching around without finding her target she encountered a slime monster.  Andel decided to fight and was knocked unconscious by an amazing dice roll by the DM.  Doobie managed to get the attention of Marv and Annalis.  Andel failed a death save just as they arrived.
   Marv made the slime monster into goo and he and Annalis returned to the hide out house with Andel's dead body.  Ellenoria made a phoenix down and with some good roles managed to brink Andel back to life.  Wanting to cash in on the opportunity Ellenoria decided to approach a cloaked figure.  She explained they had a dead body they were trying to work some necromancy on, but couldn't quite get it right.  She said she had heard the cloaked figure's order were expert nacromancers and wondered if she could learn from them.  The sham didn't work and we found ourselves in battle.

July 05, 2014

Converting Lux to Watts

Sunset in Beloit

Sunset in Beloit

   Now that Operation Lux is going there is a question: how does one convert lux data to the power generated by a solar panel?  This article will address this question.
   The first thing we need to understand are the units.  Lumens are units of total visible light, and lux is the number of lumens for an area of 1x1 meter.  Light is energy, and the amount of energy for a given amount of light is called irradiance which is typically measured in watts per meter squared (W/m2).  Converting lux to W/m2 is isn't quite one for one because irradiance is generally for a specific wavelength of light and sunlight consists of many wavelengths.  However, a scale factor of 0.0079 can be used to represent sunlight and convert lux to W/m2.
   Solar panels list their power rating in watts under Standard Test Conditions (STC).  This assumes an irradiance of 1000 W/m2 which turns out to be 126,582 lux.  That is generally considered to be an unrealistic amount of light as even in the best daylight conditions will not be this bright.  However it does give us a value at which to start.  Knowing the output and the size of the solar collector we can determine the solar panel's efficiency.  For example, I read the specifications for a 30 watt solar panel that consists of 12x 125x125 mm cells.  This gives an area of 12 * 0.125 m * 0.125 m = 0.1875 m2.  At 1000 W/m2 (as define by the STC) that means this area receives 1000 W/m2 * 0.1875 m2 = 187.5 watts of light energy.  From that amount of energy the solar panel is able to produce 30 watts of electricity, thus 30 W / 187 W = 0.16 or 16% efficiency.  This is a fairly typical value for solar panel efficiency—while they can convert light to electricity they don't convert much.
   We now have all the information we need to calculate how much power a solar panel will create at a given lux by multiplying the irradiance by the efficiency.  Say, for instance, the example 30 watt solar panel receives 15,000 lux.  That means 15000 lux * 0.0079 (W/m2)/lux = 118.5 W/m2.  Now multiply by the solar collector area: 118.5 W/m2 * 0.1875 m2 = 22.22 W.  And lastly multiply by the efficiency: 22.22 W * 0.16 = 3.555 W.  So at 15,000 lux our 30 watt solar panel will only produce 3.555 watts of power.
   The data that ππ is logging is the average lux.  If we compute the average lux for the entire day, we can compute the average wattage each day.  If the 15,000 lux example above happened to be the average for the day, that would mean 3.555 watts are available for a 24 hour period.  That is just barely enough to run the Raspberry Pi and with losses probably not enough to keep a battery charged so ππ could run during darkness.  Right now we do not know the average lux for a 24 hour period, and that is the point of this Operation Lux.  When we do have this number then it can be applied using the methods outlined here, and a solar panel selected.  So this is how you do it.  Now we must wait for data.

July 04, 2014

Operation Lux Begins

   With the housing complete and the caulk dry it was time to mount ππ on the roof.  To do this I used a scrap board and put four screws on it for feet.  The points of the screws will sink into the roof a bit to provide anchorage without going through the shingles.  A brick on this should provide enough weight the setup won't move.  Now I just needed power.  Our house has no outlets on the outside of the house, or even in the garage.  We have wanted power in the garage for a long time so I ran a feed of wire through an existing hole and put in a ground fault interrupted (GFI) outlet in the garage.  From there I ran an extension cord out a window and to the experiment on the roof.  With the additional wireless router upstairs I had no trouble getting an wireless network connection.  The experiment began to log data right away and now we just have to wait for data.

July 03, 2014

Desirae

Desirae

   The router we have in the basement has a fairly weak signal by the time you reach the second floor.  Since I had an other wireless router anyway I decided to see about adding a second source for wireless access.  Turned out to be fairly easy with DD-WRT.  Our experiment with the Raspberry Pi should benefit from this.

July 02, 2014

ππ Enclosure

   Yesterday I got the software working so ππ can log Lux over time.  Not it is time to make ππ ready to live on the roof.  I'm bad mechanically, but after talking with Xiphos some I came up with a mounting plan that should work.  ππ will go into a plastic container.  The sensor is mounted to a microscope slide using a twist tie I super-glued to the slide.  The Raspberry Pi is suspended inside a plastic container by two screws I happen to find that were small enough to get into the mounting holes.  A cut a hole into the plastic container so I could mount the slide on it, and then covered the top with caulk.  This should prevent water from coming into the housing.  I plan to leave the bottom open so air can blow past, allowing the device to stay cool (I hope).
   Pictured is the Raspberry Pi and the sensor mounted to the housing.  Once the caulk is dry I can move the computer into the housing.

July 01, 2014

ππ Measures Lux

   Now that I have all the parts I need, I decided to do some work on ππ.  Soldering took longer to gather all the pieces than to actually put on the connector.  I quick look at the pin out and I had the digital luminosity sensor connected to the Raspberry Pi.  Programming was a little more tricky.  The documentation for the device wasn't great, and the example code was written for an Arduino but a Raspberry Pi.  However, it didn't take too long to figure out how Linux deals with the I2C bus (guess what?  It's a file!).  Looking at some example code I was able to reverse engineer it so I could setup and poll the device.  Soon I had numbers on the screen that reacted to light.  And after more work I had them in the units of Lux using the example code's conversion function.  I used their function because I assumed they verified the math was correct.  I have nothing to use as a calibration test, so I didn't want to change anything I don't need to.
   After getting the input, it was time to log the data.  I setup my program to sample as fast as the device would obtain the data and average those results over a minute.  Each minute the data is saved to a CSV file.  This will allow me to see what kind of light I can reasonably expect from a location I plan to place a solar panel.  Knowing how much light is available will allow me to select a solar panel of sufficient size to power ππ.

June 30, 2014

   So today was hot and muggy, but we've managed to make it this far into the month without turning on the air conditioner.  So Xiphos and I decided we can suffer a little longer to make it July.  Then at about 5:00 pm a storm front moved in and the temperature dropped about 15 degrees.  Once cool it stayed that way and we had no problem making it until the first of July.
   Pictured is a storm from last night.  There was a tornado warning just to the west of us, but I don't know if it hit anything.  The resulting storm was almost continuous lightning and a lot of rain.