Andrew Que Sites list Photos
Projects Contact
Main
   The Red-Dragon typically wakes up each morning, does backups, and turns back off.  Sometimes, however, it fails to establish a connect to the network and just sits there.  I was looking for a script that could check to see how long the system was idle, and then shutdown.  While I found some options, the problem I ran into is that using rsync doesn't count as activity.  So instead I simply made a script that turns the computer off after an hour.  Even or worst case backups only take an 45 minutes, so this should be fine.

July 07, 2020

TW2002 Python Script

   Success in writing a Python telnet script that is able to log into TW2002 and prune planets. Lot of trial and error until I discovered the prompts can mess things up.  Python's telnet library takes some getting used to.  You can have it write something to the telnet port and then wait for a prompt.  There are two things to be careful about.  One, the prompt you are waiting for must be the first occurrence.  That means no stringing together a bunch of commands that could produce the same prompt, as the first instance of that prompt found will satisfy the wait.  Second, TW2002 uses ANSI color codes.  So you need to have your prompt text all of the same color, or a regular expression that takes into account the color change. 
   Once I had that straightened out, I was able to write a script that queried all the player's planets, and then pruned them.  In TW2002, each planet has a maximum number of colonists.  Half that maximum is the optimal number for maximum production.  But colonists reproduce and populations rise.  So you need to move people around in order to keep the planets optimal.  Now I have a script that can do this busy work.
   The script is useful but more as something to help learn the telnet library of Python.  My plans are to make a few more scripted actions as a way to have a script gathering information I can use to better understand the trading algorithm.  And for that, I need to be able to automate trading.

My UPS always reports a communications failure shortly after system start which is broadcast to all terminals.

Broadcast message from root@data-dragon (somewhere) (Sat Jun 27 15:55:16 2020):
                                                                               
Warning communications lost with UPS     

After a quick search I found solution. Edit the file /etc/apcupsd/apccontrol and change WALL=wall to WALL=logger. Now instead of broadcasting the message to all terminals, the message will be placed in a log file.

   The other day while compiling PHP on the Web-Pi, I ran it out of memory and nearly locked it up.  Need to be careful using the -j option in make on smaller devices as finite memory is much more apparent.  Still, although the system was at a crawl and not very responsive, it did eventually respond to the crtl-break, stopped the compile, and recovered fully.  That cannot be said of other systems I've run out of memory.
   The first item of fallout I had with the Web-Pi was JPEG images.  I use a very old Captcha to fend off bots, and it uses JPEG.  Turns out when I compiled PHP 5, it didn't actually find the JPEG library but didn't fail to configure either.  It took a good deal of trial and error but I finally got the JPEG library included.  I needed --with-jpeg-dir=/usr/lib in the configuration line.

July 03, 2020

Web-Pi hosting DrQue.net

Web-Pi and Ethernet Switch

Web-Pi and Ethernet Switch

   About a week ago the Web-Pi took over DrQue.net form the Sun-Dragon.  This isn't permanent, but will verify I got all the services transferred and functional.  Hopefully I didn't miss anything but so far so good.
   Pictured is the Web-Pi next to a completely full 24-port Ethernet switch.  We have a lot of computers in the house, and these connections don't cover the computers that are wireless.

July 01, 2020

Bounded Variable Sigmoid Mapping Function for Curving Random Numbers

So the other day I wrote about a curving function for random numbers that can weight them more toward high or low values. What if we want numbers more concentrated at a center? We can use a Box-Muller transform and get numbers with Gaussian distribution. The problem with that is the numbers are unbounded. That is, there is no limit to how much derivation from the center there can be. We want a fixed upper and lower limit. I've tried doing this in the past with a ranged Box-Muller transform but it isn't quite as configurable as I would like. Then I ran across this post on Stack Exchange. It is a variable Sigmoid function bound between 0 and 1. Here is an implementation:

Coefficient: unknown.
Standard deviation: unknown.

This is a fantastic function. For coefficient values less than 1 and greater than zero, the function exhibits Gausian distribution. This produces numbers around a center point (0.5) that can deviate to some maximum on either side (0 or 1) with control over how likely deviation (something akin to standard deviation). However, it is more versatile than that. For coefficients greater than 1, the function produces reverse Gaussian distribution, pushing the results to the sides and making the middle less likely. There are applications for both. However, let's first look at the transform function. The Stack Exchange post gives it as:

y = 1 - 1 1 + ( 1 x - 1 ) - c

But I like this form better as it is easier to read:

y = 1 1 + ( x - 1 - 1 ) c

It is a bounded function but is actually undefined at the end-points. I could see why x=0 didn't work (divide-by-zero), but it wasn't initially obvious why x=1 didn't work. This identity makes it more obvious:

x c = e c ln ( x )

So if x=1, the value inside the parenthesis becomes 0, and you cannot take the natural log of 0 as it is undefined at 0.

Looking at the function it is easy to deduce that at the end point y=0 when x=0 and y=1 when x=1, but we can prove it mathematically:

lim x 0 + 1 1 + ( x - 1 - 1 ) c = 0 lim x 1 - 1 1 + ( x - 1 - 1 ) c = 1

So a complete definition would be:

y = { 0 x = 0 1 1 + ( x - 1 - 1 ) c 0 < x < 1 1 x = 1 { x : 0 = x = 1 } { c : c > 0 } { y : 0 = x = 1 }

Now there must be a relation between the coefficient in this equation and the standard deviation. Sadly, I have was unable to find it before publishing this article.

June 30, 2020

Python key handler thread

I've been using Python for a lot of projects at work, and one problem I've had is dealing with the keyboard.  Typically I just want to be able to read a key and trigger some event when that happens.  Just reading a single key press isn't trivial.  There are plenty of write ups about how to change the terminal into single character mode, and that does work.  But what if you don't want the read to block so you can do an event driven system? 

In the past I have simply made a thread that puts the terminal into non-blocking single character mode.  The body of the thread reads the key.  If there isn't one, the thread sleeps.  If there is one, the key is read and dispatched.  The problem with this approach is the sleep time because it limits how fast keys respond.

I tried a different implementation the other day I'm sharing here.  For Linux machines it is more efficient because it is still blocking.  However, rather than block by just waiting on the terminal (stdin) it waits for either the terminal or a temporary file.  This is accomplished through the use of select which can block until an event occurs on one or more files.  That eliminates the need for an arbitrary sleep.

#!/usr/bin/env python3
#------------------------------------------------------------------------------
# Uses: Thread for reading the keyboard and placing characters in a queue.
# Date: 2020-02-09
# Author: Andrew Que <https://www.DrQue.net/>
#
#                              (C) Copyright 2020
#------------------------------------------------------------------------------

# For reading key in non-blocking mode.
import sys
import os

import select
import tempfile

# For thread.
import threading
import time
import queue

class KeyboardThread( threading.Thread ) :
  """
  Thread for handling single character keyboard input.

  Input:
    shutdown - Instance of `threading.Event` used to signal the thread to shutdown.
    keyQueue - Instance of `queue.Queue` where keys that have been pressed are placed.

  Queue:
    New keys are placed in the queue as they arrive.  When the thread shuts down
    `None` is placed in the queue.  This will unblock queue and there is no
    other way for `None` to be enter the queue, thus safe to assume `None` is
    a shutdown event.
  """

  #-----------------------------------------------------------------------------
  def __init__( self, shutdown : threading.Event, keyQueue : queue.Queue, start=False, *args, **kwargs ) :
    super().__init__( *args, **kwargs )
    self._shutdown = shutdown
    self._keyQueue = keyQueue

    # We use a temporary file (a pipe) to shutdown the thread.
    # Writes to the temporary file unblock the thread.  So to end the thread
    # we simply write something to the temp file.
    self._shutdownFile = os.pipe()
    threading.Thread( target=self._stopBody ).start()

    if start :
      self.start()

  #-----------------------------------------------------------------------------
  def _stopBody( self ) :
    """
    This function simply waits for the shutdown even and then unblocks the
    keyboard thread by writing something to the shutdown file.
    """

    # Wait for shutdown.
    self._shutdown.wait()

    # Signal the keyboard task by writing dummy data to the shutdown file.
    os.write( self._shutdownFile[ 1 ], b"\0" )

  #-----------------------------------------------------------------------------
  def run( self ) :
    try :
      self.runLinux()
    except ModuleNotFoundError :
      self.runWindows()

  #-----------------------------------------------------------------------------
  def runLinux( self ) :

    import tty
    import termios

    # File descriptor of stdin.
    # (Likely this will always be 0.)
    fileDescriptor = sys.stdin.fileno()

    # Set terminal in raw mode.
    savedAttributes = termios.tcgetattr( fileDescriptor )
    tty.setcbreak( fileDescriptor )

    try :
      # Wait for data from either `stdin` or the shutdown file.
      readDescriptors = [ fileDescriptor, self._shutdownFile[ 0 ] ]

      while not self._shutdown.is_set() :
        # Wait for a data.
        [ reading, writing, exceptional ] = select.select( readDescriptors, [], [] )

        # If this was a key press...
        if fileDescriptor in reading :
          # Read and queue the new key.
          # (Could be more than one character for the case of special keys)
          character = os.read( fileDescriptor, 8 ).decode()
          self._keyQueue.put( character )

    finally :
      # Restore terminal settings.
      termios.tcsetattr( fileDescriptor, termios.TCSANOW, savedAttributes )

      # Unblock queue by adding `None`.
      self._keyQueue.put( None )

  #-----------------------------------------------------------------------------
  def runWindows( self ) :
    import msvcrt

    while not self._shutdown.is_set() :
      if msvcrt.kbhit() :
        # Read and queue the new key.
        character = msvcrt.getch().decode()

        if "\0" == character :
          secondCharacter = msvcrt.getch().decode()
          character += secondCharacter

        self._keyQueue.put( character )
      else:
        # Come back latter.
        # 100ms is rather arbitrary.
        self._shutdown.wait( 0.1 )

    # Unblock queue by adding `None`.
    self._keyQueue.put( None )


#-------------------------------------------------------------------------------
if __name__ == '__main__':

  import string

  # Escape key value.
  ESC_KEY = chr( 27 )

  # Event to signal shutdown.
  shutdown = threading.Event()

  # Queue to hold keys.
  keyQueue = queue.Queue()

  class KeyUnload( threading.Thread ) :
    def __init__( self, shutdown : threading.Event, keyQueue : queue.Queue ) :
      super().__init__()
      self._shutdown = shutdown
      self._keyQueue = keyQueue

    def run( self ) :
      while not self._shutdown.is_set() :
        key = self._keyQueue.get()
        if ESC_KEY == key :
          print( "Shutdown key pressed" )
          self._shutdown.set()
        elif key is not None :
          if key[ 0 ] == chr( 27 ) :
            print( "Special key code '" + key[ 1: ] + "' pressed" )
          elif "\n" == key :
            print( "Enter key was pressed" )
          elif key.isprintable() :
            print( "The '" + key + "' key was pressed" )
          elif len( key ) > 1 :
            print( key )
          else:
            print( "Character", ord( key ), "received." )

  # Thread to unload key presses.
  keyUnload = KeyUnload( shutdown, keyQueue )
  keyUnload.start()

  # Keyboard thread.
  keyboardThread = KeyboardThread( shutdown, keyQueue )
  keyboardThread.start()

  # Timer to shutdown the example after 3 seconds.
  timer = threading.Timer( 3.0, lambda: shutdown.set() )
  #timer.start()

  # Wait for shutdown.
  print( "Hit ESC to stop, or wait 3 seconds." )
  shutdown.wait()

  print( "Shutdown" )

  # Stop timer.
  timer.cancel()

  # Wait for keyboard thread to finish.
  keyUnload.join()
  keyboardThread.join()

  print( "Done" )

What we have is a thread that takes an Event as a parameter.  The event is used to signal the thread to shutdown, typically when the program ends.  We create a pipe which is just a temporary file for which we can use select.  The body of the thread setups up the terminal for blocking single character mode.  Then it enters a loop that waits for a character from either the terminal or the shutdown file.  If there was a character from the terminal, it is placed in a queue for consumption by some other process.  A second thread is spun up that simply waits for the event to be set, and writes something to the shutdown pipe.  That will cause the main thread to unblock and see that it is time to shutdown.

This efficient process only works for POSIX system.  The class has a Windows version but it uses the less efficient timed out block method.  The whole things seems overly complected, but is completely functional.  Maybe it will help someone else having the same issue.

June 29, 2020

Function for Curving Random Numbers

Your typical Pseudo-Random Number Generator (PRNG) will generate uniform random numbers—.  That is, equal odds of any number.  This is often normalized so the random numbers are between 0 and 1.  There are times we do not wish equal odds, but rather some known odds.  One way to do this is through a mapping function that will curve the numbers.  Here is a simple curving function demo:

 

 
 
Curve position: unknown
Coefficient: unknown
Standard deviation: unknown

 

The circle can be moved around to modify the curve and it also depicts odds at that point.  Try dragging it around.  The curve will always pass through the center of the circle by generating a coefficient.  The plot below the main graph is a histogram, showing the distribution of values.  Unless the distribution is uniform (a coefficient of 1) more values will by on one side as opposed to the other.

For example, if the point is at 0.4, 0.3, that means the 40% of all the random numbers will be below 0.3.  Conversely, 70% are above 0.3.  Since this is a one-to-one map transform we can simply plug in a normalized random number to generate a weighted random result.  Since the values are normalized we can scale it up (by multiplying the result by an amplitude constant) and/or offset it (adding a constant).

The function for the mapping is very simple.

y = x c { x | 0 = x = 1 } { c | 0 < c }

This function will map any x value between 0.0 and 1.0 which is the range of normalized PRNG.  The coefficient is any positive real number above zero.  This allows the output to be weighted more toward smaller or larger numbers.  To use the method of applying a fixed-point (as demonstrated by the circle on the plot), the following equation can be used:

c = ln y ln x

Where x and y are the coordinators of the point on the curve.  This will guarantee a coefficient that  passes through the curve.

2 comments have been made.

From Noah

July 03, 2020 at 1:50 PM

Very interesting! The 1:1 mapping here makes a lot of sense, and I like how easy it is to just normalize across a new range.

I wonder if more complex probabilities could be created the same way - for example a normal distribution? Binomial? Rayleigh? As long as it's continuous there could be a lot of interesting ways to mess with PRNG output, should you need it.

From Andrew Que (http://www.DrQue.net)

Middleton, WI

July 03, 2020 at 3:32 PM

Funny you should say that.  Working on a follow-up article that talks about this.