Ran into an interesting artifact with some Python code today.  I've been using the multiprocessing unit to run processes in parallel.  These processes needed a large dictionary as part of their operation, but the size of the object was slowing down the script because it was duplicated for each running process.  I found I could create shared objects using multiprocessing.Manager which allowed the large dictionary to be shared rather then duplicated.  Oddly it had another side-effect.  When using them my print statements inside the running processes were no longer getting piped on the command line.  Confused I narrowed down the problem to the following:

#!/usr/bin/env python3

import sys
import multiprocessing

#------------------------------------------------------------------------------
def _process( name ) :
    print( "Process", name )

#------------------------------------------------------------------------------
if __name__ == "__main__" :

    _PROCESS_COUNT = 4

    # Creating an instance of `manager` data causes prints not to work.
    manager = multiprocessing.Manager()
    shared_data = manager.list()

    work = range( _PROCESS_COUNT )

    print( "Processing..." )

    with multiprocessing.Pool( _PROCESS_COUNT ) as pool :
        pool.map( _process, work )

    print( "Done." )

Now executing this without any output redirection works as expected.

$ ./test.py 
Processing...
Process 0
Process 1
Process 3
Process 2
Done.

However, if the output is redirected, the print statements from the _process function no longer come through.

$ ./test.py | cat -
Processing...
Done.

I was baffled.  Redirection should have no effect on the output.  I decided to reach out to the Stackoverflow community.  In the comments I got some feedback asking if I tried flushing the stdout after printing.  Honestly I didn't think that was going to work but gave it a test.  Sure enough, it did work. 

#!/usr/bin/env python3

import sys
import multiprocessing

#------------------------------------------------------------------------------
def _process( name ) :
    print( "Process", name, flush=True )

#------------------------------------------------------------------------------
if __name__ == "__main__" :

    _PROCESS_COUNT = 4

    # Creating an instance of `manager` data causes prints not to work.
    manager = multiprocessing.Manager()
    shared_data = manager.list()

    work = range( _PROCESS_COUNT )

    print( "Processing..." )

    with multiprocessing.Pool( _PROCESS_COUNT ) as pool :
        pool.map( _process, work )

    print( "Done." )

Now I'm more perplexed.  Why would one need to flush the output if using shared objects? If the shared_data line is removed, the output also works.  However, I have a working solution.

After my success in getting my Waveshare CoreH7XXI single-board computer’s SDRAM running it was time to get the sound running. I had previously accomplished this streaming sound from a single-board using an STM32 F4 micro-controller. It was time to port that to the H7.

While it didn’t take long to get the code ported, it wasn’t working at all. The PWM channel was working fine, but I was unable to get DMA to feed in the waveform. After some tracing I found I was getting a Transfer Error Interrupt from the DMA controller as soon as launched the DMA request. After some digging, I found out why. There are 5 RAM regions on the H7. By default, data is placed in a 128 KiB region starting 0x2000000. DMA is unable to access this region and I had to use an area marked RAM_D2 starting at 0x3000000.

That made sense and I modified the linker script so I had a way to specify this memory region. However, I found two linker script: one for RAM and one for flash. So my initial attempt failed because I modified the RAM script. I assume this is meant for running the code out of RAM. I never need to do this so I added the same lines to the flash script. That did work, and so did the DMA.

This was a longer walk than expected. Typically STM controllers are pretty straightforward, but the H7 is more complex and hence has a couple of gotchas like this. Now that I know it shouldn’t be a problem working around it.

Back in February of 2010 I wrote a simple program to count all the prime numbers between 2 and 2³².  It use pthreads so I could fully utilize the duel processor Red-Dragon.  The C11 standard introduced threads right into the language.  Several years ago I thought I'd port my prime counting program to use this because any system that supported the full C11 standard would be able to compile and run it.  Sadly I discovered that gcc had not implemented C11 threads at that time.  However, as of version GCC 9.3.0 C11 threads are present—you just have to link with the pthreads library as it uses pthreads under the hood. 

Porting didn't take too long.  However, there was one item I did not have from the start that I would need: counting semaphores.  Counting semaphores are used by the program to keep some specified number of running threads—typically one thread for each core of the CPU.  The C11 threads implementation does have mutexes and c conditional waits and those can be used to make a counting semaphore.  So I wrote a very simple header file with inline functions for counting semaphores.

The implementation is nearly identical, just using the C11 thread structures and my semaphore unit.  I don't know the speeds of the original Red Dragon, but I ran a speed test in May of 2017.  My fastest machine at the time, a duel-core 4-thread Intel i7, required 28.26 minutes.  My AMD Ryzen 7 1700, 8-core/16-thread CPU needs 16.58 minutes.

In previous articles of the series on Amiga MOD files I wrote about implementing a system to read the file format and print the notes. The goal is to be able to render audio output. Rather than directly have audio go to a sound device I decided that writing the output to an audio file would be easier. One of the simplest audio file formats is the uncompressed Waveform Audio File Format, WAVE, or just WAV. It was created in the early 90s by IBM and Microsoft—right around the time I was learning how to write my own software. So along with BMP it became one of the first file formats I reverse engineered sometime between 1994 and 1996. With ready access to the Internet it is no longer necessary to manually workout the details by trial and error. In this article I want to look at what will take me much longer to write about than it did to implement.

Although my goal was to write WAV files, the first step was to be able to read them. The format supports storing multiple chunks of data, but most of the time there is just a single chunk. This site gave me enough detail on how the header to a WAV file is laid out. To correctly read a WAV file I would really need to account for all of the chunks. But the assumption was we were only going to use a single chunk—so that is all I was concerned about.

Now there is a weird mix of little/big endian. Why anyone would do this I couldn’t say—usually you pick one or the other. Intel has always been little endian and it turns out the specification only defines headers descriptions in big endian. It is easier just to tread those as 4 characters. Right away I could most of the all the data was 32-bit aligned. The are 16-bit fields but they come in pairs. Knowing I was writing for a little endian system I could take a shortcut—I could read the header directly into a C structure. C structures are get complected due to alignment. However, this structure was already aligned so I could use the structure trick. To be sure I used bitfields, making the 16-bit words actually bit fields in 32-bit words.

Let’s look at code that simply reads the WAV file header and prints the fields.

This code was created in a very short amount of time so very little through was given to cleaning it up. What I really wanted now was to create a WAV file. Armed with the header information I wrote a program to produce a cord and write the results to a WAV file.

Here is a simple file that uses a default header for an 8-bit, mono wave file of a fixed duration. Small modifications such as to duration and sample rate can be made and tested. It simply makes a C major seventh chord with a linear fade out and writes the data to the wave file. Part of this process requires building a table of note frequencies. A typical piano has 88 keys, but I didn’t like the odd range—7.3 octaves. So I went for an extended note rage of 108 keys—a full 9 octaves, from C₀ to B₈. The frequency of each notes follows this equation:

Where f is frequency, and n is the note from 0 to 107 with 0 being C₀ and 107 being B₈. This is a slight modification of the equation found for piano key freuencies being offset of account for the large range. Most of the file is fairly self-explanatory. The note generation happens here:

First, we find the time in seconds. Then we calculate the angle of this time in radians. Multiplying this by the note frequency and take the sine and we get a the desired note. Add these notes up and we get a chord. The amplitude of each sine wave is 1, so adding four of them together will result in a maximum amplitude of 4. Thus we divide by 4. That produces a number between -1 and 1. For 8-bit PCM we need a value between 0 and 255 with 128 being the zero point. The maximum volume is 128 so we multiply by this changing our range from -128 to +128. Adding 128 will give a range between 0 and 256. 256 is too high so we clip it to 255. We now have the 8-bit PCM value that is stored in the sample buffer.

Just for fun I generated a couple of other chords, picked difference sample frequencies and duration. Naturally they are all functional as there isn’t anything special. All of this was to demonstrate I could write a valid WAV file. Now satisfied, it was time to make a library to create WAV files from a sample set.

My first library was more or less a wrapper of what is shown here. A single function that took a sample frequency and a set of samples and wrote them to a WAV file. That was enough for my first day’s work, but as the project progressed I needed better functions.

The second function I wrote simply appended samples to a WAV file. This allowed me to incrementally add data. The third and most useful function set allows a WAV file to be opened, samples added, and then closed at which time the header details are completed. The this version also included the number of channels so stereo WAV files could be created.

If I expand the MOD library to handle other formats I might also add the ability to work with 16-bit data. For now, 8-bit data is sufficient.