simple pub/sub

[Steve Howell]

Hi, I'm looking for ideas on building a simple architecture that
allows a bunch of independent Python processes to exchange data using
files and perform calculations.

One Python program would be collecting data from boat instruments on a
serial port, then writing that info out to a file, and also
transmitting data to instruments as needed (again through the serial
port).  It would be the most complex program in terms of having to
poll file descriptors, or something similar, but I want to limit its
job to pure communication handling.

Then, I would have a bunch of programs doing various calculations on
the input stream.  It's totally acceptable if they just sleep a bit,
wake up, see if there is any new data, and if so, do some
calculations, so I will probably just use this:

http://www.dabeaz.com/generators/follow.py

Some of the "calculator" programs might be looking for trends in the
data, and when a certain threshold is reached, they will need to
notify the comm program of the new message.  To avoid collisions
between all the different calculator programs, I am thinking the
simplest thing is just have them each write to their own output file.

So then the comm program has multiple data sources.  It wants to track
to the serial port, but it also wants to monitor the output files from
the various calculator processes.  I want to do this in a mostly
nonblocking way, so I would probably poll on the file descriptors.

The only problems are that select "...cannot be used on regular files
to determine whether a file has grown since it was last read," and it
only works on Unix for files.  (I'm using Unix myself, but one of the
other programmers insists on Windows, and is resisting Python as
well.)

So a variation would be that the calculator programs talk to the comm
program via sockets, which seems slightly on the heavy side, but it is
certainly feasible.

I know that something like twisted could probably solve my problem,
but I'm also wondering if there is some solution that is a little more
lightweight and batteries-included.

At the heart of the matter, I just want the programs to be able to do
the following tasks.

  1) All programs will occasionally need to write a line of text to
some data stream without having to lock it.  (I don't care if the
write blocks.)
  2) All programs will need to be able to do a non-blocking read of a
line of text from a data stream.  (And there may be multiple
consumers.)
  3) The comm program will need to be able to poll the serial port and
input data streams to see which ones are ready.

Any thoughts?