threads or queue for this task

[Alex Martelli]

robin at execulink.com wrote:
    ...
> For example, I really don't know what "proper design" I should use in
> this case. Any sample code?

IMHO, proper design of a multithreaded app in Python almost invariably
uses instances of Queue to for inter-thread communication.  I do not
understand all the details of your requirements, but, roughly speaking,
I gather: you have some dedicated working threads, each of which must
repeatedly run the same function on all "work-units" sutable for that
thread; and another thread that is able to determine what work units
are suitable for what other threads.  In this case, you can have a
separate Queue for each dedicated thread.  Each such thread peels the
next work-unit destined for it from its Queue, performs the work, then
goes back to peeling.  The supervisor thread just posts work units to
the appropriate queues.

If you can post more precise/accurate/detailed specs, I'll be glad
to try to help in more detail.

> The interface to Queue is deceptively simple. Are there constructs to
> avoid or performance trade-offs to different methods? Again, any
> explicit code (or at least explicit help as to which methods are
> preferable) would be appreciated.

The no-wait methods, or explicit timeouts on the normal methods that
do wait, are often symptoms that you're polling.  IF you can avoid
polling (that's not always possible, alas), your performance will be
better: a sleeping thread consumes close to no resources, one that is
polling consumes resources in rough proportion to the frequency with
which it's polling (and suffers response delays in inverse proportion
to it).


> I am also put off by the description of the Queue Object, particularly
> methods like empty() which says: "Return 1 if the queue is empty, 0
> otherwise. Because of multithreading semantics, this is not reliable."
> 
> If it's not reliable, then what's the point of it existing? Why is it

Because some people like "purely indicative" methods to exist.

> not reliable? (I imagine simply because the thread doing the checking
> could be superceded by the thread that implements the Queue in the
> time it takes for the method to return and be acted upon.) Is it just

Of course.  Basically, when method empty returns 1, all you know is
"there was an instant in the reasonably recent past in which this
queue instance was empty" -- this, of course, has nothing to do with
whether the queue is empty now.  Similarly, when empty returns 0, then
there was an instant in which the queue was non-empty, but, again,
this doesn't necessarily matter much.  Things may be a bit better
when your overall architecture is such that only one thread ever
posts to a given Queue, and/or only one thread ever gets stuff from
it: in such case, you may be able to prove stronger and more useful
statements.  For example, if the thread calling the empy method is
the only thread that ever posts stuff to the Queue, then if the Queue
was empty when empty() was running, and the thread knows it has
posted nothing more, then the Queue is still empty.

> in certain cases that it's not reliable? How can one code if you don't
> reliably know when a Queue is empty or full?

By just trying to post, and/or to get, stuff from the Queue, and
sleeping until the action can be performed; or, polling issues
apart, from performing e.g. a get-nowait operation in a try/except and
treating the exception, if it comes, as a VERY reliable indication
that, yes, the Queue WAS indeed empty at the time we tried to peel
an entry off it.


Alex