Queue class deadlocking application

[Gordon McMillan]

Nathan Clegg writes:

> When I had only one producer and one consumer, the Queue class was a
> great performer for me in my multithreaded (threading module)
> application.  When I went up to four producers, I found the app
> deadlocking at seemingly random locations, freezing all threads. 
> The only operations ever performed are put_nowait and get_nowait. 
> The application freezes when the queue size is anywhere from 20-50
> in size, and the data objects themselves are simple strings.  The
> max queue size is 10,000.  Again, with only one producer, no
> problems.

How do you know the queue size if you only ever use put_nowait and 
get_nowait <wink>?
 
> Has anyone ever run into this before?  Can someone say with some
> certainty that my problem lies not in Queue but another area?  I
> have only one queue and it is the only shared data, so I have no
> implemented any mutexes or signals myself.  Any ideas, hints, or
> questions?  Thanks...

I coded up something that should test your description of the problem 
- 4 producers, 1 consumer; each doing it's thing after random sleeps 
(but the consumer only slightly slower than the 4 producers). I've 
been watching the queue size (sampled every 1 sec) do some nice 
brownian motion (very much "programmer stumbling towards a 
solution"). I've tried shorter runs with different relative rates. No 
problems.

I'll leave it running for awhile, but I'd rather suspect it's 
"something else".


- Gordon