Producer/consumer Queue "trick"

[John Lenton]

On Fri, Jan 14, 2005 at 04:26:02PM -0600, Evan Simpson wrote:
> WEBoggle needs a new game board every three minutes.  Boards take an 
> unpredictable (much less than 3min, but non-trivial) amount of time to 
> generate. The system is driven by web requests, and I don't want the 
> request that happens to trigger the need for the new board to have to 
> pay the time cost of generating it.
> 
> I set up a producer thread that does nothing but generate boards and put 
> them into a length-two Queue (blocking).  At the rate that boards are 
> pulled from the Queue, it's almost always full, but my poor consumer 
> thread was still being blocked for "a long time" each time it fetched a 
> board.
> 
> At this point I realized that q.get() on a full Queue immediately wakes 
> up the producer, which has been blocked waiting to add a board to the 
> Queue.  It sets about generating the next board, and the consumer 
> doesn't get to run again until the producer blocks again or is preempted.
> 
> The solution was simple: have the producer time.sleep(0.001) when 
> q.put(board) returns.

If I had had that problem, I would probably have set up different
server processes (consumer(s), producer(s), and queue(s)), coordinated
by somthing like pyro's event server. But I don't really know the
problem, so it's probably just bad guesswork on my part---you probably
don't need to scale at all.

-- 
John Lenton (john at grulic.org.ar) -- Random fortune:
Tonight's the night: Sleep in a eucalyptus tree.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20050114/f858b0b7/attachment.sig>