Get multiprocessing.Queue to do priorities

uuid M8R-gqmgb4 at mailinator.com
Sat May 9 13:00:54 EDT 2009


I just read up and it seems that no matter the approach, it's futile to 
use multiprocessing.Queue, since there is a bug that prevents true 
FIFO. (http://bugs.python.org/issue4999)

Any recommendation on an alternate way to build a priority queue to use 
with a "one producer, many consumers" type multiprocessing setup would 
be welcomed!

:|

On 2009-05-09 18:42:34 +0200, uuid <M8R-gqmgb4 at mailinator.com> said:

> Hello,
> I was wondering whether there was a way to make multiprocessing.Queue 
> behave in a priority queue-like fashion. Subclassing with heappush and 
> heappop for put and get doesn't work the old way (multiprocessing.Queue 
> seems to use different data structures than Queue.Queue?)
> 
> Could one create a heapq within the producer as a proxy, and then feed 
> a proper queue from that? Does anyone have an idea on how to deal with 
> the queue item flow control? (Since all of the sorting has to happen 
> within the heapq, it should only pass items to the real Queue if it's 
> empty?)
> 
> Thanks in advance!





More information about the Python-list mailing list