Get multiprocessing.Queue to do priorities

uuid address_is at invalid.invalid
Sat May 9 14:23:25 EDT 2009


Scott David Daniels <Scott.Daniels at Acm.Org> wrote:
> 
> ? "one producer, many consumers" ?
> What would the priority queue do?  Choose a consumer?

Sorry, I should have provided a little more detail. There is one
producer thread, reading urls from multiple files and external input.
These urls have a certain priority, and are fed to multiple consumer
threads for fetching and further processing (XML parsing and such CPU
intensive stuff). Since the content of the urls is changing over time,
it is crucial to have a certain amount of control over the order in
which the requests occur. So, to answer the question: 
The priority queue would make sure that out of a number of
asynchronously added items, those with a high priority are fetched first
by the worker threads. Sounds like a perfect case for a heap, if only I
could :)



More information about the Python-list mailing list