Multiprocessing.Queue - I want to end.

[Cameron Simpson]

On 30Apr2009 22:57, MRAB <google at mrabarnett.plus.com> wrote:
> Luis Zarrabeitia wrote:
>> The problem: when there is no more data to process, how can I signal 
>> the consumers to consume until the queue is empty and then stop 
>> consuming? I need them to do some clean-up work after they finish (and 
>> then I need the main script to summarize the results)
[...]
>> As you can see, I'm sending one 'None' per consumer, and hoping that no 
>> consumer will read more than one None. While this particular 
>> implementation ensures that, it is very fragile. [...]
>>
> The producer could send just one None to indicate that it has finished
> producing.
>
> Each consumer could get the data from the queue, but if it's None then
> put it back in the queue for the other consumer, then clean up and
> finish.
>
> When all the consumers have finished, the queue will contain just the
> single None.

And as it happens I have an IterableQueue class right here which does
_exact_ what was just described. You're welcome to it if you like.
Added bonus is that, as the name suggests, you can use the class as
an iterator:

  for item in iterq:
    ...

The producer calls iterq.close() when it's done.

I'll clean up the formatting and add a bunch of missing docstrings if
anyone wants it...

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Indeed! But do not reject these teachings as false because I am crazy. The
reason that I am crazy is because they are true.
        - Malaclypse the Younger