Multiprocessing.Queue - I want to end.

[Hendrik van Rooyen]

"Luis Alberto Zarrabeitia Gomez" <kyrie at uh.cu> wrote:

>Quoting Hendrik van Rooyen <mail at microcorp.co.za>:

>> In fact I happen to believe that anything that does any work needs
>> one and only one input queue and nothing else, but I am peculiar
>> that way.
>
>Well, I also need some output. In my case, the outputs are files with the
result
>of the processing, that can be summarized later (hence the need to 'join' the
>processes, to know when I can summarize them).

When you know what you want to do with the output (which
I suspect is your case) then the trick is to put the individual
outputs onto the input queue of the summary process.

This will form a virtual (or real if you have different machines)
systolic array with producers feeding consumers that feed
the summary process, all running concurrently.

Ponder on it that in the description only the word "feed" is used...

And if you don't know where your output must go, then it must be
put onto the input queue of your default router, who must know
what to do with stuff. - this is in a sense what happens when
you pipe stdout to stdin in a *nix environment - you use the OS
to set up a temporary systolic array.

You only need to keep the output of the consumers in files if
you need access to it later for some reason. In your case it sounds
as if you are only interested in the output of the summary.

- Hendrik