Ending data exchange through multiprocessing pipe

[MRAB]

Michal Chruszcz wrote:
> On Apr 22, 6:33 pm, MRAB <goo... at mrabarnett.plus.com> wrote:
>> You could do this:
>>
>>      from multiprocessing.queues import Empty
>>
>>      queue = Queue()
>>      Process(target=f, args=(queue,)).start()
>>      while active_children():
>>          try:
>>              print queue.get(timeout=1)
>>          except Empty:
>>              pass
> 
> This one isn't generic. When I have tasks that all finish within 0.1
> seconds the above gives 10x overhead. On the other hand, if I know the
> results will be available after 10 hours there's no use in checking
> every second.
> 
If there is a result in the queue then it will return immediately; if
not, then it will wait for up to 1 second for a result to arrive, but if
none arrives in that time then it will raise an Empty exception. Raising
only 1 exception every second won't consume a negligible amount of
processing power (I wouldn't worry about it).