multiprocessing problems

[Nils Ruettershoff]

Hi,

Adam Tauno Williams wrote:

[...]
>> Here's the guts of my latest incarnation.
>> def ProcessBatch(files):
>>     p = []
>>     for file in files:
>>         p.append(Process(target=ProcessFile,args=file))
>>     for x in p:
>>         x.start()
>>     for x in p:
>>         x.join()
>>     p = []
>>     return
>> Now, the function calling ProcessBatch looks like this:
>> def ReplaceIt(files):
>>     processFiles = []
>>     for replacefile in files:
>>         if(CheckSkipFile(replacefile)):
>>             processFiles.append(replacefile)
>>             if(len(processFiles) == 4):
>>                 ProcessBatch(processFiles)
>>                 processFiles = []
>>     #check for left over files once main loop is done and process them
>>     if(len(processFiles) > 0):
>>         ProcessBatch(processFiles)
>>     
>
> According to this you will create files is sets of four, but an unknown
> number of sets of four.
>
>   
This is not correct, 'cause of the x.join(). This will block the parent 
process until all processes have been terminated. So as soon as the 
current set of processes have finished their job, a new set will be spawned.

Cheers,
Nils
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100121/8c4ec86d/attachment-0001.html>