multiprocessing problems

DoxaLogos doxalogos at gmail.com
Tue Jan 19 11:15:22 EST 2010


Hi,

I decided to play around with the multiprocessing module, and I'm
having some strange side effects that I can't explain.  It makes me
wonder if I'm just overlooking something obvious or not.  Basically, I
have a script parses through a lot of files doing search and replace
on key strings inside the file.  I decided the split the work up on
multiple processes on each processor core (4 total).  I've tried many
various ways doing this form using pool to calling out separate
processes, but the result has been the same: computer crashes from
endless process spawn.

Here's the guts of my latest incarnation.

def ProcessBatch(files):
    p = []
    for file in files:
        p.append(Process(target=ProcessFile,args=file))

    for x in p:
        x.start()

    for x in p:
        x.join()

    p = []
    return

Now, the function calling ProcessBatch looks like this:
def ReplaceIt(files):
    """
    All this does is walks through all the files passed to it and
verifies
    the file is a legitimate file to be processed (project file).

    @param files:  files to be processed
    """
    processFiles = []
    for replacefile in files:
        if(CheckSkipFile(replacefile)):
            processFiles.append(replacefile)
            if(len(processFiles) == 4):
                ProcessBatch(processFiles)
                processFiles = []

    #check for left over files once main loop is done and process them
    if(len(processFiles) > 0):
        ProcessBatch(processFiles)

    return

Specs:
Windows 7 64-bit
Python v2.6.2
Intel i5


Thanks



More information about the Python-list mailing list