run function in separate process

[Alex Martelli]

<malkarouri at gmail.com> wrote:
   ...
> somebody points me to a web page/reference that says how to call a
> function then reclaim the whole memory back in python.
> 
> Meanwhile, the best that I could do is fork a process, compute the
> results, and return them back to the parent process. This I

That's my favorite way to ensure that all resources get reclaimed: let
the operating system do the job.

> implemented in the following function, which is kinda working for me
> now, but I am sure it can be much improved. There should be a better
> way to return the result that a temporary file, for example. I

You can use a pipe.  I.e. (untested code):

def run_in_separate_process(f, *a, **k):
    import os, sys, cPickle
    pread, pwrite = os.pipe()
    pid = os.fork()
    if pid>0:
        os.close(pwrite)
        with os.fdopen(pread, 'rb') as f:
            return cPickle.load(f)
    else:
        os.close(pread)
        result = f(*a, **k)
        with os.fdopen(pwrite, 'wb') as f:
            cPickle.dump(f, -1)
        sys.exit()

Using cPickle instead of pickle, and a negative protocol (on the files
pedantically specified as binary:-), meaning the latest and greatest
available pickling protocol, rather than the default 0, should improve
performance.


Alex