[IPython-dev] question about multiprocessing

[Darren Dale]

Against Robert Kern's characteristically sage and eloquently
communicated advice, I contributed a feature to h5py which, at import
time, attempts to determine if the import is occurring in an IPython
session and, if so, registers a custom completer. I think Robert's
exact response was: "Eww."

It came back and bit me today, as I was starting to port some parallel
processing code from parallelpython to the multiprocessing package. A
simple test script would raise a PicklingError if the master
development branch of ipython was installed. Part of the issue was a
change in behavior of IPython.core.ipapi.get(): it used to return None
if there was no global instance of InteractiveShell, now it creates
one if it doesn't exist. That is easy enough to deal with, we can call
IPython.core.iplib.InteractiveShell.initialized() instead.

Here is a simple script that reproduces the error without importing
h5py. Note that the ipython import and call to get() are lame, and the
script will run without errors if those two lines are commented out.

from multiprocessing import Pool

import IPython.core.ipapi as ip
ip.get()

def update(i):
    print i

def f(i):
    return i*i

if __name__ == '__main__':
    pool = Pool()
    for i in range(10):
        pool.apply_async(f, [i], callback=update)
    pool.close()
    pool.join()


I don't understand the issue here. Maybe it's a situation that should
never crop up in the real world. But in case it is important, I
thought I should bring it to the devs attention.

Cheers,
Darren