problem with simple multiprocessing script on OS X

[Darren Dale]

On Aug 24, 4:32 pm, Thomas Jollans <tho... at jollybox.de> wrote:
> On Tuesday 24 August 2010, it occurred to Darren Dale to exclaim:
>
>
>
>
>
> > On Aug 23, 9:58 am, Darren Dale <dsdal... at gmail.com> wrote:
> > > The following script runs without problems on Ubuntu and Windows 7.
> > > h5py is a package wrapping the hdf5 library (http://code.google.com/p/
> > > h5py/):
>
> > > from multiprocessing import Pool
> > > import h5py
>
> > > def update(i):
> > >     print i
>
> > > def f(i):
> > >     "hello foo"
> > >     return i*i
>
> > > if __name__ == '__main__':
> > >     pool = Pool()
> > >     for i in range(10):
> > >         pool.apply_async(f, [i], callback=update)
> > >     pool.close()
> > >     pool.join()
>
> > > On OS X 10.6 (tested using python-2.6.5 from MacPorts), I have to
> > > comment out the as-yet unused h5py import, otherwise I get a
> > > traceback:
>
> What on earth is h5py doing there?  If what you're telling us is actually
> happening, and the code works 1:1 on Linux and Windows, but fails on OSX, and
> you're using the same versions of h5py and Python, then the h5py
> initialization code is not only enticing multiprocessing to try to pickle
> something other than usual, but it is also doing that due to some platform-
> dependent witchcraft, and I doubt there's very much separating the OSX
> versions from the Linux versions of anything involved.

Your analysis was spot on.

About a year ago, I contributed a patch to h5py which checks to see if
h5py is being imported into an active IPython session. If so, then a
custom tab completer is loaded to make it easier to navigate hdf5
files. In the development version of IPython, a function that used to
return None if there was no instance of an IPython interactive shell
now creates and returns a new instance. This was the cause of the
error I was reporting. If one were to install ipython from the master
branch at github or from http://ipython.scipy.org/dist/testing/ipython-dev-nightly.tgz,
then the following script will reproduce the problem. I'm not sure why
this causes an error, but I'll discuss it with the IPython devs.

Thank you Thomas and Benjamin for helping me understand the problem.

Darren


from multiprocessing import Pool

import IPython.core.ipapi as ip

ip.get()

def update(i):
    print i

def f(i):
    return i*i

if __name__ == '__main__':
    pool = Pool()
    for i in range(10):
        pool.apply_async(f, [i], callback=update)
    pool.close()
    pool.join()