problem with simple multiprocessing script on OS X

Darren Dale dsdale24 at gmail.com
Mon Aug 23 09:58:16 EDT 2010


The following script runs without problems on Ubuntu and Windows 7.
h5py is a package wrapping the hdf5 library (http://code.google.com/p/
h5py/):

from multiprocessing import Pool
import h5py

def update(i):
    print i

def f(i):
    "hello foo"
    return i*i

if __name__ == '__main__':
    pool = Pool()
    for i in range(10):
        pool.apply_async(f, [i], callback=update)
    pool.close()
    pool.join()


On OS X 10.6 (tested using python-2.6.5 from MacPorts), I have to
comment out the as-yet unused h5py import, otherwise I get a
traceback:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/multiprocessing/pool.py", line 226, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed


I've searched that pickle error and found some references to pickling
a lambda, but I don't think that is the issue. There are no lambdas in
the h5py module, and the script runs fine on windows and linux. I need
access to both multiprocessing and h5py objects in the same module, so
I can register a callback that saves the results to an hdf5 file.

Are there any suggestions as to what could be the problem, or
suggestions on how I can track it down?

Thanks,
Darren



More information about the Python-list mailing list