[Tutor] About C++ library with multiprocessing

[Peter Otten]

Yohei Uemura wrote:

> Hi all,
> 
> I'm Yohei.
> I have a problem with making a script which use both multiprocessing and
> C++ library.
> My C++ library is made by using boost python.
> The library worked well if it was imported normally.
> That is,
> "import simple as F
> A = range(0,100)
> for x in A:
>    print F.addition(x)
> "
> However, it couldn't work if it was called from multiprocessing.
> That is,
> "import simple as F
> import multiprocessing as mp
> pool = mp.Pool()
> A = range(0,100)
> results = pool.map(F.simple,A)
> "
> The error messages are shown in the attached file.
> It seems a 'pickle' problem...

Attachments may be stripped off, so it's better to include the traceback 
into the message body.

> Traceback (most recent call last):
>    File "testCXX.py", line 15, in <module>
>      results = pool.map(F.addition,A)
>    File "/usr/lib64/python2.7/multiprocessing/pool.py", line 251, in map
>      return self.map_async(func, iterable, chunksize).get()
>    File "/usr/lib64/python2.7/multiprocessing/pool.py", line 558, in get
>      raise self._value
>  cPickle.PicklingError: Can't pickle <type 'builtin_function_or_method'>:
>  attribute lookup __builtin__.builtin_function_or_method failed

The processes communicate by passing pickled objects to each other. For some 
reason F.simple() doesn't have a __module__ attribute that correctly tells 
Python its origin, and thus pickling fails. You can use

https://docs.python.org/dev/library/copyreg.html

to work around that limitation or -- even easier -- wrap it into a Python 
function that can be pickled:

...

def mysimple(arg):
    return F.simple(arg)
...

results = pool.map(mysimple, A)