[execnet-dev] pickling issue using multiprocessing "Can't pickle ... it's not found as __channelexec__.f"

[Ward vanaerschot]

Hi all,

my original problem was this: run python code "Y" remotely which imports a
module "X". On every channel.remote_exec("Y") the path to import module "X"
on the remote side changes and contains a 'slightly' different version of
X.py . However, since the server spawns several threads the consequence is
that once a module "X" is loaded it will not be loaded again, which is not
what I want.
My original solution is to startup a "process" in "Y" which imports "X" and
solves .
The multiprocessing module seems convenient since I can pass Queue objects
to it to facilitate the communication:

in "Y":
import sys, os
from functools import partial
import multiprocessing

def f(receiveQ):
   sys.path.append(os.path.join(pathToX)) # pathToX is assigned before this
code fragment
   import X

def go()
   multiprocessing.Process(target=partial(f, receiveQ))

if __name__ == '__channelexec__':
   go()

I get the following trace:

Traceback (most recent call last):
  File "c:\python27\lib\threading.py", line 530, in __bootstrap_inner
    self.run()
  File "", line 226, in run
  File "c:\python27\lib\multiprocessing\process.py", line 104, in start
    self._popen = Popen(self)
  File "c:\python27\lib\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "c:\python27\lib\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "c:\python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "c:\python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function f at 0x0537B1F0>: it's not foun
d as __channelexec__.f

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\python27\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "c:\python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "c:\python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "c:\python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError


Somehow pickle expects my function 'f' to 'live' in __channelexec__ but it
does not find it.
I suppose this is because __channelexec__ is defined when executing the code
but the code is compiled without this knowledge.
Is there a bug in gateway_base.SlaveGateway.executetask() which compiles and
executes the source such that pickling is not possible?

thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110519/ac691369/attachment.html>