From ward.vanaerschot at gmail.com  Thu May 19 12:23:54 2011
From: ward.vanaerschot at gmail.com (Ward vanaerschot)
Date: Thu, 19 May 2011 12:23:54 +0200
Subject: [execnet-dev] pickling issue using multiprocessing "Can't pickle
 ... it's not found as __channelexec__.f"
Message-ID: <BANLkTi=u3Qw6rzO_AYS9BWyBnBLDQxAWkw@mail.gmail.com>

Hi all,

my original problem was this: run python code "Y" remotely which imports a
module "X". On every channel.remote_exec("Y") the path to import module "X"
on the remote side changes and contains a 'slightly' different version of
X.py . However, since the server spawns several threads the consequence is
that once a module "X" is loaded it will not be loaded again, which is not
what I want.
My original solution is to startup a "process" in "Y" which imports "X" and
solves .
The multiprocessing module seems convenient since I can pass Queue objects
to it to facilitate the communication:

in "Y":
import sys, os
from functools import partial
import multiprocessing

def f(receiveQ):
   sys.path.append(os.path.join(pathToX)) # pathToX is assigned before this
code fragment
   import X

def go()
   multiprocessing.Process(target=partial(f, receiveQ))

if __name__ == '__channelexec__':
   go()

I get the following trace:

Traceback (most recent call last):
  File "c:\python27\lib\threading.py", line 530, in __bootstrap_inner
    self.run()
  File "", line 226, in run
  File "c:\python27\lib\multiprocessing\process.py", line 104, in start
    self._popen = Popen(self)
  File "c:\python27\lib\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "c:\python27\lib\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "c:\python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "c:\python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function f at 0x0537B1F0>: it's not foun
d as __channelexec__.f

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\python27\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "c:\python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "c:\python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "c:\python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError


Somehow pickle expects my function 'f' to 'live' in __channelexec__ but it
does not find it.
I suppose this is because __channelexec__ is defined when executing the code
but the code is compiled without this knowledge.
Is there a bug in gateway_base.SlaveGateway.executetask() which compiles and
executes the source such that pickling is not possible?

thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110519/ac691369/attachment.html>

From Ronny.Pfannschmidt at gmx.de  Thu May 19 12:55:20 2011
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Thu, 19 May 2011 12:55:20 +0200
Subject: [execnet-dev] pickling issue using multiprocessing "Can't
 pickle ... it's not found as __channelexec__.f"
In-Reply-To: <BANLkTi=u3Qw6rzO_AYS9BWyBnBLDQxAWkw@mail.gmail.com>
References: <BANLkTi=u3Qw6rzO_AYS9BWyBnBLDQxAWkw@mail.gmail.com>
Message-ID: <1305802520.422.29.camel@Klappe2>

Hi Ward,

when you remote_exec a module, it gets __name__ == "__channelexec__" and
is transient

therefore you cant import from it
which unfortunately breaks pickle loading of functions (which relies on
importing)

Is it necessary to mix execnet and multiprocessing?
it seems weird to use both to me. 

-- Ronny

On Thu, 2011-05-19 at 12:23 +0200, Ward vanaerschot wrote:
> Hi all,
> 
> 
> my original problem was this: run python code "Y" remotely which
> imports a module "X". On every channel.remote_exec("Y") the path to
> import module "X" on the remote side changes and contains a 'slightly'
> different version of X.py . However, since the server spawns several
> threads the consequence is that once a module "X" is loaded it will
> not be loaded again, which is not what I want.
> My original solution is to startup a "process" in "Y" which imports
> "X" and solves .
> The multiprocessing module seems convenient since I can pass Queue
> objects to it to facilitate the communication:
> 
> 
> in "Y":
> import sys, os
> from functools import partial
> import multiprocessing
> 
> 
> def f(receiveQ):
>    sys.path.append(os.path.join(pathToX)) # pathToX is assigned before
> this code fragment
>    import X
> 
> 
> def go()
>    multiprocessing.Process(target=partial(f, receiveQ))
> 
> 
> if __name__ == '__channelexec__':
>    go()
> 
> 
> I get the following trace:
> 
> 
> Traceback (most recent call last):
>   File "c:\python27\lib\threading.py", line 530, in __bootstrap_inner
>     self.run()
>   File "", line 226, in run
>   File "c:\python27\lib\multiprocessing\process.py", line 104, in
> start
>     self._popen = Popen(self)
>   File "c:\python27\lib\multiprocessing\forking.py", line 244, in
> __init__
>     dump(process_obj, to_child, HIGHEST_PROTOCOL)
>   File "c:\python27\lib\multiprocessing\forking.py", line 167, in dump
>     ForkingPickler(file, protocol).dump(obj)
>   File "c:\python27\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "c:\python27\lib\pickle.py", line 331, in save
>     self.save_reduce(obj=obj, *rv)
>   File "c:\python27\lib\pickle.py", line 419, in save_reduce
>     save(state)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 649, in save_dict
>     self._batch_setitems(obj.iteritems())
>   File "c:\python27\lib\pickle.py", line 681, in _batch_setitems
>     save(v)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 748, in save_global
>     (obj, module, name))
> PicklingError: Can't pickle <function f at 0x0537B1F0>: it's not foun
> d as __channelexec__.f
> 
> 
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "c:\python27\lib\multiprocessing\forking.py", line 347, in main
>     self = load(from_parent)
>   File "c:\python27\lib\pickle.py", line 1378, in load
>     return Unpickler(file).load()
>   File "c:\python27\lib\pickle.py", line 858, in load
>     dispatch[key](self)
>   File "c:\python27\lib\pickle.py", line 880, in load_eof
>     raise EOFError
> EOFError
> 
> 
> 
> 
> Somehow pickle expects my function 'f' to 'live' in __channelexec__
> but it does not find it.
> I suppose this is because __channelexec__ is defined when executing
> the code but the code is compiled without this knowledge. 
> Is there a bug in gateway_base.SlaveGateway.executetask() which
> compiles and executes the source such that pickling is not possible?
> 
> 
> thanks!
> 
> 
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110519/d3cc9527/attachment.pgp>