From cymrow at gmail.com  Tue Nov 16 04:48:52 2010
From: cymrow at gmail.com (M T)
Date: Mon, 15 Nov 2010 22:48:52 -0500
Subject: [execnet-dev] Issue running code in python2 from a python3 script
	frozen by cx_Freeze
Message-ID: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>

I'm trying to use execnet from a Python 3 script that I've made into an
executable with cx_Freeze. As a script, the code runs fine, and as an
executable, when I try to run code with a separate copy of Python 3 it works
fine. But, as an executable, when I try to run code with a copy of Python 2,
I get an exception.

It took me a long time to narrow the problem down to this code:

(test.py)
import execnet


def main():

gw = execnet.makegateway('popen//python=c:/dev/Python26/python.exe')   #
this runs with an exception

# gw = execnet.makegateway('popen//python=c:/dev/Python31/python.exe')  #
this runs fine

channel = gw.remote_exec('channel.send(1)')

value = channel.receive()   # the exception is raised here

print(value)


 if __name__ == '__main__':

main()


I am creating the executable with this:

(setup.py)
from cx_Freeze import setup, Executable

 includes = [

'execnet.multi', 'execnet.xspec', 'execnet.threadpool',

'execnet.gateway', 'execnet.gateway_base', 'execnet.gateway_socket',

'execnet.rsync', 'execnet.rsync_remote']

 setup(

name = "test",

executables = [Executable("test.py")],

options = {

'build_exe': {

'includes': includes,

}

},

 )


Here is the traceback:


Traceback (most recent call last):

  File
"C:\dev\Python31\lib\site-packages\cx_Freeze\initscripts\Console3.py", line
27, in <module>

    exec(code, m.__dict__)

  File "test.py", line 11, in <module>

    main()

  File "test.py", line 6, in main

    channel = gw.remote_exec('channel.send(1)')

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway.py",
line 116, in remote_exec

    (source, call_name, kwargs)))

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
line 652, in _send

    msg.writeto(self._io)

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
line 112, in writeto

    serialize(io, (self.msgtype, self.channelid, self.data))

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
line 939, in serialize

    _Serializer(io).save(obj)

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
line 959, in save

    self._stream.write(s)

  File
"C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
line 95, in write

    self.outfile.flush()

IOError: [Errno 22] Invalid argument


Can someone help me figure out what's going on here?


Thanks,

miguel


PS: an unrelated issue is the need to explicitly include the execnet modules
for cx_Freeze, due to the use of apipkg.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20101115/2aaf3f9b/attachment.html>

From cymrow at gmail.com  Wed Nov 17 02:15:47 2010
From: cymrow at gmail.com (M T)
Date: Tue, 16 Nov 2010 20:15:47 -0500
Subject: [execnet-dev] Issue running code in python2 from a python3
	script frozen by cx_Freeze
In-Reply-To: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>
References: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>
Message-ID: <AANLkTinNbab=WwmnqwR9LF_mR92P=3g_Forf8XOWfZfN@mail.gmail.com>

I think I've tracked down the problem. It seems execnet points a python
subprocess to the path of the execnet for the main process and tries to get
the subprocess to load that. In a frozen cx_Freeze executable, though, all
the main process' modules are compiled python byte-code, which in my case
would be python3 byte-code, so python2's zipimport can't load it.

Not sure what the best solution to this is. It would be nice if all of the
bootstrap code could be sent through sendexec. For now I will try keeping an
uncompiled copy of execnet with my frozen executable. Any other ideas would
be appreciated, though.

Thanks,
miguel

On Mon, Nov 15, 2010 at 10:48 PM, M T <cymrow at gmail.com> wrote:

> I'm trying to use execnet from a Python 3 script that I've made into an
> executable with cx_Freeze. As a script, the code runs fine, and as an
> executable, when I try to run code with a separate copy of Python 3 it works
> fine. But, as an executable, when I try to run code with a copy of Python 2,
> I get an exception.
>
> It took me a long time to narrow the problem down to this code:
>
> (test.py)
> import execnet
>
>
> def main():
>
> gw = execnet.makegateway('popen//python=c:/dev/Python26/python.exe')   #
> this runs with an exception
>
> # gw = execnet.makegateway('popen//python=c:/dev/Python31/python.exe')  #
> this runs fine
>
> channel = gw.remote_exec('channel.send(1)')
>
> value = channel.receive()   # the exception is raised here
>
> print(value)
>
>
>  if __name__ == '__main__':
>
> main()
>
>
> I am creating the executable with this:
>
> (setup.py)
> from cx_Freeze import setup, Executable
>
>  includes = [
>
> 'execnet.multi', 'execnet.xspec', 'execnet.threadpool',
>
> 'execnet.gateway', 'execnet.gateway_base', 'execnet.gateway_socket',
>
> 'execnet.rsync', 'execnet.rsync_remote']
>
>  setup(
>
> name = "test",
>
> executables = [Executable("test.py")],
>
> options = {
>
>  'build_exe': {
>
>  'includes': includes,
>
>  }
>
>  },
>
>  )
>
>
> Here is the traceback:
>
>
> Traceback (most recent call last):
>
>   File
> "C:\dev\Python31\lib\site-packages\cx_Freeze\initscripts\Console3.py", line
> 27, in <module>
>
>     exec(code, m.__dict__)
>
>   File "test.py", line 11, in <module>
>
>     main()
>
>   File "test.py", line 6, in main
>
>     channel = gw.remote_exec('channel.send(1)')
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway.py",
> line 116, in remote_exec
>
>     (source, call_name, kwargs)))
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> line 652, in _send
>
>     msg.writeto(self._io)
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> line 112, in writeto
>
>     serialize(io, (self.msgtype, self.channelid, self.data))
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> line 939, in serialize
>
>     _Serializer(io).save(obj)
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> line 959, in save
>
>     self._stream.write(s)
>
>   File
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> line 95, in write
>
>     self.outfile.flush()
>
> IOError: [Errno 22] Invalid argument
>
>
> Can someone help me figure out what's going on here?
>
>
> Thanks,
>
> miguel
>
>
> PS: an unrelated issue is the need to explicitly include the execnet
> modules for cx_Freeze, due to the use of apipkg.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20101116/f71386f2/attachment.html>

From holger at merlinux.eu  Thu Nov 18 11:07:09 2010
From: holger at merlinux.eu (holger krekel)
Date: Thu, 18 Nov 2010 11:07:09 +0100
Subject: [execnet-dev] Issue running code in python2 from a
	python3	script frozen by cx_Freeze
In-Reply-To: <AANLkTinNbab=WwmnqwR9LF_mR92P=3g_Forf8XOWfZfN@mail.gmail.com>
References: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>
	<AANLkTinNbab=WwmnqwR9LF_mR92P=3g_Forf8XOWfZfN@mail.gmail.com>
Message-ID: <20101118100709.GL1009@trillke.net>

Hi Miguel,

On Tue, Nov 16, 2010 at 20:15 -0500, M T wrote:
> I think I've tracked down the problem. It seems execnet points a python
> subprocess to the path of the execnet for the main process and tries to get
> the subprocess to load that. In a frozen cx_Freeze executable, though, all
> the main process' modules are compiled python byte-code, which in my case
> would be python3 byte-code, so python2's zipimport can't load it.

correct analysis i think.  What app/package are you working on, btw?

> Not sure what the best solution to this is. It would be nice if all of the
> bootstrap code could be sent through sendexec. For now I will try keeping an
> uncompiled copy of execnet with my frozen executable. Any other ideas would
> be appreciated, though.

If you use ssh- or socket-gateways the neccessary bootstrap will be
send - the content of gateway_base.py actually.

"popen" gateways take a shortcut by not sending this source but
just using the filesystem location which bites your usage, unfortunately.

So two bits needed to resolve this: 

* the source code of gateway_base.py needs to be kept
* a way to configure to not use the shortcut, maybe a new
  gateway type "srcpopen" or so?

As to including the packages: how does cx_Freeze discover
the modules/packages usually?   If you do a simple "execnet.__dict__"
all sub packages will be imported i think - maybe that helps?

cheers,
holger
 
> Thanks,
> miguel
> 
> On Mon, Nov 15, 2010 at 10:48 PM, M T <cymrow at gmail.com> wrote:
> 
> > I'm trying to use execnet from a Python 3 script that I've made into an
> > executable with cx_Freeze. As a script, the code runs fine, and as an
> > executable, when I try to run code with a separate copy of Python 3 it works
> > fine. But, as an executable, when I try to run code with a copy of Python 2,
> > I get an exception.
> >
> > It took me a long time to narrow the problem down to this code:
> >
> > (test.py)
> > import execnet
> >
> >
> > def main():
> >
> > gw = execnet.makegateway('popen//python=c:/dev/Python26/python.exe')   #
> > this runs with an exception
> >
> > # gw = execnet.makegateway('popen//python=c:/dev/Python31/python.exe')  #
> > this runs fine
> >
> > channel = gw.remote_exec('channel.send(1)')
> >
> > value = channel.receive()   # the exception is raised here
> >
> > print(value)
> >
> >
> >  if __name__ == '__main__':
> >
> > main()
> >
> >
> > I am creating the executable with this:
> >
> > (setup.py)
> > from cx_Freeze import setup, Executable
> >
> >  includes = [
> >
> > 'execnet.multi', 'execnet.xspec', 'execnet.threadpool',
> >
> > 'execnet.gateway', 'execnet.gateway_base', 'execnet.gateway_socket',
> >
> > 'execnet.rsync', 'execnet.rsync_remote']
> >
> >  setup(
> >
> > name = "test",
> >
> > executables = [Executable("test.py")],
> >
> > options = {
> >
> >  'build_exe': {
> >
> >  'includes': includes,
> >
> >  }
> >
> >  },
> >
> >  )
> >
> >
> > Here is the traceback:
> >
> >
> > Traceback (most recent call last):
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\cx_Freeze\initscripts\Console3.py", line
> > 27, in <module>
> >
> >     exec(code, m.__dict__)
> >
> >   File "test.py", line 11, in <module>
> >
> >     main()
> >
> >   File "test.py", line 6, in main
> >
> >     channel = gw.remote_exec('channel.send(1)')
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway.py",
> > line 116, in remote_exec
> >
> >     (source, call_name, kwargs)))
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > line 652, in _send
> >
> >     msg.writeto(self._io)
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > line 112, in writeto
> >
> >     serialize(io, (self.msgtype, self.channelid, self.data))
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > line 939, in serialize
> >
> >     _Serializer(io).save(obj)
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > line 959, in save
> >
> >     self._stream.write(s)
> >
> >   File
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > line 95, in write
> >
> >     self.outfile.flush()
> >
> > IOError: [Errno 22] Invalid argument
> >
> >
> > Can someone help me figure out what's going on here?
> >
> >
> > Thanks,
> >
> > miguel
> >
> >
> > PS: an unrelated issue is the need to explicitly include the execnet
> > modules for cx_Freeze, due to the use of apipkg.
> >
> >

> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev


-- 


From holger at merlinux.eu  Fri Nov 19 08:09:59 2010
From: holger at merlinux.eu (holger krekel)
Date: Fri, 19 Nov 2010 08:09:59 +0100
Subject: [execnet-dev] Issue running code in python2 from a
	python3	script frozen by cx_Freeze
In-Reply-To: <AANLkTinXvpqbcTtand0zxjRi-Rr10UKUxZb8sXA=nSXs@mail.gmail.com>
References: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>
	<AANLkTinNbab=WwmnqwR9LF_mR92P=3g_Forf8XOWfZfN@mail.gmail.com>
	<20101118100709.GL1009@trillke.net>
	<AANLkTinXvpqbcTtand0zxjRi-Rr10UKUxZb8sXA=nSXs@mail.gmail.com>
Message-ID: <20101119070959.GP1009@trillke.net>

On Thu, Nov 18, 2010 at 21:09 -0500, M T wrote:
> On Thu, Nov 18, 2010 at 5:07 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi Miguel,
> >
> > On Tue, Nov 16, 2010 at 20:15 -0500, M T wrote:
> > > I think I've tracked down the problem. It seems execnet points a python
> > > subprocess to the path of the execnet for the main process and tries to
> > get
> > > the subprocess to load that. In a frozen cx_Freeze executable, though,
> > all
> > > the main process' modules are compiled python byte-code, which in my case
> > > would be python3 byte-code, so python2's zipimport can't load it.
> >
> > correct analysis i think.  What app/package are you working on, btw?
> >
> 
> A hobby project, really. I'm doing some python introspection as part of a
> larger program. execnet is perfect for this, since it lets me dynamically
> access modules written for different python versions.

I am interested in a source pointer if you distribute it at some point.

> > > Not sure what the best solution to this is. It would be nice if all of
> > the
> > > bootstrap code could be sent through sendexec. For now I will try keeping
> > an
> > > uncompiled copy of execnet with my frozen executable. Any other ideas
> > would
> > > be appreciated, though.
> >
> > If you use ssh- or socket-gateways the neccessary bootstrap will be
> > send - the content of gateway_base.py actually.
> >
> > "popen" gateways take a shortcut by not sending this source but
> > just using the filesystem location which bites your usage, unfortunately.
> >
> > So two bits needed to resolve this:
> >
> > * the source code of gateway_base.py needs to be kept
> > * a way to configure to not use the shortcut, maybe a new
> >  gateway type "srcpopen" or so?
> >
> 
> I got it working by having cx_Freeze copy the .py files directly into the
> built exe (in this case, a zip file embedded in the exe). The python2
> subprocess is then actually able to import it directly from the executable
> based on the path execnet gets from __file__ (pretty cool, really). I may do
> something like srcpopen at some point just to be a little more efficient. I
> also have some minor fixes for frozen Windows exe's. Perhaps this weekend I
> can send you a pull request on bitbucket.

Saw you opened a branch already and your "expr if cond else other" style
addition ... note that execnet is meant to run on Python2.4.  You can check
compatibility by running "tox -e py24 -e py31" for example.  You might also
add a configuration into tox.ini that performs a cx_freeze related test/run.
see http://codespeak.net/tox on tox documentation.

> > As to including the packages: how does cx_Freeze discover
> > the modules/packages usually?   If you do a simple "execnet.__dict__"
> > all sub packages will be imported i think - maybe that helps?
> >
> > I'd have to look into this more. I know that the pyglet package uses lazy
> module loading and gets around it using "if False: import module". Since
> that works, I think they must be scanning the source for imports. py2exe is
> not able to discover lazy loaded modules either.

ah, i see.  good luck with your experiments!
cheers,
holger

(P.S.: re-CCed execnet-dev - thinks it's good to have this conversation
in the archives/seen by others)

> thanks,
> miguel
> 
> 
> > cheers,
> > holger
> >
> > > Thanks,
> > > miguel
> > >
> > > On Mon, Nov 15, 2010 at 10:48 PM, M T <cymrow at gmail.com> wrote:
> > >
> > > > I'm trying to use execnet from a Python 3 script that I've made into an
> > > > executable with cx_Freeze. As a script, the code runs fine, and as an
> > > > executable, when I try to run code with a separate copy of Python 3 it
> > works
> > > > fine. But, as an executable, when I try to run code with a copy of
> > Python 2,
> > > > I get an exception.
> > > >
> > > > It took me a long time to narrow the problem down to this code:
> > > >
> > > > (test.py)
> > > > import execnet
> > > >
> > > >
> > > > def main():
> > > >
> > > > gw = execnet.makegateway('popen//python=c:/dev/Python26/python.exe')
> > #
> > > > this runs with an exception
> > > >
> > > > # gw = execnet.makegateway('popen//python=c:/dev/Python31/python.exe')
> >  #
> > > > this runs fine
> > > >
> > > > channel = gw.remote_exec('channel.send(1)')
> > > >
> > > > value = channel.receive()   # the exception is raised here
> > > >
> > > > print(value)
> > > >
> > > >
> > > >  if __name__ == '__main__':
> > > >
> > > > main()
> > > >
> > > >
> > > > I am creating the executable with this:
> > > >
> > > > (setup.py)
> > > > from cx_Freeze import setup, Executable
> > > >
> > > >  includes = [
> > > >
> > > > 'execnet.multi', 'execnet.xspec', 'execnet.threadpool',
> > > >
> > > > 'execnet.gateway', 'execnet.gateway_base', 'execnet.gateway_socket',
> > > >
> > > > 'execnet.rsync', 'execnet.rsync_remote']
> > > >
> > > >  setup(
> > > >
> > > > name = "test",
> > > >
> > > > executables = [Executable("test.py")],
> > > >
> > > > options = {
> > > >
> > > >  'build_exe': {
> > > >
> > > >  'includes': includes,
> > > >
> > > >  }
> > > >
> > > >  },
> > > >
> > > >  )
> > > >
> > > >
> > > > Here is the traceback:
> > > >
> > > >
> > > > Traceback (most recent call last):
> > > >
> > > >   File
> > > > "C:\dev\Python31\lib\site-packages\cx_Freeze\initscripts\Console3.py",
> > line
> > > > 27, in <module>
> > > >
> > > >     exec(code, m.__dict__)
> > > >
> > > >   File "test.py", line 11, in <module>
> > > >
> > > >     main()
> > > >
> > > >   File "test.py", line 6, in main
> > > >
> > > >     channel = gw.remote_exec('channel.send(1)')
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway.py",
> > > > line 116, in remote_exec
> > > >
> > > >     (source, call_name, kwargs)))
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > line 652, in _send
> > > >
> > > >     msg.writeto(self._io)
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > line 112, in writeto
> > > >
> > > >     serialize(io, (self.msgtype, self.channelid, self.data))
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > line 939, in serialize
> > > >
> > > >     _Serializer(io).save(obj)
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > line 959, in save
> > > >
> > > >     self._stream.write(s)
> > > >
> > > >   File
> > > >
> > "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > line 95, in write
> > > >
> > > >     self.outfile.flush()
> > > >
> > > > IOError: [Errno 22] Invalid argument
> > > >
> > > >
> > > > Can someone help me figure out what's going on here?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > miguel
> > > >
> > > >
> > > > PS: an unrelated issue is the need to explicitly include the execnet
> > > > modules for cx_Freeze, due to the use of apipkg.
> > > >
> > > >
> >
> > > _______________________________________________
> > > execnet-dev mailing list
> > > execnet-dev at codespeak.net
> > > http://codespeak.net/mailman/listinfo/execnet-dev
> >
> >
> > --
> >

-- 


From cymrow at gmail.com  Mon Nov 22 01:36:40 2010
From: cymrow at gmail.com (M T)
Date: Sun, 21 Nov 2010 19:36:40 -0500
Subject: [execnet-dev] Issue running code in python2 from a python3
 script frozen by cx_Freeze
In-Reply-To: <20101119070959.GP1009@trillke.net>
References: <AANLkTi=ozkR3ahVnhh6BERAQQ2uWEysqLm97RdHzSbS4@mail.gmail.com>
	<AANLkTinNbab=WwmnqwR9LF_mR92P=3g_Forf8XOWfZfN@mail.gmail.com>
	<20101118100709.GL1009@trillke.net>
	<AANLkTinXvpqbcTtand0zxjRi-Rr10UKUxZb8sXA=nSXs@mail.gmail.com>
	<20101119070959.GP1009@trillke.net>
Message-ID: <AANLkTi=Mm+P75EAUNYE1N5K0S-0Lq+ngF_cCEEYMd--2@mail.gmail.com>

On Fri, Nov 19, 2010 at 2:09 AM, holger krekel <holger at merlinux.eu> wrote:

> On Thu, Nov 18, 2010 at 21:09 -0500, M T wrote:
> > On Thu, Nov 18, 2010 at 5:07 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > Hi Miguel,
> > >
> > > On Tue, Nov 16, 2010 at 20:15 -0500, M T wrote:
> > > > I think I've tracked down the problem. It seems execnet points a
> python
> > > > subprocess to the path of the execnet for the main process and tries
> to
> > > get
> > > > the subprocess to load that. In a frozen cx_Freeze executable,
> though,
> > > all
> > > > the main process' modules are compiled python byte-code, which in my
> case
> > > > would be python3 byte-code, so python2's zipimport can't load it.
> > >
> > > correct analysis i think.  What app/package are you working on, btw?
> > >
> >
> > A hobby project, really. I'm doing some python introspection as part of a
> > larger program. execnet is perfect for this, since it lets me dynamically
> > access modules written for different python versions.
>
> I am interested in a source pointer if you distribute it at some point.
>

If I ever get there, I'd be happy to share :).

>
> > > > Not sure what the best solution to this is. It would be nice if all
> of
> > > the
> > > > bootstrap code could be sent through sendexec. For now I will try
> keeping
> > > an
> > > > uncompiled copy of execnet with my frozen executable. Any other ideas
> > > would
> > > > be appreciated, though.
> > >
> > > If you use ssh- or socket-gateways the neccessary bootstrap will be
> > > send - the content of gateway_base.py actually.
> > >
> > > "popen" gateways take a shortcut by not sending this source but
> > > just using the filesystem location which bites your usage,
> unfortunately.
> > >
> > > So two bits needed to resolve this:
> > >
> > > * the source code of gateway_base.py needs to be kept
> > > * a way to configure to not use the shortcut, maybe a new
> > >  gateway type "srcpopen" or so?
> > >
> >
> > I got it working by having cx_Freeze copy the .py files directly into the
> > built exe (in this case, a zip file embedded in the exe). The python2
> > subprocess is then actually able to import it directly from the
> executable
> > based on the path execnet gets from __file__ (pretty cool, really). I may
> do
> > something like srcpopen at some point just to be a little more efficient.
> I
> > also have some minor fixes for frozen Windows exe's. Perhaps this weekend
> I
> > can send you a pull request on bitbucket.
>
> Saw you opened a branch already and your "expr if cond else other" style
> addition ... note that execnet is meant to run on Python2.4.  You can check
> compatibility by running "tox -e py24 -e py31" for example.  You might also
> add a configuration into tox.ini that performs a cx_freeze related
> test/run.
> see http://codespeak.net/tox on tox documentation.
>

Fixed. I'm used to only having to support python2.5, so I missed that. I've
also added the option to set "importdir" on the popen xspec. That way the
location of gateway_base.py can be set manually if needed. A few manual
steps are still needed to get everything in the right place for cx_Freeze,
but I don't see any simple way to automate this. Even using the 'if False'
trick, you still have to copy gateway_base.py to the right location.

>
> > > As to including the packages: how does cx_Freeze discover
> > > the modules/packages usually?   If you do a simple "execnet.__dict__"
> > > all sub packages will be imported i think - maybe that helps?
> > >
> > > I'd have to look into this more. I know that the pyglet package uses
> lazy
> > module loading and gets around it using "if False: import module". Since
> > that works, I think they must be scanning the source for imports. py2exe
> is
> > not able to discover lazy loaded modules either.
>

> ah, i see.  good luck with your experiments!
> cheers,
> holger


> (P.S.: re-CCed execnet-dev - thinks it's good to have this conversation
> in the archives/seen by others)
>
> > thanks,
> > miguel
> >
> >
> > > cheers,
> > > holger
> > >
> > > > Thanks,
> > > > miguel
> > > >
> > > > On Mon, Nov 15, 2010 at 10:48 PM, M T <cymrow at gmail.com> wrote:
> > > >
> > > > > I'm trying to use execnet from a Python 3 script that I've made
> into an
> > > > > executable with cx_Freeze. As a script, the code runs fine, and as
> an
> > > > > executable, when I try to run code with a separate copy of Python 3
> it
> > > works
> > > > > fine. But, as an executable, when I try to run code with a copy of
> > > Python 2,
> > > > > I get an exception.
> > > > >
> > > > > It took me a long time to narrow the problem down to this code:
> > > > >
> > > > > (test.py)
> > > > > import execnet
> > > > >
> > > > >
> > > > > def main():
> > > > >
> > > > > gw =
> execnet.makegateway('popen//python=c:/dev/Python26/python.exe')
> > > #
> > > > > this runs with an exception
> > > > >
> > > > > # gw =
> execnet.makegateway('popen//python=c:/dev/Python31/python.exe')
> > >  #
> > > > > this runs fine
> > > > >
> > > > > channel = gw.remote_exec('channel.send(1)')
> > > > >
> > > > > value = channel.receive()   # the exception is raised here
> > > > >
> > > > > print(value)
> > > > >
> > > > >
> > > > >  if __name__ == '__main__':
> > > > >
> > > > > main()
> > > > >
> > > > >
> > > > > I am creating the executable with this:
> > > > >
> > > > > (setup.py)
> > > > > from cx_Freeze import setup, Executable
> > > > >
> > > > >  includes = [
> > > > >
> > > > > 'execnet.multi', 'execnet.xspec', 'execnet.threadpool',
> > > > >
> > > > > 'execnet.gateway', 'execnet.gateway_base',
> 'execnet.gateway_socket',
> > > > >
> > > > > 'execnet.rsync', 'execnet.rsync_remote']
> > > > >
> > > > >  setup(
> > > > >
> > > > > name = "test",
> > > > >
> > > > > executables = [Executable("test.py")],
> > > > >
> > > > > options = {
> > > > >
> > > > >  'build_exe': {
> > > > >
> > > > >  'includes': includes,
> > > > >
> > > > >  }
> > > > >
> > > > >  },
> > > > >
> > > > >  )
> > > > >
> > > > >
> > > > > Here is the traceback:
> > > > >
> > > > >
> > > > > Traceback (most recent call last):
> > > > >
> > > > >   File
> > > > >
> "C:\dev\Python31\lib\site-packages\cx_Freeze\initscripts\Console3.py",
> > > line
> > > > > 27, in <module>
> > > > >
> > > > >     exec(code, m.__dict__)
> > > > >
> > > > >   File "test.py", line 11, in <module>
> > > > >
> > > > >     main()
> > > > >
> > > > >   File "test.py", line 6, in main
> > > > >
> > > > >     channel = gw.remote_exec('channel.send(1)')
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway.py",
> > > > > line 116, in remote_exec
> > > > >
> > > > >     (source, call_name, kwargs)))
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > > line 652, in _send
> > > > >
> > > > >     msg.writeto(self._io)
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > > line 112, in writeto
> > > > >
> > > > >     serialize(io, (self.msgtype, self.channelid, self.data))
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > > line 939, in serialize
> > > > >
> > > > >     _Serializer(io).save(obj)
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > > line 959, in save
> > > > >
> > > > >     self._stream.write(s)
> > > > >
> > > > >   File
> > > > >
> > >
> "C:\dev\Python31\lib\site-packages\execnet-1.0.8-py3.1.egg\execnet\gateway_base.py",
> > > > > line 95, in write
> > > > >
> > > > >     self.outfile.flush()
> > > > >
> > > > > IOError: [Errno 22] Invalid argument
> > > > >
> > > > >
> > > > > Can someone help me figure out what's going on here?
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > miguel
> > > > >
> > > > >
> > > > > PS: an unrelated issue is the need to explicitly include the
> execnet
> > > > > modules for cx_Freeze, due to the use of apipkg.
> > > > >
> > > > >
> > >
> > > > _______________________________________________
> > > > execnet-dev mailing list
> > > > execnet-dev at codespeak.net
> > > > http://codespeak.net/mailman/listinfo/execnet-dev
> > >
> > >
> > > --
> > >
>
> --
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20101121/9e903fce/attachment.html>

From holger at merlinux.eu  Thu Nov 25 18:03:42 2010
From: holger at merlinux.eu (holger krekel)
Date: Thu, 25 Nov 2010 18:03:42 +0100
Subject: [execnet-dev] execnet-1.0.9: small refinements
Message-ID: <20101125170342.GZ1009@trillke.net>

Hi all,

just released execnet-1.0.9, the rapid multi-Python deployment library.
1.0. contains some small additions and refinements, mostly 
implemented by Ronny Pfannschmidt. See the docs fore more info:

    http://codespeak.net/execnet

cheers,
holger

1.0.9
--------------------------------

- add gw.reconfigure() to configure per gateway options.  Currently supported:
  py2str_as_py3str and py3str_as_py2str to configure string deserialization

- channel.makefile() objects now have a isatty() returning False

- group.allocate_id(spec) allows to early-determine an (automatic) id

- internal refactorings and cleanups (thanks Ronny Pfannschmidt):
  - refactor message types into received handler functions
  - refactor b(chr(opcode)) to bchr(opcode)
  - reorder Message ctor args, rename msgtype to msgcode
  - refactor gateway.send to take message's init args instead of a message
  - inline and remove Message.writeto/readfrom
  - refactor collection loading to avoid the indirection over tuple
  - remove the unused NamedThreadPool