[execnet-dev] Sub-processes PyPy
Luis José Novoa
luisjosenovoa at gmail.com
Wed Nov 18 11:13:09 EST 2015
Thanks. I am indeed using the latest pypy version for windows under Python
2.7 (pypy 4.0.0).
One thing I noticed was that executing just one gateway and a channel under
pypy does not cause problems. The problem appears when retrieving results
from channels in two or more gws.
LJN
On Nov 18, 2015 10:57 AM, "holger krekel" <holger at merlinux.eu> wrote:
> On Wed, Nov 18, 2015 at 09:40 -0500, Luis José Novoa wrote:
> > Thanks very much for your reply. In the command line I get no error, as
> > shown in the image below.
> >
> > [image: Inline image 1]
> >
> > The error appears after executing queue = mch.make_receive_queue().
> >
> >
> > I appreciate all the help.
>
> not sure what is different wrt to pypy/cpython here.
> Make sure you are running the newest pypy versions.
> Maybe Ronny can chime in at some point.
>
> holger
>
> > LJ
> >
> > On Wed, Nov 18, 2015 at 3:52 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > On Tue, Nov 17, 2015 at 20:55 -0500, Luis José Novoa wrote:
> > > > Hi All. Thanks in advance for the time invested in reading, thinking
> > > about,
> > > > and responding to my emails.
> > > >
> > > > Right now, I have something like this:
> > > >
> > > >
> > > > import execnet as execnet
> > > > import time
> > > > import outt
> > > >
> > > >
> > > > if __name__ == "__main__":
> > > >
> > > > start = time.time()
> > > >
> > > > factor11 = 20
> > > > factor12 = 10
> > > > factor13 = 200
> > > >
> > > > factor21 = 5
> > > > factor22 = 5
> > > > factor23 = 200
> > > >
> > > >
> > > > group = execnet.Group(['popen'] * 2)
> > > >
> > > > channel1 = group['gw0'].remote_exec(outt)
> > > > channel1.send([factor11,factor12,factor13,10])
> > > > channel2 = group['gw1'].remote_exec(outt)
> > > > channel2.send([factor21,factor22,factor23,4])
> > > >
> > > > mch = execnet.MultiChannel([channel1, channel2])
> > > > queue = mch.make_receive_queue()
> > > >
> > > > results = []
> > > > for i in range(2):
> > > > results.append(queue.get())
> > > > print results
> > > >
> > > >
> > > > group.terminate()
> > > >
> > > > end = time.time() - start
> > > >
> > > > print end, 'seconds'
> > > >
> > > >
> > > > where, outt is a module containing (please do not try to make sense
> out
> > > of
> > > > whatever is done inside the multiplier function here):
> > > >
> > > > if __name__ == '__channelexec__':
> > > >
> > > > a = channel.receive()
> > > >
> > > >
> > > > def summi(p):
> > > > return p**2
> > > >
> > > >
> > > > def multiplier(channel, factor1, factor2, factor3, a):
> > > > param = a[3]
> > > > r = summi(p=param)
> > > > s = []
> > > > d = []
> > > > for i in xrange(900000):
> > > > s.append(i)
> > > > d.append(i)
> > > > channel.send({'Number':factor1+factor2, 'list': sum(i
> for i
> > > in
> > > > s)})
> > > >
> > > >
> > > > multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)
> > >
> > > I'd recommend putting the actual excuted code on both sides into
> > > functions and just having the ``if __name__ == ...`` sections call
> > > into those functions. Then you don't need to indent function
> definitions.
> > >
> > > > Im not sure if this is the best way to proceed, so if you see
> something
> > > > that does not make sense, Id really appreciate you letting me know.
> If I
> > > > run the first piece of code, I get the expected results (althought
> Im not
> > > > sure if the subprocesses are actually running in parallel), but when
> I
> > > > specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] * 2))
> I
> > > get
> > > > the following error:
> > > >
> > > >
> > > > [5976] Warning: unhandled RemoteError: Traceback (most recent call
> last):
> > > > File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 841, in _local_receive
> > > > data = loads_internal(data, channel, strconfig)
> > > > File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 1350, in loads_internal
> > > > return Unserializer(io, channelfactory, strconfig).load()
> > > > File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 1160, in load
> > > > "wire protocol corruption?" % (opcode,))
> > > > LoadError: unkown opcode '\n' - wire protocol corruption?
> > >
> > > Not sure how this comes. If you just do
> > >
> > > >>> execnet.makegateway("popen//python=pypy.exe")
> > >
> > > on the interactive prompt, what happens?
> > >
> > > holger
> > >
> > >
> > >
> > > >
> > > > Once more, thanks for all the help.
> > > >
> > > > On Tue, Nov 17, 2015 at 8:20 AM, Luis José Novoa <
> > > luisjosenovoa at gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you again for your kind reply. When passing a module, can I
> pass
> > > > > the parameters corresponding to the function in the module that
> calls
> > > > > other functions?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > LJN
> > > > > On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu>
> wrote:
> > > > >
> > > > >> On Mon, Nov 16, 2015 at 16:56 -0500, Luis José Novoa wrote:
> > > > >> > Thanks again for the kind reply. After going through the
> > > multichannels
> > > > >> > examples I have this toy example:
> > > > >> >
> > > > >> > def summi(p):
> > > > >> > return p**2
> > > > >> >
> > > > >> >
> > > > >> > def multiplier(channel, factor1, factor2, factor3):
> > > > >> > while not channel.isclosed():
> > > > >> > param = channel.receive()
> > > > >> > r = summi(p=param)
> > > > >> > channel.send(factor1 * factor2 + factor3 + r)
> > > > >> >
> > > > >> > if __name__ == "__main__":
> > > > >> >
> > > > >> > factor11 = 20
> > > > >> > factor12 = 10
> > > > >> > factor13 = 200
> > > > >> >
> > > > >> > factor21 = 5
> > > > >> > factor22 = 5
> > > > >> > factor23 = 200
> > > > >> >
> > > > >> >
> > > > >> > gw = execnet.makegateway()
> > > > >> > channel1 = gw.remote_exec(multiplier, factor1=factor11,
> > > > >> > factor2=factor12, factor3=factor13)
> > > > >> > channel1.send(1)
> > > > >> > gw2 = execnet.makegateway()
> > > > >> > channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> > > > >> > factor2=factor22, factor3=factor23)
> > > > >> > channel2.send(1)
> > > > >> > mch = execnet.MultiChannel([channel1, channel2])
> > > > >> > queue = mch.make_receive_queue()
> > > > >> > results = []
> > > > >> > for i in range(2):
> > > > >> > results.append(queue.get())
> > > > >> > print results
> > > > >> >
> > > > >> >
> > > > >> > Now, when running it I get the following error:
> > > > >> >
> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
> > > > >> ['summi']).
> > > > >> >
> > > > >> > I tried the following:
> > > > >> >
> > > > >> > def multiplier(channel, factor1, factor2, factor3):
> > > > >> > def summi(p):
> > > > >> > return p**2
> > > > >> > while not channel.isclosed():
> > > > >> > param = channel.receive()
> > > > >> > r = summi(p=param)
> > > > >> > channel.send(factor1 * factor2 + factor3 + r)
> > > > >> >
> > > > >> >
> > > > >> > and with this change I got:
> > > > >> >
> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
> ['p',
> > > > >> 'p'])
> > > > >>
> > > > >> You need to send a module rather than just a function if your
> > > > >> function depends on another function in that module.
> > > > >> execnet only sends the source of the function, otherwise.
> > > > >> See here also:
> > > > >>
> > > > >>
> > > > >>
> > >
> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
> > > > >>
> > > > >> cheers,
> > > > >> holger
> > > > >>
> > > > >>
> > > > >> >
> > > > >> >
> > > > >> > I am just trying to understand the nature of the errors
> reported,
> > > and
> > > > >> > possibly find a way around it. The function calls that I want to
> > > > >> > parallelize invoke other functions from within, just as in the
> toy
> > > > >> example.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Any clarification is much appreciated.
> > > > >> >
> > > > >> > Thank you in advance.
> > > > >> >
> > > > >> > Luis.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <
> holger at merlinux.eu>
> > > > >> wrote:
> > > > >> >
> > > > >> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
> > > > >> > > > Thanks for your reply. I was trying to replicate one of the
> > > execnet
> > > > >> > > > examples available online and had problems using PyPy due to
> > > pydev
> > > > >> (using
> > > > >> > > > aptana under windows). But now it works. Thanks.
> > > > >> > > >
> > > > >> > > > Now, this is the kind of procedure that I am trying to
> replicate
> > > > >> (which
> > > > >> > > > currently uses the multiprocessing module and uses CPython
> to
> > > solve
> > > > >> the
> > > > >> > > > subproblems in parallel):
> > > > >> > > >
> > > > >> > > > out = mp.Queue()
> > > > >> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b,
> c,
> > > d,
> > > > >> out))
> > > > >> > > for
> > > > >> > > > s in range(len(networks))]
> > > > >> > > >
> > > > >> > > > *# Run subproblems*
> > > > >> > > > for sb in subproblems:
> > > > >> > > > sb.start()
> > > > >> > > >
> > > > >> > > > *# Get results from the out queue*
> > > > >> > > > results = [out.get() for sb in subproblems]
> > > > >> > >
> > > > >> > > did you look into the multi-channel examples?
> > > > >> > >
> > > > >> > > http://codespeak.net/execnet/example/test_multi.html
> > > > >> > >
> > > > >> > > you can start a pypy subprocess like so:
> > > > >> > >
> > > > >> > > execnet.makegateway("popen//python=pypy")
> > > > >> > >
> > > > >> > > and that should work if used in the multichannel examples.
> > > > >> > >
> > > > >> > > holger
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > So, what I would like to do, is to solve the subproblems
> with
> > > PyPy,
> > > > >> as
> > > > >> > > they
> > > > >> > > > are time consuming. I would like to do this using execnet,
> but I
> > > > >> still
> > > > >> > > > haven't been able to get my head around it (Im clearly not
> even
> > > > >> close to
> > > > >> > > an
> > > > >> > > > expert in these matters). I wonder if anyone can point me
> in the
> > > > >> right
> > > > >> > > > direction, maybe through an example, on how to do this.
> > > > >> > > >
> > > > >> > > > Thanks very much in advance. I apologize for any
> inconvenience
> > > this
> > > > >> may
> > > > >> > > > cause.
> > > > >> > > >
> > > > >> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
> > > > >> fijall at gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Luis
> > > > >> > > > >
> > > > >> > > > > Things should work on pypy without any problem whatsoever,
> > > just
> > > > >> like
> > > > >> > > > > with normal python.
> > > > >> > > > >
> > > > >> > > > > Cheers,
> > > > >> > > > > fijal
> > > > >> > > > >
> > > > >> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <
> > > > >> > > luisjosenovoa at gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > > > Hi Everyone,
> > > > >> > > > > >
> > > > >> > > > > > As part of a large scale optimization application, I am
> > > using
> > > > >> the
> > > > >> > > python
> > > > >> > > > > > multiprocessing module to execute a procedure several
> times
> > > in
> > > > >> > > parallel.
> > > > >> > > > > > These executions in turn, return results used by other
> > > > >> procedure,
> > > > >> > > and the
> > > > >> > > > > > whole thing repeats iteratively. I would like to call
> PyPy
> > > only
> > > > >> for
> > > > >> > > the
> > > > >> > > > > > subprocesses (to improve performance), as in the main
> > > procedure
> > > > >> I use
> > > > >> > > > > > libraries like gurobipy, which are not compatible with
> > > PyPy. I
> > > > >> > > looked at
> > > > >> > > > > the
> > > > >> > > > > > documentation of execnet online, but I could not find
> > > examples
> > > > >> for
> > > > >> > > > > > communication between CPython and PyPy.
> > > > >> > > > > >
> > > > >> > > > > > Any suggestions for using CPython + Multiprocessing with
> > > PyPy
> > > > >> would
> > > > >> > > be
> > > > >> > > > > > highly appreciated.
> > > > >> > > > > >
> > > > >> > > > > > Have a great day.
> > > > >> > > > > >
> > > > >> > > > > > --
> > > > >> > > > > > Luis J. Novoa
> > > > >> > > > > >
> > > > >> > > > > > _______________________________________________
> > > > >> > > > > > execnet-dev mailing list
> > > > >> > > > > > execnet-dev at python.org
> > > > >> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Luis J. Novoa
> > > > >> > >
> > > > >> > > > _______________________________________________
> > > > >> > > > execnet-dev mailing list
> > > > >> > > > execnet-dev at python.org
> > > > >> > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > about me: http://holgerkrekel.net/about-me/
> > > > >> > > contracting: http://merlinux.eu
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Luis J. Novoa
> > > > >>
> > > > >> --
> > > > >> about me: http://holgerkrekel.net/about-me/
> > > > >> contracting: http://merlinux.eu
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Luis J. Novoa
> > >
> > > --
> > > about me: http://holgerkrekel.net/about-me/
> > > contracting: http://merlinux.eu
> > >
> >
> >
> >
> > --
> > Luis J. Novoa
>
>
>
> --
> about me: http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151118/368e0e00/attachment-0001.html>
More information about the execnet-dev
mailing list