[execnet-dev] Sub-processes PyPy
Luis José Novoa
luisjosenovoa at gmail.com
Tue Nov 17 20:55:07 EST 2015
Hi All. Thanks in advance for the time invested in reading, thinking about,
and responding to my emails.
Right now, I have something like this:
import execnet as execnet
import time
import outt
if __name__ == "__main__":
start = time.time()
factor11 = 20
factor12 = 10
factor13 = 200
factor21 = 5
factor22 = 5
factor23 = 200
group = execnet.Group(['popen'] * 2)
channel1 = group['gw0'].remote_exec(outt)
channel1.send([factor11,factor12,factor13,10])
channel2 = group['gw1'].remote_exec(outt)
channel2.send([factor21,factor22,factor23,4])
mch = execnet.MultiChannel([channel1, channel2])
queue = mch.make_receive_queue()
results = []
for i in range(2):
results.append(queue.get())
print results
group.terminate()
end = time.time() - start
print end, 'seconds'
where, outt is a module containing (please do not try to make sense out of
whatever is done inside the multiplier function here):
if __name__ == '__channelexec__':
a = channel.receive()
def summi(p):
return p**2
def multiplier(channel, factor1, factor2, factor3, a):
param = a[3]
r = summi(p=param)
s = []
d = []
for i in xrange(900000):
s.append(i)
d.append(i)
channel.send({'Number':factor1+factor2, 'list': sum(i for i in
s)})
multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)
Im not sure if this is the best way to proceed, so if you see something
that does not make sense, Id really appreciate you letting me know. If I
run the first piece of code, I get the expected results (althought Im not
sure if the subprocesses are actually running in parallel), but when I
specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] * 2)) I get
the following error:
[5976] Warning: unhandled RemoteError: Traceback (most recent call last):
File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 841, in _local_receive
data = loads_internal(data, channel, strconfig)
File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 1350, in loads_internal
return Unserializer(io, channelfactory, strconfig).load()
File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 1160, in load
"wire protocol corruption?" % (opcode,))
LoadError: unkown opcode '\n' - wire protocol corruption?
Once more, thanks for all the help.
On Tue, Nov 17, 2015 at 8:20 AM, Luis José Novoa <luisjosenovoa at gmail.com>
wrote:
> Thank you again for your kind reply. When passing a module, can I pass
> the parameters corresponding to the function in the module that calls
> other functions?
>
> Thank you.
>
> LJN
> On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu> wrote:
>
>> On Mon, Nov 16, 2015 at 16:56 -0500, Luis José Novoa wrote:
>> > Thanks again for the kind reply. After going through the multichannels
>> > examples I have this toy example:
>> >
>> > def summi(p):
>> > return p**2
>> >
>> >
>> > def multiplier(channel, factor1, factor2, factor3):
>> > while not channel.isclosed():
>> > param = channel.receive()
>> > r = summi(p=param)
>> > channel.send(factor1 * factor2 + factor3 + r)
>> >
>> > if __name__ == "__main__":
>> >
>> > factor11 = 20
>> > factor12 = 10
>> > factor13 = 200
>> >
>> > factor21 = 5
>> > factor22 = 5
>> > factor23 = 200
>> >
>> >
>> > gw = execnet.makegateway()
>> > channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> > factor2=factor12, factor3=factor13)
>> > channel1.send(1)
>> > gw2 = execnet.makegateway()
>> > channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> > factor2=factor22, factor3=factor23)
>> > channel2.send(1)
>> > mch = execnet.MultiChannel([channel1, channel2])
>> > queue = mch.make_receive_queue()
>> > results = []
>> > for i in range(2):
>> > results.append(queue.get())
>> > print results
>> >
>> >
>> > Now, when running it I get the following error:
>> >
>> > ValueError: ("the use of non-builtin globals isn't supported",
>> ['summi']).
>> >
>> > I tried the following:
>> >
>> > def multiplier(channel, factor1, factor2, factor3):
>> > def summi(p):
>> > return p**2
>> > while not channel.isclosed():
>> > param = channel.receive()
>> > r = summi(p=param)
>> > channel.send(factor1 * factor2 + factor3 + r)
>> >
>> >
>> > and with this change I got:
>> >
>> > ValueError: ("the use of non-builtin globals isn't supported", ['p',
>> 'p'])
>>
>> You need to send a module rather than just a function if your
>> function depends on another function in that module.
>> execnet only sends the source of the function, otherwise.
>> See here also:
>>
>>
>> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>>
>> cheers,
>> holger
>>
>>
>> >
>> >
>> > I am just trying to understand the nature of the errors reported, and
>> > possibly find a way around it. The function calls that I want to
>> > parallelize invoke other functions from within, just as in the toy
>> example.
>> >
>> >
>> >
>> > Any clarification is much appreciated.
>> >
>> > Thank you in advance.
>> >
>> > Luis.
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
>> wrote:
>> >
>> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
>> > > > Thanks for your reply. I was trying to replicate one of the execnet
>> > > > examples available online and had problems using PyPy due to pydev
>> (using
>> > > > aptana under windows). But now it works. Thanks.
>> > > >
>> > > > Now, this is the kind of procedure that I am trying to replicate
>> (which
>> > > > currently uses the multiprocessing module and uses CPython to solve
>> the
>> > > > subproblems in parallel):
>> > > >
>> > > > out = mp.Queue()
>> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
>> out))
>> > > for
>> > > > s in range(len(networks))]
>> > > >
>> > > > *# Run subproblems*
>> > > > for sb in subproblems:
>> > > > sb.start()
>> > > >
>> > > > *# Get results from the out queue*
>> > > > results = [out.get() for sb in subproblems]
>> > >
>> > > did you look into the multi-channel examples?
>> > >
>> > > http://codespeak.net/execnet/example/test_multi.html
>> > >
>> > > you can start a pypy subprocess like so:
>> > >
>> > > execnet.makegateway("popen//python=pypy")
>> > >
>> > > and that should work if used in the multichannel examples.
>> > >
>> > > holger
>> > >
>> > >
>> > > >
>> > > > So, what I would like to do, is to solve the subproblems with PyPy,
>> as
>> > > they
>> > > > are time consuming. I would like to do this using execnet, but I
>> still
>> > > > haven't been able to get my head around it (Im clearly not even
>> close to
>> > > an
>> > > > expert in these matters). I wonder if anyone can point me in the
>> right
>> > > > direction, maybe through an example, on how to do this.
>> > > >
>> > > > Thanks very much in advance. I apologize for any inconvenience this
>> may
>> > > > cause.
>> > > >
>> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
>> fijall at gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Luis
>> > > > >
>> > > > > Things should work on pypy without any problem whatsoever, just
>> like
>> > > > > with normal python.
>> > > > >
>> > > > > Cheers,
>> > > > > fijal
>> > > > >
>> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <
>> > > luisjosenovoa at gmail.com>
>> > > > > wrote:
>> > > > > > Hi Everyone,
>> > > > > >
>> > > > > > As part of a large scale optimization application, I am using
>> the
>> > > python
>> > > > > > multiprocessing module to execute a procedure several times in
>> > > parallel.
>> > > > > > These executions in turn, return results used by other
>> procedure,
>> > > and the
>> > > > > > whole thing repeats iteratively. I would like to call PyPy only
>> for
>> > > the
>> > > > > > subprocesses (to improve performance), as in the main procedure
>> I use
>> > > > > > libraries like gurobipy, which are not compatible with PyPy. I
>> > > looked at
>> > > > > the
>> > > > > > documentation of execnet online, but I could not find examples
>> for
>> > > > > > communication between CPython and PyPy.
>> > > > > >
>> > > > > > Any suggestions for using CPython + Multiprocessing with PyPy
>> would
>> > > be
>> > > > > > highly appreciated.
>> > > > > >
>> > > > > > Have a great day.
>> > > > > >
>> > > > > > --
>> > > > > > Luis J. Novoa
>> > > > > >
>> > > > > > _______________________________________________
>> > > > > > execnet-dev mailing list
>> > > > > > execnet-dev at python.org
>> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Luis J. Novoa
>> > >
>> > > > _______________________________________________
>> > > > execnet-dev mailing list
>> > > > execnet-dev at python.org
>> > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > >
>> > >
>> > > --
>> > > about me: http://holgerkrekel.net/about-me/
>> > > contracting: http://merlinux.eu
>> > >
>> >
>> >
>> >
>> > --
>> > Luis J. Novoa
>>
>> --
>> about me: http://holgerkrekel.net/about-me/
>> contracting: http://merlinux.eu
>>
>
--
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151117/968a1f15/attachment-0001.html>
More information about the execnet-dev
mailing list