[execnet-dev] Sub-processes PyPy

[holger krekel]

On Mon, Nov 16, 2015 at 16:56 -0500, Luis José Novoa wrote:
> Thanks again for the kind reply. After going through the multichannels
> examples I have this toy example:
> 
> def summi(p):
>     return p**2
> 
> 
> def multiplier(channel, factor1, factor2, factor3):
>     while not channel.isclosed():
>         param = channel.receive()
>         r = summi(p=param)
>         channel.send(factor1 * factor2 + factor3 + r)
> 
> if __name__ == "__main__":
> 
>     factor11 = 20
>     factor12 = 10
>     factor13 = 200
> 
>     factor21 = 5
>     factor22 = 5
>     factor23 = 200
> 
> 
>     gw = execnet.makegateway()
>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> factor2=factor12, factor3=factor13)
>     channel1.send(1)
>     gw2 = execnet.makegateway()
>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> factor2=factor22, factor3=factor23)
>     channel2.send(1)
>     mch = execnet.MultiChannel([channel1, channel2])
>     queue = mch.make_receive_queue()
>     results = []
>     for i in range(2):
>         results.append(queue.get())
>     print results
> 
> 
> Now, when running it I get the following error:
> 
> ValueError: ("the use of non-builtin globals isn't supported", ['summi']).
> 
> I tried the following:
> 
> def multiplier(channel, factor1, factor2, factor3):
>      def summi(p):
>          return p**2
>     while not channel.isclosed():
>         param = channel.receive()
>         r = summi(p=param)
>         channel.send(factor1 * factor2 + factor3 + r)
> 
> 
> and with this change I got:
> 
> ValueError: ("the use of non-builtin globals isn't supported", ['p', 'p'])

You need to send a module rather than just a function if your
function depends on another function in that module.
execnet only sends the source of the function, otherwise.
See here also:

http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii

cheers,
holger


> 
> 
> I am just trying to understand the nature of the errors reported, and
> possibly find a way around it. The function calls that I want to
> parallelize invoke other functions from within, just as in the toy example.
> 
> 
> 
> Any clarification is much appreciated.
> 
> Thank you in advance.
> 
> Luis.
> 
> 
> 
> 
> 
> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
> > > Thanks for your reply. I was trying to replicate one of the execnet
> > > examples available online and had problems using PyPy due to pydev (using
> > > aptana under windows). But now it works. Thanks.
> > >
> > > Now, this is the kind of procedure that I am trying to replicate (which
> > > currently uses the multiprocessing module and uses CPython to solve the
> > > subproblems in parallel):
> > >
> > > out = mp.Queue()
> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out))
> > for
> > > s in range(len(networks))]
> > >
> > > *# Run subproblems*
> > > for sb in subproblems:
> > >     sb.start()
> > >
> > > *# Get results from the out queue*
> > > results = [out.get() for sb in subproblems]
> >
> > did you look into the multi-channel examples?
> >
> >     http://codespeak.net/execnet/example/test_multi.html
> >
> > you can start a pypy subprocess like so:
> >
> >     execnet.makegateway("popen//python=pypy")
> >
> > and that should work if used in the multichannel examples.
> >
> > holger
> >
> >
> > >
> > > So, what I would like to do, is to solve the subproblems with PyPy, as
> > they
> > > are time consuming. I would like to do this using execnet, but I still
> > > haven't been able to get my head around it (Im clearly not even close to
> > an
> > > expert in these matters). I wonder if anyone can point me in the right
> > > direction, maybe through an example, on how to do this.
> > >
> > > Thanks very much in advance. I apologize for any inconvenience this may
> > > cause.
> > >
> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
> > > wrote:
> > >
> > > > Hi Luis
> > > >
> > > > Things should work on pypy without any problem whatsoever, just like
> > > > with normal python.
> > > >
> > > > Cheers,
> > > > fijal
> > > >
> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <
> > luisjosenovoa at gmail.com>
> > > > wrote:
> > > > > Hi Everyone,
> > > > >
> > > > > As part of a large scale optimization application, I am using the
> > python
> > > > > multiprocessing module to execute a procedure several times in
> > parallel.
> > > > > These executions in turn, return results used by other procedure,
> > and the
> > > > > whole thing repeats iteratively. I would like to call PyPy only for
> > the
> > > > > subprocesses (to improve performance), as in the main procedure I use
> > > > > libraries like gurobipy, which are not compatible with PyPy. I
> > looked at
> > > > the
> > > > > documentation of execnet online, but I could not find examples for
> > > > > communication between CPython and PyPy.
> > > > >
> > > > > Any suggestions for using CPython + Multiprocessing with PyPy would
> > be
> > > > > highly appreciated.
> > > > >
> > > > > Have a great day.
> > > > >
> > > > > --
> > > > > Luis J. Novoa
> > > > >
> > > > > _______________________________________________
> > > > > execnet-dev mailing list
> > > > > execnet-dev at python.org
> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Luis J. Novoa
> >
> > > _______________________________________________
> > > execnet-dev mailing list
> > > execnet-dev at python.org
> > > https://mail.python.org/mailman/listinfo/execnet-dev
> >
> >
> > --
> > about me:    http://holgerkrekel.net/about-me/
> > contracting: http://merlinux.eu
> >
> 
> 
> 
> -- 
> Luis J. Novoa

-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu