[execnet-dev] Sub-processes PyPy

[holger krekel]

On Tue, Nov 10, 2015 at 12:46 -0500, Luis José Novoa wrote:
> Thanks for your reply. I was trying to replicate one of the execnet
> examples available online and had problems using PyPy due to pydev (using
> aptana under windows). But now it works. Thanks.
> 
> Now, this is the kind of procedure that I am trying to replicate (which
> currently uses the multiprocessing module and uses CPython to solve the
> subproblems in parallel):
> 
> out = mp.Queue()
> subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out)) for
> s in range(len(networks))]
> 
> *# Run subproblems*
> for sb in subproblems:
>     sb.start()
> 
> *# Get results from the out queue*
> results = [out.get() for sb in subproblems]

did you look into the multi-channel examples?

    http://codespeak.net/execnet/example/test_multi.html

you can start a pypy subprocess like so:

    execnet.makegateway("popen//python=pypy")

and that should work if used in the multichannel examples.

holger


> 
> So, what I would like to do, is to solve the subproblems with PyPy, as they
> are time consuming. I would like to do this using execnet, but I still
> haven't been able to get my head around it (Im clearly not even close to an
> expert in these matters). I wonder if anyone can point me in the right
> direction, maybe through an example, on how to do this.
> 
> Thanks very much in advance. I apologize for any inconvenience this may
> cause.
> 
> On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> 
> > Hi Luis
> >
> > Things should work on pypy without any problem whatsoever, just like
> > with normal python.
> >
> > Cheers,
> > fijal
> >
> > On Thu, Nov 5, 2015 at 4:36 PM, Luis José Novoa <luisjosenovoa at gmail.com>
> > wrote:
> > > Hi Everyone,
> > >
> > > As part of a large scale optimization application, I am using the python
> > > multiprocessing module to execute a procedure several times in parallel.
> > > These executions in turn, return results used by other procedure, and the
> > > whole thing repeats iteratively. I would like to call PyPy only for the
> > > subprocesses (to improve performance), as in the main procedure I use
> > > libraries like gurobipy, which are not compatible with PyPy. I looked at
> > the
> > > documentation of execnet online, but I could not find examples for
> > > communication between CPython and PyPy.
> > >
> > > Any suggestions for using CPython + Multiprocessing with PyPy would be
> > > highly appreciated.
> > >
> > > Have a great day.
> > >
> > > --
> > > Luis J. Novoa
> > >
> > > _______________________________________________
> > > execnet-dev mailing list
> > > execnet-dev at python.org
> > > https://mail.python.org/mailman/listinfo/execnet-dev
> > >
> >
> 
> 
> 
> -- 
> Luis J. Novoa

> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev


-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu