From kelleyk at kelleyk.net  Thu Nov  5 06:02:11 2015
From: kelleyk at kelleyk.net (Kevin Kelley)
Date: Thu, 5 Nov 2015 03:02:11 -0800
Subject: [execnet-dev] Interest in pull requests?
In-Reply-To: <20151029090000.GD28212@merlinux.eu>
References: <CAOcMgv_pTVE9WO1=oD49LGnsu8k=o+vP-cfUN-ABROT8yRChyA@mail.gmail.com>
 <20151029090000.GD28212@merlinux.eu>
Message-ID: <CAOcMgv-1iDd8o0BV3HcHHz7NihG0RA=pB9LB=pJx5iy07yL0Ew@mail.gmail.com>

Howdy,

Sorry for the slow reply!  I have opened a pull request for 'execnet-sudo':
https://bitbucket.org/hpk42/execnet/pull-requests/20/add-initial-implementation-of-execnet-sudo/diff
(This one is pretty rough and the pull request describes some changes that
I think need to be made.)

'execnet-importhook' is a bit more separate and my implementation did not
require any changes to 'execnet' itself.  It could also use some clean-up,
but I have pushed the implementation that my project is now successfully
using: https://github.com/kelleyk/execnet-importhook

Cheers,

Kevin

On Thu, Oct 29, 2015 at 2:00 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi Kevin,
>
> On Tue, Oct 27, 2015 at 14:30 -0700, Kevin Kelley wrote:
> > Howdy!
> >
> > While working on a little project of mine, I've implemented two new
> > execnet-related features, and I'm curious if the maintainers would be
> > receptive to pull requests for them or if you guys think they would be
> > better kept separate.
> >
> > The first ("execnet-importhook") inserts itself into the import machinery
> > on the remote/slave side.  When an import is about to fail (because the
> > module is not present on the slave), the slave asks the master if the
> > master can find the module.  If it can, the master sends the slave the
> > module's source; the slave can then successfully import it.
> >
> > Right now this feature works with both absolute and relative imports.  It
> > only works with pure-Python modules (not with extension modules), though
> > adding support for those (in situations where the master and the slave
> are
> > similar enough to be using the same extension modules) would probably not
> > be hard.  It currently supports Python 3.4 and Python 3.5 on the slave
> > side.  Adding support for Python 2 is most likely possible; I would just
> > have to spend some time remembering exactly how the import machinery
> works
> > there, since it is rather different.  (I'll probably wind up doing this
> > eventually.)
> >
> > The current implementation doesn't change the source of execnet itself at
> > all; there's just an extra call right after you create the gateway.
>
> I wonder if we could introduce pluggy (a little one-module plugin system
> used by pytest and some other projects) and your importhook would
> become a plugin.  This allows you to update/fix things and also initially
> only support 3.4/3.5 if you like.
>
> Pluggy basically allows a program to add 1:N function calls (N = number
> of plugins implementing a particular function).
>
> > The second ("execnet-sudo") uses pexpect to watch for (and respond to)
> > password prompts so that you can use sudo on the remote/slave side to
> > launch a privileged interpreter.
> >
> > This requires that PopenPiped() also pipe the process's stderr and that a
> > hook be inserted into Popen2IOMaster.__init__() between the call to
> > PopenPiped() and Popen2IO.__init__().  The hook calls a function that,
> > generically, handles any connection setup; in this case, it's about 10
> > lines of pexpect code.
> >
> > I haven't tested exhaustively, but the current implementation seems to
> work
> > well with a good mix of Python 2 and Python 3 interpreters and Ubuntu and
> > OS X hosts.
> >
> > Let me know if you think it's worth my putting together pull requests for
> > any of this!
>
> Can you do PRs to show ronny and me (the current maintainers of execnet)
> the way how you implemented the two functionalities?  This is just to
> allow us to see what changed etc, not primarily aimed at a direct merge.
>
> cheers,
> holger
>
>
>
> > Cheers,
> >
> > Kevin
>
> > _______________________________________________
> > execnet-dev mailing list
> > execnet-dev at python.org
> > https://mail.python.org/mailman/listinfo/execnet-dev
>
>
> --
> about me:    http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151105/d06059f4/attachment.html>

From luisjosenovoa at gmail.com  Thu Nov  5 11:36:10 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Thu, 5 Nov 2015 11:36:10 -0500
Subject: [execnet-dev] Sub-processes PyPy
Message-ID: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>

Hi Everyone,

As part of a large scale optimization application, I am using the python
multiprocessing module to execute a procedure several times in parallel.
These executions in turn, return results used by other procedure, and the
whole thing repeats iteratively. I would like to call PyPy only for the
subprocesses (to improve performance), as in the main procedure I use
libraries like gurobipy, which are not compatible with PyPy. I looked at
the documentation of execnet online, but I could not find examples for
communication between CPython and PyPy.

Any suggestions for using CPython + Multiprocessing with PyPy would be
highly appreciated.
Have a great day.

-- 
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151105/9607103e/attachment.html>

From fijall at gmail.com  Sat Nov  7 10:02:23 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 7 Nov 2015 15:02:23 +0000
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
Message-ID: <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>

Hi Luis

Things should work on pypy without any problem whatsoever, just like
with normal python.

Cheers,
fijal

On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <luisjosenovoa at gmail.com> wrote:
> Hi Everyone,
>
> As part of a large scale optimization application, I am using the python
> multiprocessing module to execute a procedure several times in parallel.
> These executions in turn, return results used by other procedure, and the
> whole thing repeats iteratively. I would like to call PyPy only for the
> subprocesses (to improve performance), as in the main procedure I use
> libraries like gurobipy, which are not compatible with PyPy. I looked at the
> documentation of execnet online, but I could not find examples for
> communication between CPython and PyPy.
>
> Any suggestions for using CPython + Multiprocessing with PyPy would be
> highly appreciated.
>
> Have a great day.
>
> --
> Luis J. Novoa
>
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev
>

From luisjosenovoa at gmail.com  Tue Nov 10 12:46:43 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Tue, 10 Nov 2015 12:46:43 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
Message-ID: <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>

Thanks for your reply. I was trying to replicate one of the execnet
examples available online and had problems using PyPy due to pydev (using
aptana under windows). But now it works. Thanks.

Now, this is the kind of procedure that I am trying to replicate (which
currently uses the multiprocessing module and uses CPython to solve the
subproblems in parallel):

out = mp.Queue()
subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out)) for
s in range(len(networks))]

*# Run subproblems*
for sb in subproblems:
    sb.start()

*# Get results from the out queue*
results = [out.get() for sb in subproblems]


So, what I would like to do, is to solve the subproblems with PyPy, as they
are time consuming. I would like to do this using execnet, but I still
haven't been able to get my head around it (Im clearly not even close to an
expert in these matters). I wonder if anyone can point me in the right
direction, maybe through an example, on how to do this.

Thanks very much in advance. I apologize for any inconvenience this may
cause.

On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Luis
>
> Things should work on pypy without any problem whatsoever, just like
> with normal python.
>
> Cheers,
> fijal
>
> On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <luisjosenovoa at gmail.com>
> wrote:
> > Hi Everyone,
> >
> > As part of a large scale optimization application, I am using the python
> > multiprocessing module to execute a procedure several times in parallel.
> > These executions in turn, return results used by other procedure, and the
> > whole thing repeats iteratively. I would like to call PyPy only for the
> > subprocesses (to improve performance), as in the main procedure I use
> > libraries like gurobipy, which are not compatible with PyPy. I looked at
> the
> > documentation of execnet online, but I could not find examples for
> > communication between CPython and PyPy.
> >
> > Any suggestions for using CPython + Multiprocessing with PyPy would be
> > highly appreciated.
> >
> > Have a great day.
> >
> > --
> > Luis J. Novoa
> >
> > _______________________________________________
> > execnet-dev mailing list
> > execnet-dev at python.org
> > https://mail.python.org/mailman/listinfo/execnet-dev
> >
>



-- 
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151110/9848cc26/attachment.html>

From holger at merlinux.eu  Thu Nov 12 06:50:15 2015
From: holger at merlinux.eu (holger krekel)
Date: Thu, 12 Nov 2015 11:50:15 +0000
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
Message-ID: <20151112115015.GO16107@merlinux.eu>

On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> Thanks for your reply. I was trying to replicate one of the execnet
> examples available online and had problems using PyPy due to pydev (using
> aptana under windows). But now it works. Thanks.
> 
> Now, this is the kind of procedure that I am trying to replicate (which
> currently uses the multiprocessing module and uses CPython to solve the
> subproblems in parallel):
> 
> out = mp.Queue()
> subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out)) for
> s in range(len(networks))]
> 
> *# Run subproblems*
> for sb in subproblems:
>     sb.start()
> 
> *# Get results from the out queue*
> results = [out.get() for sb in subproblems]

did you look into the multi-channel examples?

    http://codespeak.net/execnet/example/test_multi.html

you can start a pypy subprocess like so:

    execnet.makegateway("popen//python=pypy")

and that should work if used in the multichannel examples.

holger


> 
> So, what I would like to do, is to solve the subproblems with PyPy, as they
> are time consuming. I would like to do this using execnet, but I still
> haven't been able to get my head around it (Im clearly not even close to an
> expert in these matters). I wonder if anyone can point me in the right
> direction, maybe through an example, on how to do this.
> 
> Thanks very much in advance. I apologize for any inconvenience this may
> cause.
> 
> On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> 
> > Hi Luis
> >
> > Things should work on pypy without any problem whatsoever, just like
> > with normal python.
> >
> > Cheers,
> > fijal
> >
> > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <luisjosenovoa at gmail.com>
> > wrote:
> > > Hi Everyone,
> > >
> > > As part of a large scale optimization application, I am using the python
> > > multiprocessing module to execute a procedure several times in parallel.
> > > These executions in turn, return results used by other procedure, and the
> > > whole thing repeats iteratively. I would like to call PyPy only for the
> > > subprocesses (to improve performance), as in the main procedure I use
> > > libraries like gurobipy, which are not compatible with PyPy. I looked at
> > the
> > > documentation of execnet online, but I could not find examples for
> > > communication between CPython and PyPy.
> > >
> > > Any suggestions for using CPython + Multiprocessing with PyPy would be
> > > highly appreciated.
> > >
> > > Have a great day.
> > >
> > > --
> > > Luis J. Novoa
> > >
> > > _______________________________________________
> > > execnet-dev mailing list
> > > execnet-dev at python.org
> > > https://mail.python.org/mailman/listinfo/execnet-dev
> > >
> >
> 
> 
> 
> -- 
> Luis J. Novoa

> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev


-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu

From luisjosenovoa at gmail.com  Mon Nov 16 16:56:34 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Mon, 16 Nov 2015 16:56:34 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <20151112115015.GO16107@merlinux.eu>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
Message-ID: <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>

Thanks again for the kind reply. After going through the multichannels
examples I have this toy example:

def summi(p):
    return p**2


def multiplier(channel, factor1, factor2, factor3):
    while not channel.isclosed():
        param = channel.receive()
        r = summi(p=param)
        channel.send(factor1 * factor2 + factor3 + r)

if __name__ == "__main__":

    factor11 = 20
    factor12 = 10
    factor13 = 200

    factor21 = 5
    factor22 = 5
    factor23 = 200


    gw = execnet.makegateway()
    channel1 = gw.remote_exec(multiplier, factor1=factor11,
factor2=factor12, factor3=factor13)
    channel1.send(1)
    gw2 = execnet.makegateway()
    channel2 = gw2.remote_exec(multiplier, factor1=factor21,
factor2=factor22, factor3=factor23)
    channel2.send(1)
    mch = execnet.MultiChannel([channel1, channel2])
    queue = mch.make_receive_queue()
    results = []
    for i in range(2):
        results.append(queue.get())
    print results


Now, when running it I get the following error:

ValueError: ("the use of non-builtin globals isn't supported", ['summi']).

I tried the following:

def multiplier(channel, factor1, factor2, factor3):
     def summi(p):
         return p**2
    while not channel.isclosed():
        param = channel.receive()
        r = summi(p=param)
        channel.send(factor1 * factor2 + factor3 + r)


and with this change I got:

ValueError: ("the use of non-builtin globals isn't supported", ['p', 'p'])



I am just trying to understand the nature of the errors reported, and
possibly find a way around it. The function calls that I want to
parallelize invoke other functions from within, just as in the toy example.



Any clarification is much appreciated.

Thank you in advance.

Luis.





On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu> wrote:

> On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> > Thanks for your reply. I was trying to replicate one of the execnet
> > examples available online and had problems using PyPy due to pydev (using
> > aptana under windows). But now it works. Thanks.
> >
> > Now, this is the kind of procedure that I am trying to replicate (which
> > currently uses the multiprocessing module and uses CPython to solve the
> > subproblems in parallel):
> >
> > out = mp.Queue()
> > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out))
> for
> > s in range(len(networks))]
> >
> > *# Run subproblems*
> > for sb in subproblems:
> >     sb.start()
> >
> > *# Get results from the out queue*
> > results = [out.get() for sb in subproblems]
>
> did you look into the multi-channel examples?
>
>     http://codespeak.net/execnet/example/test_multi.html
>
> you can start a pypy subprocess like so:
>
>     execnet.makegateway("popen//python=pypy")
>
> and that should work if used in the multichannel examples.
>
> holger
>
>
> >
> > So, what I would like to do, is to solve the subproblems with PyPy, as
> they
> > are time consuming. I would like to do this using execnet, but I still
> > haven't been able to get my head around it (Im clearly not even close to
> an
> > expert in these matters). I wonder if anyone can point me in the right
> > direction, maybe through an example, on how to do this.
> >
> > Thanks very much in advance. I apologize for any inconvenience this may
> > cause.
> >
> > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
> > wrote:
> >
> > > Hi Luis
> > >
> > > Things should work on pypy without any problem whatsoever, just like
> > > with normal python.
> > >
> > > Cheers,
> > > fijal
> > >
> > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> luisjosenovoa at gmail.com>
> > > wrote:
> > > > Hi Everyone,
> > > >
> > > > As part of a large scale optimization application, I am using the
> python
> > > > multiprocessing module to execute a procedure several times in
> parallel.
> > > > These executions in turn, return results used by other procedure,
> and the
> > > > whole thing repeats iteratively. I would like to call PyPy only for
> the
> > > > subprocesses (to improve performance), as in the main procedure I use
> > > > libraries like gurobipy, which are not compatible with PyPy. I
> looked at
> > > the
> > > > documentation of execnet online, but I could not find examples for
> > > > communication between CPython and PyPy.
> > > >
> > > > Any suggestions for using CPython + Multiprocessing with PyPy would
> be
> > > > highly appreciated.
> > > >
> > > > Have a great day.
> > > >
> > > > --
> > > > Luis J. Novoa
> > > >
> > > > _______________________________________________
> > > > execnet-dev mailing list
> > > > execnet-dev at python.org
> > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > >
> > >
> >
> >
> >
> > --
> > Luis J. Novoa
>
> > _______________________________________________
> > execnet-dev mailing list
> > execnet-dev at python.org
> > https://mail.python.org/mailman/listinfo/execnet-dev
>
>
> --
> about me:    http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
>



-- 
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151116/0d553da0/attachment.html>

From holger at merlinux.eu  Tue Nov 17 04:12:04 2015
From: holger at merlinux.eu (holger krekel)
Date: Tue, 17 Nov 2015 09:12:04 +0000
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
Message-ID: <20151117091204.GW16107@merlinux.eu>

On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
> Thanks again for the kind reply. After going through the multichannels
> examples I have this toy example:
> 
> def summi(p):
>     return p**2
> 
> 
> def multiplier(channel, factor1, factor2, factor3):
>     while not channel.isclosed():
>         param = channel.receive()
>         r = summi(p=param)
>         channel.send(factor1 * factor2 + factor3 + r)
> 
> if __name__ == "__main__":
> 
>     factor11 = 20
>     factor12 = 10
>     factor13 = 200
> 
>     factor21 = 5
>     factor22 = 5
>     factor23 = 200
> 
> 
>     gw = execnet.makegateway()
>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> factor2=factor12, factor3=factor13)
>     channel1.send(1)
>     gw2 = execnet.makegateway()
>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> factor2=factor22, factor3=factor23)
>     channel2.send(1)
>     mch = execnet.MultiChannel([channel1, channel2])
>     queue = mch.make_receive_queue()
>     results = []
>     for i in range(2):
>         results.append(queue.get())
>     print results
> 
> 
> Now, when running it I get the following error:
> 
> ValueError: ("the use of non-builtin globals isn't supported", ['summi']).
> 
> I tried the following:
> 
> def multiplier(channel, factor1, factor2, factor3):
>      def summi(p):
>          return p**2
>     while not channel.isclosed():
>         param = channel.receive()
>         r = summi(p=param)
>         channel.send(factor1 * factor2 + factor3 + r)
> 
> 
> and with this change I got:
> 
> ValueError: ("the use of non-builtin globals isn't supported", ['p', 'p'])

You need to send a module rather than just a function if your
function depends on another function in that module.
execnet only sends the source of the function, otherwise.
See here also:

http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii

cheers,
holger


> 
> 
> I am just trying to understand the nature of the errors reported, and
> possibly find a way around it. The function calls that I want to
> parallelize invoke other functions from within, just as in the toy example.
> 
> 
> 
> Any clarification is much appreciated.
> 
> Thank you in advance.
> 
> Luis.
> 
> 
> 
> 
> 
> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> > > Thanks for your reply. I was trying to replicate one of the execnet
> > > examples available online and had problems using PyPy due to pydev (using
> > > aptana under windows). But now it works. Thanks.
> > >
> > > Now, this is the kind of procedure that I am trying to replicate (which
> > > currently uses the multiprocessing module and uses CPython to solve the
> > > subproblems in parallel):
> > >
> > > out = mp.Queue()
> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d, out))
> > for
> > > s in range(len(networks))]
> > >
> > > *# Run subproblems*
> > > for sb in subproblems:
> > >     sb.start()
> > >
> > > *# Get results from the out queue*
> > > results = [out.get() for sb in subproblems]
> >
> > did you look into the multi-channel examples?
> >
> >     http://codespeak.net/execnet/example/test_multi.html
> >
> > you can start a pypy subprocess like so:
> >
> >     execnet.makegateway("popen//python=pypy")
> >
> > and that should work if used in the multichannel examples.
> >
> > holger
> >
> >
> > >
> > > So, what I would like to do, is to solve the subproblems with PyPy, as
> > they
> > > are time consuming. I would like to do this using execnet, but I still
> > > haven't been able to get my head around it (Im clearly not even close to
> > an
> > > expert in these matters). I wonder if anyone can point me in the right
> > > direction, maybe through an example, on how to do this.
> > >
> > > Thanks very much in advance. I apologize for any inconvenience this may
> > > cause.
> > >
> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <fijall at gmail.com>
> > > wrote:
> > >
> > > > Hi Luis
> > > >
> > > > Things should work on pypy without any problem whatsoever, just like
> > > > with normal python.
> > > >
> > > > Cheers,
> > > > fijal
> > > >
> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> > luisjosenovoa at gmail.com>
> > > > wrote:
> > > > > Hi Everyone,
> > > > >
> > > > > As part of a large scale optimization application, I am using the
> > python
> > > > > multiprocessing module to execute a procedure several times in
> > parallel.
> > > > > These executions in turn, return results used by other procedure,
> > and the
> > > > > whole thing repeats iteratively. I would like to call PyPy only for
> > the
> > > > > subprocesses (to improve performance), as in the main procedure I use
> > > > > libraries like gurobipy, which are not compatible with PyPy. I
> > looked at
> > > > the
> > > > > documentation of execnet online, but I could not find examples for
> > > > > communication between CPython and PyPy.
> > > > >
> > > > > Any suggestions for using CPython + Multiprocessing with PyPy would
> > be
> > > > > highly appreciated.
> > > > >
> > > > > Have a great day.
> > > > >
> > > > > --
> > > > > Luis J. Novoa
> > > > >
> > > > > _______________________________________________
> > > > > execnet-dev mailing list
> > > > > execnet-dev at python.org
> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Luis J. Novoa
> >
> > > _______________________________________________
> > > execnet-dev mailing list
> > > execnet-dev at python.org
> > > https://mail.python.org/mailman/listinfo/execnet-dev
> >
> >
> > --
> > about me:    http://holgerkrekel.net/about-me/
> > contracting: http://merlinux.eu
> >
> 
> 
> 
> -- 
> Luis J. Novoa

-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu

From opensource at ronnypfannschmidt.de  Tue Nov 17 04:34:30 2015
From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt)
Date: Tue, 17 Nov 2015 10:34:30 +0100
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <20151117091204.GW16107@merlinux.eu>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
Message-ID: <CB86333F-C9C8-4B4C-B4F9-7CAECD41F254@ronnypfannschmidt.de>

I believe there is a bug in the sanity check when inner functions are passed inside a remote function
We should fix that

Am 17. November 2015 10:12:04 MEZ, schrieb holger krekel <holger at merlinux.eu>:
>On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
>> Thanks again for the kind reply. After going through the
>multichannels
>> examples I have this toy example:
>> 
>> def summi(p):
>>     return p**2
>> 
>> 
>> def multiplier(channel, factor1, factor2, factor3):
>>     while not channel.isclosed():
>>         param = channel.receive()
>>         r = summi(p=param)
>>         channel.send(factor1 * factor2 + factor3 + r)
>> 
>> if __name__ == "__main__":
>> 
>>     factor11 = 20
>>     factor12 = 10
>>     factor13 = 200
>> 
>>     factor21 = 5
>>     factor22 = 5
>>     factor23 = 200
>> 
>> 
>>     gw = execnet.makegateway()
>>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> factor2=factor12, factor3=factor13)
>>     channel1.send(1)
>>     gw2 = execnet.makegateway()
>>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> factor2=factor22, factor3=factor23)
>>     channel2.send(1)
>>     mch = execnet.MultiChannel([channel1, channel2])
>>     queue = mch.make_receive_queue()
>>     results = []
>>     for i in range(2):
>>         results.append(queue.get())
>>     print results
>> 
>> 
>> Now, when running it I get the following error:
>> 
>> ValueError: ("the use of non-builtin globals isn't supported",
>['summi']).
>> 
>> I tried the following:
>> 
>> def multiplier(channel, factor1, factor2, factor3):
>>      def summi(p):
>>          return p**2
>>     while not channel.isclosed():
>>         param = channel.receive()
>>         r = summi(p=param)
>>         channel.send(factor1 * factor2 + factor3 + r)
>> 
>> 
>> and with this change I got:
>> 
>> ValueError: ("the use of non-builtin globals isn't supported", ['p',
>'p'])
>
>You need to send a module rather than just a function if your
>function depends on another function in that module.
>execnet only sends the source of the function, otherwise.
>See here also:
>
>http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>
>cheers,
>holger
>
>
>> 
>> 
>> I am just trying to understand the nature of the errors reported, and
>> possibly find a way around it. The function calls that I want to
>> parallelize invoke other functions from within, just as in the toy
>example.
>> 
>> 
>> 
>> Any clarification is much appreciated.
>> 
>> Thank you in advance.
>> 
>> Luis.
>> 
>> 
>> 
>> 
>> 
>> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
>wrote:
>> 
>> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
>> > > Thanks for your reply. I was trying to replicate one of the
>execnet
>> > > examples available online and had problems using PyPy due to
>pydev (using
>> > > aptana under windows). But now it works. Thanks.
>> > >
>> > > Now, this is the kind of procedure that I am trying to replicate
>(which
>> > > currently uses the multiprocessing module and uses CPython to
>solve the
>> > > subproblems in parallel):
>> > >
>> > > out = mp.Queue()
>> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
>out))
>> > for
>> > > s in range(len(networks))]
>> > >
>> > > *# Run subproblems*
>> > > for sb in subproblems:
>> > >     sb.start()
>> > >
>> > > *# Get results from the out queue*
>> > > results = [out.get() for sb in subproblems]
>> >
>> > did you look into the multi-channel examples?
>> >
>> >     http://codespeak.net/execnet/example/test_multi.html
>> >
>> > you can start a pypy subprocess like so:
>> >
>> >     execnet.makegateway("popen//python=pypy")
>> >
>> > and that should work if used in the multichannel examples.
>> >
>> > holger
>> >
>> >
>> > >
>> > > So, what I would like to do, is to solve the subproblems with
>PyPy, as
>> > they
>> > > are time consuming. I would like to do this using execnet, but I
>still
>> > > haven't been able to get my head around it (Im clearly not even
>close to
>> > an
>> > > expert in these matters). I wonder if anyone can point me in the
>right
>> > > direction, maybe through an example, on how to do this.
>> > >
>> > > Thanks very much in advance. I apologize for any inconvenience
>this may
>> > > cause.
>> > >
>> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski
><fijall at gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Luis
>> > > >
>> > > > Things should work on pypy without any problem whatsoever, just
>like
>> > > > with normal python.
>> > > >
>> > > > Cheers,
>> > > > fijal
>> > > >
>> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
>> > luisjosenovoa at gmail.com>
>> > > > wrote:
>> > > > > Hi Everyone,
>> > > > >
>> > > > > As part of a large scale optimization application, I am using
>the
>> > python
>> > > > > multiprocessing module to execute a procedure several times
>in
>> > parallel.
>> > > > > These executions in turn, return results used by other
>procedure,
>> > and the
>> > > > > whole thing repeats iteratively. I would like to call PyPy
>only for
>> > the
>> > > > > subprocesses (to improve performance), as in the main
>procedure I use
>> > > > > libraries like gurobipy, which are not compatible with PyPy.
>I
>> > looked at
>> > > > the
>> > > > > documentation of execnet online, but I could not find
>examples for
>> > > > > communication between CPython and PyPy.
>> > > > >
>> > > > > Any suggestions for using CPython + Multiprocessing with PyPy
>would
>> > be
>> > > > > highly appreciated.
>> > > > >
>> > > > > Have a great day.
>> > > > >
>> > > > > --
>> > > > > Luis J. Novoa
>> > > > >
>> > > > > _______________________________________________
>> > > > > execnet-dev mailing list
>> > > > > execnet-dev at python.org
>> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Luis J. Novoa
>> >
>> > > _______________________________________________
>> > > execnet-dev mailing list
>> > > execnet-dev at python.org
>> > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >
>> >
>> > --
>> > about me:    http://holgerkrekel.net/about-me/
>> > contracting: http://merlinux.eu
>> >
>> 
>> 
>> 
>> -- 
>> Luis J. Novoa
>
>-- 
>about me:    http://holgerkrekel.net/about-me/
>contracting: http://merlinux.eu
>_______________________________________________
>execnet-dev mailing list
>execnet-dev at python.org
>https://mail.python.org/mailman/listinfo/execnet-dev

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151117/44a95aec/attachment.html>

From holger at merlinux.eu  Tue Nov 17 07:22:56 2015
From: holger at merlinux.eu (holger krekel)
Date: Tue, 17 Nov 2015 12:22:56 +0000
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CB86333F-C9C8-4B4C-B4F9-7CAECD41F254@ronnypfannschmidt.de>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CB86333F-C9C8-4B4C-B4F9-7CAECD41F254@ronnypfannschmidt.de>
Message-ID: <20151117122256.GY16107@merlinux.eu>

On Tue, Nov 17, 2015 at 10:34 +0100, Ronny Pfannschmidt wrote:
> I believe there is a bug in the sanity check when inner functions are passed inside a remote function
> We should fix that

but the below example is not an inner function.

holger

> Am 17. November 2015 10:12:04 MEZ, schrieb holger krekel <holger at merlinux.eu>:
> >On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
> >> Thanks again for the kind reply. After going through the
> >multichannels
> >> examples I have this toy example:
> >> 
> >> def summi(p):
> >>     return p**2
> >> 
> >> 
> >> def multiplier(channel, factor1, factor2, factor3):
> >>     while not channel.isclosed():
> >>         param = channel.receive()
> >>         r = summi(p=param)
> >>         channel.send(factor1 * factor2 + factor3 + r)
> >> 
> >> if __name__ == "__main__":
> >> 
> >>     factor11 = 20
> >>     factor12 = 10
> >>     factor13 = 200
> >> 
> >>     factor21 = 5
> >>     factor22 = 5
> >>     factor23 = 200
> >> 
> >> 
> >>     gw = execnet.makegateway()
> >>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> >> factor2=factor12, factor3=factor13)
> >>     channel1.send(1)
> >>     gw2 = execnet.makegateway()
> >>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> >> factor2=factor22, factor3=factor23)
> >>     channel2.send(1)
> >>     mch = execnet.MultiChannel([channel1, channel2])
> >>     queue = mch.make_receive_queue()
> >>     results = []
> >>     for i in range(2):
> >>         results.append(queue.get())
> >>     print results
> >> 
> >> 
> >> Now, when running it I get the following error:
> >> 
> >> ValueError: ("the use of non-builtin globals isn't supported",
> >['summi']).
> >> 
> >> I tried the following:
> >> 
> >> def multiplier(channel, factor1, factor2, factor3):
> >>      def summi(p):
> >>          return p**2
> >>     while not channel.isclosed():
> >>         param = channel.receive()
> >>         r = summi(p=param)
> >>         channel.send(factor1 * factor2 + factor3 + r)
> >> 
> >> 
> >> and with this change I got:
> >> 
> >> ValueError: ("the use of non-builtin globals isn't supported", ['p',
> >'p'])
> >
> >You need to send a module rather than just a function if your
> >function depends on another function in that module.
> >execnet only sends the source of the function, otherwise.
> >See here also:
> >
> >http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
> >
> >cheers,
> >holger
> >
> >
> >> 
> >> 
> >> I am just trying to understand the nature of the errors reported, and
> >> possibly find a way around it. The function calls that I want to
> >> parallelize invoke other functions from within, just as in the toy
> >example.
> >> 
> >> 
> >> 
> >> Any clarification is much appreciated.
> >> 
> >> Thank you in advance.
> >> 
> >> Luis.
> >> 
> >> 
> >> 
> >> 
> >> 
> >> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
> >wrote:
> >> 
> >> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> >> > > Thanks for your reply. I was trying to replicate one of the
> >execnet
> >> > > examples available online and had problems using PyPy due to
> >pydev (using
> >> > > aptana under windows). But now it works. Thanks.
> >> > >
> >> > > Now, this is the kind of procedure that I am trying to replicate
> >(which
> >> > > currently uses the multiprocessing module and uses CPython to
> >solve the
> >> > > subproblems in parallel):
> >> > >
> >> > > out = mp.Queue()
> >> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
> >out))
> >> > for
> >> > > s in range(len(networks))]
> >> > >
> >> > > *# Run subproblems*
> >> > > for sb in subproblems:
> >> > >     sb.start()
> >> > >
> >> > > *# Get results from the out queue*
> >> > > results = [out.get() for sb in subproblems]
> >> >
> >> > did you look into the multi-channel examples?
> >> >
> >> >     http://codespeak.net/execnet/example/test_multi.html
> >> >
> >> > you can start a pypy subprocess like so:
> >> >
> >> >     execnet.makegateway("popen//python=pypy")
> >> >
> >> > and that should work if used in the multichannel examples.
> >> >
> >> > holger
> >> >
> >> >
> >> > >
> >> > > So, what I would like to do, is to solve the subproblems with
> >PyPy, as
> >> > they
> >> > > are time consuming. I would like to do this using execnet, but I
> >still
> >> > > haven't been able to get my head around it (Im clearly not even
> >close to
> >> > an
> >> > > expert in these matters). I wonder if anyone can point me in the
> >right
> >> > > direction, maybe through an example, on how to do this.
> >> > >
> >> > > Thanks very much in advance. I apologize for any inconvenience
> >this may
> >> > > cause.
> >> > >
> >> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski
> ><fijall at gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Luis
> >> > > >
> >> > > > Things should work on pypy without any problem whatsoever, just
> >like
> >> > > > with normal python.
> >> > > >
> >> > > > Cheers,
> >> > > > fijal
> >> > > >
> >> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> >> > luisjosenovoa at gmail.com>
> >> > > > wrote:
> >> > > > > Hi Everyone,
> >> > > > >
> >> > > > > As part of a large scale optimization application, I am using
> >the
> >> > python
> >> > > > > multiprocessing module to execute a procedure several times
> >in
> >> > parallel.
> >> > > > > These executions in turn, return results used by other
> >procedure,
> >> > and the
> >> > > > > whole thing repeats iteratively. I would like to call PyPy
> >only for
> >> > the
> >> > > > > subprocesses (to improve performance), as in the main
> >procedure I use
> >> > > > > libraries like gurobipy, which are not compatible with PyPy.
> >I
> >> > looked at
> >> > > > the
> >> > > > > documentation of execnet online, but I could not find
> >examples for
> >> > > > > communication between CPython and PyPy.
> >> > > > >
> >> > > > > Any suggestions for using CPython + Multiprocessing with PyPy
> >would
> >> > be
> >> > > > > highly appreciated.
> >> > > > >
> >> > > > > Have a great day.
> >> > > > >
> >> > > > > --
> >> > > > > Luis J. Novoa
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > execnet-dev mailing list
> >> > > > > execnet-dev at python.org
> >> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Luis J. Novoa
> >> >
> >> > > _______________________________________________
> >> > > execnet-dev mailing list
> >> > > execnet-dev at python.org
> >> > > https://mail.python.org/mailman/listinfo/execnet-dev
> >> >
> >> >
> >> > --
> >> > about me:    http://holgerkrekel.net/about-me/
> >> > contracting: http://merlinux.eu
> >> >
> >> 
> >> 
> >> 
> >> -- 
> >> Luis J. Novoa
> >
> >-- 
> >about me:    http://holgerkrekel.net/about-me/
> >contracting: http://merlinux.eu
> >_______________________________________________
> >execnet-dev mailing list
> >execnet-dev at python.org
> >https://mail.python.org/mailman/listinfo/execnet-dev
> 
> -- 
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu

From opensource at ronnypfannschmidt.de  Tue Nov 17 07:47:30 2015
From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt)
Date: Tue, 17 Nov 2015 13:47:30 +0100
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <20151117122256.GY16107@merlinux.eu>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CB86333F-C9C8-4B4C-B4F9-7CAECD41F254@ronnypfannschmidt.de>
 <20151117122256.GY16107@merlinux.eu>
Message-ID: <79797FC8-94CF-4491-BB01-5694B4B31FFE@ronnypfannschmidt.de>

I'm sorry for not clarifying, I was referring to his second example, which does use a inner function,

Am 17. November 2015 13:22:56 MEZ, schrieb holger krekel <holger at merlinux.eu>:
>On Tue, Nov 17, 2015 at 10:34 +0100, Ronny Pfannschmidt wrote:
>> I believe there is a bug in the sanity check when inner functions are
>passed inside a remote function
>> We should fix that
>
>but the below example is not an inner function.
>
>holger
>
>> Am 17. November 2015 10:12:04 MEZ, schrieb holger krekel
><holger at merlinux.eu>:
>> >On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
>> >> Thanks again for the kind reply. After going through the
>> >multichannels
>> >> examples I have this toy example:
>> >> 
>> >> def summi(p):
>> >>     return p**2
>> >> 
>> >> 
>> >> def multiplier(channel, factor1, factor2, factor3):
>> >>     while not channel.isclosed():
>> >>         param = channel.receive()
>> >>         r = summi(p=param)
>> >>         channel.send(factor1 * factor2 + factor3 + r)
>> >> 
>> >> if __name__ == "__main__":
>> >> 
>> >>     factor11 = 20
>> >>     factor12 = 10
>> >>     factor13 = 200
>> >> 
>> >>     factor21 = 5
>> >>     factor22 = 5
>> >>     factor23 = 200
>> >> 
>> >> 
>> >>     gw = execnet.makegateway()
>> >>     channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> >> factor2=factor12, factor3=factor13)
>> >>     channel1.send(1)
>> >>     gw2 = execnet.makegateway()
>> >>     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> >> factor2=factor22, factor3=factor23)
>> >>     channel2.send(1)
>> >>     mch = execnet.MultiChannel([channel1, channel2])
>> >>     queue = mch.make_receive_queue()
>> >>     results = []
>> >>     for i in range(2):
>> >>         results.append(queue.get())
>> >>     print results
>> >> 
>> >> 
>> >> Now, when running it I get the following error:
>> >> 
>> >> ValueError: ("the use of non-builtin globals isn't supported",
>> >['summi']).
>> >> 
>> >> I tried the following:
>> >> 
>> >> def multiplier(channel, factor1, factor2, factor3):
>> >>      def summi(p):
>> >>          return p**2
>> >>     while not channel.isclosed():
>> >>         param = channel.receive()
>> >>         r = summi(p=param)
>> >>         channel.send(factor1 * factor2 + factor3 + r)
>> >> 
>> >> 
>> >> and with this change I got:
>> >> 
>> >> ValueError: ("the use of non-builtin globals isn't supported",
>['p',
>> >'p'])
>> >
>> >You need to send a module rather than just a function if your
>> >function depends on another function in that module.
>> >execnet only sends the source of the function, otherwise.
>> >See here also:
>> >
>>
>>http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>> >
>> >cheers,
>> >holger
>> >
>> >
>> >> 
>> >> 
>> >> I am just trying to understand the nature of the errors reported,
>and
>> >> possibly find a way around it. The function calls that I want to
>> >> parallelize invoke other functions from within, just as in the toy
>> >example.
>> >> 
>> >> 
>> >> 
>> >> Any clarification is much appreciated.
>> >> 
>> >> Thank you in advance.
>> >> 
>> >> Luis.
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> On Thu, Nov 12, 2015 at 6:50 AM, holger krekel
><holger at merlinux.eu>
>> >wrote:
>> >> 
>> >> > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
>> >> > > Thanks for your reply. I was trying to replicate one of the
>> >execnet
>> >> > > examples available online and had problems using PyPy due to
>> >pydev (using
>> >> > > aptana under windows). But now it works. Thanks.
>> >> > >
>> >> > > Now, this is the kind of procedure that I am trying to
>replicate
>> >(which
>> >> > > currently uses the multiprocessing module and uses CPython to
>> >solve the
>> >> > > subproblems in parallel):
>> >> > >
>> >> > > out = mp.Queue()
>> >> > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c,
>d,
>> >out))
>> >> > for
>> >> > > s in range(len(networks))]
>> >> > >
>> >> > > *# Run subproblems*
>> >> > > for sb in subproblems:
>> >> > >     sb.start()
>> >> > >
>> >> > > *# Get results from the out queue*
>> >> > > results = [out.get() for sb in subproblems]
>> >> >
>> >> > did you look into the multi-channel examples?
>> >> >
>> >> >     http://codespeak.net/execnet/example/test_multi.html
>> >> >
>> >> > you can start a pypy subprocess like so:
>> >> >
>> >> >     execnet.makegateway("popen//python=pypy")
>> >> >
>> >> > and that should work if used in the multichannel examples.
>> >> >
>> >> > holger
>> >> >
>> >> >
>> >> > >
>> >> > > So, what I would like to do, is to solve the subproblems with
>> >PyPy, as
>> >> > they
>> >> > > are time consuming. I would like to do this using execnet, but
>I
>> >still
>> >> > > haven't been able to get my head around it (Im clearly not
>even
>> >close to
>> >> > an
>> >> > > expert in these matters). I wonder if anyone can point me in
>the
>> >right
>> >> > > direction, maybe through an example, on how to do this.
>> >> > >
>> >> > > Thanks very much in advance. I apologize for any inconvenience
>> >this may
>> >> > > cause.
>> >> > >
>> >> > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski
>> ><fijall at gmail.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Luis
>> >> > > >
>> >> > > > Things should work on pypy without any problem whatsoever,
>just
>> >like
>> >> > > > with normal python.
>> >> > > >
>> >> > > > Cheers,
>> >> > > > fijal
>> >> > > >
>> >> > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
>> >> > luisjosenovoa at gmail.com>
>> >> > > > wrote:
>> >> > > > > Hi Everyone,
>> >> > > > >
>> >> > > > > As part of a large scale optimization application, I am
>using
>> >the
>> >> > python
>> >> > > > > multiprocessing module to execute a procedure several
>times
>> >in
>> >> > parallel.
>> >> > > > > These executions in turn, return results used by other
>> >procedure,
>> >> > and the
>> >> > > > > whole thing repeats iteratively. I would like to call PyPy
>> >only for
>> >> > the
>> >> > > > > subprocesses (to improve performance), as in the main
>> >procedure I use
>> >> > > > > libraries like gurobipy, which are not compatible with
>PyPy.
>> >I
>> >> > looked at
>> >> > > > the
>> >> > > > > documentation of execnet online, but I could not find
>> >examples for
>> >> > > > > communication between CPython and PyPy.
>> >> > > > >
>> >> > > > > Any suggestions for using CPython + Multiprocessing with
>PyPy
>> >would
>> >> > be
>> >> > > > > highly appreciated.
>> >> > > > >
>> >> > > > > Have a great day.
>> >> > > > >
>> >> > > > > --
>> >> > > > > Luis J. Novoa
>> >> > > > >
>> >> > > > > _______________________________________________
>> >> > > > > execnet-dev mailing list
>> >> > > > > execnet-dev at python.org
>> >> > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >> > > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Luis J. Novoa
>> >> >
>> >> > > _______________________________________________
>> >> > > execnet-dev mailing list
>> >> > > execnet-dev at python.org
>> >> > > https://mail.python.org/mailman/listinfo/execnet-dev
>> >> >
>> >> >
>> >> > --
>> >> > about me:    http://holgerkrekel.net/about-me/
>> >> > contracting: http://merlinux.eu
>> >> >
>> >> 
>> >> 
>> >> 
>> >> -- 
>> >> Luis J. Novoa
>> >
>> >-- 
>> >about me:    http://holgerkrekel.net/about-me/
>> >contracting: http://merlinux.eu
>> >_______________________________________________
>> >execnet-dev mailing list
>> >execnet-dev at python.org
>> >https://mail.python.org/mailman/listinfo/execnet-dev
>> 
>> -- 
>> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail
>gesendet.

MFG Ronny

From luisjosenovoa at gmail.com  Tue Nov 17 08:20:11 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Tue, 17 Nov 2015 08:20:11 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <20151117091204.GW16107@merlinux.eu>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
Message-ID: <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>

Thank you again for your kind reply. When passing a module, can I pass the
parameters corresponding to the function in the module that calls other
functions?

Thank you.

LJN
On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu> wrote:

> On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
> > Thanks again for the kind reply. After going through the multichannels
> > examples I have this toy example:
> >
> > def summi(p):
> >     return p**2
> >
> >
> > def multiplier(channel, factor1, factor2, factor3):
> >     while not channel.isclosed():
> >         param = channel.receive()
> >         r = summi(p=param)
> >         channel.send(factor1 * factor2 + factor3 + r)
> >
> > if __name__ == "__main__":
> >
> >     factor11 = 20
> >     factor12 = 10
> >     factor13 = 200
> >
> >     factor21 = 5
> >     factor22 = 5
> >     factor23 = 200
> >
> >
> >     gw = execnet.makegateway()
> >     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> > factor2=factor12, factor3=factor13)
> >     channel1.send(1)
> >     gw2 = execnet.makegateway()
> >     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> > factor2=factor22, factor3=factor23)
> >     channel2.send(1)
> >     mch = execnet.MultiChannel([channel1, channel2])
> >     queue = mch.make_receive_queue()
> >     results = []
> >     for i in range(2):
> >         results.append(queue.get())
> >     print results
> >
> >
> > Now, when running it I get the following error:
> >
> > ValueError: ("the use of non-builtin globals isn't supported",
> ['summi']).
> >
> > I tried the following:
> >
> > def multiplier(channel, factor1, factor2, factor3):
> >      def summi(p):
> >          return p**2
> >     while not channel.isclosed():
> >         param = channel.receive()
> >         r = summi(p=param)
> >         channel.send(factor1 * factor2 + factor3 + r)
> >
> >
> > and with this change I got:
> >
> > ValueError: ("the use of non-builtin globals isn't supported", ['p',
> 'p'])
>
> You need to send a module rather than just a function if your
> function depends on another function in that module.
> execnet only sends the source of the function, otherwise.
> See here also:
>
>
> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>
> cheers,
> holger
>
>
> >
> >
> > I am just trying to understand the nature of the errors reported, and
> > possibly find a way around it. The function calls that I want to
> > parallelize invoke other functions from within, just as in the toy
> example.
> >
> >
> >
> > Any clarification is much appreciated.
> >
> > Thank you in advance.
> >
> > Luis.
> >
> >
> >
> >
> >
> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> > > > Thanks for your reply. I was trying to replicate one of the execnet
> > > > examples available online and had problems using PyPy due to pydev
> (using
> > > > aptana under windows). But now it works. Thanks.
> > > >
> > > > Now, this is the kind of procedure that I am trying to replicate
> (which
> > > > currently uses the multiprocessing module and uses CPython to solve
> the
> > > > subproblems in parallel):
> > > >
> > > > out = mp.Queue()
> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
> out))
> > > for
> > > > s in range(len(networks))]
> > > >
> > > > *# Run subproblems*
> > > > for sb in subproblems:
> > > >     sb.start()
> > > >
> > > > *# Get results from the out queue*
> > > > results = [out.get() for sb in subproblems]
> > >
> > > did you look into the multi-channel examples?
> > >
> > >     http://codespeak.net/execnet/example/test_multi.html
> > >
> > > you can start a pypy subprocess like so:
> > >
> > >     execnet.makegateway("popen//python=pypy")
> > >
> > > and that should work if used in the multichannel examples.
> > >
> > > holger
> > >
> > >
> > > >
> > > > So, what I would like to do, is to solve the subproblems with PyPy,
> as
> > > they
> > > > are time consuming. I would like to do this using execnet, but I
> still
> > > > haven't been able to get my head around it (Im clearly not even
> close to
> > > an
> > > > expert in these matters). I wonder if anyone can point me in the
> right
> > > > direction, maybe through an example, on how to do this.
> > > >
> > > > Thanks very much in advance. I apologize for any inconvenience this
> may
> > > > cause.
> > > >
> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
> fijall at gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Luis
> > > > >
> > > > > Things should work on pypy without any problem whatsoever, just
> like
> > > > > with normal python.
> > > > >
> > > > > Cheers,
> > > > > fijal
> > > > >
> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> > > luisjosenovoa at gmail.com>
> > > > > wrote:
> > > > > > Hi Everyone,
> > > > > >
> > > > > > As part of a large scale optimization application, I am using the
> > > python
> > > > > > multiprocessing module to execute a procedure several times in
> > > parallel.
> > > > > > These executions in turn, return results used by other procedure,
> > > and the
> > > > > > whole thing repeats iteratively. I would like to call PyPy only
> for
> > > the
> > > > > > subprocesses (to improve performance), as in the main procedure
> I use
> > > > > > libraries like gurobipy, which are not compatible with PyPy. I
> > > looked at
> > > > > the
> > > > > > documentation of execnet online, but I could not find examples
> for
> > > > > > communication between CPython and PyPy.
> > > > > >
> > > > > > Any suggestions for using CPython + Multiprocessing with PyPy
> would
> > > be
> > > > > > highly appreciated.
> > > > > >
> > > > > > Have a great day.
> > > > > >
> > > > > > --
> > > > > > Luis J. Novoa
> > > > > >
> > > > > > _______________________________________________
> > > > > > execnet-dev mailing list
> > > > > > execnet-dev at python.org
> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Luis J. Novoa
> > >
> > > > _______________________________________________
> > > > execnet-dev mailing list
> > > > execnet-dev at python.org
> > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > >
> > >
> > > --
> > > about me:    http://holgerkrekel.net/about-me/
> > > contracting: http://merlinux.eu
> > >
> >
> >
> >
> > --
> > Luis J. Novoa
>
> --
> about me:    http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151117/99faa94d/attachment-0001.html>

From luisjosenovoa at gmail.com  Tue Nov 17 20:55:07 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Tue, 17 Nov 2015 20:55:07 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>
Message-ID: <CAFKi5F-6nYLXQ3Yu+Rec_RXrmeDK8tJhgZcgW-u0q4w916ynUA@mail.gmail.com>

Hi All. Thanks in advance for the time invested in reading, thinking about,
and responding to my emails.

Right now, I have something like this:


import execnet as execnet
import time
import outt


if __name__ == "__main__":

    start = time.time()

    factor11 = 20
    factor12 = 10
    factor13 = 200

    factor21 = 5
    factor22 = 5
    factor23 = 200


    group = execnet.Group(['popen'] * 2)

    channel1 = group['gw0'].remote_exec(outt)
    channel1.send([factor11,factor12,factor13,10])
    channel2 = group['gw1'].remote_exec(outt)
    channel2.send([factor21,factor22,factor23,4])

    mch = execnet.MultiChannel([channel1, channel2])
    queue = mch.make_receive_queue()

    results = []
    for i in range(2):
        results.append(queue.get())
    print results


    group.terminate()

    end = time.time() - start

    print end, 'seconds'


where, outt is a module containing (please do not try to make sense out of
whatever is done inside the multiplier function here):

if __name__ == '__channelexec__':

    a = channel.receive()


    def summi(p):
        return p**2


    def multiplier(channel, factor1, factor2, factor3, a):
            param = a[3]
            r = summi(p=param)
            s = []
            d = []
            for i in xrange(900000):
                s.append(i)
                d.append(i)
            channel.send({'Number':factor1+factor2, 'list': sum(i for i in
s)})


    multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)



Im not sure if this is the best way to proceed, so if you see something
that does not make sense, Id really appreciate you letting me know. If I
run the first piece of code, I get the expected results (althought Im not
sure if the subprocesses are actually running in parallel), but when I
specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] * 2)) I get
the following error:


[5976] Warning: unhandled RemoteError: Traceback (most recent call last):
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 841, in _local_receive
    data = loads_internal(data, channel, strconfig)
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 1350, in loads_internal
    return Unserializer(io, channelfactory, strconfig).load()
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 1160, in load
    "wire protocol corruption?" % (opcode,))
LoadError: unkown opcode '\n' - wire protocol corruption?


Once more, thanks for all the help.

On Tue, Nov 17, 2015 at 8:20 AM, Luis Jos? Novoa <luisjosenovoa at gmail.com>
wrote:

> Thank you again for your kind reply. When passing a module, can I pass
> the  parameters corresponding to the function in the module that calls
> other functions?
>
> Thank you.
>
> LJN
> On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu> wrote:
>
>> On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
>> > Thanks again for the kind reply. After going through the multichannels
>> > examples I have this toy example:
>> >
>> > def summi(p):
>> >     return p**2
>> >
>> >
>> > def multiplier(channel, factor1, factor2, factor3):
>> >     while not channel.isclosed():
>> >         param = channel.receive()
>> >         r = summi(p=param)
>> >         channel.send(factor1 * factor2 + factor3 + r)
>> >
>> > if __name__ == "__main__":
>> >
>> >     factor11 = 20
>> >     factor12 = 10
>> >     factor13 = 200
>> >
>> >     factor21 = 5
>> >     factor22 = 5
>> >     factor23 = 200
>> >
>> >
>> >     gw = execnet.makegateway()
>> >     channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> > factor2=factor12, factor3=factor13)
>> >     channel1.send(1)
>> >     gw2 = execnet.makegateway()
>> >     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> > factor2=factor22, factor3=factor23)
>> >     channel2.send(1)
>> >     mch = execnet.MultiChannel([channel1, channel2])
>> >     queue = mch.make_receive_queue()
>> >     results = []
>> >     for i in range(2):
>> >         results.append(queue.get())
>> >     print results
>> >
>> >
>> > Now, when running it I get the following error:
>> >
>> > ValueError: ("the use of non-builtin globals isn't supported",
>> ['summi']).
>> >
>> > I tried the following:
>> >
>> > def multiplier(channel, factor1, factor2, factor3):
>> >      def summi(p):
>> >          return p**2
>> >     while not channel.isclosed():
>> >         param = channel.receive()
>> >         r = summi(p=param)
>> >         channel.send(factor1 * factor2 + factor3 + r)
>> >
>> >
>> > and with this change I got:
>> >
>> > ValueError: ("the use of non-builtin globals isn't supported", ['p',
>> 'p'])
>>
>> You need to send a module rather than just a function if your
>> function depends on another function in that module.
>> execnet only sends the source of the function, otherwise.
>> See here also:
>>
>>
>> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>>
>> cheers,
>> holger
>>
>>
>> >
>> >
>> > I am just trying to understand the nature of the errors reported, and
>> > possibly find a way around it. The function calls that I want to
>> > parallelize invoke other functions from within, just as in the toy
>> example.
>> >
>> >
>> >
>> > Any clarification is much appreciated.
>> >
>> > Thank you in advance.
>> >
>> > Luis.
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
>> wrote:
>> >
>> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
>> > > > Thanks for your reply. I was trying to replicate one of the execnet
>> > > > examples available online and had problems using PyPy due to pydev
>> (using
>> > > > aptana under windows). But now it works. Thanks.
>> > > >
>> > > > Now, this is the kind of procedure that I am trying to replicate
>> (which
>> > > > currently uses the multiprocessing module and uses CPython to solve
>> the
>> > > > subproblems in parallel):
>> > > >
>> > > > out = mp.Queue()
>> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
>> out))
>> > > for
>> > > > s in range(len(networks))]
>> > > >
>> > > > *# Run subproblems*
>> > > > for sb in subproblems:
>> > > >     sb.start()
>> > > >
>> > > > *# Get results from the out queue*
>> > > > results = [out.get() for sb in subproblems]
>> > >
>> > > did you look into the multi-channel examples?
>> > >
>> > >     http://codespeak.net/execnet/example/test_multi.html
>> > >
>> > > you can start a pypy subprocess like so:
>> > >
>> > >     execnet.makegateway("popen//python=pypy")
>> > >
>> > > and that should work if used in the multichannel examples.
>> > >
>> > > holger
>> > >
>> > >
>> > > >
>> > > > So, what I would like to do, is to solve the subproblems with PyPy,
>> as
>> > > they
>> > > > are time consuming. I would like to do this using execnet, but I
>> still
>> > > > haven't been able to get my head around it (Im clearly not even
>> close to
>> > > an
>> > > > expert in these matters). I wonder if anyone can point me in the
>> right
>> > > > direction, maybe through an example, on how to do this.
>> > > >
>> > > > Thanks very much in advance. I apologize for any inconvenience this
>> may
>> > > > cause.
>> > > >
>> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
>> fijall at gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Luis
>> > > > >
>> > > > > Things should work on pypy without any problem whatsoever, just
>> like
>> > > > > with normal python.
>> > > > >
>> > > > > Cheers,
>> > > > > fijal
>> > > > >
>> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
>> > > luisjosenovoa at gmail.com>
>> > > > > wrote:
>> > > > > > Hi Everyone,
>> > > > > >
>> > > > > > As part of a large scale optimization application, I am using
>> the
>> > > python
>> > > > > > multiprocessing module to execute a procedure several times in
>> > > parallel.
>> > > > > > These executions in turn, return results used by other
>> procedure,
>> > > and the
>> > > > > > whole thing repeats iteratively. I would like to call PyPy only
>> for
>> > > the
>> > > > > > subprocesses (to improve performance), as in the main procedure
>> I use
>> > > > > > libraries like gurobipy, which are not compatible with PyPy. I
>> > > looked at
>> > > > > the
>> > > > > > documentation of execnet online, but I could not find examples
>> for
>> > > > > > communication between CPython and PyPy.
>> > > > > >
>> > > > > > Any suggestions for using CPython + Multiprocessing with PyPy
>> would
>> > > be
>> > > > > > highly appreciated.
>> > > > > >
>> > > > > > Have a great day.
>> > > > > >
>> > > > > > --
>> > > > > > Luis J. Novoa
>> > > > > >
>> > > > > > _______________________________________________
>> > > > > > execnet-dev mailing list
>> > > > > > execnet-dev at python.org
>> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Luis J. Novoa
>> > >
>> > > > _______________________________________________
>> > > > execnet-dev mailing list
>> > > > execnet-dev at python.org
>> > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > >
>> > >
>> > > --
>> > > about me:    http://holgerkrekel.net/about-me/
>> > > contracting: http://merlinux.eu
>> > >
>> >
>> >
>> >
>> > --
>> > Luis J. Novoa
>>
>> --
>> about me:    http://holgerkrekel.net/about-me/
>> contracting: http://merlinux.eu
>>
>


-- 
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151117/968a1f15/attachment-0001.html>

From holger at merlinux.eu  Wed Nov 18 03:52:30 2015
From: holger at merlinux.eu (holger krekel)
Date: Wed, 18 Nov 2015 08:52:30 +0000
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F-6nYLXQ3Yu+Rec_RXrmeDK8tJhgZcgW-u0q4w916ynUA@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>
 <CAFKi5F-6nYLXQ3Yu+Rec_RXrmeDK8tJhgZcgW-u0q4w916ynUA@mail.gmail.com>
Message-ID: <20151118085230.GA16107@merlinux.eu>

On Tue, Nov 17, 2015 at 20:55 -0500, Luis Jos? Novoa wrote:
> Hi All. Thanks in advance for the time invested in reading, thinking about,
> and responding to my emails.
> 
> Right now, I have something like this:
> 
> 
> import execnet as execnet
> import time
> import outt
> 
> 
> if __name__ == "__main__":
> 
>     start = time.time()
> 
>     factor11 = 20
>     factor12 = 10
>     factor13 = 200
> 
>     factor21 = 5
>     factor22 = 5
>     factor23 = 200
> 
> 
>     group = execnet.Group(['popen'] * 2)
> 
>     channel1 = group['gw0'].remote_exec(outt)
>     channel1.send([factor11,factor12,factor13,10])
>     channel2 = group['gw1'].remote_exec(outt)
>     channel2.send([factor21,factor22,factor23,4])
> 
>     mch = execnet.MultiChannel([channel1, channel2])
>     queue = mch.make_receive_queue()
> 
>     results = []
>     for i in range(2):
>         results.append(queue.get())
>     print results
> 
> 
>     group.terminate()
> 
>     end = time.time() - start
> 
>     print end, 'seconds'
> 
> 
> where, outt is a module containing (please do not try to make sense out of
> whatever is done inside the multiplier function here):
> 
> if __name__ == '__channelexec__':
> 
>     a = channel.receive()
> 
> 
>     def summi(p):
>         return p**2
> 
> 
>     def multiplier(channel, factor1, factor2, factor3, a):
>             param = a[3]
>             r = summi(p=param)
>             s = []
>             d = []
>             for i in xrange(900000):
>                 s.append(i)
>                 d.append(i)
>             channel.send({'Number':factor1+factor2, 'list': sum(i for i in
> s)})
> 
> 
>     multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)

I'd recommend putting the actual excuted code on both sides into
functions and just having the ``if __name__ == ...`` sections call
into those functions.  Then you don't need to indent function definitions.
 
> Im not sure if this is the best way to proceed, so if you see something
> that does not make sense, Id really appreciate you letting me know. If I
> run the first piece of code, I get the expected results (althought Im not
> sure if the subprocesses are actually running in parallel), but when I
> specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] * 2)) I get
> the following error:
> 
> 
> [5976] Warning: unhandled RemoteError: Traceback (most recent call last):
>   File "C:\Users\Luis J
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> line 841, in _local_receive
>     data = loads_internal(data, channel, strconfig)
>   File "C:\Users\Luis J
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> line 1350, in loads_internal
>     return Unserializer(io, channelfactory, strconfig).load()
>   File "C:\Users\Luis J
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> line 1160, in load
>     "wire protocol corruption?" % (opcode,))
> LoadError: unkown opcode '\n' - wire protocol corruption?

Not sure how this comes.  If you just do 

    >>> execnet.makegateway("popen//python=pypy.exe")

on the interactive prompt, what happens?

holger



> 
> Once more, thanks for all the help.
> 
> On Tue, Nov 17, 2015 at 8:20 AM, Luis Jos? Novoa <luisjosenovoa at gmail.com>
> wrote:
> 
> > Thank you again for your kind reply. When passing a module, can I pass
> > the  parameters corresponding to the function in the module that calls
> > other functions?
> >
> > Thank you.
> >
> > LJN
> > On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu> wrote:
> >
> >> On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
> >> > Thanks again for the kind reply. After going through the multichannels
> >> > examples I have this toy example:
> >> >
> >> > def summi(p):
> >> >     return p**2
> >> >
> >> >
> >> > def multiplier(channel, factor1, factor2, factor3):
> >> >     while not channel.isclosed():
> >> >         param = channel.receive()
> >> >         r = summi(p=param)
> >> >         channel.send(factor1 * factor2 + factor3 + r)
> >> >
> >> > if __name__ == "__main__":
> >> >
> >> >     factor11 = 20
> >> >     factor12 = 10
> >> >     factor13 = 200
> >> >
> >> >     factor21 = 5
> >> >     factor22 = 5
> >> >     factor23 = 200
> >> >
> >> >
> >> >     gw = execnet.makegateway()
> >> >     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> >> > factor2=factor12, factor3=factor13)
> >> >     channel1.send(1)
> >> >     gw2 = execnet.makegateway()
> >> >     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> >> > factor2=factor22, factor3=factor23)
> >> >     channel2.send(1)
> >> >     mch = execnet.MultiChannel([channel1, channel2])
> >> >     queue = mch.make_receive_queue()
> >> >     results = []
> >> >     for i in range(2):
> >> >         results.append(queue.get())
> >> >     print results
> >> >
> >> >
> >> > Now, when running it I get the following error:
> >> >
> >> > ValueError: ("the use of non-builtin globals isn't supported",
> >> ['summi']).
> >> >
> >> > I tried the following:
> >> >
> >> > def multiplier(channel, factor1, factor2, factor3):
> >> >      def summi(p):
> >> >          return p**2
> >> >     while not channel.isclosed():
> >> >         param = channel.receive()
> >> >         r = summi(p=param)
> >> >         channel.send(factor1 * factor2 + factor3 + r)
> >> >
> >> >
> >> > and with this change I got:
> >> >
> >> > ValueError: ("the use of non-builtin globals isn't supported", ['p',
> >> 'p'])
> >>
> >> You need to send a module rather than just a function if your
> >> function depends on another function in that module.
> >> execnet only sends the source of the function, otherwise.
> >> See here also:
> >>
> >>
> >> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
> >>
> >> cheers,
> >> holger
> >>
> >>
> >> >
> >> >
> >> > I am just trying to understand the nature of the errors reported, and
> >> > possibly find a way around it. The function calls that I want to
> >> > parallelize invoke other functions from within, just as in the toy
> >> example.
> >> >
> >> >
> >> >
> >> > Any clarification is much appreciated.
> >> >
> >> > Thank you in advance.
> >> >
> >> > Luis.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <holger at merlinux.eu>
> >> wrote:
> >> >
> >> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> >> > > > Thanks for your reply. I was trying to replicate one of the execnet
> >> > > > examples available online and had problems using PyPy due to pydev
> >> (using
> >> > > > aptana under windows). But now it works. Thanks.
> >> > > >
> >> > > > Now, this is the kind of procedure that I am trying to replicate
> >> (which
> >> > > > currently uses the multiprocessing module and uses CPython to solve
> >> the
> >> > > > subproblems in parallel):
> >> > > >
> >> > > > out = mp.Queue()
> >> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b, c, d,
> >> out))
> >> > > for
> >> > > > s in range(len(networks))]
> >> > > >
> >> > > > *# Run subproblems*
> >> > > > for sb in subproblems:
> >> > > >     sb.start()
> >> > > >
> >> > > > *# Get results from the out queue*
> >> > > > results = [out.get() for sb in subproblems]
> >> > >
> >> > > did you look into the multi-channel examples?
> >> > >
> >> > >     http://codespeak.net/execnet/example/test_multi.html
> >> > >
> >> > > you can start a pypy subprocess like so:
> >> > >
> >> > >     execnet.makegateway("popen//python=pypy")
> >> > >
> >> > > and that should work if used in the multichannel examples.
> >> > >
> >> > > holger
> >> > >
> >> > >
> >> > > >
> >> > > > So, what I would like to do, is to solve the subproblems with PyPy,
> >> as
> >> > > they
> >> > > > are time consuming. I would like to do this using execnet, but I
> >> still
> >> > > > haven't been able to get my head around it (Im clearly not even
> >> close to
> >> > > an
> >> > > > expert in these matters). I wonder if anyone can point me in the
> >> right
> >> > > > direction, maybe through an example, on how to do this.
> >> > > >
> >> > > > Thanks very much in advance. I apologize for any inconvenience this
> >> may
> >> > > > cause.
> >> > > >
> >> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
> >> fijall at gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Luis
> >> > > > >
> >> > > > > Things should work on pypy without any problem whatsoever, just
> >> like
> >> > > > > with normal python.
> >> > > > >
> >> > > > > Cheers,
> >> > > > > fijal
> >> > > > >
> >> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> >> > > luisjosenovoa at gmail.com>
> >> > > > > wrote:
> >> > > > > > Hi Everyone,
> >> > > > > >
> >> > > > > > As part of a large scale optimization application, I am using
> >> the
> >> > > python
> >> > > > > > multiprocessing module to execute a procedure several times in
> >> > > parallel.
> >> > > > > > These executions in turn, return results used by other
> >> procedure,
> >> > > and the
> >> > > > > > whole thing repeats iteratively. I would like to call PyPy only
> >> for
> >> > > the
> >> > > > > > subprocesses (to improve performance), as in the main procedure
> >> I use
> >> > > > > > libraries like gurobipy, which are not compatible with PyPy. I
> >> > > looked at
> >> > > > > the
> >> > > > > > documentation of execnet online, but I could not find examples
> >> for
> >> > > > > > communication between CPython and PyPy.
> >> > > > > >
> >> > > > > > Any suggestions for using CPython + Multiprocessing with PyPy
> >> would
> >> > > be
> >> > > > > > highly appreciated.
> >> > > > > >
> >> > > > > > Have a great day.
> >> > > > > >
> >> > > > > > --
> >> > > > > > Luis J. Novoa
> >> > > > > >
> >> > > > > > _______________________________________________
> >> > > > > > execnet-dev mailing list
> >> > > > > > execnet-dev at python.org
> >> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Luis J. Novoa
> >> > >
> >> > > > _______________________________________________
> >> > > > execnet-dev mailing list
> >> > > > execnet-dev at python.org
> >> > > > https://mail.python.org/mailman/listinfo/execnet-dev
> >> > >
> >> > >
> >> > > --
> >> > > about me:    http://holgerkrekel.net/about-me/
> >> > > contracting: http://merlinux.eu
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Luis J. Novoa
> >>
> >> --
> >> about me:    http://holgerkrekel.net/about-me/
> >> contracting: http://merlinux.eu
> >>
> >
> 
> 
> -- 
> Luis J. Novoa

-- 
about me:    http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu

From luisjosenovoa at gmail.com  Wed Nov 18 11:13:09 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Wed, 18 Nov 2015 11:13:09 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <20151118155746.GH16107@merlinux.eu>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>
 <CAFKi5F-6nYLXQ3Yu+Rec_RXrmeDK8tJhgZcgW-u0q4w916ynUA@mail.gmail.com>
 <20151118085230.GA16107@merlinux.eu>
 <CAFKi5F_bS9jMTs34yQS2+yiHAynu7F2A=wnuVrJkJbQEP+wEag@mail.gmail.com>
 <20151118155746.GH16107@merlinux.eu>
Message-ID: <CAFKi5F8i860_tkw6LH2N8nKt+Pwngm9xgWA_Bh0LNOVQZb1MYA@mail.gmail.com>

Thanks. I am indeed using the latest pypy version for windows under Python
2.7 (pypy 4.0.0).

One thing I noticed was that executing just one gateway and a channel under
pypy does not cause problems. The problem appears when retrieving results
from channels in two or more gws.

LJN
On Nov 18, 2015 10:57 AM, "holger krekel" <holger at merlinux.eu> wrote:

> On Wed, Nov 18, 2015 at 09:40 -0500, Luis Jos? Novoa wrote:
> > Thanks very much for your reply. In the command line I get no error, as
> > shown in the image below.
> >
> > [image: Inline image 1]
> >
> > The error appears after executing queue = mch.make_receive_queue().
> >
> >
> > I appreciate all the help.
>
> not sure what is different wrt to pypy/cpython here.
> Make sure you are running the newest pypy versions.
> Maybe Ronny can chime in at some point.
>
> holger
>
> > LJ
> >
> > On Wed, Nov 18, 2015 at 3:52 AM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > > On Tue, Nov 17, 2015 at 20:55 -0500, Luis Jos? Novoa wrote:
> > > > Hi All. Thanks in advance for the time invested in reading, thinking
> > > about,
> > > > and responding to my emails.
> > > >
> > > > Right now, I have something like this:
> > > >
> > > >
> > > > import execnet as execnet
> > > > import time
> > > > import outt
> > > >
> > > >
> > > > if __name__ == "__main__":
> > > >
> > > >     start = time.time()
> > > >
> > > >     factor11 = 20
> > > >     factor12 = 10
> > > >     factor13 = 200
> > > >
> > > >     factor21 = 5
> > > >     factor22 = 5
> > > >     factor23 = 200
> > > >
> > > >
> > > >     group = execnet.Group(['popen'] * 2)
> > > >
> > > >     channel1 = group['gw0'].remote_exec(outt)
> > > >     channel1.send([factor11,factor12,factor13,10])
> > > >     channel2 = group['gw1'].remote_exec(outt)
> > > >     channel2.send([factor21,factor22,factor23,4])
> > > >
> > > >     mch = execnet.MultiChannel([channel1, channel2])
> > > >     queue = mch.make_receive_queue()
> > > >
> > > >     results = []
> > > >     for i in range(2):
> > > >         results.append(queue.get())
> > > >     print results
> > > >
> > > >
> > > >     group.terminate()
> > > >
> > > >     end = time.time() - start
> > > >
> > > >     print end, 'seconds'
> > > >
> > > >
> > > > where, outt is a module containing (please do not try to make sense
> out
> > > of
> > > > whatever is done inside the multiplier function here):
> > > >
> > > > if __name__ == '__channelexec__':
> > > >
> > > >     a = channel.receive()
> > > >
> > > >
> > > >     def summi(p):
> > > >         return p**2
> > > >
> > > >
> > > >     def multiplier(channel, factor1, factor2, factor3, a):
> > > >             param = a[3]
> > > >             r = summi(p=param)
> > > >             s = []
> > > >             d = []
> > > >             for i in xrange(900000):
> > > >                 s.append(i)
> > > >                 d.append(i)
> > > >             channel.send({'Number':factor1+factor2, 'list': sum(i
> for i
> > > in
> > > > s)})
> > > >
> > > >
> > > >     multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)
> > >
> > > I'd recommend putting the actual excuted code on both sides into
> > > functions and just having the ``if __name__ == ...`` sections call
> > > into those functions.  Then you don't need to indent function
> definitions.
> > >
> > > > Im not sure if this is the best way to proceed, so if you see
> something
> > > > that does not make sense, Id really appreciate you letting me know.
> If I
> > > > run the first piece of code, I get the expected results (althought
> Im not
> > > > sure if the subprocesses are actually running in parallel), but when
> I
> > > > specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] * 2))
> I
> > > get
> > > > the following error:
> > > >
> > > >
> > > > [5976] Warning: unhandled RemoteError: Traceback (most recent call
> last):
> > > >   File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 841, in _local_receive
> > > >     data = loads_internal(data, channel, strconfig)
> > > >   File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 1350, in loads_internal
> > > >     return Unserializer(io, channelfactory, strconfig).load()
> > > >   File "C:\Users\Luis J
> > > >
> > >
> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
> > > > line 1160, in load
> > > >     "wire protocol corruption?" % (opcode,))
> > > > LoadError: unkown opcode '\n' - wire protocol corruption?
> > >
> > > Not sure how this comes.  If you just do
> > >
> > >     >>> execnet.makegateway("popen//python=pypy.exe")
> > >
> > > on the interactive prompt, what happens?
> > >
> > > holger
> > >
> > >
> > >
> > > >
> > > > Once more, thanks for all the help.
> > > >
> > > > On Tue, Nov 17, 2015 at 8:20 AM, Luis Jos? Novoa <
> > > luisjosenovoa at gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you again for your kind reply. When passing a module, can I
> pass
> > > > > the  parameters corresponding to the function in the module that
> calls
> > > > > other functions?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > LJN
> > > > > On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu>
> wrote:
> > > > >
> > > > >> On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
> > > > >> > Thanks again for the kind reply. After going through the
> > > multichannels
> > > > >> > examples I have this toy example:
> > > > >> >
> > > > >> > def summi(p):
> > > > >> >     return p**2
> > > > >> >
> > > > >> >
> > > > >> > def multiplier(channel, factor1, factor2, factor3):
> > > > >> >     while not channel.isclosed():
> > > > >> >         param = channel.receive()
> > > > >> >         r = summi(p=param)
> > > > >> >         channel.send(factor1 * factor2 + factor3 + r)
> > > > >> >
> > > > >> > if __name__ == "__main__":
> > > > >> >
> > > > >> >     factor11 = 20
> > > > >> >     factor12 = 10
> > > > >> >     factor13 = 200
> > > > >> >
> > > > >> >     factor21 = 5
> > > > >> >     factor22 = 5
> > > > >> >     factor23 = 200
> > > > >> >
> > > > >> >
> > > > >> >     gw = execnet.makegateway()
> > > > >> >     channel1 = gw.remote_exec(multiplier, factor1=factor11,
> > > > >> > factor2=factor12, factor3=factor13)
> > > > >> >     channel1.send(1)
> > > > >> >     gw2 = execnet.makegateway()
> > > > >> >     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
> > > > >> > factor2=factor22, factor3=factor23)
> > > > >> >     channel2.send(1)
> > > > >> >     mch = execnet.MultiChannel([channel1, channel2])
> > > > >> >     queue = mch.make_receive_queue()
> > > > >> >     results = []
> > > > >> >     for i in range(2):
> > > > >> >         results.append(queue.get())
> > > > >> >     print results
> > > > >> >
> > > > >> >
> > > > >> > Now, when running it I get the following error:
> > > > >> >
> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
> > > > >> ['summi']).
> > > > >> >
> > > > >> > I tried the following:
> > > > >> >
> > > > >> > def multiplier(channel, factor1, factor2, factor3):
> > > > >> >      def summi(p):
> > > > >> >          return p**2
> > > > >> >     while not channel.isclosed():
> > > > >> >         param = channel.receive()
> > > > >> >         r = summi(p=param)
> > > > >> >         channel.send(factor1 * factor2 + factor3 + r)
> > > > >> >
> > > > >> >
> > > > >> > and with this change I got:
> > > > >> >
> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
> ['p',
> > > > >> 'p'])
> > > > >>
> > > > >> You need to send a module rather than just a function if your
> > > > >> function depends on another function in that module.
> > > > >> execnet only sends the source of the function, otherwise.
> > > > >> See here also:
> > > > >>
> > > > >>
> > > > >>
> > >
> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
> > > > >>
> > > > >> cheers,
> > > > >> holger
> > > > >>
> > > > >>
> > > > >> >
> > > > >> >
> > > > >> > I am just trying to understand the nature of the errors
> reported,
> > > and
> > > > >> > possibly find a way around it. The function calls that I want to
> > > > >> > parallelize invoke other functions from within, just as in the
> toy
> > > > >> example.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Any clarification is much appreciated.
> > > > >> >
> > > > >> > Thank you in advance.
> > > > >> >
> > > > >> > Luis.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <
> holger at merlinux.eu>
> > > > >> wrote:
> > > > >> >
> > > > >> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
> > > > >> > > > Thanks for your reply. I was trying to replicate one of the
> > > execnet
> > > > >> > > > examples available online and had problems using PyPy due to
> > > pydev
> > > > >> (using
> > > > >> > > > aptana under windows). But now it works. Thanks.
> > > > >> > > >
> > > > >> > > > Now, this is the kind of procedure that I am trying to
> replicate
> > > > >> (which
> > > > >> > > > currently uses the multiprocessing module and uses CPython
> to
> > > solve
> > > > >> the
> > > > >> > > > subproblems in parallel):
> > > > >> > > >
> > > > >> > > > out = mp.Queue()
> > > > >> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a, b,
> c,
> > > d,
> > > > >> out))
> > > > >> > > for
> > > > >> > > > s in range(len(networks))]
> > > > >> > > >
> > > > >> > > > *# Run subproblems*
> > > > >> > > > for sb in subproblems:
> > > > >> > > >     sb.start()
> > > > >> > > >
> > > > >> > > > *# Get results from the out queue*
> > > > >> > > > results = [out.get() for sb in subproblems]
> > > > >> > >
> > > > >> > > did you look into the multi-channel examples?
> > > > >> > >
> > > > >> > >     http://codespeak.net/execnet/example/test_multi.html
> > > > >> > >
> > > > >> > > you can start a pypy subprocess like so:
> > > > >> > >
> > > > >> > >     execnet.makegateway("popen//python=pypy")
> > > > >> > >
> > > > >> > > and that should work if used in the multichannel examples.
> > > > >> > >
> > > > >> > > holger
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > So, what I would like to do, is to solve the subproblems
> with
> > > PyPy,
> > > > >> as
> > > > >> > > they
> > > > >> > > > are time consuming. I would like to do this using execnet,
> but I
> > > > >> still
> > > > >> > > > haven't been able to get my head around it (Im clearly not
> even
> > > > >> close to
> > > > >> > > an
> > > > >> > > > expert in these matters). I wonder if anyone can point me
> in the
> > > > >> right
> > > > >> > > > direction, maybe through an example, on how to do this.
> > > > >> > > >
> > > > >> > > > Thanks very much in advance. I apologize for any
> inconvenience
> > > this
> > > > >> may
> > > > >> > > > cause.
> > > > >> > > >
> > > > >> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
> > > > >> fijall at gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Luis
> > > > >> > > > >
> > > > >> > > > > Things should work on pypy without any problem whatsoever,
> > > just
> > > > >> like
> > > > >> > > > > with normal python.
> > > > >> > > > >
> > > > >> > > > > Cheers,
> > > > >> > > > > fijal
> > > > >> > > > >
> > > > >> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
> > > > >> > > luisjosenovoa at gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > > > Hi Everyone,
> > > > >> > > > > >
> > > > >> > > > > > As part of a large scale optimization application, I am
> > > using
> > > > >> the
> > > > >> > > python
> > > > >> > > > > > multiprocessing module to execute a procedure several
> times
> > > in
> > > > >> > > parallel.
> > > > >> > > > > > These executions in turn, return results used by other
> > > > >> procedure,
> > > > >> > > and the
> > > > >> > > > > > whole thing repeats iteratively. I would like to call
> PyPy
> > > only
> > > > >> for
> > > > >> > > the
> > > > >> > > > > > subprocesses (to improve performance), as in the main
> > > procedure
> > > > >> I use
> > > > >> > > > > > libraries like gurobipy, which are not compatible with
> > > PyPy. I
> > > > >> > > looked at
> > > > >> > > > > the
> > > > >> > > > > > documentation of execnet online, but I could not find
> > > examples
> > > > >> for
> > > > >> > > > > > communication between CPython and PyPy.
> > > > >> > > > > >
> > > > >> > > > > > Any suggestions for using CPython + Multiprocessing with
> > > PyPy
> > > > >> would
> > > > >> > > be
> > > > >> > > > > > highly appreciated.
> > > > >> > > > > >
> > > > >> > > > > > Have a great day.
> > > > >> > > > > >
> > > > >> > > > > > --
> > > > >> > > > > > Luis J. Novoa
> > > > >> > > > > >
> > > > >> > > > > > _______________________________________________
> > > > >> > > > > > execnet-dev mailing list
> > > > >> > > > > > execnet-dev at python.org
> > > > >> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Luis J. Novoa
> > > > >> > >
> > > > >> > > > _______________________________________________
> > > > >> > > > execnet-dev mailing list
> > > > >> > > > execnet-dev at python.org
> > > > >> > > > https://mail.python.org/mailman/listinfo/execnet-dev
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > about me:    http://holgerkrekel.net/about-me/
> > > > >> > > contracting: http://merlinux.eu
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Luis J. Novoa
> > > > >>
> > > > >> --
> > > > >> about me:    http://holgerkrekel.net/about-me/
> > > > >> contracting: http://merlinux.eu
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Luis J. Novoa
> > >
> > > --
> > > about me:    http://holgerkrekel.net/about-me/
> > > contracting: http://merlinux.eu
> > >
> >
> >
> >
> > --
> > Luis J. Novoa
>
>
>
> --
> about me:    http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151118/368e0e00/attachment-0001.html>

From luisjosenovoa at gmail.com  Wed Nov 18 19:56:29 2015
From: luisjosenovoa at gmail.com (=?UTF-8?Q?Luis_Jos=C3=A9_Novoa?=)
Date: Wed, 18 Nov 2015 19:56:29 -0500
Subject: [execnet-dev] Sub-processes PyPy
In-Reply-To: <CAFKi5F8i860_tkw6LH2N8nKt+Pwngm9xgWA_Bh0LNOVQZb1MYA@mail.gmail.com>
References: <CAFKi5F_WVP1YJuHKGC9rhdJ2nkZcs3EB0iBMZE4pU80RDdCu7g@mail.gmail.com>
 <CAK5idxTQUgNjyRucPtPmAmMXSfkwL8Q7uh8pnfcHjj2XW5wRSw@mail.gmail.com>
 <CAFKi5F9q2DwORAqKXNpGH_JwfNw6d6iE=+WbzGt3tOZvhC6o8Q@mail.gmail.com>
 <20151112115015.GO16107@merlinux.eu>
 <CAFKi5F_RLe1Btd6fYbe30zLE5PqWu6jmOkPB3i5c0Rs2+Pkwsw@mail.gmail.com>
 <20151117091204.GW16107@merlinux.eu>
 <CAFKi5F9qbBg3AxTUVum_kfp9XaBOThXc0hF_V0Cm421rw=_MPw@mail.gmail.com>
 <CAFKi5F-6nYLXQ3Yu+Rec_RXrmeDK8tJhgZcgW-u0q4w916ynUA@mail.gmail.com>
 <20151118085230.GA16107@merlinux.eu>
 <CAFKi5F_bS9jMTs34yQS2+yiHAynu7F2A=wnuVrJkJbQEP+wEag@mail.gmail.com>
 <20151118155746.GH16107@merlinux.eu>
 <CAFKi5F8i860_tkw6LH2N8nKt+Pwngm9xgWA_Bh0LNOVQZb1MYA@mail.gmail.com>
Message-ID: <CAFKi5F9rK9mVqDSVLjXmVgn88-sB=aDYtCCEfwkdBCtYHAhKSw@mail.gmail.com>

Hi All. Another illustration of the problem can be seen here:

import execnet
import time
import outt


if __name__ == "__main__":

    start = time.time()

    factor11 = 20
    factor12 = 10
    factor13 = 200

    factor21 = 5
    factor22 = 5
    factor23 = 200

    gw1 = execnet.makegateway('popen//python=pypy.exe')
    gw2 = execnet.makegateway('popen//python=pypy.exe')


    channel1 = gw1.remote_exec(outt)
    channel1.send([factor11,factor12,factor13,10])
    channel2 = gw2.remote_exec(outt)
    channel2.send([factor21,factor22,factor23,4])

    res1 = channel1.receive()
    print res1

    res2 = channel2.receive()
    print res2

where, again, outt.py corresponds to

def summi(p):
    return p**2

def multiplier(channel, factor1, factor2, factor3, a):
        param = a[3]
        r = summi(p=param)
        s = []
        for i in xrange(900000):
            s.append(i)
        channel.send({'Number':factor1+factor2, 'list': sum(i for i in s)})

if __name__ == '__channelexec__':
    a = channel.receive()
    multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)


When running the first script I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 685, in runfile
    execfile(filename, namespace)
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 71, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)
  File "C:/Users/Luis J Novoa/Documents/Courses/I. Sixth
Semester/Research/CVRPTW/CODE/PExecnet.py", line 56, in <module>
    res2 = channel2.receive()
  File "C:\Users\Luis J
Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
line 737, in receive
    raise self._getremoteerror() or EOFError()
EOFError


If this change is made:

gw2 = execnet.makegateway('popen')

then the expected results are obtained, but I can see that the pypy.exe
remains open. I hope this can help identify the issue.

Again, I enormously appreciate your help.

On Wed, Nov 18, 2015 at 11:13 AM, Luis Jos? Novoa <luisjosenovoa at gmail.com>
wrote:

> Thanks. I am indeed using the latest pypy version for windows under Python
> 2.7 (pypy 4.0.0).
>
> One thing I noticed was that executing just one gateway and a channel
> under pypy does not cause problems. The problem appears when retrieving
> results from channels in two or more gws.
>
> LJN
> On Nov 18, 2015 10:57 AM, "holger krekel" <holger at merlinux.eu> wrote:
>
>> On Wed, Nov 18, 2015 at 09:40 -0500, Luis Jos? Novoa wrote:
>> > Thanks very much for your reply. In the command line I get no error, as
>> > shown in the image below.
>> >
>> > [image: Inline image 1]
>> >
>> > The error appears after executing queue = mch.make_receive_queue().
>> >
>> >
>> > I appreciate all the help.
>>
>> not sure what is different wrt to pypy/cpython here.
>> Make sure you are running the newest pypy versions.
>> Maybe Ronny can chime in at some point.
>>
>> holger
>>
>> > LJ
>> >
>> > On Wed, Nov 18, 2015 at 3:52 AM, holger krekel <holger at merlinux.eu>
>> wrote:
>> >
>> > > On Tue, Nov 17, 2015 at 20:55 -0500, Luis Jos? Novoa wrote:
>> > > > Hi All. Thanks in advance for the time invested in reading, thinking
>> > > about,
>> > > > and responding to my emails.
>> > > >
>> > > > Right now, I have something like this:
>> > > >
>> > > >
>> > > > import execnet as execnet
>> > > > import time
>> > > > import outt
>> > > >
>> > > >
>> > > > if __name__ == "__main__":
>> > > >
>> > > >     start = time.time()
>> > > >
>> > > >     factor11 = 20
>> > > >     factor12 = 10
>> > > >     factor13 = 200
>> > > >
>> > > >     factor21 = 5
>> > > >     factor22 = 5
>> > > >     factor23 = 200
>> > > >
>> > > >
>> > > >     group = execnet.Group(['popen'] * 2)
>> > > >
>> > > >     channel1 = group['gw0'].remote_exec(outt)
>> > > >     channel1.send([factor11,factor12,factor13,10])
>> > > >     channel2 = group['gw1'].remote_exec(outt)
>> > > >     channel2.send([factor21,factor22,factor23,4])
>> > > >
>> > > >     mch = execnet.MultiChannel([channel1, channel2])
>> > > >     queue = mch.make_receive_queue()
>> > > >
>> > > >     results = []
>> > > >     for i in range(2):
>> > > >         results.append(queue.get())
>> > > >     print results
>> > > >
>> > > >
>> > > >     group.terminate()
>> > > >
>> > > >     end = time.time() - start
>> > > >
>> > > >     print end, 'seconds'
>> > > >
>> > > >
>> > > > where, outt is a module containing (please do not try to make sense
>> out
>> > > of
>> > > > whatever is done inside the multiplier function here):
>> > > >
>> > > > if __name__ == '__channelexec__':
>> > > >
>> > > >     a = channel.receive()
>> > > >
>> > > >
>> > > >     def summi(p):
>> > > >         return p**2
>> > > >
>> > > >
>> > > >     def multiplier(channel, factor1, factor2, factor3, a):
>> > > >             param = a[3]
>> > > >             r = summi(p=param)
>> > > >             s = []
>> > > >             d = []
>> > > >             for i in xrange(900000):
>> > > >                 s.append(i)
>> > > >                 d.append(i)
>> > > >             channel.send({'Number':factor1+factor2, 'list': sum(i
>> for i
>> > > in
>> > > > s)})
>> > > >
>> > > >
>> > > >     multiplier(channel,factor1=a[0],factor2=a[1],factor3=a[2], a=a)
>> > >
>> > > I'd recommend putting the actual excuted code on both sides into
>> > > functions and just having the ``if __name__ == ...`` sections call
>> > > into those functions.  Then you don't need to indent function
>> definitions.
>> > >
>> > > > Im not sure if this is the best way to proceed, so if you see
>> something
>> > > > that does not make sense, Id really appreciate you letting me know.
>> If I
>> > > > run the first piece of code, I get the expected results (althought
>> Im not
>> > > > sure if the subprocesses are actually running in parallel), but
>> when I
>> > > > specify PyPy (group = execnet.Group(['popen//python=pypy.exe'] *
>> 2)) I
>> > > get
>> > > > the following error:
>> > > >
>> > > >
>> > > > [5976] Warning: unhandled RemoteError: Traceback (most recent call
>> last):
>> > > >   File "C:\Users\Luis J
>> > > >
>> > >
>> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
>> > > > line 841, in _local_receive
>> > > >     data = loads_internal(data, channel, strconfig)
>> > > >   File "C:\Users\Luis J
>> > > >
>> > >
>> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
>> > > > line 1350, in loads_internal
>> > > >     return Unserializer(io, channelfactory, strconfig).load()
>> > > >   File "C:\Users\Luis J
>> > > >
>> > >
>> Novoa\Anaconda\lib\site-packages\execnet-1.4.1-py2.7.egg\execnet\gateway_base.py",
>> > > > line 1160, in load
>> > > >     "wire protocol corruption?" % (opcode,))
>> > > > LoadError: unkown opcode '\n' - wire protocol corruption?
>> > >
>> > > Not sure how this comes.  If you just do
>> > >
>> > >     >>> execnet.makegateway("popen//python=pypy.exe")
>> > >
>> > > on the interactive prompt, what happens?
>> > >
>> > > holger
>> > >
>> > >
>> > >
>> > > >
>> > > > Once more, thanks for all the help.
>> > > >
>> > > > On Tue, Nov 17, 2015 at 8:20 AM, Luis Jos? Novoa <
>> > > luisjosenovoa at gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Thank you again for your kind reply. When passing a module, can I
>> pass
>> > > > > the  parameters corresponding to the function in the module that
>> calls
>> > > > > other functions?
>> > > > >
>> > > > > Thank you.
>> > > > >
>> > > > > LJN
>> > > > > On Nov 17, 2015 4:12 AM, "holger krekel" <holger at merlinux.eu>
>> wrote:
>> > > > >
>> > > > >> On Mon, Nov 16, 2015 at 16:56 -0500, Luis Jos? Novoa wrote:
>> > > > >> > Thanks again for the kind reply. After going through the
>> > > multichannels
>> > > > >> > examples I have this toy example:
>> > > > >> >
>> > > > >> > def summi(p):
>> > > > >> >     return p**2
>> > > > >> >
>> > > > >> >
>> > > > >> > def multiplier(channel, factor1, factor2, factor3):
>> > > > >> >     while not channel.isclosed():
>> > > > >> >         param = channel.receive()
>> > > > >> >         r = summi(p=param)
>> > > > >> >         channel.send(factor1 * factor2 + factor3 + r)
>> > > > >> >
>> > > > >> > if __name__ == "__main__":
>> > > > >> >
>> > > > >> >     factor11 = 20
>> > > > >> >     factor12 = 10
>> > > > >> >     factor13 = 200
>> > > > >> >
>> > > > >> >     factor21 = 5
>> > > > >> >     factor22 = 5
>> > > > >> >     factor23 = 200
>> > > > >> >
>> > > > >> >
>> > > > >> >     gw = execnet.makegateway()
>> > > > >> >     channel1 = gw.remote_exec(multiplier, factor1=factor11,
>> > > > >> > factor2=factor12, factor3=factor13)
>> > > > >> >     channel1.send(1)
>> > > > >> >     gw2 = execnet.makegateway()
>> > > > >> >     channel2 = gw2.remote_exec(multiplier, factor1=factor21,
>> > > > >> > factor2=factor22, factor3=factor23)
>> > > > >> >     channel2.send(1)
>> > > > >> >     mch = execnet.MultiChannel([channel1, channel2])
>> > > > >> >     queue = mch.make_receive_queue()
>> > > > >> >     results = []
>> > > > >> >     for i in range(2):
>> > > > >> >         results.append(queue.get())
>> > > > >> >     print results
>> > > > >> >
>> > > > >> >
>> > > > >> > Now, when running it I get the following error:
>> > > > >> >
>> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
>> > > > >> ['summi']).
>> > > > >> >
>> > > > >> > I tried the following:
>> > > > >> >
>> > > > >> > def multiplier(channel, factor1, factor2, factor3):
>> > > > >> >      def summi(p):
>> > > > >> >          return p**2
>> > > > >> >     while not channel.isclosed():
>> > > > >> >         param = channel.receive()
>> > > > >> >         r = summi(p=param)
>> > > > >> >         channel.send(factor1 * factor2 + factor3 + r)
>> > > > >> >
>> > > > >> >
>> > > > >> > and with this change I got:
>> > > > >> >
>> > > > >> > ValueError: ("the use of non-builtin globals isn't supported",
>> ['p',
>> > > > >> 'p'])
>> > > > >>
>> > > > >> You need to send a module rather than just a function if your
>> > > > >> function depends on another function in that module.
>> > > > >> execnet only sends the source of the function, otherwise.
>> > > > >> See here also:
>> > > > >>
>> > > > >>
>> > > > >>
>> > >
>> http://codespeak.net/execnet/example/test_info.html#remote-exec-a-module-avoiding-inlined-source-part-ii
>> > > > >>
>> > > > >> cheers,
>> > > > >> holger
>> > > > >>
>> > > > >>
>> > > > >> >
>> > > > >> >
>> > > > >> > I am just trying to understand the nature of the errors
>> reported,
>> > > and
>> > > > >> > possibly find a way around it. The function calls that I want
>> to
>> > > > >> > parallelize invoke other functions from within, just as in the
>> toy
>> > > > >> example.
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > Any clarification is much appreciated.
>> > > > >> >
>> > > > >> > Thank you in advance.
>> > > > >> >
>> > > > >> > Luis.
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Thu, Nov 12, 2015 at 6:50 AM, holger krekel <
>> holger at merlinux.eu>
>> > > > >> wrote:
>> > > > >> >
>> > > > >> > > On Tue, Nov 10, 2015 at 12:46 -0500, Luis Jos? Novoa wrote:
>> > > > >> > > > Thanks for your reply. I was trying to replicate one of the
>> > > execnet
>> > > > >> > > > examples available online and had problems using PyPy due
>> to
>> > > pydev
>> > > > >> (using
>> > > > >> > > > aptana under windows). But now it works. Thanks.
>> > > > >> > > >
>> > > > >> > > > Now, this is the kind of procedure that I am trying to
>> replicate
>> > > > >> (which
>> > > > >> > > > currently uses the multiprocessing module and uses CPython
>> to
>> > > solve
>> > > > >> the
>> > > > >> > > > subproblems in parallel):
>> > > > >> > > >
>> > > > >> > > > out = mp.Queue()
>> > > > >> > > > subproblems = [mp.Process(target=MyFunction, args=(s, a,
>> b, c,
>> > > d,
>> > > > >> out))
>> > > > >> > > for
>> > > > >> > > > s in range(len(networks))]
>> > > > >> > > >
>> > > > >> > > > *# Run subproblems*
>> > > > >> > > > for sb in subproblems:
>> > > > >> > > >     sb.start()
>> > > > >> > > >
>> > > > >> > > > *# Get results from the out queue*
>> > > > >> > > > results = [out.get() for sb in subproblems]
>> > > > >> > >
>> > > > >> > > did you look into the multi-channel examples?
>> > > > >> > >
>> > > > >> > >     http://codespeak.net/execnet/example/test_multi.html
>> > > > >> > >
>> > > > >> > > you can start a pypy subprocess like so:
>> > > > >> > >
>> > > > >> > >     execnet.makegateway("popen//python=pypy")
>> > > > >> > >
>> > > > >> > > and that should work if used in the multichannel examples.
>> > > > >> > >
>> > > > >> > > holger
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > >
>> > > > >> > > > So, what I would like to do, is to solve the subproblems
>> with
>> > > PyPy,
>> > > > >> as
>> > > > >> > > they
>> > > > >> > > > are time consuming. I would like to do this using execnet,
>> but I
>> > > > >> still
>> > > > >> > > > haven't been able to get my head around it (Im clearly not
>> even
>> > > > >> close to
>> > > > >> > > an
>> > > > >> > > > expert in these matters). I wonder if anyone can point me
>> in the
>> > > > >> right
>> > > > >> > > > direction, maybe through an example, on how to do this.
>> > > > >> > > >
>> > > > >> > > > Thanks very much in advance. I apologize for any
>> inconvenience
>> > > this
>> > > > >> may
>> > > > >> > > > cause.
>> > > > >> > > >
>> > > > >> > > > On Sat, Nov 7, 2015 at 10:02 AM, Maciej Fijalkowski <
>> > > > >> fijall at gmail.com>
>> > > > >> > > > wrote:
>> > > > >> > > >
>> > > > >> > > > > Hi Luis
>> > > > >> > > > >
>> > > > >> > > > > Things should work on pypy without any problem
>> whatsoever,
>> > > just
>> > > > >> like
>> > > > >> > > > > with normal python.
>> > > > >> > > > >
>> > > > >> > > > > Cheers,
>> > > > >> > > > > fijal
>> > > > >> > > > >
>> > > > >> > > > > On Thu, Nov 5, 2015 at 4:36 PM, Luis Jos? Novoa <
>> > > > >> > > luisjosenovoa at gmail.com>
>> > > > >> > > > > wrote:
>> > > > >> > > > > > Hi Everyone,
>> > > > >> > > > > >
>> > > > >> > > > > > As part of a large scale optimization application, I am
>> > > using
>> > > > >> the
>> > > > >> > > python
>> > > > >> > > > > > multiprocessing module to execute a procedure several
>> times
>> > > in
>> > > > >> > > parallel.
>> > > > >> > > > > > These executions in turn, return results used by other
>> > > > >> procedure,
>> > > > >> > > and the
>> > > > >> > > > > > whole thing repeats iteratively. I would like to call
>> PyPy
>> > > only
>> > > > >> for
>> > > > >> > > the
>> > > > >> > > > > > subprocesses (to improve performance), as in the main
>> > > procedure
>> > > > >> I use
>> > > > >> > > > > > libraries like gurobipy, which are not compatible with
>> > > PyPy. I
>> > > > >> > > looked at
>> > > > >> > > > > the
>> > > > >> > > > > > documentation of execnet online, but I could not find
>> > > examples
>> > > > >> for
>> > > > >> > > > > > communication between CPython and PyPy.
>> > > > >> > > > > >
>> > > > >> > > > > > Any suggestions for using CPython + Multiprocessing
>> with
>> > > PyPy
>> > > > >> would
>> > > > >> > > be
>> > > > >> > > > > > highly appreciated.
>> > > > >> > > > > >
>> > > > >> > > > > > Have a great day.
>> > > > >> > > > > >
>> > > > >> > > > > > --
>> > > > >> > > > > > Luis J. Novoa
>> > > > >> > > > > >
>> > > > >> > > > > > _______________________________________________
>> > > > >> > > > > > execnet-dev mailing list
>> > > > >> > > > > > execnet-dev at python.org
>> > > > >> > > > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > >> > > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > > Luis J. Novoa
>> > > > >> > >
>> > > > >> > > > _______________________________________________
>> > > > >> > > > execnet-dev mailing list
>> > > > >> > > > execnet-dev at python.org
>> > > > >> > > > https://mail.python.org/mailman/listinfo/execnet-dev
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > about me:    http://holgerkrekel.net/about-me/
>> > > > >> > > contracting: http://merlinux.eu
>> > > > >> > >
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > --
>> > > > >> > Luis J. Novoa
>> > > > >>
>> > > > >> --
>> > > > >> about me:    http://holgerkrekel.net/about-me/
>> > > > >> contracting: http://merlinux.eu
>> > > > >>
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Luis J. Novoa
>> > >
>> > > --
>> > > about me:    http://holgerkrekel.net/about-me/
>> > > contracting: http://merlinux.eu
>> > >
>> >
>> >
>> >
>> > --
>> > Luis J. Novoa
>>
>>
>>
>> --
>> about me:    http://holgerkrekel.net/about-me/
>> contracting: http://merlinux.eu
>>
>


-- 
Luis J. Novoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20151118/6ba4c3ac/attachment-0001.html>