Exploiting Dual Core's with Py_NewInterpreter's separated GIL ?

[Filip Wasilewski]

robert wrote:
> I'd like to use multiple CPU cores for selected time consuming Python computations (incl. numpy/scipy) in a frictionless manner.
>
> Interprocess communication is tedious and out of question, so I thought about simply using a more Python interpreter instances (Py_NewInterpreter) with extra GIL in the same process.
> I expect to be able to directly push around Python Object-Trees between the 2 (or more) interpreters by doing some careful locking.

I don't want to discourage you but what about reference counting/memory
management for shared objects? Doesn't seem fun for me.


Take a look at IPython1 and it's parallel computing capabilities [1,
2]. It is designed to run on multiple systems or a single system with
multiple CPU/multi-core. It's worker interpreters (engines) are loosely
coupled and can utilize several MPI modules, so there is no low-level
messing with GIL. Although it is work in progress it already looks
quite awesome.

[1] http://ipython.scipy.org/moin/Parallel_Computing
[2] http://ipython.scipy.org/moin/Parallel_Computing/Tutorial

fw