xmlrpc idea for getting around the GIL

[Patrick Stinson]

On Mon, Nov 23, 2009 at 1:01 AM, Carl Banks <pavlovevidence at gmail.com> wrote:
> On Nov 22, 10:58 pm, Patrick Stinson <patrickstinson.li... at gmail.com>
> wrote:
>> On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch <de... at nospam.web.de> wrote:
> icating) the multiprocessing module would be ideal.
>> > The problem is that the OP has a embedded application running threads.
>> > multiprocssing doesn't help there.
>>
>> that's right. I cannot make CPython calls from my original C-based threads.
>
>
> It's quite possible to do that.  A thread started from C can make
> calls to Python if it first calls PyGILState_Ensure, although you'd
> have to make sure that the Python interpreter has been previously
> initialized.  See PEP 311 for details.
>
> http://www.python.org/dev/peps/pep-0311/
>
> I also suggest that if you want people to be more receptive to write
> your replies after the quoted text.  That is the custom in this
> newsgroup/mailing list.
>
>
> Carl Banks
> --
> http://mail.python.org/mailman/listinfo/python-list
>

thanks for the tip.

What I meant was that I am *not allowed* to make calls to the CPython
API from the threads I currently have because these threads are high
priority and are never allowed to make blocking calls. Fortunately,
each of these threads can have a completely separate interpreter, so
my idea was to create a daemon process for each thread. This daemon
would contain the libpython symbols and would make the CPython calls.
This would keep my current threads from having to contend over the
single GIL.

My question was whether or not anyone has done anything like this in
C/C++. This situation is slightly unique in that I am trying to
maintain separate interpreters within a single app (which is currently
kind of hacked out using a single interpreter, but it's ugly), but I
could see how this sort of thing would be useful for other C/C++ apps
that implement an embedded scripting engine.

My reference to multiprocessing was based on the idea that the library
hides the details fo the process management, shared memory, and rpc
mechanisms. Again, I can't use multiprocessing because it runs *in*
python I need this to be implemented *outside* of python to avoid
acquiring the GIL. complex, I know.

Naturally, the most intimidating part of perusing this kind of idea is
the overhead of the processess management and the RPC. Further,
libpython executes callable objects using C function pointers, and I
can't think of a way that this would be able to re-gain access to the
original app's respective functions if the interpreter was living in
another processes.

That's not to mention the impossibility of debugging.

Damn you, Gil.