More than one interpreter per process?

[Graham Dumpleton]

On Dec 19, 3:07 pm, Roger Binns <rog... at rogerbinns.com> wrote:
> Graham Dumpleton wrote:
> > When using mod_wsgi there is no problem with C extension modules which
> > use simplified GIL API provided that one configures mod_wsgi to
> > delegate that specific application to run in the context of the first
> > interpreter instance created by Python.
>
> Graham, I've asked you before but never quite got a straight answer.

Maybe because it isn't that simple. :-)

> What *exactly* should extension authors change their code to in order to
> be fully compatible?  For example how should the C function below be
> changed:
>
> void somefunc(void)
> {
>   PyGILState_STATE gilstate=PyGILState_Ensure();
>
>   abc();
>
>   Py_BEGIN_ALLOW_THREADS
>      def();
>   Py_END_ALLOW_THREADS
>
>   ghi();
>
>   PyGILState_Release(gilstate);
>
> }

What you do depends on what the overall C extension module does. It
isn't really possible to say how the above may need to be changed as
there is a great deal of context which is missing as far as knowing
how that function comes to be called. Presented with that function in
isolation I can only say that using simplified GIL API is probably the
only way of doing it and therefore can only be used safely against the
first interpreter created by Python.

If the direction of calling for a C extension module is always Python
code into C code and that is far as it goes, then none of this is an
issue as you only need to use Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS.

The problem case is where C code needs to callback into Python code
and you are not using simplified GIL API in order to be able to
support multiple sub interpreters. The easiest thing to do here is to
cache a thread state object for the interpreter instance when you
first obtained the handle for the object which allows you to interact
with C extension module internals. Later when a callback from C to
Python code needs to occur then you lookup the cached thread state
object and use that as the argument to PyEval_AcquireThread().

As example see:

  http://svn.dscpl.com.au/ose/trunk/software/python/opydispatch.cc

The thread state object is cached when handle to an instance is first
created. Any callbacks which are registered remember the interpreter
pointer and then that is used as key to lookup up the cached thread
state.

This code was done a long time ago. It is possible that it needs to be
revised based on what has been learnt about simplified GIL API.

This way of doing things will also not work where it is possible that
sub interpreters are created and then later destroyed prior to process
exit because of the fact that the interpreter pointer is cached. But
then, enough C extension modules have this problem that recycling sub
interpreters in a process isn't practical anyway.

Note that the indicated file uses a global cache. The agent.hh/
opyagent.cc files at that same location implement a more complicated
caching system based on individual objects.

The software which this is from is probably a more extreme example of
what is required, but your example was too simple to draw any
conclusions from.

Graham