C API design flaw (was: Re: Multiple independent Python interpreters in a C/C++ program?)

[sturlamolden]

On Apr 12, 7:05 pm, sturlamolden <sturlamol... at yahoo.no> wrote:

> In theory, a GIL private to each (sub)interpreter would make Python
> more scalable. The current GIL behaves like the BKL in earlier Linux
> kernels. However, some third-party software, notably Apache's
> mod_python, is claimed to depend on this behaviour.

I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_Get()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environment" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensure would take the interpreter
returned by Py_NewInterpreter as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensure does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensure, and almost every function in Python's C API, is the
heart of the problem.