Multiple interpreters and the global lock

[Daniel Dittmar]

Problem:

One topic that occasionally crops up is the wish to keep
multiple interpreters completely separated.  The standard
answer is a reference to the 'global lock', which allows
only one interpreter to run at a time.  This of course
limits scalability of Python on the server side, especially
on multi processor machines.

While searching DejaNews for the topic, I found that GvR is
not opposed to removing the lock, provided that there is no
negative impact on stability and performance.

What I hope to get as answers:
- is it possible?
- would patches be accepted, welcomed, frenetically applauded?
- what would be the smoothest path to completition, breaking
  the least code?


Core interpreter:

   All (most) routines will require an additional parameter,
the interpreter (PyThreadState *).  It is probably a good
idea to give these routines new names.  The old names are
still around as wrappers to the new routines, they take the
'current' interpreter from a global variable (read on to
'keeping extension modules binary compatible' why this is
useful).  Would the additional parameter have a 'negative
impact on performance'?

Because objects aren't shared between interpreter,
no special handling of reference counting is necessary.
(I know there are exceptions to this)


Extension modules:

New and adapted modules:
will call the new names, passing the interpreter object
along.  They get the interpreter as the first argument to an
extension function.  Such functions are marked by the
INTERPRETER_PROTECTED flag in the PyMethodDef struct.  They
would still use something like Py_BEGIN_ALLOW_THREADS etc.
for multiple threads inside a single interpreter, but this
lock would be interpreter local.

Binary compatibility:
because older extension functions lack the
INTERPRETER_PROTECTED flag, the 'current' interpreter knows
that they still rely on a global interpreter.  The 'current'
interpreter acquires the global lock (it's still around),
sets the global interpreter to itself and calls the
function.  Upon returning, the global lock is released and
the global interpreter pointer is cleared.  Would this
actually work for a module compiled for the current Python?
Or would there be a need to recompile these, thus providing
only source compatibility?


Problems:

Static objects (mostly type objects):

    PyTypeObject variables are often static in an extension
module.  This of course makes them shared between multiple
interpreters.  As reference counting is not (should not) be
synchronized, this could lead to a zero count and
destruction of a static object, which is never a good idea.
Does anyone has an idea how to solve this?  (I think it was
Frederik Lundh who pointed to this problem in a similar
thread).

memory management:

   having to synchronize on a single heap limits
scalability, especially when lots of objects get allocated
and released.  The right way would be to have each
interpreter have it's own heap.  As objects aren't shared,
this would be a solution.

   But is a bit more difficult to remain binary compatible
for 'old' modules.  This would require replacing 'malloc'
with a Python specific implementation.  Unfortunately, this
will not work in the Windows world, where extension DLLs are
linked to get their malloc from a specific runtime DLL.

   One intermediate step would be to provide a allocator
object as part of the interpreter and to encourage everyone
to use it.  The first version will only call malloc and
free, thus preserving compatibility.  But this gives
everyone a chance to prepare their sources for an upcoming
incompatible release (Python 2.0), where functions without
INTERPRETER_PROTECTED are no longer accepted.


Disclaimer:

   The cause for this post is not to advance mankind, but to
advance a commercial product.

   Any references to implementation details of Perl and
JavaScript are only partly for information.  They serve
mostly to stir competition.

Daniel Dittmar
daniel.dittmar at sap.com
SAP AG, Basis Entwicklung Berlin