[Python-Dev] multiple interpreters and extension modules

[Guido van Rossum]

I think you understand exactly what is happening. It is happening for
good reasons. Rather than asking for a change in semantics, I
recommend that you deal with it, either in your Python code, or in
your extension. It's not likely to change.

--Guido

On 12/22/06, Jeremy Kloth <jeremy.kloth at 4suite.org> wrote:
> [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython
> development enough to merit posting here ]]
>
> I have received a bug report for 4Suite that involves a PyObject_IsInstance
> check failing for what appears to be the correct class, that is, the class
> names match.  With some investigating, I have found that the true problem is
> with multiple interpreters.  The reason for this is that each sub-interpreter
> has a "new" copy of any pure Python module. The following issues are also
> true for modules that have been reloaded, but I think that is common
> knowledge.  I mention it only for completeness.
>
> Here is where it crosses into CPython development.
>
> Now extension modules do not get reloaded into each sub-interpreter, but a
> copy of the module's dict from when it was first loaded.  This particular
> extension, when initialized, loads the class in question from the pure Python
> module and stores in in a global variable for quick access. I know I can
> change this specific case to reload the class each time and for this exact
> case, it is most likely what I'll do.  However, we have other extensions that
> make use of this same technique and their performance would suffer greatly by
> doing the lookup for each use of the class.
>
> Also performance isn't the only issue here as we also have C types that
> inherit from a Python class and those types will obviously fail an
> isinstance() check in Python code run in different interpreters as the base
> class is "frozen" at the first import.
>
> So I guess the questions are:
> 1) is subclassing Python classes in C as a static type supported? Even if they
> would be declared on the heap, they would be bound to the first loaded Python
> class.
>
> 2) would it be worthwhile to have an extension loading function similar to
> DllMain() for Windows' DLLs which is called multiple times with a flag
> stating the reason for the call? For reference, see:
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp
> I envision the reasons as: 'module init', 'module fini', 'interpreter
> init', 'interpreter fini'. I see no current need for special treatment of
> threads as they share the same interpreter space.
>
> As to question 2, I'm in the process of prototyping that approach to see if it
> is feasible in our code and if so, (and people find it useful) I'll attempt
> to write a PEP for that feature.
>
> My current thoughts on that are instead of the current init<module name> entry
> point in C modules, a new entry point is defined, and if found, used in its
> place. That way, this proposal would be optional for extension writers and
> existing modules would continue to work without change.
>
> Feedback appreciated.
>
> --
> Jeremy Kloth
> http://4suite.org/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)