[Python-Dev] Pre-PEP: Redesigning extension modules

[Stefan Behnel]

Nick Coghlan, 24.08.2013 16:22:
> On 24 August 2013 23:19, Stefan Behnel wrote:
>> Nick Coghlan, 24.08.2013 13:36:
>>> On 24 August 2013 15:51, Nick Coghlan wrote:
>>>> My current plan is to create an experimental prototype of this
>>>> approach this weekend. That will include stdlib test cases, so it will
>>>> also show how it looks from the extension developer's point of view.
>>>
>>> I prototyped as much as I could without PEP 451's ModuleSpec support here:
>>>
>>>     https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
>>
>> Cool. I'll take a look.
> 
> The new _PyImport_CreateAndExecExtensionModule function does the heavy lifting:
> 
>     https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65
> 
> One key point to note is that it *doesn't* call
> _PyImport_FixupExtensionObject, which is the API that handles all the
> PEP 3121 per-module state stuff. Instead, the idea will be for modules
> that don't need additional C level state to just implement
> PyImportExec_NAME, while those that *do* need C level state implement
> PyImportCreate_NAME and return a custom object (which may or may not
> be a module subtype).

Is it really a common case for an extension module not to need any C level
state at all? I mean, this might work for very simple accelerator modules
with only a few stand-alone functions. But anything non-trivial will almost
certainly have some kind of global state, cache, external library, etc.,
and that state is best stored at the C level for safety reasons.


> Such modules can still support reloading (e.g.
> to pick up reloaded or removed module dependencies) by providing
> PyImportExec_NAME as well.
> 
> (in a PEP 451 world, this would likely be split up as two separate
> functions, one for create, one for exec)

Can't we just always require extension modules to implement their own type?
Sure, it's a lot of boiler plate code, but that could be handled by a
simple C code generator or maybe even a copy&paste example in the docs. I
would like to avoid making it too easy for users in the future to get
anything wrong with reloading or sub-interpreters. Most people won't test
these things for their own code and the harder it is to make them not work,
the more likely it is that a given set of dependencies will properly work
in a sub-interpreter.

If users are required to implement their own type, I think it would be more
obvious where to put global module state, how to define functions (i.e.
module methods), how to handle garbage collection at the global module
level, etc.


>>> On systems that use dynload_shlib (at least Linux & the BSDs), this
>>> branch allows extension modules to be imported if they provide a
>>> PyImportExec_NAME hook. The new hook is preferred to the existing
>>> PyInit_NAME hook, so extension modules using the stable ABI can
>>> provide both and degrade to the legacy initialisation API on older
>>> versions of Python.
>>
>> Hmm, right, good call. Since both init schemes have to be part of the
>> stable ABI, we can's rely on people compiling out one or the other. So
>> using the old one as a fallback should work. However, only actual usage in
>> code will tell us how it feels on user side. Supporting both in the same
>> binary will most likely complicate things quite a bit.
> 
> It shouldn't be too bad - the PyInit_NAME fallback would just need to
> do the equivalent of calling PyImportCreate_NAME (or PyModule_Create
> if not using a custom object), call PyImportExec_NAME on it, and then
> return the result.
> 
> Modules that genuinely *needed* the new behaviour wouldn't be able to
> provide a sensible fallback, and would thus be limited to Python 3.4+

Right. I only saw it from the POV of Cython, which *will* have to support
both, and *will* use the new feature in Py3.4+. No idea how that is going
to work, but we've found so many tricks and work-arounds in the past that
I'm sure it'll work somehow. Module level properties are just way too
tempting not to make use of them.

Stefan