[Python-Dev] Pre-PEP: Redesigning extension modules

[Nick Coghlan]

On 24 August 2013 23:19, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Nick Coghlan, 24.08.2013 13:36:
>> On 24 August 2013 15:51, Nick Coghlan wrote:
>>> My current plan is to create an experimental prototype of this
>>> approach this weekend. That will include stdlib test cases, so it will
>>> also show how it looks from the extension developer's point of view.
>>
>> I prototyped as much as I could without PEP 451's ModuleSpec support here:
>>
>>     https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
>
> Cool. I'll take a look.

The new _PyImport_CreateAndExecExtensionModule function does the heavy lifting:

    https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65

One key point to note is that it *doesn't* call
_PyImport_FixupExtensionObject, which is the API that handles all the
PEP 3121 per-module state stuff. Instead, the idea will be for modules
that don't need additional C level state to just implement
PyImportExec_NAME, while those that *do* need C level state implement
PyImportCreate_NAME and return a custom object (which may or may not
be a module subtype). Such modules can still support reloading (e.g.
to pick up reloaded or removed module dependencies) by providing
PyImportExec_NAME as well.

(in a PEP 451 world, this would likely be split up as two separate
functions, one for create, one for exec)

>> On systems that use dynload_shlib (at least Linux & the BSDs), this
>> branch allows extension modules to be imported if they provide a
>> PyImportExec_NAME hook. The new hook is preferred to the existing
>> PyInit_NAME hook, so extension modules using the stable ABI can
>> provide both and degrade to the legacy initialisation API on older
>> versions of Python.
>
> Hmm, right, good call. Since both init schemes have to be part of the
> stable ABI, we can's rely on people compiling out one or the other. So
> using the old one as a fallback should work. However, only actual usage in
> code will tell us how it feels on user side. Supporting both in the same
> binary will most likely complicate things quite a bit.

It shouldn't be too bad - the PyInit_NAME fallback would just need to
do the equivalent of calling PyImportCreate_NAME (or PyModule_Create
if not using a custom object), call PyImportExec_NAME on it, and then
return the result.

Modules that genuinely *needed* the new behaviour wouldn't be able to
provide a sensible fallback, and would thus be limited to Python 3.4+

>> The PyImportExec hook is called with a pre-created module object that
>> the hook is then expected to populate. To aid in this task, I added
>> two new APIs:
>>
>>     PyModule_SetDocString
>>     PyModule_AddFunctions
>>
>> These cover setting the docstring and adding module level functions,
>> tasks that are handled through the PyModule_Create API when using the
>> PyInit_NAME style hook.
>
> What are those needed for? If you subtype the module type, or provide an
> arbitrary extension type as implementation, you'd get these for free,
> wouldn't you? It's in no way different from setting up an extension type.

The idea is to let people use an import system provided module object
if they don't define a custom PyImportCreate_NAME hook.

Setting the docstring and adding module level functions were the two
things that PyModule_Create previously handled neatly through the
Py_ModuleDef struct. The two new API functions just break out those
subsets as separate operations to call on the import system provided
module.

>> The _testimportexec.c module
>
> Where can I find that module?

Oops, forgot to add it to the repo. Uploaded now:

https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Modules/_testimportexec.c?at=new_extension_imports

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia