[Import-SIG] PEP 489: Redesigning extension module loading

[Petr Viktorin]

On 04/17/2015 08:51 AM, Stefan Behnel wrote:
> Petr Viktorin schrieb am 16.04.2015 um 13:05:
>> A wart I added is "singleton modules", necessary for
>> "PyState_FindModule"-like functionality. I wouldn't mind not including
>> this, but it would mean the new API can't replace all use cases of the
>> old PyInit_.
>>
>> Singleton Modules
>> -----------------
>>
>> Modules defined by PyModuleDef may be registered with PyState_AddModule,
>> and later retrieved with PyState_FindModule.
>>
>> Under the new API, there is no one-to-one mapping between PyModuleSpec
>> and the module created from it.
>> In particular, multiple modules may be loaded from the same description.
>
> Is that because a single shared library (which is what the module spec
> refers to, right?) can contain multiple modules? Or are you referring to
> something else here?

By using Loader.create_module/Loader.exec_module directly, you can load 
an extension module without adding it to sys.modules. You can do this as 
many times as you like, and you always get a new, independent module object.


>> This means that there is no "global" instance of a module object.
>> Any C-level callbacks that need access to the module state need to be passed
>> a reference to the module object, either directly or indirectly.
>>
>>
>> However, there are some modules that really need to be only loaded once:
>> typically ones that wrap a C library with global state.
>> These modules should set the PyModule_EXPORT_SINGLETON flag
>> in PyModuleDesc.flags. When this flag is set, loading an additional
>> copy of the module after it has been loaded once will return the previously
>> loaded object.
>> This will be done on a low level, using _PyImport_FixupExtensionObject.
>> Additionally, the module will be automatically registered using
>> PyState_AddSingletonModule (see below) after execution slots are processed.
>>
>> Singleton modules can be retrieved, registered or unregistered with
>> the interpreter state using three new functions, which parallel their
>> PyModuleDef counterparts, PyState_FindModule, PyState_AddModule,
>> and PyState_RemoveModule::
>>
>>      PyObject* PyState_FindSingletonModule(PyModuleDesc *desc)
>>      int PyState_AddSingletonModule(PyObject *module, PyModuleDesc *desc)
>>      int PyState_RemoveSingletonModule(PyModuleDesc *desc)
>>
>>
>> .. note::
>>
>>      If PyModuleDef is used instead of PyModuleDesc, the flag would be specified
>>      as a slot with NULL value, i.e. ``{Py_mod_flag_singleton, NULL}``.
>>      In this case, PyState_FindModule, PyState_AddModule and
>>      PyState_RemoveModule can be used instead of the new functions.
>>
>> .. note::
>>
>>      Another possibility is to use PyModuleDef_Base in PyModuleDesc, and
>>      have PyState_FindModule and friends work with either of the two structures.
>
> Yes, this is totally a wart. However, I'm not sure I understand the actual
> use case from what is written above. Could you clarify why this whole
> special case is needed?

Normally, you need to pass module object to any C-level callbacks that 
need the module state in any way. Since in the new scheme of things 
there may be multiple modules, you can't just attach a module object to 
interpreter state and then look it up.

However, consider wrapping a C library with global state. The library 
might not allow you to pass arbitrary data to your callbacks, so there's 
no proper way to get to the module object.
So you want to load the module only once, returning the same object when 
it's created from the same slots, and a way to get to your module object 
from anywhere. That's what Python does currently, with 
PyState_FindModule for finding the module.

Well, that's the use case as I understand it.

It would read a bit better if PyModule_Def is reused – in that variant 
it's a way to keep PyState_FindModule working. The more I look at this 
though, the more I see using PyState_FindModule as something that should 
just be discontinued when converting a module to the new API.

Perhaps it'll be better to remove the flag; there's always a possibility 
to add it in the future.

-- 
Petr Viktorin