[Python-Dev] Make extension module initialisation more like Python module initialisation

[Stefan Behnel]

Stefan Behnel, 08.11.2012 14:20:
> M.-A. Lemburg, 08.11.2012 14:01:
>> On 08.11.2012 13:47, Stefan Behnel wrote:
>>> I suspect that this will be put into a proper PEP at some point, but I'd
>>> like to bring this up for discussion first. This came out of issues 13429
>>> and 16392.
>>>
>>> http://bugs.python.org/issue13429
>>>
>>> http://bugs.python.org/issue16392
>>>
>>> Stefan
>>>
>>>
>>> The problem
>>> ===========
>>>
>>> Python modules and extension modules are not being set up in the same way.
>>> For Python modules, the module is created and set up first, then the module
>>> code is being executed. For extensions, i.e. shared libraries, the module
>>> init function is executed straight away and does both the creation and
>>> initialisation. This means that it knows neither the __file__ it is being
>>> loaded from nor its package (i.e. its FQMN). This hinders relative imports
>>> and resource loading. In Py3, it's also not being added to sys.modules,
>>> which means that a (potentially transitive) re-import of the module will
>>> really try to reimport it and thus run into an infinite loop when it
>>> executes the module init function again. And without the FQMN, it's not
>>> trivial to correctly add the module to sys.modules either.
>>>
>>> We specifically run into this for Cython generated modules, for which it's
>>> not uncommon that the module init code has the same level of complexity as
>>> that of any 'regular' Python module. Also, the lack of a FQMN and correct
>>> file path hinders the compilation of __init__.py modules, i.e. packages,
>>> especially when relative imports are being used at module init time.
>>>
>>> The proposal
>>> ============
>>>
>>> ... [callbacks] ...
>>>
>>> Alternatives
>>> ============
>>> ...
>>> 3) The callback could be registered statically in the PyModuleDef struct by
>>> adding a new field. This is not trivial to do in a backwards compatible way
>>> because the struct would grow longer without explicit initialisation by
>>> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but
>>> would still break at least binary compatibility.
>>
>> I think the above is the cleaner approach than the callback mechanism.
> 
> Oh, definitely.
> 
> 
>> There's no problem in adding new slots to the end of the PyModuleDef struct
>> - we've been doing that for years in many other structs :-)
>>
>> All you have to do is bump the Python API version number.
>>
>> (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details)
> 
> The difference is that this specific struct is provided by user code and
> (typically) initialised statically. There is no guarantee that user code
> that does not expect the additional field will initialise it to 0. Failing
> that, I don't see how we could trust its value in any way.

Hmm - you're actually right. In C, uninitialised fields in a static struct
are set to 0 automatically. Same case as the type structs. That makes your
objection perfectly valid. I'll rewrite and shorten the proposal.

Thanks!

Stefan