[Python-Dev] Make extension module initialisation more like Python module initialisation

[M.-A. Lemburg]

On 08.11.2012 13:47, Stefan Behnel wrote:
> Hi,
> 
> I suspect that this will be put into a proper PEP at some point, but I'd
> like to bring this up for discussion first. This came out of issues 13429
> and 16392.
> 
> http://bugs.python.org/issue13429
> 
> http://bugs.python.org/issue16392
> 
> Stefan
> 
> 
> The problem
> ===========
> 
> Python modules and extension modules are not being set up in the same way.
> For Python modules, the module is created and set up first, then the module
> code is being executed. For extensions, i.e. shared libraries, the module
> init function is executed straight away and does both the creation and
> initialisation. This means that it knows neither the __file__ it is being
> loaded from nor its package (i.e. its FQMN). This hinders relative imports
> and resource loading. In Py3, it's also not being added to sys.modules,
> which means that a (potentially transitive) re-import of the module will
> really try to reimport it and thus run into an infinite loop when it
> executes the module init function again. And without the FQMN, it's not
> trivial to correctly add the module to sys.modules either.
> 
> We specifically run into this for Cython generated modules, for which it's
> not uncommon that the module init code has the same level of complexity as
> that of any 'regular' Python module. Also, the lack of a FQMN and correct
> file path hinders the compilation of __init__.py modules, i.e. packages,
> especially when relative imports are being used at module init time.
> 
> The proposal
> ============
> 
> ... [callbacks] ...
>
> Alternatives
> ============
> ...
> 3) The callback could be registered statically in the PyModuleDef struct by
> adding a new field. This is not trivial to do in a backwards compatible way
> because the struct would grow longer without explicit initialisation by
> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but
> would still break at least binary compatibility.

I think the above is the cleaner approach than the callback mechanism.
There's no problem in adding new slots to the end of the PyModuleDef struct
- we've been doing that for years in many other structs :-)

All you have to do is bump the Python API version number.

(Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 08 2012)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/