[Python-Dev] nested extension modules?

[Guido van Rossum]

> Using the source (Luke), I was trying to figure out the best way to add a
> nested submodule from within an extension module. I noticed that the module
> initialization code will set the module name from the package context (if
> set), altogether discarding any name passed explicitly:
> 
> [modsupport.c: Py_InitModule4()]
> 
>  ...
>  if (_Py_PackageContext != NULL) {
>   char *p = strrchr(_Py_PackageContext, '.');
>   if (p != NULL && strcmp(name, p+1) == 0) {
>    name = _Py_PackageContext;
>    _Py_PackageContext = NULL;
>   }
>  }
> 
> This _Py_PackageContext is set up from within _PyImport_LoadDynamicModule
> [importdl.c:]
> 
>  ...
>  oldcontext = _Py_PackageContext;
>  _Py_PackageContext = packagecontext;
>  (*p)();
>  _Py_PackageContext = oldcontext;
> 
> IIUC, this means that when an extension module is loaded as part of a
> package, any submodules I create my calling Py_InitModule<whatever> will
> come out with the same name.
> 
> Questions:
> 
> a. Have I got the analysis right?

Not quite, if I understand what you're saying.  The package context,
despite its name, is not the package name, but the full name of the
*module*, when the shared library is found inside a package.

If, e.g., a package directory P contains an extension module file
E.so, the package context is set to "P.E".  The initE() function is
supposed to call Py_InitModule4() with "E" as the module name.
Py_InitModule4() then sees that this is the last component of the
package context, and changes the module name to "P.E".  It also nulls
out the package context.

The checkin comment I made back in 1997 explains this:

    Fix importing of shared libraries from inside packages.
    This is a bit of a hack: when the shared library is loaded, the
    module name is "package.module", but the module calls
    Py_InitModule*() with just "module" for the name.  The shared
    library loader squirrels away the true name of the module in
    _Py_PackageContext, and Py_InitModule*() will substitute this (if
    the name actually matches).

> b. Is there a more-sanctioned way around this other than touching
> _Py_PackageContext (which seems to be intended to be private)

I think using _Py_PackageContext is your only hope.  If you contribute
some docs for it we'll gladly add them to the API docs.

--Guido van Rossum (home page: http://www.python.org/~guido/)