[Import-SIG] PEP 489: Redesigning extension module loading; version 4

[Petr Viktorin]

On Thu, May 7, 2015 at 5:35 PM, Petr Viktorin <encukou at gmail.com> wrote:
> Hello!
>
> Based on previous discussions, particularly the lacks of objections to
> repurposing ModuleDef.m_reload, I've sent an updated version of PEP 489
> to the editors. I'm including a copy below.
>
> The implementation is nearly finished, with several things missing:
> - Support for non-Linuxy platforms
> - PyImport_Inittab, see below
> - Documentation
> - porting "xx" and "xxsubtype" modules (but "xxlimited" is done)
[...]
>
>
> Some further thoughts:
>
> The docstring and methods are initialized in the creation step, rather
> than exec. I don't think it's important enough to do this in exec, and
> this way the implementation is easier (with respect to NULL slots, and
> backwards compatibility with PyInit-based modules where Exec is a no-op).
>
> As I was implementing this, I ran into PyImport_Inittab. I'll need to
> add a similar list of PyModuleDefs.

And here I'm somewhat stumped, can someone help me find the right direction?

There's a tool called freeze, which (among other things) generates the
PyImport_Inittab, in the file config.c which looks a bit like this:

extern PyObject* PyInit__thread(void);
extern PyObject* PyInit__signal(void);
[... and so on for the other modules ...]

struct _inittab _PyImport_Inittab[] = {
    {"_thread", PyInit__thread},
    {"_signal", PyInit__signal},
    [... and so on for the other modules ...]
};

This file is generated just from a list of module names, without
loading them. So, it can't easily determine whether a module uses
PyInit_*, or PyModuleExport_*. But it needs to choose the hook name
correctly, otherwise the program will fail to link.

I can see three solutions for this problem.
I could modify freeze to inspect the modules somehow. I'm wary of
writing platform-specific code for such an edge case, though, and I'm
not sure if freeze always has access to the modules it processes,
rather than just their names.

I could introduce some way to specify which hook is used out-of band.
But that's just passing the problem on to users, not solving it.
Also, freeze is pretty minimal and I'm vaguely aware of third-party
tools that do something similar (cx_freeze, py2exe, py2app); I might
need to coordinate with them.

Or, I could keep the "PyInit_*" hook name, and allow it to return
PyModuleDef instead of a module. This is obviously a hack, and would
force me to get back down to the drawing board, but considering the
options it seems best to explore this option.
(PyInit_* and PyModuleExport_* signatures are technically compatible,
since a PyModuleDef is a PyObject)

I'd welcome your thoughts.