[Import-SIG] PEP 489: Redesigning extension module loading

[Petr Viktorin]

On 03/26/2015 05:25 AM, Nick Coghlan wrote:
> On 25 March 2015 at 23:36, Petr Viktorin <encukou at gmail.com> wrote:
>> On 03/25/2015 01:11 PM, Nick Coghlan wrote:
>>>
>>> On 25 March 2015 at 02:34, Petr Viktorin <encukou at gmail.com> wrote:
>>>>
>>>> I'll share my notes on an API with PEP 384-style slots, before attempting
>>>> to
>>>> write it out in PEP language.
>>>>
>>>> I struggled to find a good name for the "PyType_Spec" equivalent, since
>>>> ModuleDef and ModuleSpec are both taken, but then I realized that, if the
>>>> docstring is put in a slot, I just need an array of slots...
>>>
>>>
>>> Because we're looking for an exported symbol, I think there's value in
>>> having a more clearly defined top level structure rather than just an
>>> array.
>>
>>
>> OK.
>> I'm not sure on cross-platform support of data rather than functions
>> exported from shared libraries, so kept the hook as a function.
>> Perhaps I'm being too paranoid here?
>
> Given that http://bugs.python.org/issue23743 came across my inbox this
> morning, I'm going to go with "No, you're not being too paranoid once
> we take C++ compilers and linkers into account".

Yeah. I'm usually against boilerplate but here the extra function will 
probably save headaches later.

> Perhaps we could make it use a new PyExport prefix though and drop the
> integer IDs in favour of exporting additional symbols? That is, have
> the hook be "PyExport_spam" with a separate "PyExport_spam_methods"?
>
> That opens the door to potentially having *other* export APIs in the
> future, like "PyExport_spam_codecs", "PyExport_spam_types",
> "PyExport_spam_constants_str".

I'm not convinced. I think the dynamic loading machinery is too 
sensitive to weird platform-specific compiler/linker details to add bits 
like this whenever they're needed. Having everything packaged up as a 
module, which registers whatever it wants when it's loaded, seems better 
to me.
Also, I believe all these extension APIs should preferably include 
module names. The PyCapsule_Import naming convention is a good idea. 
Makes it easier to know where things come from.

[...]

> Pulling this idea for your full extension example:
>
> static PyExportDef_Method spam_methods[] = {
>      {"demo", (PyCFunction)spam_demo,  ...},
>      {NULL, NULL}
> };
>
> static PyExportDef_ModuleState spam_statedef = {
>      sizeof(spam_state_t),
>      spam_state_traverse,
>      spam_state_clear,
>      spam_state_free
>      /* any of those three can be NULL if not needed */
> }
>
> static PyExportDef_Module spam_module = {
>      PyDoc_STR("A spammy module"),
>      spam_exec,
>      spam_statedef
> }
>
> PyExportDef_Module *PyExport_spam {
>      return spam_module;
> }
>
> PyExportDef_Method *PyExport_spam_methods {
>      return spam_methods;
> }
>
> Using slots instead, the last part (from spam_module down) would
> revert to being closer to your example:
>
> static PyExportDef_ModuleSlot spam_slots[] = {
>      {Py_m_doc, PyDoc_STR("A spammy module")},
>      {Py_m_methods, spam_methods},
>      {Py_m_statedef, spam_statedef},
>      {Py_m_exec, spam_exec},
>      {0, NULL}
> }
>
> PyExportDef_ModuleSlot *PyExport_spam {
>      return spam_slots;
> }
>
> So actually writing that down suggests numeric slots may still be a
> better idea.

Yes, I think so as well. Also consider:
- Python can fail hard on unknown slot numbers. Checking for unknown 
exports would require enumerating exported symbols, which is definitely 
not something I can code for every platform (if that's even possible).
- Additional exported functions are not actually more type-safe – the 
exported symbols would be void*, Python would still need to cast to an 
appropriate function type.


> I like the "PyExport" and "PyExportDef" prefixes though.

As above, I think modules are the only thing that should be exported, 
and with that, "PyModuleExport" sounds better.