[Cython] PEP 489 impacts "public" declarations (generated module header file)

[Stefan Behnel]

Hi all!

Here's a question to cython-users. Is anyone of you using the "public"
modifier to export symbols to external code? And are you calling the module
init function from them directly, e.g. when embedding a Cython module in a
C/C++ application? I'd like to get some feedback on how people use this
feature.

Here's why: I'm pretty much done implementing PEP 489 for CPython 3.5+,
which splits the extension module initialisation into multiple steps to
enable more Python-like module features. It changes the behaviour of the
module init function, which no longer does the complete module creation and
initialisation, but only returns a metadata object (PyModuleDef) that
steers the init process.

https://www.python.org/dev/peps/pep-0489/

My implementation is here:

https://github.com/cython/cython/pull/1794

One problem that I found: The docs on `public` declarations show how to
call the (Py2) module init function in order to initialise the module when
embedding:

http://docs.cython.org/en/latest/src/userguide/external_C_code.html#public-declarations

That is no longer enough in Py3.5+. The PEP contains a minimal example of
what needs to be done now, in addition to some new C-API functions that
help doing it:

https://www.python.org/dev/peps/pep-0489/#legacy-init

https://www.python.org/dev/peps/pep-0489/#new-functions

I can see two ways to fix the new situation: let users adapt their code
(assuming that it's probably a rare use case), or export a new public
function that implements the module initialisation process similar to what
Cython does in pre-Py3.5, but probably losing most of the new features,
such as setting a proper value for "__file__" at initialisation time,
although we could accept an (optional?) ModuleSpec as argument.

Exporting such a function adds convenience, at least for some users, at the
price of exporting another public name from Cython modules (at least those
that use public symbols). But it's not clear that such a function would
cover the user's needs, because some might want to influence the process
(that's what the PEP was written for, after all). And since I expect the
number of users to be low, adding a convenience feature for even less of
them might be unhelpful.

I'd also rather avoid putting this function into the header file itself,
because that makes code that uses it highly dependent on the internals,
which might change with newer Cython/Python versions.

So, does anyone use this feature and could explain a bit how you use it to
help us understand what we should consider "convenient" for users?

Stefan