From ncoghlan at gmail.com  Sun Jul 20 06:46:28 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 20 Jul 2014 14:46:28 +1000
Subject: [Import-SIG] Volunteer needed: PEP to migrate extension module
 imports to PEP 451 for Python 3.5
Message-ID: <CADiSq7c4Ot7t6KvUY9mjcg=tJ1bDrJTJogbVVhiU+=40B7DzKg@mail.gmail.com>

Lobbing this out there in case someone else has the time or
inclination to run with it before I do - I know Stefan has previously
gone as far as creating a proto-PEP for the idea
(https://mail.python.org/pipermail/python-dev/2013-August/128087.html),
so perhaps he'd be willing to take up the gauntlet again, since Cython
will hopefully be a key beneficiary here.

I, uh, kinda have a lot on my plate already - and this is much easier
to hand over to someone else than the complicated mess that is the
interpreter startup sequence, let alone any of the other things (like
metadata 2.0) that I don't think I could fully delegate even if I
wanted to. So, my apologies for being the bottleneck on this for so
long, this is me offering to step out of the way if someone else wants
to run with the task of thrashing out the implementation and API
details :)

Since I had to do a bit of digging to rediscover some of the relevant
details, a refresher course on where we got to prior to the 3.4
release:

Back during the PEP 451 discussions, one of the things I pushed for
was the ability for loaders to customise module *creation* in addition
to initialisation. This was specifically to support migration of C
extension module imports from the previous import hook design to the
new one - currently, the module init function inside the extension is
responsible for creating the module object, so migrating that in a
backwards compatible fashion means being able to return that module as
part of the loading process, rather than relying on one being
automatically created by the Python interpreter.

I posted a possible C level API design in Stefan's proto-PEP last year
(https://mail.python.org/pipermail/python-dev/2013-August/128098.html):

    PyObject * PyImportCreate_<modulename>(PyObject *spec); /* Optional */
    int PyImportExec_<modulename>(PyObject *mod);

And I created an initial prototype implementation as described in
https://mail.python.org/pipermail/python-dev/2013-August/128101.html

https://mail.python.org/pipermail/python-dev/2013-August/128113.html
covers some subsequent discussions I had with Stefan about the
technical details.

And then https://mail.python.org/pipermail/python-dev/2013-August/128234.html
covered some of the limitations of the prototype, since it predated
the actual implementation of PEP 451. In particular, reload handling
still isn't well defined.

PJE covered some of those reloading related concerns in more detail at
https://mail.python.org/pipermail/python-dev/2013-August/128129.html -
we haven't really fully resolved them at this point, but a full
extension modules migration PEP would need to do so.

I still like the basic concept of the PyImportCreate/Exec module as
the new way to implement extension modules. The theory would be that
taking an existing Python module and compiling it as a C extension
instead could rely entirely on implementing PyImportExec (populating a
standard module object with dynamically created types and function
objects), and let the interpreter take care of providing the module
object. One key advantage of this is that extension modules created
this way would *automatically* support subinterpreters properly, since
they wouldn't have any C level global state. Reloading of such modules
would also be *much* closer in behaviour to reloading of their Python
counterparts (since the PyImportExec function could just be rerun in
the existing module, as happens with normal Python modules)

PyImportCreate would then come into play in cases where there *was* C
level global state that needed to be managed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia