[Cython] dynamically compile and import pure python modules

[Stefan Behnel]

mark florisson, 22.03.2012 19:50:
> For the fused type runtime dispatch I found it very convenient to use
> the with statement, but that is not supported in Python 2.4. However,
> the compiler could dynamically compile compiler code with the compiler
> itself and import it (pyximport), if it is not needed to compile
> Cython itself. I gave it a try and it seems to work like a charm (but
> probably needs more testing :):
> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2

The advantages are limited, so I'm leaning towards seeing the drawbacks, of
which there are at least some. For one, *running* Cython (as opposed to
installing it) becomes more complex and involves a (much) higher first time
overhead. We'd also start putting shared libraries into user directories
without asking them first. Might be a problem on shared installations with
many users.

Note also that Cython no longer compiles itself when installing in PyPy (at
all), but that would be easy to special case here (and PyPy obviously has
features like the "with" statement).

Next, I think it would tempt us to split source files into separate modules
just because that way we can use a specific feature in one of them because
it'll get compiled (and the other half is needed for bootstrapping). That
would be bad design. OTOH, it might be worth taking a general look at the
code base to see what's really required for bootstrapping, or maybe for
compiling pure Python code in general. Factoring that out, and thus
separating the Python compiler from the Cython specific language features
might (might!) be an actual improvement. (Then again, there's .pxd
overriding and the "cython" module, which add Cython features back in, and
those two make Cython much more attactive...)

I also started to dislike pyximport because it's way outdated, fragile,
complicated and its features are overly hacked together (and I'm not
entirely innocent in that regard). I would love to see a rewrite that also
supports compiling packages properly. Not a GSoC by itself, but it's
certainly a worthy project. What about a project that aims for separating
out a Python compiler and rewriting pyximport as a jitty frontend for it?
Maybe not the greatest use case ever, but a fun project, I'd say.

Stefan