[Cython] dynamically compile and import pure python modules

[mark florisson]

On 22 March 2012 20:07, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 22.03.2012 19:50:
>> For the fused type runtime dispatch I found it very convenient to use
>> the with statement, but that is not supported in Python 2.4. However,
>> the compiler could dynamically compile compiler code with the compiler
>> itself and import it (pyximport), if it is not needed to compile
>> Cython itself. I gave it a try and it seems to work like a charm (but
>> probably needs more testing :):
>> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2
>
> The advantages are limited, so I'm leaning towards seeing the drawbacks, of
> which there are at least some. For one, *running* Cython (as opposed to
> installing it) becomes more complex and involves a (much) higher first time
> overhead. We'd also start putting shared libraries into user directories
> without asking them first. Might be a problem on shared installations with
> many users.

The overhead would only be for certain python versions that try to use
certain functionality, in this case, python2.4 and fused types. To be
honest, the overhead isn't very large. As for compiling shared
libraries, I don't think people will complain about shared libraries,
that's the only way in which Python and Cython can be used.

> Note also that Cython no longer compiles itself when installing in PyPy (at
> all), but that would be easy to special case here (and PyPy obviously has
> features like the "with" statement).
>
> Next, I think it would tempt us to split source files into separate modules
> just because that way we can use a specific feature in one of them because
> it'll get compiled (and the other half is needed for bootstrapping). That
> would be bad design.

Possibly. In the case of fused types, the code of the fused node is
nearly 800 lines, which is probably good to separate from the other,
typically smaller nodes, especially considering it's kind of a
specific feature. In my case, and I wouldn't mind limiting the
functionality until further discussion to that case only, using the
with statement really helps keeping track of blocks, and the resulting
code is much more readable than it would otherwise be.

> OTOH, it might be worth taking a general look at the
> code base to see what's really required for bootstrapping, or maybe for
> compiling pure Python code in general. Factoring that out, and thus
> separating the Python compiler from the Cython specific language features
> might (might!) be an actual improvement. (Then again, there's .pxd
> overriding and the "cython" module, which add Cython features back in, and
> those two make Cython much more attactive...)
>
> I also started to dislike pyximport because it's way outdated, fragile,
> complicated and its features are overly hacked together (and I'm not
> entirely innocent in that regard). I would love to see a rewrite that also
> supports compiling packages properly. Not a GSoC by itself, but it's
> certainly a worthy project. What about a project that aims for separating
> out a Python compiler and rewriting pyximport as a jitty frontend for it?
> Maybe not the greatest use case ever, but a fun project, I'd say.

Yeah, the code is pretty terrible, but it seems to work.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel