[Cython] dynamically compile and import pure python modules

[mark florisson]

On 29 April 2012 05:50, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.04.2012 21:55:
>> On 28 April 2012 19:39, Stefan Behnel wrote:
>>> mark florisson, 28.04.2012 20:18:
>>>> On 28 April 2012 18:55, Stefan Behnel wrote:
>>>>> mark florisson, 27.04.2012 22:38:
>>>>>> On 27 April 2012 21:16, Stefan Behnel wrote:
>>>>>>> What about this deal: we remove the hard bootstrap dependency on the fused
>>>>>>> types code (and maybe other Cython specific features) and require its
>>>>>>> compilation at install time in Py2.4 (and maybe even 2.5). That would allow
>>>>>>> us to use newer Python syntax (and even Cython supported syntax) there
>>>>>>> (except for fused types, obviously). Failure to compile the module in
>>>>>>> Python 2.4/5 at install time would then abort the installation. Bad luck
>>>>>>> for the user, but easy to fix by installing a newer Python version.
>>>>>>>
>>>>>>> That would give us the advantage of not needing to pollute user home
>>>>>>> directories with shared libraries at runtime (which I would consider a very
>>>>>>> annoying property).
>>>>>>
>>>>>> I think it's fine to require compiling in the installed case (or will
>>>>>> that be a problem for some package managers?). In the non-installed
>>>>>> case with python versions smaller than needed, would you prefer a
>>>>>> pyximport or an error message telling you to install Cython? Because
>>>>>> for development you really don't want to install every time.
>>>>>
>>>>> I think it's fine to require at least Python 2.6 for Cython core
>>>>> development. Just the installation (basically, what we test in Jenkins
>>>>> anyway) should work in Py2.4 and shouldn't require any rebuilds at runtime.
>>>>
>>>> Well, sometimes stuff works in say, 2.7 but fails in 2.4. In that case
>>>> you really have to test with the failing python versions, which means
>>>> you'd have to reinstall every time you want to try the tests again
>>>> (this is handled automatically for py3k, which runs the 2to3 tool).
>>>
>>> The number of times I recently ran tests in Py2.4 myself is really not
>>> worth mentioning. Most of the time, when something fails there, the error I
>>> get in Jenkins is so obvious that I just commit an untested fix for it.
>>
>> In my experience that still fails quite often, you may still forget
>> some test or accidentally add some whitespace, and then you're going
>> to build everything on Jenkins only to realize one hour later that
>> something is still broken. Maybe it's because of the buffer
>> differences and because 2.4 is the first thing that runs on Jenkins,
>> but I test quite often in 2.4.
>
> That may be the reason.
>
> Still, how much overhead is it really to run "setup.py build_ext -i"?
> Especially compared to getting the exact same compilation overhead with an
> import time triggered rebuild?
>

The compilation overhead is irrelevant, the important difference is
that one is *automatic* and hence preferable. Knowing myself, I will
forget that module X, Y and Z need to be compiled, which means when I
change the code I don't see the changes take effect, because I forgot
to rebuild the modules. After a while I will start to question my
sanity and insert a print statement at module level, only to release
that I had to rebuild after every single change.

>>> I think it's really acceptable to require a run of "setup.py build_ext -i"
>>> for local developer testing in Py2.4.
>>>
>>>> I'm also sure many users just clone from git, add the directory to
>>>> PYTHONPATH and work from there.
>>>
>>> I'm sure there are close to no users who try to do that with Py2.4 these
>>> days. Maybe there are some who do it with Py2.5, but we are not currently
>>> considering to break a plain Python run there, AFAICT.
>>>
>>> I think the normal way users employ Cython is after a proper installation.
>>>
>>>> So I guess what I'm saying is, it's fine to mandate compilation at
>>>> compile time (don't allow flags to disable compilation), and (for me),
>>>> pyximport is totally fine, but Cython must be workable (all
>>>> functionality), without needing to install or build, in all versions.
>>>
>>> Workable, ok. But if fused types are only available in a compiled installed
>>> version under Python 2.4, that's maybe not optimal but certainly
>>> acceptable. Users of Py2.4 should be used to suffering anyway.
>>
>> That's a really confusing statement. If they are used to suffering,
>> they why can't they bear a runtime-compiled module if they didn't
>> install? :) If a user installs she will have at least one compiled
>> module in the system, but if she doesn't install, she will also have
>> one compiled module.
>>
>> I'm kind of wondering though, if this is really such a big problem,
>> then it means we can also never cache any user JIT-compiled code (like
>> e.g. OpenCL)?
>
> That's a different situation because it's the user's decision to use a
> (caching) JIT compiler in the first place. We don't enforce that.
>
> Basically, what I'm saying is: why do something complicated at runtime when
> we can just do everything normally at install time and be done? Once Cython
> is installed, it should just work, without needing to rebuild parts of
> itself. I'm convinced that that will make it a lot more acceptable for
> package maintainers, sys admins and users.
>
> Keep in mind that the installation under PyPy doesn't compile anything and
> just installs the plain Python modules. Enforcing compilation in Py2.4 is
> just saying that sys admins of very old systems (which we still want to
> support, at least in the generated C code) will no longer get the benefit
> of the pure Python installation feature. It doesn't impact the users at
> all, that's a feature.
>
>
>>>> That means either not using the with statement, or compiling with
>>>> pyximport in certain versions in certain situations only (i.e., only
>>>> in incompatible python version in case the user neither built nor
>>>> installed Cython). I don't think that's a problem, if people don't
>>>> like to have those shared library modules (will they even notice?) in
>>>> their user directories, they can install or build Cython.
>>>
>>> Why require the use of pyximport at runtime when we can do everything
>>> during installation? I really don't see an advantage.
>>
>> We will do everything during installation, but not mandate an installation.
>
> I think we should in the case at hand.
>
>
>>>> Finally, my attachment to the with statement here is mostly to the
>>>> aesthetics of the resulting code, rewriting my pull request is not so
>>>> much work, so we can leave that out of consideration here.
>>>
>>> If it's not too much work, that would obviously make things go a lot smoother.
>>
>> I'm not sure why we're always making such a fuss over the little
>> things, but I suppose I'd prefer mandating a compile in 2.4 over not
>> having the with statement.
>
> Ok.
>
>
>> Unless you want to mandate installing in
>> every version, this still means we can't write any parts of the
>> compiler in actual Cython code.
>
> We're doing just that for Py2.4 now by starting to use illegal syntax.
>
> However, since you're changing topics here already, do you actually see any
> advantage in starting to use non-Python code? I don't think that would be a
> good idea at all. I think we've gotten along quite happily with plain
> Python code for the compiler itself so far and I don't see an interest in
> changing that. In particular, I can't see the compiler being in need of any
> of its own non-Python language features.

I don't currently either, although we do compile the compiler for
speed. So I can see use cases where pxd overlays  (which are imho one
of the worst features of Cython as they suddenly can turn your classes
into cdef classes without you being aware when working in the .py
file) fall short for more time-consuming parts of the compiler.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Anyway, we clearly don't agree, so I suppose I'll change my code as I
think installing is worse than writing 2.4-compatible code, and hope
to retain the readability.