[Cython] dynamically compile and import pure python modules

[mark florisson]

On 28 April 2012 19:39, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.04.2012 20:18:
>> On 28 April 2012 18:55, Stefan Behnel wrote:
>>> mark florisson, 27.04.2012 22:38:
>>>> On 27 April 2012 21:16, Stefan Behnel wrote:
>>>>> What about this deal: we remove the hard bootstrap dependency on the fused
>>>>> types code (and maybe other Cython specific features) and require its
>>>>> compilation at install time in Py2.4 (and maybe even 2.5). That would allow
>>>>> us to use newer Python syntax (and even Cython supported syntax) there
>>>>> (except for fused types, obviously). Failure to compile the module in
>>>>> Python 2.4/5 at install time would then abort the installation. Bad luck
>>>>> for the user, but easy to fix by installing a newer Python version.
>>>>>
>>>>> That would give us the advantage of not needing to pollute user home
>>>>> directories with shared libraries at runtime (which I would consider a very
>>>>> annoying property).
>>>>
>>>> I think it's fine to require compiling in the installed case (or will
>>>> that be a problem for some package managers?). In the non-installed
>>>> case with python versions smaller than needed, would you prefer a
>>>> pyximport or an error message telling you to install Cython? Because
>>>> for development you really don't want to install every time.
>>>
>>> I think it's fine to require at least Python 2.6 for Cython core
>>> development. Just the installation (basically, what we test in Jenkins
>>> anyway) should work in Py2.4 and shouldn't require any rebuilds at runtime.
>>
>> Well, sometimes stuff works in say, 2.7 but fails in 2.4. In that case
>> you really have to test with the failing python versions, which means
>> you'd have to reinstall every time you want to try the tests again
>> (this is handled automatically for py3k, which runs the 2to3 tool).
>
> The number of times I recently ran tests in Py2.4 myself is really not
> worth mentioning. Most of the time, when something fails there, the error I
> get in Jenkins is so obvious that I just commit an untested fix for it.

In my experience that still fails quite often, you may still forget
some test or accidentally add some whitespace, and then you're going
to build everything on Jenkins only to realize one hour later that
something is still broken. Maybe it's because of the buffer
differences and because 2.4 is the first thing that runs on Jenkins,
but I test quite often in 2.4.

> I think it's really acceptable to require a run of "setup.py build_ext -i"
> for local developer testing in Py2.4.
>
>
>> I'm also sure many users just clone from git, add the directory to
>> PYTHONPATH and work from there.
>
> I'm sure there are close to no users who try to do that with Py2.4 these
> days. Maybe there are some who do it with Py2.5, but we are not currently
> considering to break a plain Python run there, AFAICT.
>
> I think the normal way users employ Cython is after a proper installation.
>
>
>> So I guess what I'm saying is, it's fine to mandate compilation at
>> compile time (don't allow flags to disable compilation), and (for me),
>> pyximport is totally fine, but Cython must be workable (all
>> functionality), without needing to install or build, in all versions.
>
> Workable, ok. But if fused types are only available in a compiled installed
> version under Python 2.4, that's maybe not optimal but certainly
> acceptable. Users of Py2.4 should be used to suffering anyway.
>

That's a really confusing statement. If they are used to suffering,
they why can't they bear a runtime-compiled module if they didn't
install? :) If a user installs she will have at least one compiled
module in the system, but if she doesn't install, she will also have
one compiled module.

I'm kind of wondering though, if this is really such a big problem,
then it means we can also never cache any user JIT-compiled code (like
e.g. OpenCL)?

>> That means either not using the with statement, or compiling with
>> pyximport in certain versions in certain situations only (i.e., only
>> in incompatible python version in case the user neither built nor
>> installed Cython). I don't think that's a problem, if people don't
>> like to have those shared library modules (will they even notice?) in
>> their user directories, they can install or build Cython.
>
> Why require the use of pyximport at runtime when we can do everything
> during installation? I really don't see an advantage.
>

We will do everything during installation, but not mandate an installation.

>> Finally, my attachment to the with statement here is mostly to the
>> aesthetics of the resulting code, rewriting my pull request is not so
>> much work, so we can leave that out of consideration here.
>
> If it's not too much work, that would obviously make things go a lot smoother.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I'm not sure why we're always making such a fuss over the little
things, but I suppose I'd prefer mandating a compile in 2.4 over not
having the with statement. Unless you want to mandate installing in
every version, this still means we can't write any parts of the
compiler in actual Cython code.