[Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Stefan Behnel]

Nathaniel Smith, 09.04.2013 15:46:
> On Tue, Apr 9, 2013 at 2:11 PM, Stefan Behnel wrote:
>> Nathaniel Smith, 09.04.2013 15:00:
>>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote:
>>>> Nathaniel Smith, 09.04.2013 14:25:
>>>> There's also the problem of dependency hell and getting rid of old modules
>>>> once they are no longer used on the user side. And also, how to get them
>>>> there in the first place. Having one package overwrite the files of
>>>> another during its installation is just asking for trouble.
>>>
>>> The system I described does not involve the addition of any new files to
>>> any package.
>>
>> I take it then that you were envisaging a separate "cython-runtime" package
>> on PyPI that Cython compiled modules would have to depend on?
> 
> No, I must have been unclear -- I envisage that each cython-compiled
> module continues to contain all the code it depends on (just like
> now). But, the first cython-compiled module that is imported tweaks
> sys.modules *as if* there were a separate cython-runtime package, and
> sticks the necessary pieces (like the memoryview TypeObject) into this
> module, and later modules import it from there.
> 
> There's no additional dependencies anywhere. You have to be careful to
> make sure you version the runtime module appropriately so that you
> don't end up with a module compiled with cython version X using the
> runtime injected by a module compiled with the incompatible cython
> version Y, but this is no harder than any other versioning problem.
> Just make sure to change the runtime module name whenever you break
> compatibility.

Ok, got it now. That solves the distribution problem, assuming that all
installed runtimes with a given version are equivalent. Basically, we'd
move the code out and then cimport the stuff back that we need, which would
then let the first import of a given runtime version insert it into
sys.modules.

I so can't wait seeing the surprise in the eyes of our users when they let
their IDE reorganise their import order and their code starts crashing. :]

Stefan