[Cython] Shared Cython runtime

[Stefan Behnel]

Robert Bradshaw, 09.04.2013 20:16:
> On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Stefan Behnel, 09.04.2013 17:12:
>>> mark florisson, 09.04.2013 16:51:
>>>> On 9 April 2013 15:47, Stefan Behnel wrote:
>>>>
>>>>> mark florisson, 09.04.2013 16:32:
>>>>>> On 9 April 2013 14:55, Nikita Nemkin wrote:
>>>>>>> One alternative for code reuse in large Cython projects
>>>>>>> could be packaging multiple modules into one shared library.
>>>>>>
>>>>>> We have 'include'! :) Seriously though, that wouldn't work well with the
>>>>>> import mechanism, and probably not for C compile time either.
>>>>>
>>>>> CPython has support for it. It's actually quite easy, you basically just
>>>>> register all module init functions explicitly at init time and let CPython
>>>>> do the rest.
>>>>>
>>>>> So, compiling a package into one big Cython module with shared internal
>>>>> types and utility code should be quite simple and would IMHO be a very cool
>>>>> feature.
>>>>
>>>> I was unaware of that, it'd be a nice feature then (though still not imho
>>>> the right solution to a runtime system). How does this work? Feel free to
>>>> point me at some documentation.
>>>
>>> It's the inittab mechanism, as described starting here:
>>>
>>> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab
>>
>> Oh, and even simpler, you can just stick multiple Cython modules with their
>> module init functions into one big shared library and have the "main" init
>> function of that module call the others. That will stick them into
>> sys.modules as a side effect, so it's not really different from going
>> through Python's import machinery for each module separately (just way more
>> efficient - at least if you really need all of the modules...).
> 
> Yep. We could even create stub .so files for the included ones that
> did nothing but import the big one to avoid having to worry about
> importing things in the right order.
> 
> The big question is what API to specify that things should be linked
> together. An option to cythonize(...) could be used, but for some
> projects (e.g. Sage) one would want more control over which modules
> get bundled together rather than all-or-nothing.

No-one forces you to use a plain cythonize("**/*.pyx"). Just be more
specific about what files to include in each pattern that you pass to in.
And you can always call cythonize() more than once if you need to. Once for
each meta-module should usually be acceptable, given that the bundled
source modules would tend to have a common configuration anyway.

Also, I'd really encourage users to merge things at a package level, rather
than compiling half of a package into one module and then having separate
Python or Cython modules lying around next to it. It keeps projects easier
to understand when you see their source directory layout. Shouldn't be the
only option, though.


>> There may be glitches with stuff like "__file__" and friends, but that
>> should not be any worse than the current way of workings.
> 
> This can probably be manipulated by Cython, though it's unclear what
> the best value would be.

I added an option to pyximport that simply uses the file path of the source
file. It makes sense there, but less so in other contexts. I actually think
that it's impossible to generally fix in the current infrastructure.

http://bugs.python.org/issue13429

Stefan