[Cython] Utilities, cython.h, libcython

[Stefan Behnel]

mark florisson, 05.10.2011 15:53:
> On 5 October 2011 08:16, Stefan Behnel wrote:
>> mark florisson, 04.10.2011 23:19:
>>> Another issue is that Cython compile time is increasing with the
>>> addition of control flow and cython utilities. If you use fused types
>>> you're also going to combinatorially add more compile time.
>>
>> I don't see that locally - a compiled Cython is hugely fast for me. In
>> comparison, the C compiler literally takes ages to compile the result. An
>> external shared library may or may not help with both - in particular, it is
>> not clear to me what makes the C compiler slow. If the compile time is
>> dominated by the number of inlined functions (which is not unlikely), a
>> shared library + header file will not make a difference.
>
> Have you tried with the memoryviews merged?

No. I didn't expect the difference to be quite that large.


> e.g. if I have this code:
>
> from libc.stdlib cimport malloc
> cdef int[:] slice =<int[:10]>  <int *>  malloc(sizeof(int) * 10)
>
> [0] [14:45] ~  ➤ time cython test.pyx
> cython test.pyx  2.61s user 0.08s system 99% cpu 2.695 total
> [0] [14:45] ~  ➤ time zsh compile
> zsh compile  1.88s user 0.06s system 99% cpu 1.946 total
>
> where 'compile' is the script that invoked the same gcc command
> distutils uses.  As you can see it took more than 2.5 seconds to
> compile this code (simply because the memoryview utilities get
> included).

Ok, that hints at serious performance problems. Could you profile it to see 
where the issues are? Is it more that the code is loaded from an external 
file? Or the fact that more utility code is parsed than necessary?

It's certainly not obvious why the inclusion of static code, even from an 
external file, should make any difference.

That being said, it's not we were lacking the infrastructure for making 
Python code run faster ...


>>> I'm sure
>>> this came up earlier, but I really think we should have a libcython
>>> and a cython.h. libcython (a shared library) should contain any common
>>> Cython-specific code not meant to be inlined, and cython.h any types,
>>> macros and inline functions etc.
>>
>> This has a couple of implications though. In order to support this on the
>> user side, we have to build one shared library per installed package in
>> order to avoid any Cython versioning issues. Just installing a versioned
>> "libcython_x.y.z.so" globally isn't enough, especially during development,
>> but also at deployment time. Different packages may use different CFLAGS or
>> Cython options, which may have an impact on the result. Encoding all
>> possible factors in the file name will be cumbersome and may mean that we
>> still end up with a number of installed Cython libraries that correlates
>> with the number of installed Cython based packages.
>
> Hm, I think the CFLAGS are important so long as they are compatible
> with Python. When the user compiles a Cython extension module with
> extra CFLAGS, this doesn't affect libpython. Similarly, the Cython
> utilities are really not the user's responsibility, so libcython
> doesn't need to be compiled with the same flags as the extension
> module. If still wanted, the user could either recompile python with
> different CFLAGS (which means libcython will get those as well), or
> not use libcython at all. CFLAGS should really only pertain to user
> code, not to the Cython library, which the user shouldn't be concerned
> about.

Well, it's either the user or the OS distribution that installs (and 
potentially builds) the libraries. That already makes it two responsible 
entities for many systems that have to agree on what gets installed in what 
way. I'm just saying, don't underestimate the details in world wide 
deployments.

Stefan