[Cython] Utilities, cython.h, libcython

[Vitja Makarov]

2011/10/7 mark florisson <markflorisson88 at gmail.com>:
> On 6 October 2011 21:56, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/10/6 mark florisson <markflorisson88 at gmail.com>:
>>> On 6 October 2011 07:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 05.10.2011 15:53:
>>>>>
>>>>> On 5 October 2011 08:16, Stefan Behnel wrote:
>>>>>>
>>>>>> mark florisson, 04.10.2011 23:19:
>>>>>>>
>>>>>>> Another issue is that Cython compile time is increasing with the
>>>>>>> addition of control flow and cython utilities. If you use fused types
>>>>>>> you're also going to combinatorially add more compile time.
>>>>>>
>>>>>> I don't see that locally - a compiled Cython is hugely fast for me. In
>>>>>> comparison, the C compiler literally takes ages to compile the result. An
>>>>>> external shared library may or may not help with both - in particular, it
>>>>>> is
>>>>>> not clear to me what makes the C compiler slow. If the compile time is
>>>>>> dominated by the number of inlined functions (which is not unlikely), a
>>>>>> shared library + header file will not make a difference.
>>>>>
>>>>> Have you tried with the memoryviews merged?
>>>>
>>>> No. I didn't expect the difference to be quite that large.
>>>>
>>>>
>>>>> e.g. if I have this code:
>>>>>
>>>>> from libc.stdlib cimport malloc
>>>>> cdef int[:] slice =<int[:10]>  <int *>  malloc(sizeof(int) * 10)
>>>>>
>>>>> [0] [14:45] ~  ➤ time cython test.pyx
>>>>> cython test.pyx  2.61s user 0.08s system 99% cpu 2.695 total
>>>>> [0] [14:45] ~  ➤ time zsh compile
>>>>> zsh compile  1.88s user 0.06s system 99% cpu 1.946 total
>>>>>
>>>>> where 'compile' is the script that invoked the same gcc command
>>>>> distutils uses.  As you can see it took more than 2.5 seconds to
>>>>> compile this code (simply because the memoryview utilities get
>>>>> included).
>>>>
>>>> Ok, that hints at serious performance problems. Could you profile it to see
>>>> where the issues are? Is it more that the code is loaded from an external
>>>> file? Or the fact that more utility code is parsed than necessary?
>>>
>>> I haven't profiled it yet (I'll do that), but I'm fairly sure it's the
>>> parsing of Cython utility files (not the loading). Maybe Tempita also
>>> adds to the overhead, I'll find out.
>>>
>>
>> Compiling this regex gives 5ms instead of 10ms on my machine
>>
>> https://github.com/cython/cython/blob/master/Cython/Compiler/Code.py#L85
>>
>> And on your example gives 3% speedup
>>
>
> Sorry, which code gets you 10ms? Also, is this about loading + regex
> matching, or just about compiling the pattern?
>

I've added decorator for load_utilities_from_file that prints time for
current call and total sum for this function and total gives 10ms.

Btw that's not that much.


> In any case, libcython would solve these issues. Profiling will still
> be useful though.
>
>>>> It's certainly not obvious why the inclusion of static code, even from an
>>>> external file, should make any difference.
>>>>
>>>> That being said, it's not we were lacking the infrastructure for making
>>>> Python code run faster ...
>>>>
>>>
>>> Heh, indeed. In this case I think caching will solve all our problems.
>>>
>>>>>>> I'm sure
>>>>>>> this came up earlier, but I really think we should have a libcython
>>>>>>> and a cython.h. libcython (a shared library) should contain any common
>>>>>>> Cython-specific code not meant to be inlined, and cython.h any types,
>>>>>>> macros and inline functions etc.
>>>>>>
>>>>>> This has a couple of implications though. In order to support this on the
>>>>>> user side, we have to build one shared library per installed package in
>>>>>> order to avoid any Cython versioning issues. Just installing a versioned
>>>>>> "libcython_x.y.z.so" globally isn't enough, especially during
>>>>>> development,
>>>>>> but also at deployment time. Different packages may use different CFLAGS
>>>>>> or
>>>>>> Cython options, which may have an impact on the result. Encoding all
>>>>>> possible factors in the file name will be cumbersome and may mean that we
>>>>>> still end up with a number of installed Cython libraries that correlates
>>>>>> with the number of installed Cython based packages.
>>>>>
>>>>> Hm, I think the CFLAGS are important so long as they are compatible
>>>>> with Python. When the user compiles a Cython extension module with
>>>>> extra CFLAGS, this doesn't affect libpython. Similarly, the Cython
>>>>> utilities are really not the user's responsibility, so libcython
>>>>> doesn't need to be compiled with the same flags as the extension
>>>>> module. If still wanted, the user could either recompile python with
>>>>> different CFLAGS (which means libcython will get those as well), or
>>>>> not use libcython at all. CFLAGS should really only pertain to user
>>>>> code, not to the Cython library, which the user shouldn't be concerned
>>>>> about.
>>>>
>>>> Well, it's either the user or the OS distribution that installs (and
>>>> potentially builds) the libraries. That already makes it two responsible
>>>> entities for many systems that have to agree on what gets installed in what
>>>> way. I'm just saying, don't underestimate the details in world wide
>>>> deployments.
>>>>
>>
>>
>>
>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>



-- 
vitja.