[Cython] Utilities, cython.h, libcython

[Vitja Makarov]

2011/10/7 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/10/7 mark florisson <markflorisson88 at gmail.com>:
>> On 6 October 2011 21:56, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2011/10/6 mark florisson <markflorisson88 at gmail.com>:
>>>> On 6 October 2011 07:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>> mark florisson, 05.10.2011 15:53:
>>>>>>
>>>>>> On 5 October 2011 08:16, Stefan Behnel wrote:
>>>>>>>
>>>>>>> mark florisson, 04.10.2011 23:19:
>>>>>>>>
>>>>>>>> Another issue is that Cython compile time is increasing with the
>>>>>>>> addition of control flow and cython utilities. If you use fused types
>>>>>>>> you're also going to combinatorially add more compile time.
>>>>>>>
>>>>>>> I don't see that locally - a compiled Cython is hugely fast for me. In
>>>>>>> comparison, the C compiler literally takes ages to compile the result. An
>>>>>>> external shared library may or may not help with both - in particular, it
>>>>>>> is
>>>>>>> not clear to me what makes the C compiler slow. If the compile time is
>>>>>>> dominated by the number of inlined functions (which is not unlikely), a
>>>>>>> shared library + header file will not make a difference.
>>>>>>
>>>>>> Have you tried with the memoryviews merged?
>>>>>
>>>>> No. I didn't expect the difference to be quite that large.
>>>>>
>>>>>
>>>>>> e.g. if I have this code:
>>>>>>
>>>>>> from libc.stdlib cimport malloc
>>>>>> cdef int[:] slice =<int[:10]>  <int *>  malloc(sizeof(int) * 10)
>>>>>>
>>>>>> [0] [14:45] ~  ➤ time cython test.pyx
>>>>>> cython test.pyx  2.61s user 0.08s system 99% cpu 2.695 total
>>>>>> [0] [14:45] ~  ➤ time zsh compile
>>>>>> zsh compile  1.88s user 0.06s system 99% cpu 1.946 total
>>>>>>
>>>>>> where 'compile' is the script that invoked the same gcc command
>>>>>> distutils uses.  As you can see it took more than 2.5 seconds to
>>>>>> compile this code (simply because the memoryview utilities get
>>>>>> included).
>>>>>
>>>>> Ok, that hints at serious performance problems. Could you profile it to see
>>>>> where the issues are? Is it more that the code is loaded from an external
>>>>> file? Or the fact that more utility code is parsed than necessary?
>>>>
>>>> I haven't profiled it yet (I'll do that), but I'm fairly sure it's the
>>>> parsing of Cython utility files (not the loading). Maybe Tempita also
>>>> adds to the overhead, I'll find out.
>>>>
>>>
>>> Compiling this regex gives 5ms instead of 10ms on my machine
>>>
>>> https://github.com/cython/cython/blob/master/Cython/Compiler/Code.py#L85
>>>
>>> And on your example gives 3% speedup
>>>
>>
>> Sorry, which code gets you 10ms? Also, is this about loading + regex
>> matching, or just about compiling the pattern?
>>
>
> I've added decorator for load_utilities_from_file that prints time for
> current call and total sum for this function and total gives 10ms.
>
> Btw that's not that much.
>
>

Here is small comparison on compiling urllib.py with cython:

((e8527c5...)) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
../cython.py urllib.py

real	0m1.699s
user	0m1.650s
sys	0m0.040s
(master) vitja at mchome:~/work/cython-vitek-git/zzz$ time python
../cython.py urllib.py

real	0m2.830s
user	0m2.790s
sys	0m0.030s


It's about 1.5 times slower.

-- 
vitja.