[Cython] PyCon-DE wrap-up by Kay Hayen

[Robert Bradshaw]

Thanks for the update and link. Sounds like PyCon-DE went well.

On Mon, Oct 10, 2011 at 1:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 09.10.2011 19:57:
>>
>> On 9 October 2011 18:35, Stefan Behnel wrote:
>>>
>>> One of the impressions I took out of the technical discussions with Kay
>>> is
>>> that there isn't really a good reason why Cython should refuse to
>>> duplicate
>>> some of the inner mechanics of CPython for optimisation purposes. Nuitka
>>> appears to be somewhat more aggressive here, partly because Kay doesn't
>>> currently care all that much about portability (e.g. to Python 3).
>>
>> Interesting. What kind of (significant) optimizations could be made by
>> duplicating code? Do you want to duplicate entire functions or do you
>> want to inline parts of those?
>
> I was mainly referring to things like direct access to type/object struct
> fields and little things like that. They can make a difference especially in
> loops, compared to calling into a generic C-API function. For example, we
> could have our own interned implementation of PyDict_Next(). I'm not very
> impressed by the performance of that C-API function - repeated calls to
> GetItem can be faster than looping over a dict with PyDict_Next()!
>
> That being said, I wasn't referring to any specific changes. It was more of
> a general remark about the invisible line that we currently draw in Cython.

CPython, especially the internals, is a slow enough moving target that
I'm not too concerned about reaching into the internals if there is a
clear benefit. If we're flexible enough to support 2.x and 3.x, I
think we can handle 3.(x+1) when it comes.

>> I actually think we should not get too tied to CPython, e.g. what if
>> PyPy gets a CPython compatible API
>
> It already implements a part of the C-API:
>
> http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html
>
> However, if we really want to support it at that level, there's likely more
> to do than just removing low-level optimisations. And that would take the
> normal route that we always use: macros and conditionally compiled inline
> functions. The mere fact that we try to support different targets doesn't
> mean that we should stop optimising for specific targets.

+1

> The same is true
> for different versions of CPython, where we often use better optimisations
> in newer releases, without sacrificing backwards compatibility.
>
> Personally, I think that supporting PyPy at the Python level is a lot more
> interesting, although it may be easier to get it working at the cpyext
> level.
>
>
>> or possibly a subset like PEP 384?
>
> That's currently not very interesting since there are basically no C
> extensions around (generated or hand written) that restrict themselves to
> that API. Supporting it in Cython would mean that we have to rewrite huge
> parts of the generated C code. It's not even clear to me yet that we *can*
> implement all of Cython's features based on PEP 384. For example, fast
> indexing into lists and tuples is basically a no-no in the restricted C-API.
> There are tons of rather unexpected restrictions like this.

I agree, PEP 384 is a nice idea, but it seems to be a rather lot of
work for an unclear/small benefit (compared to other stuff we could be
doing.)

- Robert