[Cython] some pull requests for 0.16?

[Stefan Behnel]

mark florisson, 20.03.2012 17:40:
> On 18 March 2012 11:58, Stefan Behnel wrote:
>> I put up two new pull requests on github:
>>
>> Implementation of PEP 380 (yield from):
>> https://github.com/cython/cython/pull/96
>>
>> Rewrite of dict iteration:
>> https://github.com/cython/cython/pull/95
>>
>> Given that the release of 0.16 has currently slowed down a bit, and given
>> that these are really nice features, could someone (and especially Mark, as
>> the responsible release manager) take a look at them and give an opinion
>> regarding their suitability for 0.16?
> 
> Sorry about the slowdown, my BT internet connection at home got closed
> off, so working on a release is rather hard, hopefully that will be
> fixed within the next few days. Anyway, during the sprints Dag worked
> on the numpy attributes accesses to generate non-deprecated numpy
> code, I'm not sure how far along that is and if we can get that in as
> well, or have it wait until 0.16.1.

I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
not too long from now. There will certainly be things to fix after the
release anyway.


> I worked on enhancing fused types runtime dispatching (e.g. select the
> right specialization at runtime from the argument types), especially
> on matching buffers. It's also a bit of a rewrite as the complexity
> grew quite a bit and ndarray and fused types were broken in subtle
> ways (I would do the pull request now if I hadn't left the code at
> home :).

I took a tiny look at that when I wrote a test case and was surprised to
see that it imports "sys" inside of the generated function. Looks like
there's a bit left to optimise.

I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
"generic functions" module. It uses type dispatching as well, and he wrote
about the ways he found to tune that. Might be an interesting read.


> Finally, there is a serious bug in the reversed(range()) optimization
> that Mike Graham discovered when working on the range() optimization
> with a runtime step, which is basically wrong for any step that is not
> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
> actually has to figure out what the actual last value was in the range
> and use that as the start instead). I personally consider this a
> blocker, so I'll try contacting Mike and see if he is still
> (interested in) working on that.

Interesting. Yes, I can well imagine that being wrong. These things are
exceedingly tricky.

However, it's easy to disable the optimisation for a specific case - likely
easier than fixing it.

Also, I don't consider this a blocker. It's not like it works in the
current release, so it's not a regression compared to that.


> As for the two pull requests, both are quite large, but the dict
> iteration rewrite is more like an enhancement whereas the 'yield from'
> is really a new (somewhat big) feature. I would personally prefer to
> merge only enhancements and bug fixes, but not new features.

Funny. I would have proposed the opposite, because I consider the "yield
from" implementation (as a new language feature) safer than the dict
iteration (which impacts existing code). As I said, I wouldn't mind
postponing both to 0.16.1, but I think it would be better to get them out
and have people use them, so that we can also fix the bugs earlier.


> If you
> feel confident enough that the dict iteration pull request is ready to
> be merged, then please do so.

I'll take another look over it and decide. Thanks.


> We can do an 0.16.1 release with the
> 'yield from', the 'lnotab' line number mapping from code objects to
> Cython source

Note that that's not useful yet - it requires changes to the code that
*uses* the code objects as well, to a) actually reuse the code objects (iff
C line numbers are disabled in traces) and b) write the proper relative
line into the byte code offset field. I just put it up to have a place
where we can discuss further changes.


> and probably the boundscheck and wraparound
> optimizations I started working on (and other things if and when they
> come up). We can release 0.16.1 somewhat shortly after releasing 0.16.
> I think a shorter release cycle with less changes would be beneficial
> and give us swifter feedback about bugs, missing features, wanted
> features, etc.

+1


> So I propose that when we merged dict iteration, fused types
> enhancements and the range bug fix, we do another beta release. We (I)
> also need to look into the test failures on Gentoo.

Whatever you choose.

In case the PyPy developers manage to get one change in that I'd like to
see on their side (two new C-API functions for exception handling), I'll
also push some more Cython changes based on it that I have lying around.
That would hugely improve the compatibility for both.

In general, I started using the "CYTHON_COMPILING_IN_..." #defines wherever
they made sense, and specifically where we rely on CPython internals and
also in some places where we use borrowed references, especially long-lived
ones. PyPy can't handle those very well (and certainly not efficiently) due
to its garbage collector and its emulated ref-counting. I can well imagine
that that's similar for IronPython.

Stefan