[Cython] PEP 380 approved: syntax for generator delegation

[Stefan Behnel]

Stefan Behnel, 05.03.2012 15:26:
> Vitja Makarov, 27.06.2011 22:31:
>> 2011/6/27 Stefan Behnel:
>>> PEP 380 ("yield from") recently went through the approval gate.
>>>
>>> http://www.python.org/dev/peps/pep-0380/
>>>
>>> The implementation in CPython isn't rounded up yet and lots of tests are
>>> missing from the test suite. But it's agreed that it'll make it into 3.3,
>>> and we should try to support it as well. It's a really nice feature.
>>
>> Wow! Interesting thing.
>> I think that could be implemented but I think we should try to
>> optimize cython generators first.
> 
> Coming back to this, the way it's now implemented in CPython is using a new
> field "f_yieldfrom" in the frame that the coroutine implementation
> automatically delegates to. Cython's coroutine implementation could do the
> same thing, and even copy much of the original implementation, by adding a
> new field to the coroutine class and updating the send(), throw() and
> close() methods accordingly. The interesting changes are those in
> Objects/genobject.c and Python/ceval.c in this patch:
> 
> http://bugs.python.org/file24214/f8349cbc1b26.diff
> 
> That makes it look like most of the work is done for us already.

Just to report some progress here: I got the implementation almost done and
it even seems to be quite portable (StopIteration propagates a return value
now, but that also works in older Pythons so far). There are a couple of
bugs that I'll have to look into, but I should have something ready in a
couple of days (or nights...). The CPython tests proved to be quite
helpful, but I'll need the usual Cython specific tests as well.

If anyone wants to write up some code that you think should work with
"yield from" in Cython, I'll happily take it. One additional feature that
would be really cool is support for delegating to C arrays and C++
iterators, so that you could wrap C++ iterators with simple values in
straight code like this:

  def gen():
     yield from some_cpp_iterable

Anyway, that's pretty far off for now as it requires specialised code for
the different C/C++ source types.

Stefan