[Cython] Recent bugs in generators

[Stefan Behnel]

Vitja Makarov, 20.04.2011 11:50:
> 2011/4/20 Stefan Behnel:
>> Vitja Makarov, 20.04.2011 10:26:
>>> 2011/4/18 Stefan Behnel:
>>>> generators have their own lifetime frame in CPython, and
>>>> exceptions don't leak from that. So, whenever it's the generator (or code
>>>> called by it) that raises an exception, that must be kept local to the
>>>> generator.
>>>
>>> Please review:
>>>
>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>
>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>> before resume switch and before return from yield. It swaps generator
>>> exception state with thread local.
>>
>> Looks good to me. I assume this fixes the problem? Then please push it into
>> mainline.
>
> old pull request is still there ;)
>
> https://github.com/cython/cython/pull/25
>
> Does __Pyx_ExceptionReset() steal references to args, so they should
> not be decrefed later?

Hmm, good call. The refcounting looks correct over the two function calls, 
but it would be nicer if it could be avoided instead. We are really just 
swapping around pointers here, no refcounting is needed.

I think it's worth using a dedicated utility function for this.

Oh, and one more thing: what should be done when exiting the generator 
normally? Would that just raise GeneratorExit anyway, so that we can 
swallow any original outer exception? I think there might be a problem with 
Python 3, where the GeneratorExit should be raised in the context of the 
original exception. Worth testing how CPython behaves here...

Stefan