[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

[Jacob Holm]

Guido van Rossum wrote:
> On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <jh at improva.dk> wrote:
>   
>> The argument that we have no value to send before we have yielded is wrong.
>> The generator containing the "yield-from" could easily have a value to send
>> (or throw), and if iter(EXPR) returns a coroutine or a non-generator it
>> could easily be ready to accept it.  That is the idea behind my attempted
>> fixes to the @coroutine issue.
>>     
>
> I think it's simpler to refrain from yield-from in that case and spell
> it out. If the value to send doesn't come from outside the outer
> generator, yield-from is not the solution.
>   

But it *could* come from outside.  If it is a coroutine calling another 
coroutine, it could have done any number of yields first, the last of 
which would return the value to be sent to the inner one.

It bothers me a lot if you cannot use yield-from with coroutines, 
because most other uses I can see are just as easily written as 
for-loops.  I'll think a bit more about this.

>> This is where the fun begins.  In an earlier thread we concluded that if the
>> thrown exception is a StopIteration and the *same* StopIteration instance
>> escapes the throw() call, it should be reraised rather than caught and
>> turned into a RETVAL.  The reasoning was the following example:
>>
>> def inner():
>>     for i in xrange(10):
>>         yield i
>>
>> def outer():
>>     yield from inner()
>>     print "if StopIteration is thrown in we shouldn't get here"
>>
>> Which we wanted to be equivalent to:
>>
>> def outer():
>>     for i in xrange(10):
>>         yield i
>>     print "if StopIteration is thrown in we shouldn't get here"
>>
>> The same argument goes for ReturnFromGenerator, so the expansion at this
>> stage should be more like:
>>     
> [snip]
>
> This example and reasoning are invalid. You shouldn't be throwing
> StopIteration (or ReturnFromGenerator) *into* a generator. That's
> something that should only come *out*.
>   

I am not claiming that you *should* be throwing StopIteration to a 
generator, just that there is nothing that prevents you from doing it, 
so we need to consider what should happen if you do.  The above 
reasoning based on the refactoring principle lead to one choice, which I 
happen to like.  If you only focus on getting the expansion in the PEP 
as simple as possible you will probably make another choice.

Note that if you don't handle StopIteration this way but just treat it 
as a normal StopIteration you open up for interesting ways to abuse 
yield-from, exactly by throwing StopIteration.   In particular, if you 
use an iterator without throw or close methods you can break out of the 
innermost yield-from and even set the value to be returned.

I don't mind either way.  I just thought I would mention this in case 
you missed it.


[snip my examples where the return value was swallowed]
>> I have previously suggested attaching the return value to the (re)raised
>> GeneratorExit, and/or saving the return value on the generator and making
>> close return the value each time it is called.  We could also choose to
>> define this as broken behavior and raise a RuntimeError, although it seems a
>> bit strange to have yield-from treat it as an error when close doesn't.
>> Silently having the yield-from construct swallow the returned value is my
>> least favored option.
>>     
>
> Attaching it to the GeneratorExit is just plain wrong -- this is an
> exception you throw *in*, not something that is thrown out (except
> when it bounces back).
>   

I expanded a little bit on the idea in my reply to Greg that must have 
crossed your mail.  Listed a number of possible solutions that had been 
discussed in my order of preference.  I don't see a problem in having 
the language construct "yield-from" raise GeneratorExit with a value as 
a result of GeneratorExit.

> One solution is not to use yield-from but write it out using yield and
> send (just like the full expansion, but you can probably drop most of
> the complexity for any particular example).
>
> Another solution is not to use close() and GeneratorExit but some
> application-specific exception to signal the end.
>   

That doesn't really solve the issue of what should happen if you write 
such code.  What bothers me most is that the return value is silently 
swallowed.

> But perhaps it would be okay to change the GeneratorExit handler in
> the expansion so that it passes through the return value with a
> StopIteration exception:
>
> rv = it.close()
> if rv is None:
>   raise StopIteration(rv)    # Or ReturnFromGenerator(rv)
> else:
>   raise
>
> Alternatively, simpler:
>
> it.throw(GeneratorExit)
> # We only get here if it yielded a value
> raise RuntimeError(...)
>
> (Though this isn't exactly if we were to use duck typing.)
>
> We could then write the first version of outer() like this:
>
> def outer():
>   try:
>     yield from inner()
>   except StopIteration as e:
>     ...access return value as e.value...
>
> and I think the second (trivial) outer() will return inner()'s return
> value just fine, since it just passes through as a StopIteration
> value.
>   

I don't mind catching an exception to get the value in this case.  I 
just think it should be GeneratorExit i should catch.

This is related to the question of what should happen if you throw 
StopIteration.   If you don't special-case StopIteration in throw, using 
StopIteration for this is fine.

> FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a
> nuisance, and that it's actually fine to allow "return value" inside a
> generator to mean "raise StopIteration(value)" (well not quite at that
> point in the code but once we are about to clean up the frame). Maybe
> I've overstated the case for preventing beginners' mistakes. After all
> they'll notice that their generator returns prematurely when they
> include any kind of return value. Also if the StopIteration ends being
> printed as a traceback the value will be printed, which is the kind of
> hint newbies love.
>   

Ok, then I will stop worrying about ReturnFromGenerator until it is 
brought up again.

Best regards
- Jacob