[Python-ideas] Yield-From: Finalization guarantees

[Jacob Holm]

Greg Ewing wrote:
> Jacob Holm wrote:
>> in most cases this will be code that is breaking the rule about
> > not catching KeyboardInterrupt and SystemExit.
>
> Not necessarily, it could be doing
>
>   except GeneratorExit:
>     return

I said *most* cases, not all.  I don't have any proof of this, just a 
gut feeling that the majority of generators that convert GeneratorExit 
to StopIteration do so because they are using a return in a finally clause.

>
>> If you use such a generator in a yield-from expression, you will get 
>> a RuntimeError('generator ignored GeneratorExit') on close, telling 
>> you that something is wrong.
>
> But it won't be at all clear *what* is wrong or what to
> do about it. The caller is making a perfectly ordinary
> yield-from call, and he's calling what looks to all the
> world like a perfectly well-behaved iterator. Where's
> the mistake?

If this was documented in the PEP, I would say the mistake was in using 
such a generator in yield-from that wasn't the final yield.  Note that 
it is perfectly ok to use such a generator in a yield-from as long as no 
outer generator yields afterwards.

>
> Remember that the generator being called may have been
> written by someone else. The caller may not know anything
> about its internals or be in a position to fix them if
> he did.

Right, that makes it harder to fix the source of the problem.

>
> > I think that getting a RuntimeError on close is sufficient indication
> > that such a generator should not be used in yield-from.
>
> But it's a perfectly valid generator by current standards.
> I don't want to declare some existing class of generators
> as being second-class citizens with respect to yield-from,
> especially based on some internal implementation detail
> unknowable to its caller.
>

I get that.

As I see it we have the following options, listed in my order of preference:

   1. Don't throw GeneratorExit to the subiterator but raise it in the
      outer generator, and don't explicitly call close.  This is the
      only version where sharing a subgenerator does not require special
      care.  It has the problem that it behaves differently in
      refcounting and non-refcounting implementations due to the
      implicit close that would happen after the yield-from in
      refcounting implementations.  It also breaks the inlining
      principle in the case of throw(GeneratorExit).
   2. Do throw GeneratorExit and don't try to reraise it.  This is the
      version that most closely follows the inlining principle.  It has
      the problem that generators that convert GeneratorExit to
      StopIteration can only be used in a yield-from if none of the
      outer generators do a yield afterwards.  Breaking this rule gives
      a RuntimeError('generator ignored GeneratorExit') on close.
   3. Do throw GeneratorExit to the subiterator, and explicitly reraise
      it if it was converted to a StopIteration.  It has the problem
      that it breaks the inlining principle for generators that convert
      GeneratorExit to StopIteration.
   4. Don't throw GeneratorExit to the subiterator, instead explicitly
      call close before raising it in the outer generator.  This is the
      behavior that #1 would have for non-shared generators in a
      refcounting implementation.  Same problem as #3 and hides the
      GeneratorExit from non-generators.

My guess is that your preference is more like 4, 3, 2, 1.  #3 is closest 
to what is in the current PEP, and is probably what it meant to say. 
(The PEP checks if the thrown exception was GeneratorExit, then does a 
bare raise instead of raising the thrown exception).

- Jacob