[Python-Dev] Generator details

[Tim Peters]

[Guido, sketches 112 ways to implement one-frame generators today <wink>]

I'm glad you're having fun too!  I won't reply in detail here; it's enough
for now to happily agree that adding a one-frame generator isn't much of a
stretch for the current implementation of the PVM.

> Loose end: what to do when there's a try/finally around a suspend?
> E.g.
>
> generator foo(l):
>     try:
>         for i in l:
>            suspend i+1
>     finally:
>         print "Done"
>
> The second translation variant would cause "Done" to be printed on
> each suspend *and* on the final return.  This is confusing (and in
> fact I think resuming the frame would be a problem since the return
> breaks down the try-finally blocks).

There are several things to be said about this:

+ A suspend really can't ever go thru today's normal "return" path, because
(among other things) that wipes out the frame's value stack!

	while (!EMPTY()) {
		v = POP();
		Py_XDECREF(v);
	}

A SUSPEND opcode would let it do what it needs to do without mixing that
into the current return path.  So my answer to:

> I'm not sure which is better; the version without call_generator()
> allows you to create your own generator without using the 'generator'
> and 'suspend' keywords, calling get_frame() explicitly.

is "both" <wink>:  get_frame() is beautifully clean, but it still needs
something like SUSPEND to keep everything straight.  Maybe this just amounts
to setting "why" to a new WHY_SUSPEND and sorting it all out after the eval
loop; OTOH, that code is pretty snaky already.

+ I *expect* the example code to print "Done" len(l)+1 times!  The generator
mechanics are the same as the current for/__getitem__ protocol in this
respect:  if you have N items to enumerate, the enumeration routine will get
called N+1 times, and that's life.  That is, the fact is that the generator
"gets to" execute code N+1 times, and the only reason your original example
seems surprising at first is that it doesn't happen to do anything (except
exit the "try" block) on the last of those times.  Change it to

generator foo(l):
    try:
        for i in l:
           suspend i+1
        cleanup()   # new line
    finally:
        print "Done"

and then you'd be surprised *not* to see "Done" printed len(l)+1 times.  So
I think the easiest thing is also the right thing in this case.

OTOH, the notion that the "finally" clause should get triggered at all the
first len(l) times is debatable.  If I picture it as a "resumable function"
then, sure, it should; but if I picture the caller as bouncing control back
& forth with the generator, coroutine style, then suspension is a just a
pause in the generator's execution.  The latter is probably the more natural
way to picture it, eh?

Which feeds into:

> Then of course we create another loose end: what if the for loop
> contains a break?  Then the frame will never be resumed and its
> finally clause will never be executed!  This sounds bad.  Perhaps the
> destructor of the frame should look at the 'resumable' bit and if set,
> resume the frame with a system exception, "Killed", indicating an
> abortion?  (This is like the kill() call in Generator.py.)  We can
> increase the likelihood that the frame's desctructor is called at the
> expected time (right when the for loop terminates), by deleting
> __frame at the end of the loop.  If the resumed frame raises another
> exception, we ignore it.  Its return value is ignored.  If it suspends
> itself again, we resume it with the "Killed" exception again until it
> dies (thoughts of the Blank Knight come to mind).

This may leave another loose end <wink>:  what if the for loop doesn't
contain a break, but dies because of an exception in some line unrelated to
the generator?  Or someone has used an explicit get_frame() in any case and
that keeps a ref to the frame alive?  If the semantic is that the generator
must be shut down no matter what, then the invoker needs code more like

    value, frame = generator(args)
    try:
        while frame:
            etc
		value, frame = resume_frame(frame)
    finally:
        if frame:
             shut_frame_down(frame)

OTOH, the possibility that someone *can* do an explicit get_frame suggests
that "for" shouldn't assume it's the master of the universe <wink>.  Perhaps
the user's intent was to generate the first 100 values in a for loop, then
break out, analyze the results, and decide whether to resume it again by
hand (I've done stuff like that ...).  So there's also a case to be made for
saying that a "finally" clause wrapping a generator body will only be
executed if the generator body raises an exception or the generator itself
decides it's done; i.e. iff it triggers while the generator is actively
running.

Just complicating things there <wink>.  It actually sounds pretty good to
raise a Killed exception in the frame destructor!  The destructor has to do
*something* to trigger the code that drains the frame's value stack anyway,
"finally" blocks or not (frame_dealloc doesn't do that now, since there's
currently no way to get out of eval_code2 with a non-empty stack).

> ...
> I am beginning to like this idea.  (Not that I have time for an
> implementation...  But it could be done without Christian's patches.)

Or with them too <wink>.  If stuff is implemented via continuations, the
same concerns about try/finally blocks pop up everywhere a continuation is
invoked:  you (probably) leave the current frame, and may or may not ever
come back.  So if there's a "finally" clause pending and you don't ever come
back, it's a surprise there too.

So while you thought you were dealing with dirt-simple one-frame generators,
you were *really* thinking about how to make general continuations play nice
<wink>.

solve-one-mystery-and-you-solve-'em-all-ly y'rs  - tim