generators and exceptions

[Tim Peters]

[lark C. Evans]
> Thank you for responding Tim, as you expected my question is less
> about what generator behavior is, as it is about what generator
> behavior with regard to exceptions could be.

For Python, it's right the way it is <0.5 wink>.  Python doesn't have any
concept of resumable exceptions, and if you wanted a generator to be
resumable after raising an exception that shouldn't be a gimmick unique to
generators, but should follow from a general scheme for resuming after
exceptions regardless of context.  There isn't one now.

> Right.  I did, but this still didn't stop my expectations.  I was wishing
> that I could resume the generator after the exception was thrown.

I'm not sure that you really would, if you had that.  Note, for example,
that

    raise ZeroDivisionError

has exactly the same effects as executing

    i = j/k

when k happens to be 0, and there's really no difference between your
MyException and the builtin ZeroDivisionError in this respect either:  all
exceptions escaping a function end the function's execution.  It didn't
*need* to be that way, but it is, and it all ties together.  You can't
change any part of it in isolation without introducing special-case warts.

> ...
> Is there a serious technical reason for this behavior?

Several, but I've got no interest in changing it, so I'm not motivated to
spend time on it.  Luckily, you have the source code too.

> Could it have worked some other way,

Yes, but not easily.

> i.e., why not let a generator survive an exception,

For starters, because no function survives an exception.  For another,
because you're picturing the case where you explicitly chose to raise a
special exception as a way to communicate an expected outcome, but Python
doesn't know that's what you're doing, and all exceptions would be treated
the same way (whether or not you "expected" them to occur).  If an
"unexpected" exception occurred in a generator, most people would want that
generator to be killed off as a result, and for the same reason they want
any function to be terminated then:  that piece of their code has gone
insane.

> as the above iterator does so well.

In that case you made a fresh function call each time; the "survival" was in
the framework you wrote to keep making fresh function calls even in the
presence of exceptions, not in a property of iterators.

> A generator is fundamentally different from a function;

Perhaps remarkably, it's not.  In the implementation, the primary difference
between a function return and a generator yield is that the former
decrements the reference count on the execution frame and clears the frame's
stack, while the latter doesn't.  In most other respects (including how they
react to exceptions), they're exactly the same.  The code object for a
generator function has the CO_GENERATOR flag set in co_flags:  search
ceval.c, and you'll find only one reference to that flag.  Surprisingly,
it's not in the eval loop:  the interpreter loop has no idea whether it's
executing a generator or a non-generator function.  It could, of course, be
taught the difference, but there's been no need for that so far.

> I tend to picture it as great syntax sugar for making iterators.

Yes, that was the intent.

> However, and unfortunately, a great many of my iterators raise
> "recoverable exceptions".

I guess they're not written in Python, then <wink>.