[Python-3000] Exception re-raising woes

[Antoine Pitrou]

Hello all,

Trying to fix #2507 (Exception state lives too long in 3.0) has uncovered new
issues with the bare "raise" statement when in used in exception block nesting
situations (see #2833: __exit__ silences the active exception). I say
"uncovered" rather than "crated" since, as Amaury points out in the latter bug
entry, re-raising behaviour has always been a bit limited or non-obvious.

Witness the following code:

   try:
      raise Exception("foo")
   except Exception:
      try: raise KeyError("caught")
      except KeyError: pass
      raise

With python 2.x and py3k pre-r62847, it would re-raise KeyError("caught")
(whereas the intuitive behaviour would be to re-raise Exception("foo")).
With py3k post-r62847, it now raises a "RuntimeError: No active
exception to reraise".

Note that in py3k at least, we can get the "correct" behaviour by writing
instead:

   try:
      raise Exception("foo")
   except Exception as e:
      try: raise KeyError("caught")
      except KeyError: pass
      raise e

The only slight annoyance being that the re-raising statement ("raise e") is
added at the end of the original traceback.

There are other funny situations. Just try (with any Python version):

def except_yield():
    try:
        raise Exception("foo")
    except:
        yield 1
        raise
list(except_yield())


The problem with properly fixing the bare "raise" statement is that right now,
the saved exception state is a member of the frame object. That is, there is no
proper stacking of exception states when some lexically nested exception
handlers are involved in the same frame.

Now perhaps it is time to think about fixing that problem, without losing the
expected properties of exceptions in py3k. I propose the following changes:

- an "except" block now also becomes a block in ceval.c terms, that is, a
specific PyTryBlock is pushed at its beginning (please note that right now
SETUP_EXCEPT, despite its name, encloses the "try" block rather than any
"except" statement)
- this specific PyTryBlock - let's name it EXCEPT_HANDLER - is created
implicitly, not explicitly through an opcode; this is necessary because it must
be created *before* setting the current exception state to the caught exception,
waiting for an opcode to be executed would be too late
- before pushing this EXCEPT_HANDLER on the block stack, the current thread's
exception state (that is, before the exception is caught) is saved on the frame
stack (that is, the three objects representing the type, value and traceback
respectively)
- an EXCEPT_HANDLER block is unwinded explicitly with a dedicated POP_EXCEPT
opcode at the end of the exception handler; this opcode, not only unwinds the
block as POP_BLOCK does, but also pops and restores the exception state which
was saved on the stack before pushing the block
- an EXCEPT_HANDLER block, when it is unwinded implicitly because of a control
transfer (e.g. "return" or "continue" or "break" or "raise"), follows the same
treatment as in the POP_EXCEPT opcode: that is, in addition to unwinding the
block, it also pops and restores the previous exception state
- the current set_exc_info() / reset_exc_info() machinery is yanked, since it is
not useful anymore; this also probably removes three fields in the frame object,
because it does not need to contain the previous exception state anymore

I've not studied the "with" statement implementation. Chances are it should
also be adapted to follow the principles above. I may also be missing other
annoying "details" :-)

What do you think?

Regards

Antoine.