[New-bugs-announce] [issue43683] Handle generator (and coroutine) state in the bytecode.

[Mark Shannon]

New submission from Mark Shannon <mark at hotpy.org>:

Every time we send, or throw, to a generator, the C code in genobject.c needs to check what state the generator is in. 
This is inefficient and couples the generator code, which should just be a thin wrapper around the interpreter, to the internals of the interpreter.

The state of the generator is known to the compiler. It should emit appropriate bytecodes to handle the different behavior for the different states.

While the main reason this is robustness and maintainability, removing the complex C code between Python caller and Python callee also opens up the possibility of some worthwhile optimizations.

There are three changes I want to make:

1. Add a new bytecode to handle starting a generator. This `GEN_START` bytecode would pop TOS, raising an exception if it is not None.
This adds some overhead for the first call to iter()/send() but speeds up all the others.

2. Handle the case of exhausted generators. This is a bit more fiddly, and involves putting an infinite loop at the end of the generator. Something like:

   CLEAR_FRAME
label:
   GEN_RETURN (Like RETURN_VALUE None, but does not discard the frame)
   JUMP label


This removes a lot of special case code for corner cases of exhausted generators and coroutines.

3. Handle throw() on `YIELD_FROM`. The problem here is that we need to differentiate between exceptions triggered by throw, which must call throw() on sub-generators, and exceptions propagating out of sub-generators which should be passed up the stack. By splitting the opcode into two (or more), it is clear which case is being handled in the interpreter without complicated logic in genobject.c

----------
messages: 389919
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: Handle generator (and coroutine) state in the bytecode.

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43683>
_______________________________________