Reducing "yield from" overhead in recursive generators

[Chris Angelico]

On Sat, 19 Mar 2022 at 14:06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> On 19/03/22 9:40 am, Rathmann wrote:
> > The other challenge/question would be to see if there is a way to implement
> > this basic approach with lower overhead.
>
> My original implementation of yield-from didn't have this problem,
> because it didn't enter any of the intermediate frames -- it just
> ran down a chain of C pointers to find the currently active one.
>
> At some point this was changed, I think so that all the frames
> would show up in the traceback in the event of an exception.
> This was evidently seen as more important than having efficient
> resumption of nested generators.
>
> I'm not sure exactly how it works now, but I believe it involves
> re-executing the YIELD_FROM bytecode in each generator down the
> chain whenever a resumption occurs. This is nice and simple, but
> not very efficient.
>
> Maybe another way could be found of preserving the traceback,
> such as temporarily splicing the chain of frames into the call
> stack and then resuming the last one.
>

This is a fundamentally difficult problem with async/await too, so if
there's any good solution, it would be worth carrying across.

ChrisA