Yielding a chain of values

[Peter Hansen]

Talin wrote:
> I'm finding that a lot of places within my code, I want to return the 
> output of a generator from another generator. Currently the only method 
> I know of to do this is to explicitly loop over the results from the 
> inner generator, and yield each one:
> 
>        for x in inner():
>            yield x
> 
> I was wondering if there was a more efficient and concise way to do 
> this. And if there isn't, then what about extending the * syntax used 
> for lists, i.e.:
> 
>    yield *inner()

It's not the first time it's been suggested.  You can check the archives 
perhaps for past discussions.

I think syntax like that could be a good idea too, for readability 
reasons perhaps, but I don't think it's really important for efficiency 
reasons.  If you think about it, this involves one stack frame being set 
up for the call to the generator, and then a series of quick context 
switches (or whatever they're called in this situation) between the 
current stack frame and the inner() one, as each yielded value is 
produced, yield, then re-yielded up to the calling frame (with another 
quick context switch).  No additional stack frames are generated, and 
very few byte codes are involved:

 >>> def f():
...   for x in inner():
...     yield x
...
 >>> dis.dis(f)
   2           0 SETUP_LOOP              21 (to 24)
               3 LOAD_GLOBAL              0 (inner)
               6 CALL_FUNCTION            0
               9 GET_ITER
         >>   10 FOR_ITER                10 (to 23)
              13 STORE_FAST               0 (x)

   3          16 LOAD_FAST                0 (x)
              19 YIELD_VALUE
              20 JUMP_ABSOLUTE           10
...

Yes, that is some overhead, but unless you are going many levels deep 
(and that's usually a design smell of some kind) this isn't likely to be 
noticed amongst the rest of the code which is presumably doing something 
non-trivial to produce the values in the first place, and to consume 
them ultimately.

The basic loop could be handled entirely from C with an appropriate 
syntax addition as you suggest, but executing those four byte code 
instructions is very quick, and there is no repeated (Python) function 
call overhead as you would expect if inner() were not a generator.

-Peter