yield_all needed in Python

[Nick Coghlan]

Jeremy Bowers wrote:
> At first I liked this, but the reason that is a syntax error is that it is
> "supposed" to be
> 
> def f():
>     yield (x for x in gen1(arg))
> 
> which today on 2.4 returns a generator instance which will in turn
> yield one generator instance from the genexp

And it would continue to do so in the future. On the other hand, removing the 
parens makes it easy to write things like tree traversal algorithms:

def left_to_right_traverse(node):
   yield x for x in node.left
   yield node .value
   yield x for x in node.right

In reality, I expect yielding each item of a sub-iterable to be more common than 
building a generator that yields generators.

> , and I am quite uncomfortable
> with the difference between the proposed behaviors with and without the
> parens.

Why? Adding parentheses can be expected to have significant effects when it 
causes things to be parsed differently. Like the example I posted originally:

   [x for x in iterable]  # List comp (no parens == eval in place)
   [(x for x in iterable)] # Parens - generator goes in list

Or, for some other cases where parentheses severely affect parsing:

   print x, y
   print (x, y)

   assert x, y
   assert (x, y)

If we want to pass an iterator into a function, we use a generator expression, 
not extended call syntax. It makes sense to base a sub-iterable yield syntax on 
the former, rather than the latter.

> Moreover, since "yield" is supposed to be analogous to "return", what does
> 
>     return x for x in gen1(arg)
> 
> do? Both "it returns a list" and "it returns a generator" have some
> arguments in their favor.

No, it would translate to:

   for x in gen1(arg):
     return x

Which is nonsense, so you would never make it legal.

> And I just now note that any * syntax, indeed, any syntax at all will
> break this.

As you noted, this argument is specious because it applies to *any* change to 
the yield syntax - yield and return are fundamentally different, since yield 
allows resumption of processing on the next call to next().

 > You know, given the marginal gains this gives anyway,

I'm not so sure the gains will be marginal. Given the penalties CPython imposes 
on recursive calls, eliminating the nested "next()" invocations could 
significantly benefit any code that uses nested iterators.

An interesting example where this could apply is:

def flatten(iterable):
   for item in iterable:
     if item is iterable:
       # Do the right thing for self-iterative things
       # like length 1 strings
       yield iterable
       raise StopIteration
     try:
       itr = iter(item):
     except TypeError:
       yield item
     else:
       yield x for x in flatten(item)

Cheers,
Nick.

P.S. Which looks more like executable pseudocode?

def traverse(node):
   yield *node.left
   yield node .value
   yield *node.right

def traverse(node):
   yield x for x in node.left
   yield node .value
   yield x for x in node.right

-- 
Nick Coghlan   |   ncoghlan at email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net