generator functions: why won't this work?

[Gabriel Genellina]

En Tue, 01 Apr 2008 23:56:50 -0300, <zillow20 at googlemail.com> escribió:

> I'm trying to understand generator functions and the yield keyword.
> I'd like to understand why the following code isn't supposed to work.
> (What I would have expected it to do is, for a variable number of
> arguments composed of numbers, tuples of numbers, tuples of tuples,
> etc., the function would give me the next number "in sequence")
> ####################################
> def getNextScalar(*args):
>    for arg in args:
>       if ( isinstance(arg, tuple)):
>          getNextScalar(arg)
>       else:
>          yield arg
> ####################################

You're not the first one in getting confused. After all, this schema works  
well for other recursive constructs.
Perhaps a progression of working code samples will help to understand what  
happens here. The simplest thing would be to just print the items as  
they're encountered, in a recursive call:

py> data = (1, 2, (3,4,(5,6),7))
py>
py> print "1) using print"
1) using print
py>
py> def getNextScalar(args):
...    for arg in args:
...       if isinstance(arg, tuple):
...          getNextScalar(arg)
...       else:
...          print arg
...
py> getNextScalar(data)
1
2
3
4
5
6
7

Now one could try to collect the numbers in a list:

py> print "2) using extend"
2) using extend
py>
py> def getNextScalar(args):
...    result = []
...    for arg in args:
...       if isinstance(arg, tuple):
...          result.extend(getNextScalar(arg))
...       else:
...          result.append(arg)
...    return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

Note that we use two different list methods: for individual items, we use  
"append", but for tuples we use "extend" in the recursive call. If extend  
weren't available, we could emulate it with append:

py> print "3) using append"
3) using append
py>
py> def getNextScalar(args):
...    result = []
...    for arg in args:
...       if isinstance(arg, tuple):
...          for item in getNextScalar(arg):
...              result.append(item)
...       else:
...          result.append(arg)
...    return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]

See how we need an additional loop to iterate over the results that we get  
 from the recursive call.
Now instead of building an intermediate result list, we delegate such task  
over the caller, and we use a generator that just yields items; this way,  
we remove all references to the result list and all result.append calls  
become yield statements. The inner loop has to remain the same. The yield  
statement acts somewhat as an "append" over an outer list created by the  
generator's caller.

py> print "4) using yield"
4) using yield
py>
py> def getNextScalar(args):
...    for arg in args:
...       if isinstance(arg, tuple):
...          for item in getNextScalar(arg):
...              yield item
...       else:
...          yield arg
...
py> getNextScalar(data)
<generator object at 0x00A3AE68>
py> list(getNextScalar(data))
[1, 2, 3, 4, 5, 6, 7]

I hope it's more clear now why you have to use yield on the recursive call  
too.

<idea mode="raw">
Perhaps this:

   yield *iterable

could be used as a shortcut for this:

   for __temp in iterable: yield __temp

</idea>

-- 
Gabriel Genellina