generator functions: why won't this work?

[Steve Holden]

zillow10 at googlemail.com wrote:
> On Apr 2, 4:42 am, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:
>> En Tue, 01 Apr 2008 23:56:50 -0300, <zillo... at googlemail.com> escribió:
>>
>>> I'm trying to understand generator functions and the yield keyword.
>>> I'd like to understand why the following code isn't supposed to work.
>>> (What I would have expected it to do is, for a variable number of
>>> arguments composed of numbers, tuples of numbers, tuples of tuples,
>>> etc., the function would give me the next number "in sequence")
>>> ####################################
>>> def getNextScalar(*args):
>>>    for arg in args:
>>>       if ( isinstance(arg, tuple)):
>>>          getNextScalar(arg)
>>>       else:
>>>          yield arg
>>> ####################################
>> You're not the first one in getting confused. After all, this schema works
>> well for other recursive constructs.
>> Perhaps a progression of working code samples will help to understand what
>> happens here. The simplest thing would be to just print the items as
>> they're encountered, in a recursive call:
>>
>> py> data = (1, 2, (3,4,(5,6),7))
>> py>
>> py> print "1) using print"
>> 1) using print
>> py>
>> py> def getNextScalar(args):
>> ...    for arg in args:
>> ...       if isinstance(arg, tuple):
>> ...          getNextScalar(arg)
>> ...       else:
>> ...          print arg
>> ...
>> py> getNextScalar(data)
>> 1
>> 2
>> 3
>> 4
>> 5
>> 6
>> 7
>>
>> Now one could try to collect the numbers in a list:
>>
>> py> print "2) using extend"
>> 2) using extend
>> py>
>> py> def getNextScalar(args):
>> ...    result = []
>> ...    for arg in args:
>> ...       if isinstance(arg, tuple):
>> ...          result.extend(getNextScalar(arg))
>> ...       else:
>> ...          result.append(arg)
>> ...    return result
>> ...
>> py> getNextScalar(data)
>> [1, 2, 3, 4, 5, 6, 7]
>>
>> Note that we use two different list methods: for individual items, we use
>> "append", but for tuples we use "extend" in the recursive call. If extend
>> weren't available, we could emulate it with append:
>>
>> py> print "3) using append"
>> 3) using append
>> py>
>> py> def getNextScalar(args):
>> ...    result = []
>> ...    for arg in args:
>> ...       if isinstance(arg, tuple):
>> ...          for item in getNextScalar(arg):
>> ...              result.append(item)
>> ...       else:
>> ...          result.append(arg)
>> ...    return result
>> ...
>> py> getNextScalar(data)
>> [1, 2, 3, 4, 5, 6, 7]
>>
>> See how we need an additional loop to iterate over the results that we get
>>  from the recursive call.
>> Now instead of building an intermediate result list, we delegate such task
>> over the caller, and we use a generator that just yields items; this way,
>> we remove all references to the result list and all result.append calls
>> become yield statements. The inner loop has to remain the same. The yield
>> statement acts somewhat as an "append" over an outer list created by the
>> generator's caller.
>>
>> py> print "4) using yield"
>> 4) using yield
>> py>
>> py> def getNextScalar(args):
>> ...    for arg in args:
>> ...       if isinstance(arg, tuple):
>> ...          for item in getNextScalar(arg):
>> ...              yield item
>> ...       else:
>> ...          yield arg
>> ...
>> py> getNextScalar(data)
>> <generator object at 0x00A3AE68>
>> py> list(getNextScalar(data))
>> [1, 2, 3, 4, 5, 6, 7]
>>
>> I hope it's more clear now why you have to use yield on the recursive call
>> too.
>>
>> <idea mode="raw">
>> Perhaps this:
>>
>>    yield *iterable
>>
>> could be used as a shortcut for this:
>>
>>    for __temp in iterable: yield __temp
>>
>> </idea>
>>
>> --
>> Gabriel Genellina
> 
> Thanks to everyone for your very helpful replies. I think I was trying
> to use generator functions without first having really read about
> iterators (something I still haven't done, although I've managed to
> extrapolate some details based on your comments), and therefore really
> "putting the cart before the horse". I was interpreting
> getNextScalar(arg) on line 3 as a function call in the usual sense
> rather than an object that was being created but never used.
> 
> I have a question about the other issue that was pointed out. With
> reference to the following (corrected) code:
> 
> def getNextScalar(*args):
> 	for arg in args:
> 		if(isinstance(arg, tuple)):
> 			for f in getNextScalar(*arg): # why not getNextScalar(arg)?
> 				yield f
> 		else:
> 			yield arg
> 
> although I've verified that the line 4 needs to be "for f in
> getNextScalar(*arg)" rather than "for f in getNextScalar(arg)" when
> passing multiple arguments to the function, I'd just like to test my
> understanding of this. Suppose I create the following generator
> object:
> 
> g = getNextScalar(1, 2, (3, 4), 5)
> 
> when the iterator reaches the tuple argument (3, 4) then, according to
> Steve and George, the * in *arg causes this tuple to be expanded into
> positional arguments, and it makes sense to do it this way. But what
> happens when getNextScalar(arg) is used instead? I presume that (3,4)
> is passed as a single tuple argument, if(isinstance(arg, tuple)) is
> true, and the tuple is passed again as an the argument to
> getNextScalar()... ad infinitum. The alternative is to define a
> function that takes a tuple, as Gabriel has done.
> 
Either approach would work - it just seemed to make more sense to use 
existing functionality and recurse (at least to me). You should try 
removing the * from the recursive call. I expect you would find that you 
(eventually) experience a stack overflow due to the infinite nature of 
the recursion.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/