generator functions: why won't this work?

[zillow10 at googlemail.com]

On Apr 2, 4:42 am, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:
> En Tue, 01 Apr 2008 23:56:50 -0300, <zillo... at googlemail.com> escribió:
>
> > I'm trying to understand generator functions and the yield keyword.
> > I'd like to understand why the following code isn't supposed to work.
> > (What I would have expected it to do is, for a variable number of
> > arguments composed of numbers, tuples of numbers, tuples of tuples,
> > etc., the function would give me the next number "in sequence")
> > ####################################
> > def getNextScalar(*args):
> >    for arg in args:
> >       if ( isinstance(arg, tuple)):
> >          getNextScalar(arg)
> >       else:
> >          yield arg
> > ####################################
>
> You're not the first one in getting confused. After all, this schema works
> well for other recursive constructs.
> Perhaps a progression of working code samples will help to understand what
> happens here. The simplest thing would be to just print the items as
> they're encountered, in a recursive call:
>
> py> data = (1, 2, (3,4,(5,6),7))
> py>
> py> print "1) using print"
> 1) using print
> py>
> py> def getNextScalar(args):
> ...    for arg in args:
> ...       if isinstance(arg, tuple):
> ...          getNextScalar(arg)
> ...       else:
> ...          print arg
> ...
> py> getNextScalar(data)
> 1
> 2
> 3
> 4
> 5
> 6
> 7
>
> Now one could try to collect the numbers in a list:
>
> py> print "2) using extend"
> 2) using extend
> py>
> py> def getNextScalar(args):
> ...    result = []
> ...    for arg in args:
> ...       if isinstance(arg, tuple):
> ...          result.extend(getNextScalar(arg))
> ...       else:
> ...          result.append(arg)
> ...    return result
> ...
> py> getNextScalar(data)
> [1, 2, 3, 4, 5, 6, 7]
>
> Note that we use two different list methods: for individual items, we use
> "append", but for tuples we use "extend" in the recursive call. If extend
> weren't available, we could emulate it with append:
>
> py> print "3) using append"
> 3) using append
> py>
> py> def getNextScalar(args):
> ...    result = []
> ...    for arg in args:
> ...       if isinstance(arg, tuple):
> ...          for item in getNextScalar(arg):
> ...              result.append(item)
> ...       else:
> ...          result.append(arg)
> ...    return result
> ...
> py> getNextScalar(data)
> [1, 2, 3, 4, 5, 6, 7]
>
> See how we need an additional loop to iterate over the results that we get
>  from the recursive call.
> Now instead of building an intermediate result list, we delegate such task
> over the caller, and we use a generator that just yields items; this way,
> we remove all references to the result list and all result.append calls
> become yield statements. The inner loop has to remain the same. The yield
> statement acts somewhat as an "append" over an outer list created by the
> generator's caller.
>
> py> print "4) using yield"
> 4) using yield
> py>
> py> def getNextScalar(args):
> ...    for arg in args:
> ...       if isinstance(arg, tuple):
> ...          for item in getNextScalar(arg):
> ...              yield item
> ...       else:
> ...          yield arg
> ...
> py> getNextScalar(data)
> <generator object at 0x00A3AE68>
> py> list(getNextScalar(data))
> [1, 2, 3, 4, 5, 6, 7]
>
> I hope it's more clear now why you have to use yield on the recursive call
> too.
>
> <idea mode="raw">
> Perhaps this:
>
>    yield *iterable
>
> could be used as a shortcut for this:
>
>    for __temp in iterable: yield __temp
>
> </idea>
>
> --
> Gabriel Genellina

Thanks to everyone for your very helpful replies. I think I was trying
to use generator functions without first having really read about
iterators (something I still haven't done, although I've managed to
extrapolate some details based on your comments), and therefore really
"putting the cart before the horse". I was interpreting
getNextScalar(arg) on line 3 as a function call in the usual sense
rather than an object that was being created but never used.

I have a question about the other issue that was pointed out. With
reference to the following (corrected) code:

def getNextScalar(*args):
	for arg in args:
		if(isinstance(arg, tuple)):
			for f in getNextScalar(*arg): # why not getNextScalar(arg)?
				yield f
		else:
			yield arg

although I've verified that the line 4 needs to be "for f in
getNextScalar(*arg)" rather than "for f in getNextScalar(arg)" when
passing multiple arguments to the function, I'd just like to test my
understanding of this. Suppose I create the following generator
object:

g = getNextScalar(1, 2, (3, 4), 5)

when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead? I presume that (3,4)
is passed as a single tuple argument, if(isinstance(arg, tuple)) is
true, and the tuple is passed again as an the argument to
getNextScalar()... ad infinitum. The alternative is to define a
function that takes a tuple, as Gabriel has done.

Regards,
AK