Dangerous behavior of list(generator)

[Terry Reedy]

On 12/13/2009 11:33 PM, exarkun at twistedmatrix.com wrote:
> But if you mistakenly don't catch it, and you're trying to debug your
> code to find this mistake, you probably won't be aided in this pursuit
> by the exception-swallowing behavior of generator expressions.

As I remember, it was the call to list that swalled the exception, not 
the generator expression. List() takes an iterable as arg and stopping 
on StopIteration is what it does and how it knows to stop and return the 
new list.

> The behavior of list comprehensions is pretty good. The behavior of
> constructing a list out of a generator expression isn't as good.

I think you are confused. A generator expression is a shorthand for a 
def statement that defines a generator function followed by a call to 
the generator function to get a generator followed by deletion of the 
function. When you call list() to make a list, it constructs the list 
from the generator, not from the expression itself. List has no idea 
that you used a generator expression or even that it was passed a 
generator. Leaving error checks out, it operates something like

def list(it):
   res = []
   it = iter(it)
   for item in it: # stops whenever it raises StopIteration
     res.append(item)
   return res

> The
> behavior which is more desirable is for a StopIteration raised out of
> the `expression` part of a `generator_expression` to not be treated
> identically to the way a StopIteration raised out of the `genexpr_for`
> part is.

It is not. StopIteration in for part stops the for loop in the 
generator. StopIteration in the expression part stops the loop in the 
list() call (sooner than it would have been otherwise). When the 
generator raises StopIteration, list() has no idea what statement within 
the body raised it. It MUST stop.

> This could provide behavior roughly equivalent to the behavior
> of a list comprehension.

Impossible. The only serious option for consistency is to special case 
list comps to also trap StopIteration raised in the expression part, but 
the devs decided not to do this as doing do is arguably a bug.

Terry Jan Reedy