Different behaviour in list comps and generator expressions

[Wolfgang Maier]

On 08.11.2014 22:31, Wolfgang Maier wrote:
> On 08.11.2014 02:50, Steven D'Aprano wrote:
>> The following list comprehension and generator expression are almost, but
>> not quite, the same:
>>
>> [expr for x in iterable]
>>
>> list(expr for x in iterable)
>>
>>
>> The difference is in the handling of StopIteration raised inside the
>> expr.
>> Generator expressions consume them and halt, while comprehensions allow
>> them to leak out.
>
> This is not the right description of what's happening. It is not the
> generator expression that consumes it, but the list constructor and, of
> course, list can't tell at which level in the inside code StopIteration
> got raised.
> So, yes this had me confused some times, too, but it is really not
> surprising.
>
>> A simple example:
>>
>> iterable = [iter([])]
>> list(next(x) for x in iterable)
>> => returns []
>>
>> But:
>>
>> [next(x) for x in iterable]
>> => raises StopIteration
>>
>
> Yes, but the equivalent to list(next(x) for x in iterable) is:
>
> l = []
> for item in (next(x) for x in iterable):
>      l.append(item)
>
> => returns [] because the for consumes the StopIteration
>
> this, on the other hand raises StopIteration
>
> l = []
> for item in [next(x) for x in iterable]:
>      l.append(item)
>
> because the error gets raised already when for causes iter() to be
> called on the list comprehension.
>
>>
>> Has anyone come across this difference in the wild? Was it a problem?
>> Do you
>> rely on that difference, or is it a nuisance? Has it caused difficulty in
>> debugging code?
>>
>> If you had to keep one behaviour, which would you keep?
>>
>
> The point is: there is no difference in the behavior of comprehensions
> and generator expressions, it is just how you access them:
> for anything that follows the iterator protocol, a comprehension (since
> it is evaluated immediately) raises during the iter() phase, while a
> generator expression (which gets evaluated lazily) raises during the
> next() phase where it gets swallowed.
>
> So in your example you would have to use the silly
>
> list([next(x) for x in iterable])
>
> if you want the error to get raised.
>
> I agree this is all rather non-intuitive, but how would you change it ?
>
>

Ah, I came across the related thread on python-ideas only now and from 
this I can see that, as I expected, you know everything I've written 
above already.

In light of the discussion on the other list:
I did find it annoying occasionally that raising StopIteration inside a 
generator expression conveys a different behavior than elsewhere. It did 
take me quite a while to understand why that is so, but after that it 
did not cause me much of a headache anymore.

I would say that Guido's suggestion of transforming StopIteration raised 
inside a generator into some other error to eliminate ambiguity would 
really help in some situations, but then I'm not sure whether that's 
worth breaking existing code.

Best,
Wolfgang