Dangerous behavior of list(generator)

[exarkun at twistedmatrix.com]

On 04:11 am, steven at remove.this.cybersource.com.au wrote:
>On Sun, 13 Dec 2009 22:45:58 +0000, exarkun wrote:
>>On 08:18 pm, steve at remove-this-cybersource.com.au wrote:
>>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>>>StopIteration is intended to be used only within the .__next__ 
>>>>>method
>>>>>of
>>>>>iterators. The devs know that other 'off-label' use results in the
>>>>>inconsistency you noted, but their and my view is 'don't do that'.
>>>>
>>>>Which is unfortunate, because it's not that hard to get 
>>>>StopIteration
>>>>without explicitly raising it yourself and this behavior makes it
>>>>difficult to debug such situations.
>>>
>>>I can't think of any way to get StopIteration without explicitly 
>>>raising
>>>it yourself. It's not like built-ins or common data structures 
>>>routinely
>>>raise StopIteration. I don't think I've *ever* seen a StopIteration 
>>>that
>>>I didn't raise myself.
>>
>>Call next on an iterator.  For example:  iter(()).next()
>
>Or in more recent versions of Python, next(iter(())).
>
>Good example. But next() is a special case, and since next() is
>documented as raising StopIteration if you call it and it raises
>StopIteration, you have raised it yourself. Just not explicitly.

But if you mistakenly don't catch it, and you're trying to debug your 
code to find this mistake, you probably won't be aided in this pursuit 
by the exception-swallowing behavior of generator expressions.
>
>>>>What's with this view, exactly?  Is it just that it's hard to 
>>>>implement
>>>>the more desirable behavior?
>>>
>>>What is that "more desirable behaviour"? That StopIteration is used 
>>>to
>>>signal that Python should stop iterating except when you want it to 
>>>be
>>>ignored? Unfortunately, yes, it's quite hard to implement "do what 
>>>the
>>>caller actually wants, not what he asked for" behaviour -- and even 
>>>if
>>>it were possible, it goes against the grain of the Zen of Python.
>>>
>>>If you've ever had to debug faulty "Do What I Mean" software, you'd 
>>>see
>>>this as a good thing.
>>
>>I have plenty of experience developing and debugging software, Steven.
>>Your argument is specious, as it presupposes that only two 
>>possibilities
>>exist: the current behavior of some kind of magical faerie land
>>behavior.
>>
>>I'm surprised to hear you say that the magical faerie land behavior
>>isn't desirable either, though.  I'd love a tool that did what I 
>>wanted,
>>not what I asked.  The only serious argument against this, I think, is
>>that it is beyond our current ability to create (and so anyone 
>>claiming
>>to be able to do it is probably mistaken).
>
>I'd argue that tools that do what you want rather than what you ask for
>are not just currently impossible, but always will be -- no matter how
>good the state of the art of artificial intelligent mind-reading 
>software
>becomes.

That may be true.  I won't try to make any predictions about the 
arbitrarily distant future, though.
>>You chopped out all the sections of this thread which discussed the 
>>more
>>desirable behavior.  You can go back and read them in earlier messages
>>if you need to be reminded.  I'm not talking about anything beyond
>>what's already been raised.
>
>I'm glad for you. But would you mind explaining for those of us aren't
>mind-readers what YOU consider the "more desirable behaviour"?

The behavior of list comprehensions is pretty good.  The behavior of 
constructing a list out of a generator expression isn't as good.  The 
behavior which is more desirable is for a StopIteration raised out of 
the `expression` part of a `generator_expression` to not be treated 
identically to the way a StopIteration raised out of the `genexpr_for` 
part is.  This could provide behavior roughly equivalent to the behavior 
of a list comprehension.
>
>If you're talking the list constructor and list comprehensions treating
>StopIteration the same, then I don't think it is at all self-evident 
>that
>the current behaviour is a bad thing, nor that the only reason for it 
>is
>that to do otherwise would be hard.

I don't expect it to be self-evident.  I wasn't even trying to convince 
anyone that it's desirable (although I did claim it, so I won't fault 
anyone for making counter-arguments).  The only thing I asked was what 
the motivation for the current behavior is.  If the motivation is that 
it is self-evident that the current behavior is the best possible 
behavior, then someone just needs to say that and my question is 
answered. :)

Jean-Paul