[Python-ideas] while conditional in list comprehension ??

[Oscar Benjamin]

On 29 January 2013 15:34, Zachary Ware <zachary.ware+pyideas at gmail.com> wrote:
>
> On Jan 29, 2013 9:26 AM, "Oscar Benjamin" <oscar.j.benjamin at gmail.com>
> wrote:
>>
>> On 29 January 2013 11:51, yoav glazner <yoavglazner at gmail.com> wrote:
>> > Here is very similar version that works (tested on python27)
>> >>>> def stop():
>> > next(iter([]))
>> >
>> >>>> list((i if i<50 else stop()) for i in range(100))
>> > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
>> > 20,
>> > 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
>> > 39,
>> > 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
>>
>> That's a great idea. You could also do:
>> >>> list(i for i in range(100) if i<50 or stop())
>>
>> It's a shame it doesn't work for list/set/dict comprehensions, though.
>>
>
> I know I'm showing my ignorance here, but how are list/dict/set
> comprehensions and generator expressions implemented differently that one's
> for loop will catch a StopIteration and the others won't? Would it make
> sense to reimplement list/dict/set comprehensions as an equivalent generator
> expression passed to the appropriate constructor, and thereby allow the
> StopIteration trick to work for each of them as well?

A for loop is like a while loop with a try/except handler for
StopIteration. So the following are roughly equivalent:

# For loop
for x in iterable:
    func1(x)
else:
    func2()

# Equivalent loop
it = iter(iterable)
while True:
    try:
        x = next(it)
    except StopIteration:
        func2()
        break
    func1(x)

A list comprehension is just like an implicit for loop with limited
functionality so it looks like:

# List comp
results = [func1(x) for x in iterable if func2(x)]

# Equivalent loop
results = []
it = iter(iterable)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    # This part is outside the try/except
    if func2(x):
        results.append(func1(x))

The problem in the above is that we only catch StopIteration around
the call to next(). So if either of func1 or func2 raises
StopIteration the exception will propagate rather than terminate the
loop. (This may mean that it terminates a for loop higher in the call
stack - which can lead to confusing bugs - so it's important to always
catch StopIteration anywhere it might get raised.)

The difference with the list(generator) version is that func1() and
func2() are both called inside the call to next() from the perspective
of the list() function. This means that if they raise StopIteration
then the try/except handler in the enclosing list function will catch
it and terminate its loop.

# list(generator)
results = list(func1(x) for x in iterable if func2(c))

# Equivalent loop:
def list(iterable):
    it = iter(iterable)
    results = []
    while True:
        try:
            # Now func1 and func2 are both called in next() here
            x = next(it)
        except StopIteration:
            break
        results.append(x)
    return results

results_gen = (func1(x) for x in iterable if func2(x))
results = list(results_gen)


Oscar