Proposal: [... for ... while cond(x)]

[Nick Mellor]

Hi all,

It might be a risk that we create some hairy or subtle semantics, but perhaps this idea has legs. The 'pythonic' syntax is very attractive.

# skip the silence at the start
data = (amp for amp in signal from abs(amp) > 0.05)

# drop files that are less than 600 bytes
# (already sorted into ascending order of size)
files = (file for file in filelist from os.stat(file).st_size > 600)

# Analyse bunch of log lines up to a certain timestamp
cutoff = datetime.datetime.now() - datetime.timedelta(hours=1)
checklogs = (entry for entry in log while entry.timestamp <= cutoff)

In use it feels along the same lines as else: clauses in try/finally and for loops, both of which I like-- it seems natural in Python, keeps the meaning clear and consistent, avoids complicated, iterative, inexpressive code.

Terry Reedy wrote:
> You're right. The syntax is ambiguous. I agree it's not a good idea,
now. :)

When while, if and from are made mutually exclusive, not mixed, on a comprehension clause, it looks to me like the ambiguity disappears:

> x for x in xs while cond(x) if blah(x)

not legal Python-- if and while are mutually exclusive on the same comprehension clause. On the one hand it's like writing x for x in xs if f(x) < 10 if g(x) > 100, the illegality of which no-one is complaining about. Just write:

x for x in xs while cond(x) and cond2(x)

or

x for x in xs if cond(x) and cond2(x)

If there is a need to filter using if as well as while, then use two comprehensions:

intermediate = [x for x in xs while cond(x)]
final = [x for x in intermediate if cond2(x)]

or the same comprehension applied to itself:

y for y in x for x in xs if cond(x) while cond2(y)

The if and the while operation *should* be kept separate-- they're different stages of processing the data.

> > x*y for x in xs while cond(x) for y in ys if cond(y)

legal Python-- the 'if cond(y)' unambiguously refers to the inner comprehension. 


> x for x in xs if blah(x) while cond(x)

not legal Python-- if and while are mutually exclusive in the same comprehension clause.

The objection to "while" as an abbreviation:

> e(x) for x in xs if cond(x)
> 
> is an abbreviation of
> 
> for x in xs:
>   if cond(x)
>     yield e(x)
> 
> and similarly with more for and if clauses,
> whereas the analogous expansion of your proposal
> 
> for x in xs:
>   while cond(x):
>      yield e(x)
> 
> is an infinite loop and not at all what you mean.

That's because while is being used here with a different semantic to the while loop. It actually means:

for x in xs:
    if not cond(x):
        break
    yield e(x)

and for 'from':


emit = False
for x in xs:
    if not emit:
        if cond(x):
            emit = True
    else:
        yield e(x)


Cheers,

Nick