[Python-Dev] RE: list comprehensions (was parsers and compilers for 2.0)

[Tim Peters]

[Tim]
>> List comprehensions are one of the best-loved features of Haskell
>> (really!), and Greg/Skip/Ping's patch implements as an exact a
>> parallel to Haskell's syntax and semantics as is possible in Python.

[Thomas Wouters]
> I don't see "it's cool in language X" as a particular good reason
> to include a feature... We don't add special syntax for regular
> expressions, support for continuations or direct access to hardware
> because of that, do we ?

As Guido (overly!) modestly says, the only language idea he ever invented
was an "else" clause on loops.  He decided listcomps "were Pythonic" before
knowing anything about Haskell (or SETL, from which Haskell took the idea).
Given that he *already* liked them, the value in looking at Haskell is for
its actual experience with them.  It would be pretty stupid *not* to look at
experience with other languages that already have it!  And that's whether
you're pro or con.

So it's not "cool in language X" that drives it at all, it's "cool in
language X" that serves to confirm or refute the prior judgment that "it's
Pythonic, period".  When, e.g., Eric predicts it will bite us hard someday,
I can point to Haskell and legitimately ask "why here and not there?".

There was once a great push for adding some notion of "protected" class
members to Python.  Guido was initially opposed, but tempted to waffle
because proponents kept pointing to C++.  Luckily, Stroustrup had something
to say about this in his "Design and Evolution of C++" book, including that
he thought adding "protected" was a mistake, driven by relentless "good
arguments" that opposed his own initial intuition.  So in that case,
*really* looking at C++ may have saved Guido from making the same mistake.

As another example, few arguments are made more frequently than that Python
should add embedded assignments in conditionals.  *Lots* of other languages
have that -- but they mostly serve to tell us it's a bug factory in
practice!  The languages that avoid the bugs point to ways to get the effect
safely (although none yet Pythonically enough for Guido).

So this is a fact:  language design is very little about wholesale
invention, and that's especially true of Python.  It's a mystically
difficult blending of borrowed ideas, and it's rare as readable Perl <wink>
that an idea will get borrowed if it didn't even work well in its native
home.  listcomps work great where they came from, and that plus "hey, Guido
likes 'em!" makes it 99% a done deal.

> My main beef with the syntax is that it is, in my eyes, unpythonic.
> It has an alien, forced feel to it, much more so than the 'evil'
> map/filter/reduce.  It doesn't 'fit' into Python the way most of
> the other features do;

Guido feels exactly the opposite:  the business about "alien, forced feel,
not fitting" is exactly what he's said about map/filter/reduce/lambda on
many occasions.  listcomps strike him (me too, for that matter) as much more
Pythonic than those.

> it's simply syntactic sugar for a specific kind of for-loop. It
> doesn't add any extra functionality,

All overwhelmingly true of augmented assignments, and range literals, and
three-argument getattr, and list.pop, etc etc etc too.  Python has lots of
syntactic sugar -- making life pleasant is not a bad thing.

> and for that large a syntactic change, I guess that scares me.

The only syntactic change is to add a new form of list constructor.  It's
isolated and self-contained, and so "small" in that sense.

> Those doubts were why I was glad you were going to write the PEP. I
> was looking forward to you explaining why I had those doubts and
> giving sound arguments against them :-)

There is no convincing argument to made either way on whether "it's
Pythonic", which I think is your primary worry.  People *never* reach
consensus on whether a given feature X "is Pythonic".  That's why it's
always Guido's job.  You've been here long enough to see that -1 and +1 are
about evenly balanced, except on (in recent memory) "import x as y" -- which
I conviently neglected to mention had been dismissed as unPythonic by Guido
just a couple weeks ago <wink -- but he didn't really mean it then,
according to me>.

> ...
> but that's mostly because I was expecting, like ESR, a huge debate
> on its syntax.

Won't happen, because it already did.  This was debated to death long ago,
and more than once, and Guido likes what he likes now.  Greg Wilson made the
only new point on listcomps I've seen since two weeks after they were first
proposed by Greg Ewing (i.e., that the ";" notation *really* sucked).

> Lets say that most my doubts arose after playing with it for
> a while. I fear people will start using it in odd construct, and
> in odd ways,

Unlike, say, filter, map and reduce <wink>?

> expecting other aspects of for-loops to be included
> in list comprehensions (break, else, continue, etc.)

Those ideas were rejected long ago too (and that Haskell and SETL also
rejected them independently shows that, whether we can explain it or not,
they're simply bad ideas).

> And then there's the way it's hard to parse because of the
> lack of punctuation in it:
>
> [((a,b)*c, (spam(d)%34)^e) for a in [(x, y) for x in L for y in
> S] for b in [b for b in B if mean(b)] for b,c in C for a,d in D
> for e in [Egg(a, b, c, d, e) for e in E]]

That isn't a serious argument, to my eyes.  Write that as a Lisp one-liner
and see what it looks like then -- nuts is nuts, and a "scare example" could
just as easily be concocted out of insane nesting of subscripts and/or
function calls and/or parenthesized arithmetic.  Idiotic nesting is a
possibility for any construct that nests!  BTW, you're missing the
possibility to nest listcomps in "the expression" position too, a la

>>> [[1 for i in range(n)] for n in range(10)]
[[],
 [1],
 [1, 1],
 [1, 1, 1],
 [1, 1, 1, 1],
 [1, 1, 1, 1, 1],
 [1, 1, 1, 1, 1, 1],
 [1, 1, 1, 1, 1, 1, 1],
 [1, 1, 1, 1, 1, 1, 1, 1],
 [1, 1, 1, 1, 1, 1, 1, 1, 1]]
>>>

I know you missed that possibility above because, despite your claim of
being hard to parse, it's dead easy to spot where your listcomps begin:  "["
is easy for the eye to find.

> I hope anyone writing something like that (notice the shadowing of
> some of the outer vrbls in the inner loops)

You can find the same in nested lambdas littering map/reduce/etc today.

> will either add some newlines and indentation by themselves, or
> will be hunted down and shot (or at least winged) by the PSU.

Nope!  We just shun them.  Natural selection will rid the Earth of them
without violence <wink>.

> I'm not arguing to remove list comprehensions. I think they are cool
> features that can replace map/filter, I just don't think they're that
> much better than the use of map/filter.

Haskell programmers have map/filter too, and Haskell programmers routinely
favor using listcomps.  This says something about what people who have both
prefer.  I predict that once you're used to them, you'll find them much more
expressive:  "[" tells you immediately you're getting a list, then the next
thing you see is what the list is built out of, and then there's a bunch of
lower-level detail.  It's great.

> Write-that-PEP-Tim-it-will-look-good-on-your-resume-ly y'rs,

except-i'm-too-old-to-need-a-resume-anymore<wink>-ly y'rs  - tim