Proposed new syntax

[Steve D'Aprano]

On Fri, 18 Aug 2017 04:16 pm, Paul Rubin wrote:

[...]
> Similarly we occasionally have to be aware of the procedural nature
> of Python list comprehensions, but most of the time we think of them
> in terms of the mathematical abstraction they are designed to resemble.


Thanks Paul, you've given me an insight into the disagreement. I strongly
disagree with your "most of the time" but I can see some justification for the
view.

I think you have hit the nail on the head regarding the argument over
comprehensions. Can we understand comprehensions as an abstraction? Yes we can:

result = [ ... magic happens in here ... ]

We can ignore the insides of the comprehension (I don't mean the implementation,
I mean literally the stuff between the square brackets) and treat it as a black
box that returns a list. Just as we can ignore the insides of a function call:

result = make_list(...)

Of course we can gloss over the fact that comprehensions are explicitly written
as iteration and ignore the details, just as we can ignore the details of any
piece of code:

result = []  # Oh, it's going to be a list.
for x in stuff:
    ...magic happens here...
print(result)  # Some sort of list. Probably.

and of course we gloss over code all the time, ignoring the details that aren't
important at the moment.

"Details, details, don't bother me with details!"

So from *that* perspective of taking a birds-eye view of the code, I'll grant
that if you don't care about the details of what the comprehension returns, we
can gloss over it and treat it as a magic black box that returns a list, and we
don't care how it was generated: recursively, iteratively, using GOTO or an
unrolled loop, or black magic, it doesn't matter and we don't care.

But only to a point. If all you care about is "it returns a list", then that's
fine. But of course people don't really care about *only* that, they also care
about what it contains, and that the comprehension or generator expression
iterates over its argument in a specific order. If we write:

[process(a) for a in (1, 2, 3)]

then we typically expect 1 to be processed before 2, and 3 last of all. If
you've ever iterated over a file in a comprehension, you have relied on that
fact, whether you realised or not, and even if process() has no side-effects
and you don't care about the order of evaluation per se, in general we care
about the order that the results are returned.

I dare say that there are cases where people would happily replace their list
comprehension with a parallel map() that doesn't guarantee either the order of
evaluation or the order the results are returned, given some sufficient speed
up. But that's a special case, not a universal: most of the time, we expect our
data to be processed in the order we give it, not in some arbitrary order, and
even if a parallel map was available we'd prefer iteration because it is more
easily understood and debugged and matches most people's expectations of
sequential processing.

For example, given:

[format(linenum) + line for (linenum, line) in enumerate(myfile)]

I think that most people would be disturbed if the returned list didn't match
the order of lines in the file.

With the simplest comprehensions, those with a single for loop, its easily to
let your eyes slide over the explicit iteration syntax and read it as some sort
of funny map syntax:

result = [expression ... iterable]  # [] means "map"

But for more complex comprehensions, that's much harder, if not impossible, and
you cannot get away from the execution details, and *that* is explicitly
written as iteration using explicit for-loops and conditional if statements.
Not as nested maps, or as recursion, or as a parallel set of function calls.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.