[Python-Dev] listcomp / par-for (summary)

[Ka-Ping Yee]

On Wed, 12 Jul 2000, Thomas Wouters wrote:
> As said, the currently implemented list-comprehension syntax was too
> unclear, especially when nesting the 'for's and 'if's too deeply. One of the
> proposals was to enforce the use of ()'s and add commas to make the 'for's
> and 'if's more readable:
> 
> [(x,y) for x in a, y in b, if y > 3]
> 
> That is not going to work.

Actually, it will (read on).

> Enforcing the parentheses is possible, but only
> if you *always* enforce them, even if they don't create a tuple:
> 
> [([x,y]) for x in a]

They don't always have to be enforced (read on).

> Secondly, the use of commas to seperate the for and if statements is not
> possible. The parser will see the commas as part of the iter-list of the
> previous for statement.

No, this really can be done.

I was kinda suspicious when i read the above statements, so i
proceeded to hack the grammar to see if i could implement the
syntax i proposed.  It is possible, and it turns out it's quite
easy and straightforward.

(No personal attack intended, Thomas -- sorry -- just wanted to
show that it *is* possible.)

All of the following syntaxes are possible (i've tried and
tested a Python capable of each one):

   1. [ <test> <for-or-if-stmt1> <for-or-if-stmt2> <...> ]
   2. [ <test>, <for-or-if-stmt1>, <for-or-if-stmt2>, <...> ]
   3. [ <test>; <for-or-if-stmt1>, <for-or-if-stmt2>, <...> ]
   4. [ <test>; <for-or-if-stmt1>; <for-or-if-stmt2>; <...> ]

For example, using a Python with "comma-comma" syntax (number 2), i get:

    >>> a = range(5)
    >>> b = range(10, 50, 10)
    >>> a
    [0, 1, 2, 3, 4]
    >>> b
    [10, 20, 30, 40]
    >>> [x*2, for x in a]
    [0, 2, 4, 6, 8]
    >>> [x, y, for x in a, for y in b, if y > 3]
      File "<stdin>", line 1
        [x, y, for x in a, for y in b, if y > 3]
                 ^
    SyntaxError: invalid syntax
    >>> [(x, y), for x in a, for y in b, if y > 3]
    [(0, 10), (0, 20), (0, 30), (0, 40), (1, 10), (1, 20), (1, 30), (1, 40), (2, 10), (2, 20), (2, 30), (2, 40), (3, 10), (3, 20), (3, 30), (3, 40), (4, 10), (4, 20), (4, 30), (4, 40)]
    >>> [[x, y], for x in a]
    [[0, 40], [1, 40], [2, 40], [3, 40], [4, 40]]
    >>> [([x, y]), for x in a]
    [[0, 40], [1, 40], [2, 40], [3, 40], [4, 40]]
    >>> [([x, y],), for x in a]
    [([0, 40],), ([1, 40],), ([2, 40],), ([3, 40],), ([4, 40],)]
    >>> def marry(*args): return apply(map, (None,)+args)[:min(map(len, args))]
    ... 
    >>> [x + y, for x in a, for y in b]
    [10, 20, 30, 40, 11, 21, 31, 41, 12, 22, 32, 42, 13, 23, 33, 43, 14, 24, 34, 44]
    >>> [x + y, for x, y in marry(a, b)]
    [10, 21, 32, 43]
    >>> 

The above shows that we can indeed enforce parentheses around the
initial test, while not requiring extra parentheses when no tuple
is being constructed.

Similarly, as you would expect, using a comma between clauses
requires you to add parens if the expression after "in" contains
bare commas.  Otherwise everything works normally.

I also tested stuff like [] [1] [1,] [1,,] [1,2] [1,2,] [1,2,,]
and the modified Pythons behaved fine.

The grammar rules for comma-comma syntax are:

    atom: '(' [testlist] ')' | '[' [test [',' (testlist | list_iter)]] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+
    list_iter: (list_for | list_if)
    list_for: 'for' exprlist 'in' test [',' list_iter]
    list_if: 'if' test [',' list_iter]

Or for semicolon-comma syntax (number 3 above):

    atom: '(' [testlist] ')' | '[' [testlist [';' list_iter]] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+
    list_iter: (list_for | list_if)
    list_for: 'for' exprlist 'in' test [',' list_iter]
    list_if: 'if' test [',' list_iter]

I won't post the changes to Python/compile.c for each case here,
but they were pretty minor changes from Greg Ewing's original patch.

Comma-comma syntax is my current favourite, with semicolon-comma
a close second (but definitely second).


-- ?!ng