Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

[Steven D'Aprano]

On Fri, 21 Mar 2014 19:06:06 -0700, Rustom Mody wrote:

> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior
> when lambdas are put in a comprehension

I don't know why you say the behaviour in Python is a problem. It's 
somewhat unexpected if you don't think about what's going on, but it's 
actually quite logical. On the contrary, I find the Haskell version weird.


>>>> fl = [lambda y : x+y for x in [1,2,3]] 
>>>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

This makes perfect sense: by the time you call the functions, the name x 
has been rebound to the value 3. If x were a global, or if comprehensions 
leaked their variable, that would be obvious: you could print x and see 
exactly what value it has. But visible or not, that's exactly what 
happens. Since x is 3, you're adding 3+2 and should get 5 no matter which 
function you call.

Unroll the first comprehension, and it's obvious what is going on:

def unrolled():
    fl = []
    x = 0
    def lambda_(y):
        return x + y
    fl.append(lambda_)
    x = 1
    def lambda_(y):
        return x + y
    fl.append(lambda_)
    x = 2
    def lambda_(y):
        return x + y
    fl.append(lambda_)
    return [f(2) for f in fl]

Don't be fooled by the coincidence that the three "lambda_" functions 
have the same name and body. They could have different names and 
different bodies, Either way, it is completely natural that they all 
closures over the same x -- that's how you wrote them.


But the Haskell version, on the other hand, is just weird:

> The same in haskell:
> 
> Prelude> let fl = [\ y -> x + y | x <- [1,2,3]] 
> Prelude> [(fl!!i) 0 | i<- [0,1,2]]
> [1,2,3]

For this to be the case, the functions in fl have to somehow "look back 
in time" to see the value of x, not as it is *now* when the function is 
called, but how it *was* when it was created. That's very weird indeed. 
If x were a global, it would be astonishing. The fact that x comes from a 
closure instead makes it no less surprising. Unroll the loop, and the 
magic is obvious.

Now I'm not sure precisely how Haskell implements this trick, but it 
suggests to me that it creates a different closure each time around the 
loop of the comprehension. That could end up being very expensive.



-- 
Steven D'Aprano
http://import-that.dreamwidth.org/