[Python-ideas] Explicit variable capture list

[Guido van Rossum]

On Sat, Jan 23, 2016 at 3:53 AM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> Andrew Barnert via Python-ideas writes:
>
>  >     powers = [lambda x: x**i for i in range(10)]
>
>  > This gives you ten functions that all return x**9, which is
>  > probably not what you wanted.
>
>  > The reason this is a problem is that Python uses "late binding",
>  > which in this context means that each of those functions is a
>  > closure that captures the variable i in a way that looks up the
>  > value of i at call time. All ten functions capture the same
>  > variable, and when you later call them, that variable's value is
>  > 9.
>

Actually it doesn't look up the value at call time, but each time it's
used. This technicality matters if in between uses you call something that
has write access to the same variable (typically using nonlocal) and
modifies it.


> But this explanation going to confuse people who understand the
> concept of variable in Python to mean names that are bound and
> re-bound to objects.  The comprehension's binding of i disappears
> before any element of powers can be called.  So from their point of
> view, either that expression is an error, or powers[i] closes over a
> new binding of the name "i", specific to "the lambda's scope" (see
> below), to the current value of i in the comprehension.
>

But this seems to refer to a very specific definition of "binding" that
doesn't have root in Python's semantic model. I suppose it may come from
Lisp (which didn't influence Python quite as much as people think :-).

So I think what you're saying here comes down that it will confuse people
who misunderstand Python's variables. Given that the misunderstanding
you're supposing here is pretty specific (it's not just due to people
who've never thought much about variables) I'm not sure I care much.


> Of course the same phenomenon is observable with other scopes.  In
> particular global scope behaves this way, as importing this file
>
>     i = 0
>     def f(x):
>         return x + i
>     i = 1
>
> and calling f(0) will demonstrate.  But changing the value of a
> global, used the way i is here, within a library module is a rather
> unusual thing to do; I doubt people will observe it.
>

I disagree again: in interactive mode most of what you do is global and you
will see this quite often.

And all scopes in Python behave the same way.


> Also, once again the semantics of lambda (specifically, that unlike
> def it doesn't create a scope)


Uh, what? I can sort of guess what you are referring to here (namely, that
no syntactic construct permissible in a lambda can assign to a local
variable -- or any variable, for that matter) but it certainly has a scope
(to hold the arguments, which are just variables, as one quickly learns
from experimenting with the arguments to a function defined using def).


> seem to be a source of confusion more
> than anything else.  Maybe it's possible to exhibit the same issue
> with def, but the def equivalent to the above lambda
>
>     >>> def make_increment(i):
>     ...  def _(x):
>     ...   return x + i
>     ...  return _
>     ...
>     >>> funcs = [make_increment(j) for j in range(3)]
>     >>> [f(0) for f in funcs]
>     [0, 1, 2]
>
> closes over i in the expected way.  (Of course in practicality, it's
> way more verbose, and in purity, it's not truly equivalent since
> there's at least one extra nesting of scope involved.)


It's such a strawman that I'm surprised you bring it up. Who would even
*think* of using that idiom as equivalent to the simple lambda?

If I were to deconstruct the original statement, I would start by replacing
the list comprehension with a plain old for loop. That would also not be
truly equivalent because the comprehension introduces a scope while the for
loop doesn't, but the difference only matters if it stomps on another
variable -- the semantics relative to the lambda are exactly the same. In
particular, this example exhibits the same phenomenon without using a
comprehension:

powers = []
for i in range(10):
    powers.append(lambda x: x**i)

This in turn can be rewritten without changing the semantics related to
scopes using a def that's equivalent (truly equivalent except for its
__name__ attribute!):

powers = []
for i in range(10):
    def f(x):
        return x**i
    powers.append(f)

(Note that the leakage of f here is irrelevant to the problem.)

This has the same problem, without being distracted by lambda or
comprehensions, and we can now explore its semantics through
experimentation. We could even unroll the for loop and get the same issue:

powers = []

i = 0
def f(x):
    return x**i
powers.append(f)

i = 1
def f(x):
    return x**i
powers.append(f)

# Etc.


> While
>
>     >>> def make_increment():
>     ...  def _(x):
>     ...   return x + i
>     ...  return _
>     ...
>     >>> funcs = [make_increment() for i in range(3)]
>     >>> [f(0) for f in funcs]
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>       File "<stdin>", line 1, in <listcomp>
>       File "<stdin>", line 3, in _
>     NameError: name 'i' is not defined
>     >>> i = 6
>     >>> [f(0) for f in funcs]
>     [6, 6, 6]
>
> doesn't make closures at all, but rather retains the global binding.
>

Totally different idiom again -- another strawman.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160123/6aa65264/attachment-0001.html>