[Python-ideas] Explicit variable capture list

[Andrew Barnert]

On Jan 25, 2016, at 15:21, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> Excellent summary, thank you, but I want to take exception to something 
> you wrote. I fear that you have inadvertently derailed the thread into a 
> considerably narrower focus than it should have.
> 
>> On Fri, Jan 22, 2016 at 08:50:52PM -0800, Andrew Barnert wrote:
>> 
>> What the thread is ultimately looking for is a solution to the 
>> "closures capturing loop variables" problem. This problem has been in 
>> the official programming FAQ[1] for decades, as "Why do lambdas 
>> defined in a loop with different values all return the same result"?
> 
> The issue is not loop variables, or rather, it's not *only* loop 
> variables, and so any solution which focuses on fixing loop variables is 
> only half a solution.

I think it really _is_ only loop variables--or at least 95% loop variables.

> ... Outside of such toys, we often find ourselves closing 
> over at least one variable which is derived from the loop variable, but 
> not the loop variable itself:

But, depending on how you write that, either (a) it already works the way you'd naively expect, or (b) the only reason you'd expect it to work is if you don't understand Python scoping (that is, you think every block is a scope).

That's different from the case with loop variables: even people who know Python scoping still regularly make the mistake with loop variables, swear at themselves, and write the default-value trick on the first debug pass. (Novices, of course, swear at themselves, try 28 random changes, then post their code on StackOverflow titled "Why Python closures does suck this way?")

It's the loop variable problem that's in the FAQ. And it does in fact come up all the time in some kinds of programs, like Tkinter code that wants to create callbacks for each of 10 buttons. And again, looking at other languages, it's the loop variable problem that's in their FAQs, and the new-variable-per-instance solution would work across most of them, and is actually used in some.

Again, I definitely acknowledge that Python's non-granular scopes make the issue much less clear-cut than in those languages where "key = API_KEYS[provider]" would actually work. That's why I said that if there's one mainstream language that _shouldn't_ use my solution, it's Python.

And, ultimately, I'm still -0 about any change--the default-value solution has worked for decades, everyone who uses Python understands it, and there's no serious problem with it.

But I think "capture by value" or "capture early" would, outside the loop-variable case, be more often an attractive nuisance for code you shouldn't be writing than a help for code you should.

If you think we _should_ solve the problem with "loop-body-local" variables, that would definitely be an argument for Nick's "define and call a function" implementation over the new-cell implementation, because his version does actually define a new scope, and can easily be written to make those variables actually loop-body-local. 

However, I think that, if we wanted that, it would be better to have a more general solution--maybe a "scope" statement that defines a new scope for its suite, or even a "let" statement that defines a new variable only until the end of the current suite.

Or, of course, we could toss this on the large pile of "problems that would be solved by light-weight multi-line lambda" (and I think it doesn't add nearly enough weight to that pile to make the problem worth solving, either).

>> The OP proposed that we should add some syntax, borrowed from C++, to 
>> function definitions that specifies that some things get captured by 
>> value.
> [...]
> 
> Regardless of the syntax chosen, this has a few things to recommend it:
> 
> - It's completely explicit. If you want a value captured, you 
> have to say so explicitly, otherwise you will get the normal variable 
> lookup behaviour that Python uses now.

Surely "for new i" is just as explicit about the fact that the variable is "special" as "def f(x; i):" or "sharedlocal i"? The difference is only _where_ it's marked, not _whether_ it's marked.

> - It's general. We can capture locals, nonlocals, globals or builtins, 
> not just loop variables.

Sure, but it may be an overly-general solution to a very specific problem. If not, then great, but... Do you really have code that would be clearer if you could capture a global variable by value?

(Of course there's code that does that as an optimization--but that's not to make the code clearer; it's to make the code slightly faster despite being less clear.)

> - It allows us to avoid the "default argument" idiom, in cases where we 
> really don't want the argument, we just want to capture the value. There 
> are a lot of functions which have their parameter list polluted by 
> extraneous arguments that should never be used by the caller simply 
> because that's the only way to get early binding/value capturing.

It's not the _only_ way. When you really want a new scope, you can always define and call a local function. Or, usually better, refactor things so you're calling a global function, or using an object, or some other solution. The default-value idiom is just the most _concise_ way.

Meanwhile, have you ever actually had a bug where someone passed an override for the i=i or len=len parameter? I suspect if people really were worried about that, they would use "*, _spam=spam", but they never do. (The only place I've seen anything like that is in generated code--e.g., a currying macro.)

>> For 
>> backward-compatibility reasons, this might have to be optional, which 
>> means new syntax; he proposed "for new i in range(10):".
> 
> I would not like to see "new" become a keyword. I have a lot of code 
> using new (and old) as a variable.

I've even got some 2.5 code that runs in 3.3+ thanks to modernize, but still uses the "new" module. :)

Of course it could become a context-sensitive keyword, like async. But yeah, that seems more like a last-resort idea than something to emulate wherever possible...