[Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Mon, May 07, 2018 at 12:48:53PM +1000, Chris Angelico wrote:
> On Mon, May 7, 2018 at 12:34 PM, Tim Peters <tim.peters at gmail.com> wrote:

> > There's a difference, though:  if `y` "leaks", BFD.  Who cares? ;-)
> > If `y` remains inaccessible, there's no way around that.
> 
> That's Steve D'Aprano's view - why not just let them ALL leak? I don't
> like it though.

I know popular opinion is against me, and backward compatibility and all 
that, but I wish that generator expressions and comprehensions ran
in their surrounding scope, like regular for statements.

(Yes, I know that makes generator expressions tricky to implement. As 
the guy who doesn't have to implement it, I don't have to care :-)

Calling it a "leak" assumes that it is a bad thing. I don't think it is 
a bad thing. It's not often that I want to check the value of a 
comprehension loop, but when I do, I have to tear the comprehension 
apart into a for-loop. Even if it is only temporarily, for debugging, 
then put the comprehension back together.

The only time I can see it is a bad thing is if I blindly copy and paste 
a comprehension out of one piece of code and dump it into another piece 
of code without checking to see that it slots in nicely without blowing 
away existing variables.

But if you're in the habit of blindly and carelessly pasting into your 
code base without looking it over, this is probably the least of your 
worries... *wink*

But what's done is done, and while there are plenty of windmills I am 
willing to tilt at, reversing the comprehensions scope decision is not 
one of them.


[...]
> Sorry, I meant "local to the comprehension's scope". We can't know the
> user's intention. We have to create semantics before the user's
> intention even exists.

Surely that's backwards? We ought to find out what people want before 
telling them that they can't have it :-)


> > But the point above remains:  if they don't leak, contexts that want
> > them to leak have no recourse.  If they do leak, then the other uses
> > would still work fine, but they'd possibly be annoyed by a leak they
> > didn't want.

Indeed.


> Then let's revert the Py3 change that put comprehensions into
> functions, and put them back to the vanilla transformation:

You know we can't do that. But we do have a choice with binding 
expressions. *Either way*, whatever we do, we're going to upset 
somebody, so we simply have to decide who that will be.

Actually we have at least three choices:


(1) Consistency Über Alles (whether foolish or not)

Now that comprehensions are their own scope, be consistent about it. 
Binding expressions inside the comprehension will be contained to the 
comprehension. I'll hate it, but at least it is consistent and easy to 
remember: the entities which create a new scope are modules, classes, 
functions, plus comprehensions.

That's going to cut out at least one motivating example though. See 
below.


(2) Binding assignments are *defined* as "leaking", or as I prefer, 
defined as existing in the lexical scope that contains the 
comprehension. Hence:

# module level
[(x := a) for a in [98, 99]]
assert x == 99

# class level
class X:
    [(x := a) for a in [98, 99]]
    
assert X.x == 99

# function level
def func():
    [(x := a) for a in [98, 99]]
    assert x == 99

Note that in *all* of these cases, the variable a does not "leak".

This will helpfully support the "running total" use-case that began this 
whole saga:

    total = 0
    running_totals = [(total := total + x) for x in [98, 99]]
    assert total == 197

(I have to say, given that this was THE motivating use-case that began 
this discussion, I think it is ironic and not a little sad that the PEP 
has evolved in a direction that leaves this use-case unsatisfied.)


(3) A compromise: binding assignments are scoped local to the 
comprehension, but they are initialised from their surrounding scope.

This would be similar to the way Lua works, as well as function 
parameter defaults.

I have some vague ideas about implementation, but there's no point 
discussing that unless people actually are interested in this option.

This will *half* satisfy the running-total example:

    total = 0
    running_totals = [(total := total + x) for x in [98, 99]]
    assert total == 0


Guaranteed to generate at least two Stackoverflow posts a month 
complaining about it, but better than nothing :-)



> Having
> the iteration variable NOT leak means it's a self-contained unit that
> simply says "that thing we're iterating over".

Assuming there are no side-effects to any of the operations inside the 
comprehension.



> Part of it is just that people seem to be fighting for the sake of
> fighting.

Them's fightin' words! *wink*

Honestly Chris, I know this must be frustrating, but I'm not fighting 
for the sake of it, and I doubt Tim is either. I'm arguing because there 
are real use-cases which remain unmet if binding-variables inside 
comprehensions are confined to the comprehension.



-- 
Steve