[Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Mon, May 07, 2018 at 10:38:51PM +1000, Chris Angelico wrote:
> On Mon, May 7, 2018 at 9:42 PM, Steven D'Aprano <steve at pearwood.info> wrote:
[...]
> > (3) A compromise: binding assignments are scoped local to the
> > comprehension, but they are initialised from their surrounding scope.
[...]
> Does it HAVE to be initialised from the surrounding scope? What if the
> surrounding scope doesn't have that variable?

No. Then it's just an uninitialised local, and it is a NameError to try 
to evaluate it before something gets bound to it.


> stuff = [spam for x in items if (spam := f(x)) < 0]
> 
> Is this going to NameError if you haven't defined spam?

It shouldn't be an error, because by the time the comprehension looks up 
the value of "spam", it has been bound by the binding-expression.


> Or is the
> compiler to somehow figure out whether or not to pull in a value?
> Never mind about implementation - what are the semantics?

Variable "spam" is not defined in any surrounding scope, so these ought 
to all be NameError (or UnboundLocalError):

    [spam for a in it]  # obviously
    [(spam := spam + a) for a in it]
    [spam if True else (spam := a) for a in it]
    [spam for a in it if True or (spam := a)]

They are errors because the name "spam" is unbound when you do a lookup. 
This is not an error, because the name is never looked up:

    [True or spam for a in it if True or (spam := a)]

Although "spam" never gets bound, neither does it get looked up, so no 
error. The next one is also okay, because "spam" gets bound before the 
first lookup:

    [(spam := spam+1) for a in it if (spam := a*2) > 0]


Here's a sketch of how I think locals are currently handled:

1. When a function is compiled, the compiler does a pass over the
   source and determines which locals are needed.

2. The compiler builds an array of slots, one for each local, 
   and sets the initial value of the slot to "empty" (undefined).

3. When the function is called, if it tries reading from a local's
   slot which is still empty, it raises UnboundLocalError.

(am I close?)

Here's the change I would suggest:

2. The compiler builds an array of slots, one for each local:

2a. For locals that are the target of binding-expression only:

    - look up the target in the current scope (that is, not the
      comprehension's scope, but the scope that the comprehension
      is inside) using the normal lookup rules, as if you were
      compiling "lambda x=x: None" and needed the value of x;

    - if the target is undefined, then swallow the error and leave
      the slot as empty;

    - otherwise store a reference to that value in the slot.

2b. For all other locals, proceed as normal.




> >> Part of it is just that people seem to be fighting for the sake of
> >> fighting.
> >
> > Them's fightin' words! *wink*
> >
> > Honestly Chris, I know this must be frustrating, but I'm not fighting
> > for the sake of it, and I doubt Tim is either. I'm arguing because there
> > are real use-cases which remain unmet if binding-variables inside
> > comprehensions are confined to the comprehension.
> 
> I don't think you are and I don't think Tim is. But do you honestly
> want to say that about EVERY person in these threads?

I'm going to assume good faith, no matter the evidence :-)


-- 
Steve