[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Nick Coghlan]

On 23 June 2018 at 13:48, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:
> [...]
>> plenty of
>> functional-language-inspired documentation to instead encourage folks
>> to view comprehensions as tightly encapsulated declarative container
>> construction syntax.
>
> I can't say I've done a broad survey, but the third-party documentation
> I've read on comprehensions typically glosses over the scoping issues
> without mentioning them. To the extent that scoping is even hinted at,
> comprehensions are treated as expressions which are exactly equivalent
> to re-writing them as a for-loop in the current scope.
>
> This is a typical example, found as the top result on googling for
> "python comprehensions":
>
> https://www.google.com/search?q=python+comprehensions
>
> http://www.pythonforbeginners.com/basics/list-comprehensions-in-python
>
> Nothing is mentioned about scope, and it repeats the inaccurate but
> simple equivalency:
>
> for item in list:
>     if conditional:
>         expression
>
> But perhaps that tutorial is too old. Okay this recent one is only a
> little more than a year old:
>
> https://hackernoon.com/list-comprehension-in-python-8895a785550b
>
> Again, no mention of scoping issues, comprehensions are simply
> expressions which presumably run in the same scope as any other
> expression.
>
> I think you over-estimate how many newcomers to Python are even aware
> that the scope of comprehensions is something to consider.

I put quite a bit of work into making it possible for folks to gloss
over the distinction and still come to mostly-correct conclusions
about how particular code snippets would behave.

I was only able to achieve it because the folks that designed lexical
scoping before me had made *read* access to lexical scopes almost
entirely transparent, and because generator expressions were designed
to fail fast if there was a bug in the expression defining the
outermost iterable (which meant that even at class scope, the
outermost iterable expression still had access to class level
variables, because it was evaluated *outside* the nested scope).

*Write* access to lexically nested scopes, by contrast, was omitted
entirely from the original lexical scoping design, and when it was
later added by https://www.python.org/dev/peps/pep-3104/, it was done
using an explicit "nonlocal" declaration statement (expressly akin to
"global"), and PEP 3099 explicitly ruled out the use of ":=" to
implicitly declare the target name as being non-local.

PEP 572 is thus taking the position that:

- we now want to make write access to outer scopes implicit (despite
PEP 3099 explicitly ruling that out as desired design feature)
- but only in comprehensions and generator expressions (not lambda
expressions, and not full nested functions)
- and only for assignment expressions, not for loop iteration variables
- and we want it to implicitly choose between a "global NAME"
declaration and a "nonlocal NAME" declaration based on where the
comprehension is defined
- and this is OK because "nobody" actually understands how
comprehensions hide the iteration variable in practice, and
"everybody" thinks they're still a simple for loop like they were in
Python 2
- the fact that the language reference, the behaviour at class scopes,
the disassembly output, and the behaviour in a debugger all indicate
that comprehensions are full nested scopes isn't important

This level of additional complication and complexity in the scoping
semantics simply isn't warranted for such a minor readability
enhancement as assignment expressions.

Cheers,
Nick.

P.S. "You did such a good job of minimising the backwards
compatibility breakage when we changed the semantics of scoping in
comprehensions that we now consider your opinion on reasonable scoping
semantics for comprehensions to be irrelevant, because everybody else
still thinks they work the same way as they did in Python 2" is such a
surreal position for folks to be taking that I'm having trouble
working out how to effectively respond to it.

Guido has complained that "I keep changing my mind about what I want",
but that's not what's actually going on: what I want is to keep folks
from taking our already complicated scoping semantics and making it
close to impossible for anyone to ever infer how they work from
experimentation at the interactive prompt. That goal has pretty much
stayed consistent since the parent local scoping proposal was first
put forward.

What keeps changing is my tactics in pursuing that goal, based on my
current perception of what the folks pushing that proposal are
actually trying to achieve (which seems to be some combination of "We
want to pretend that the Python 3 scoping changes in comprehensions
never happened, but we still want to avoid leaking the iteration
variables somehow" and "We want to enable new clever tricks with state
export from comprehensions and generator expressions"), as well as
repeatedly asking myself what *actually* bothers me about the proposal
(which I've now distilled down to just the comprehension scoping
issue, and the reliance on an arbitrary syntactic restriction against
top level usage to avoid competing with traditional assignment
statements).

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia