unintuitive for-loop behavior

[Steve D'Aprano]

On Sun, 2 Oct 2016 09:41 pm, Chris Angelico wrote:

> The only way to prove that something is a new binding is to
> demonstrate that, when this binding is removed, a previous one becomes
> visible. In Python, that only ever happens across namespaces, and in 
> CPython, the only way I can make it happen is with globals->builtins.

It isn't necessary for a previous binding to become visible, because there
might not be any previous binding. There's no builtin 'x', but there is a
builtin 'str'. Hence:

x = str = 1
assert x == 1 and str == 1
del x, str
assert str  # succeeds
assert x  # NameError
x = str = 2  # create new bindings, or update existing ones?

Is it our conclusion that therefore Python creates a new binding for str
but not for x? Or that the evidence for x is "inconclusive"? Either answer
is illogical.

NameError should be sufficient to prove that the binding to 'x' is gone and
therefore any binding to 'x' is semantically a new binding, not an update.



[...]
>> I'm not entirely sure that Guido would say No. I think that "fail early
>> if the local doesn't exist" as CPython does would be permitted rather
>> than mandatory. But I could be wrong.
> 
> Interesting. The real question, then, is: Are function-local names
> inherently special? I can't make anything else do this. Class
> namespaces don't shadow globals or builtins, even during construction:

Actually, they do:

py> class Example:
...     str = 1
...     x = str(999)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Example
TypeError: 'int' object is not callable


Locals are certainly special. The language definition singles out binding[1]
to a variable inside a function as the trigger to treat it as a local, so
there are distinct "lookup name which is a local" and "lookup name which is
not a local" modes. Those distinct modes go back to at least Python 1.5.

"Local lookup" (LOAD_FAST) only looks for a local, and fails fast if it
isn't found, rather than searching globals and builtins; "not a local
lookup" never looks for a local (even if one exists -- see below).


>> By the way, here's an example showing that IronPython does allowing
>> writing to locals to affect the local namespace:
>>
>>>>> def func():
>> ...     locals()['x'] = 1
>> ...     print(x)
>> ...     if False:
>> ...         x = 9999
>> ...
>>>>> func()
>> 1
>>
>> And *that* behaviour is most definitely allowed -- the fact that writing
>> to locals() isn't supported by CPython is most definitely an
>> implementation- specific limitation, not a feature.
> 
> Yep. Would the same have happened if you'd omitted the "if False"
> part, though - that is, if 'x' were not a local name due to
> assignment, could it *become* local through mutation of locals()?

Yes the local would have been created, but you wouldn't see it from an
ordinary lookup of name x. You would need to use

    locals()['x']

to see that it exists. For x to return it, you have to convince the compiler
to use LOAD_FAST bytecode rather than LOAD_whatever.

IronPython doesn't use LOAD_* bytecodes, it compiles to whatever .Net's CLR
uses, but it follows the same semantics. Likewise Jython.

My feeling is that we're in a rather murky part of Python's behaviour here,
one where a clean name lookup model has been most definitely put aside in
favour of speed, but it isn't quite clear whether this is defined language
behaviour, reference behaviour that all implementations are required to
follow, or implementation-specific behaviour that may be varied but so far
hasn't been, at least not by any of the Big Four implementations (CPython,
PyPy, IronPython, Jython).





[1] For the purposes of deciding whether something is local or not,
unbinding (del) is considered a binding.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.