Ask for help about class variable scope (Re: Why doesn't a dictionary work in classes?)

[eryk sun]

On 12/27/18, Chris Angelico <rosuav at gmail.com> wrote:
>
> Class scope is special, and a generator expression within that class
> scope is special too. There have been proposals to make these kinds of
> things less special, but the most important thing to remember is that
> when you create a generator expression, it is actually a function.
> Remember that a function inside a class statement becomes a method,
> and that inside the method, you have to use "self.X" rather than just
> "X" to reference class attributes. That's what's happening here.

A generator expression is implemented internally as a generator
function that takes an iterator as its only parameter (named ".0") and
gets called immediately to get a generator object. There's some inline
bytecode in the defining scope that sets this up.

A generator object has iterator methods (__iter__, __next__) and
close, send, and throw methods. Its code and execution state in
CPython uses a code object and a frame object:

    >>> g = (c for c in 'spam')
    >>> g.gi_code.co_varnames
    ('.0', 'c')

Initially it hasn't run, so there's no 'c' value yet:

    >>> sorted(g.gi_frame.f_locals)
    ['.0']

'c' is defined after it executes up to the first yield:

    >>> next(g)
    's'
    >>> sorted(g.gi_frame.f_locals)
    ['.0', 'c']

Unlike a function object, a generator object is not a descriptor (i.e.
it has no __get__ method) that could in principle be bound as either a
class or instance method. Anyway, since the class doesn't exist yet,
trying to bind and call a method at this point can't work.

In contrast generator functions are commonly used for methods that can
access class attributes. But this is confusing matters since the class
__dict__ (and certainly not the instance __dict__) that's used for
attributes is not the locals() of the initial class statement
execution. In other words, class attribute access is not a closure
over the class statement scope. For methods, it depends on early
binding of the bound object to a method's __self__ attribute, which is
implicitly passed as the first argument to its __func__ function when
the method is called.

The execution locals of a class statement is a temporary namespace
that gets copied when the class object is instantiated. For example:

    class C:
        exec_dict = locals()

    >>> C.s = 'spam'
    >>> sorted(C.__dict__)
    ['__dict__', '__doc__', '__module__', '__weakref__', 'exec_dict', 's']

The original locals():

    >>> sorted(C.exec_dict)
    ['__module__', '__qualname__', 'exec_dict']

What could be done is for the compiler to introduce nonlocal free
variables (like what's already done with __class__), and then capture
the current value to the locals() dict after it's done executing. For
example (a clumsy one; in practice it would be implicit, expanding on
how __class__ is implemented):

    def make_Foo():
        XS = None
        class Foo:
            nonlocal XS
            XS = [15] * 4
            Z5 = sum(XS[i] for i in range(len(XS)))
            locals()['XS'] = XS
        return Foo

    >>> Foo = make_Foo()
    >>> Foo.Z5
    60
    >>> Foo.XS
    [15, 15, 15, 15]

However, this would be confusing in general, since there's no way to
keep the class attribute in sync with the cell variable. So a function
that's called as a class method may see different values for XS and
cls.XS. This is a bad idea.

This is straying off topic, but note that a consequence of late
binding is that super() can be broken by rebinding __class__:

    class C:
        def f(self):
            nonlocal __class__
            __class__ = None
            super()

    >>> C().f()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 5, in f
    RuntimeError: super(): __class__ is not a type (NoneType)

It's not a bug or design flaw; just an example of code shooting itself
in the foot by stepping on an informally reserved dunder name.