Special attributes added to classes on creation

[eryk sun]

On Sun, Jul 3, 2016 at 4:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Is there any documentation for exactly what keys are added to classes when?

It should be documented that the namespace that's passed to the
metaclass contains __module__ and __qualname__, and optionally __doc__
if the class has a docstring.

These names are also available as local variables while executing the
class body:

    >>> class C:
    ...     'eggs and spam'
    ...     print('__module__:', __module__)
    ...     print('__qualname__:', __qualname__)
    ...     print('__doc__:', __doc__)
    ...
    __module__: __main__
    __qualname__: C
    __doc__: eggs and spam

This is potentially an (unlikely) issue because the class body
preamble overwrites whatever custom values of these attributes were
added by the metaclass __prepare__ method. For example:

    class Meta(type):
        @classmethod
        def __prepare__(mcls, name, bases, **kwds):
            return {'__module__': 'spam',
                    '__qualname__': 'eggs.spam.' + name,
                    '__doc__': 'spam spam spam',
                    'x': 42}

    class C(metaclass=Meta):
        'docstring'

    >>> C.__module__
    '__main__'
    >>> C.__qualname__
    'C'
    >>> C.__doc__
    'docstring'
    >>> C.x
    42

I don't think additional documentation is required for the __dict__
and __weakref__ descriptors. They're described with respect to
__slots__. They can't be added until after the metaclass processes
__slots__, if present.

You can begin unraveling how CPython 3.x class creation works by
disassembling the bytecode for the following example:

    def f():
        class Q(Base, metaclass=Meta):
            'example class'
            x = 1
            def method(self):
                __class__ # or use super()

The initial work of defining a class is implemented via
builtins.__build_class__, which is referenced by the bytecode
instruction LOAD_BUILD_CLASS. This built-in function takes two
required positional arguments: the argumentless function that
implements the class body and the class name. Optionally, the base
class(es) are passed as the remaining positional arguments, and the
metaclass is passed as a keyword argument. Additional keyword
arguments can be passed in the class statement, but this requires a
custom metaclass with __new__ and __init__ methods that accept the
extra arguments. The metaclass __prepare__ method also gets passed the
name, bases and extra keyword arguments.

Here's the bytecode setup to call __build_class__ for the above example:

    >>> dis.dis(f)
    2    0 LOAD_BUILD_CLASS
         2 LOAD_CONST          1 (<code object Q ...>)
         4 LOAD_CONST          2 ('Q')
         6 MAKE_FUNCTION       0
         8 LOAD_CONST          2 ('Q')
        10 LOAD_GLOBAL         0 (Base)
        12 LOAD_CONST          3 ('metaclass')
        14 LOAD_GLOBAL         1 (Meta)
        16 EXTENDED_ARG        1
        18 CALL_FUNCTION     259 (3 positional,
                                  1 keyword pair)
        20 STORE_FAST          0 (Q)

        22 LOAD_CONST          0 (None)
        24 RETURN_VALUE

The locals mapping for the class body comes from the metaclass
__prepare__ method, if defined, and is otherwise an empty dict.
There's a default type.__prepare__, which returns an empty dict:

    >>> type.__prepare__(1, 2, 3, spam='whatever')
    {}

The code for the class body defines the __module__, __qualname__, and
__doc__ attributes. It also defines the __class__ closure when a class
has methods that use super() or reference __class__.

The return value of the class body is either the __class__ cell
object, if defined, or None. Returning the __class__ closure cell
allows __build_class__  to set the new class as the value of the cell.

Here's the bytecode for the body of class Q:

    >>> dis.dis(f.__code__.co_consts[1])
    2        0 LOAD_NAME           0 (__name__)
             2 STORE_NAME          1 (__module__)
             4 LOAD_CONST          0 ('f.<locals>.Q')
             6 STORE_NAME          2 (__qualname__)
    3        8 LOAD_CONST          1 ('example class')
            10 STORE_NAME          3 (__doc__)
    4       12 LOAD_CONST          2 (1)
            14 STORE_NAME          4 (x)
    5       16 LOAD_CLOSURE        0 (__class__)
            18 BUILD_TUPLE         1
            20 LOAD_CONST          3 (<code object method ...>)
            22 LOAD_CONST          4 ('f.<locals>.Q.method')
            24 MAKE_FUNCTION       8
            26 STORE_NAME          5 (method)
            28 LOAD_CLOSURE        0 (__class__)
            30 RETURN_VALUE

    >>> f.__code__.co_consts[1].co_cellvars
    ('__class__',)

The rest of the process is all written in C.

compiler_class, which compiles a class statement to the above bytecode:
https://hg.python.org/cpython/file/v3.6.0a2/Python/compile.c#l1809

builtins.__build_class__:
https://hg.python.org/cpython/file/v3.6.0a2/Python/bltinmodule.c#l53

type.__new__:
https://hg.python.org/cpython/file/v3.6.0a2/Objects/typeobject.c#l2268

In type_new, the class dict gets initialized starting on line 2516.
__module__, if not already set, is the value of __name__ in the
current globals. __qualname__, if not already set, defaults to the
class name.

The getset descriptors for __dict__ and __wreakref__, if present, are
added to the class on line 2625, but they're not added to the class
dict until the type is made ready via PyType_Ready. On line 4886,
add_getset is called to add them. The names "__dict__" and
"__weakref__" are defined by subtype_getsets_full, etc, starting on
line 2154. Also in PyType_Ready, on line 4937, __doc__ is added if
it's not already present in the class dict.