Difference in behavior of GenericMeta between 3.6.0 and 3.6.1

[Oren Ben-Kiki]

TL;DR: We need improved documentation of the way meta-classes behave for
generic classes, and possibly reconsider the way "__setattr__" and
"__getattribute__" behave for such classes.

I am using meta-programming pretty heavily in one of my projects.
It took me a while to figure out the dance between meta-classes and generic
classes in Python 3.6.0.

I couldn't find good documentation for any of this (if anyone has a good
link, please share...), but with a liberal use of "print" I managed to
reverse engineer how this works. The behavior isn't intuitive but I can
understand the motivation (basically, "type annotations shall not change
the behavior of the program").

For the uninitiated:

* It turns out that there are two kinds of instances of generic classes:
the "unspecialized" class (basically ignoring type parameters), and
"specialized" classes (created when you write "Foo[Bar]", which know the
type parameters, "Bar" in this case).

* This means the meta-class "__new__" method is called sometimes to create
the unspecialized class, and sometimes to create a specialized one - in the
latter case, it is called with different arguments...

* No object is actually an instance of the specialized class; that is, the
"__class__" of an instance of "Foo[Bar]" is actually the unspecialized
"Foo" (which means you can't get the type parameters by looking at an
instance of a generic class).

So far, so good, sort of. I implemented my meta-classes to detect whether
they are creating a "specialized" or "unspecialized" class and behave
accordingly.

However, these meta-classes stopped working when switching to Python 3.6.1.
The reason is that in Python 3.6.1, a "__setattr__" implementation was
added to "GenericMeta", which redirects the setting of an attribute of a
specialized class instance to set the attribute of the unspecialized class
instance instead.

This causes code such as the following (inside the meta-class) to behave in
a mighty confusing way:

    if is-not-specialized:
        cls._my_attribute = False
    else:  # Is specialized:
        cls._my_attribute = True
        assert cls._my_attribute  # Fails!

As you can imagine, this caused us some wailing and gnashing of teeth,
until we figured out (1) that this was the problem and (2) why it was
happening.

Looking into the source code in "typing.py", I see that I am not the only
one who had this problem. Specifically, the implementers of the "abc"
module had the exact same problem. Their solution was simple: the
"GenericMeta.__setattr__" code explicitly tests whether the attribute name
starts with "_abc_", in which case it maintains the old behavior.

Obviously, I should not patch the standard library typing.py to preserve
"_my_attribute". My current workaround is to derive from GenericMeta,
define my own "__setattr__", which preserves the old behavior for
"_my_attribute", and use that instead of the standard GenericMeta
everywhere.

My code now works in both 3.6.0 and 3.6.1. However, I think the following
points are worth fixing and/or discussion:

* This is a breaking change, but it isn't listed in
https://www.python.org/downloads/release/python-361/ - it should probably
be listed there.

* In general it would be good to have some documentation on the way that
meta-classes and generic classes interact with each other, as part of the
standard library documentation (apologies if it is there and I missed it...
link?)

* I'm not convinced the new behavior is a better default. I don't recall
seeing a discussion about making this change, possibly I missed it (link?)

* There is a legitimate need for the old behavior (normal per-instance
attributes). For example, it is needed by the "abc" module (as well as my
project). So, some mechanism should be recommended (in the documentation)
for people who need the old behavior.

* Separating between "really per instance" attributes and "forwarded to the
unspecialized instance" attributes based on their prefix seems to violate
"explicit is better than implicit". For example, it would have been
explicit to say "cls.__unspecialized__.attribute" (other explicit
mechanisms are possible).

* Perhaps the whole notion of specialized vs. unspecialized class instances
needs to be made more explicit in the GenericMeta API...

* Finally and IMVHO most importantly, it is *very* confusing to override
"__setattr__" and not override "__getattribute__" to match. This gives rise
to code like "cls._foo = True; assert cls._foo" failing. This feels
wrong.... And presumably fixing the implementation so that
"__getattribute__" forwards the same set of attributes to the
"unspecialized" instance wouldn't break any code... Other than code that
already broken due to the new functionality, that is.