[Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Thu, Apr 2, 2015 at 4:46 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 2 April 2015 at 16:38, PJ Eby <pje at telecommunity.com> wrote:
>>
>> IOW, there's no need to modify the core just to have *that* feature,
>> since if you control the base class you can already do what PEP 487
>> does in essentially every version of Python, ever.  If that's all PEP
>> 487 is going to do, it should just be a PyPI package on a
>> stdlib-inclusion track, not a change to core Python.  It's not
>> actually adding back any of the dynamicness (dynamicity?
>> hookability?) that PEP 3115 took away.
>
> The specific feature that PEP 487 is adding is the ability to
> customise creation of subclasses without risking the introduction of a
> metaclass conflict.  That allows it to be used in situations where
> adopting any of the existing metaclass based mechanisms would require
> a potential compatibility break

But metaclass conflicts are *also* fixable in end-user code, and have
been since 2.2.  All you need to do is use a metaclass *function* that
automatically merges the metaclasses involved, which essentially
amounts to doing `class MergedMeta(base1.__class__,
base2.__class__,...)`.  (Indeed, I've had a library for doing just
that since 2002, that originally ran on Python 2.2,.)

On Python 3, it's even easier to use that approach, because you can
just use something like `class whatever(base1, base2,
metaclass=noconflict)` whenever a conflict comes up.  (And because the
implementation wouldn't have to deal with classic classes or
__metaclass__, as my Python 2 implementation has to.)

IOW, *all* of PEP 487 is straightforward to implement in userspace as
a metaclass and a function that already exist off-the-shelf in Python
2...  and whose implementations would be simplified by porting them to
Python 3, and dropping any extraneous features:

* http://svn.eby-sarna.com/PEAK/src/peak/util/Meta.py?view=markup
(the `makeClass` function does what my hypothetical `noconflict` above
does, with a slightly different API, and support for classic classes,
__metaclass__, etc., that could all be stripped out)

* http://svn.eby-sarna.com/DecoratorTools/peak/util/decorators.py?view=markup
 (see the `classy_class` metaclass and `classy` mixin base that
implement features similar to `__init_subclass__`, plus others that
could be stripped out)

Basically, you can pull out those functions/classes (and whatever else
they use in those modules), port 'em to Python 3, make any API changes
deemed suitable, and call it a day.  And the resulting code could go
to a stdlib metaclass utility module after a reasonable break-in
period.


> (as well as being far more
> approachable as a mechanism than the use of custom metaclasses).

Sure, nobody's arguing that it's not a desirable feature.  I
*implemented* that mechanism for Python 2 (eight years ago) because
it's easier to use even for those of us who are fully versed in the
dark metaclass arts. ;-)   Here's the documentation:

    http://peak.telecommunity.com/DevCenter/DecoratorTools#meta-less-classes

So the feature doesn't even require *stdlib* adoption, let alone
changes to Python core.  (Heck, I wasn't even the first to implement
this feature: Zope had it for Python *1.5.2*, in their
ExtensionClass.)

It's a totally solved problem in Python 2, although the solution is
admittedly not widely known.  If the PEP 487 metaclass library,
however, were to just port some bits of my code to Python 3 this could
be a done deal already and available in *all* versions of Python 3,
not just the next one.


> The gap I agree this approach leaves is a final
> post-namespace-execution step that supports establishing any class
> level invariants implied by decorators and other functions used in the
> class body. Python 2 allowed that to be handled with a dynamically
> generated __metaclass__ and PEP 422 through __autodecorate__, while
> PEP 487 currently has no equivalent mechanism.

Right.  And it's *only* having such a mechanism available by *default*
that requires a language change.  Conversely, if we *are* making a
language change, then adding a hook that allows method decorators to
access the just-defined class provides roughly the same generality
that Python 2 had in this respect.

All I want is the ability for method decorators to find out what class
they were added to, at the time the class is built, rather than having
to wait for an access or invocation that may never come.  This could
be as simple as __build_class__ or type.__call__ looking through the
new class's dictionary for objects with a `__used_in_class__(cls,
name)` method, e.g.:

for k, v in dict.items():
    if hasattr(v, '__used_in_class__'):
        v.__used_in_class__(cls, k)

This doesn't do what PEP 487 or 422 do, but it's the bare minimum for
what I need, and it actually allows this type of decorator to avoid
any frame inspection because they can just add a __used_in_class__
attribute to the functions being decorated.  (It actually also
eliminates another common use for metaclasses and class decorators,
e.g. in ORM and other structure-like classes that need properties to
know their names without duplicating code, as in  `x = field(int)` vs
`x = field('x', int)`.  I have a fair amount of code that does this
sort of thing, too, though it's not in urgent need of porting to
Python 3.)