[Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[PJ Eby]

On Fri, Apr 3, 2015 at 4:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> That means I'm now OK with monkeypatching __build_class__ being the
> only way to get dynamic hooking of the class currently being defined
> from the class body - folks that really want that behaviour can
> monkeypatch it in, while folks that think it's a bad idea don't need
> to worry about.

I'd still prefer to only do that as an emulation of an agreed-upon
descriptor notification protocol, such that it's a backport of an
approved PEP, so I hope we can work that out.  But I guess if not,
then whatever works.  I just wish you'd been okay with it in 2012, as
there was more than once in the last few years where I had some
downtime and thought about trying to do some porting work.  :-(

And in the meantime, the only alternative Python implementation I know
of that's made *any* headway on Python 3 in the last few years (i.e.,
PyPy 3) *includes* a compatibly monkeypatchable __build_class__.  It
appears that the *other* obstacles to making a compatible Python 3
implementation are a lot tougher for implementers to get over than
compatibility with __build_class__.  ;-)


> Neither PEP 422 nor 487 are designed to eliminate metaclass conflicts
> in general, they're primarily designed to let base classes run
> arbitrary code after the namespace has been executed in a subclass
> definition *without* needing a custom metaclass.

And yet the argument was being made that the lack of custom metaclass
was a feature because it avoided conflict.  I'm just trying to point
out that if avoiding conflict is desirable, building *every possible
metaclass feature* into the Python core isn't a scalable solution.  At
this point, co-operative inheritance is a well-understood model in
Python, so providing an API to automatically mix metaclasses
(explicitly, at first) seems like a good step towards solving the
metaclass conflict problem in general.

When Guido introduced the new MRO scheme in Python 2.2, he noted that
the source he'd gotten that scheme from had explained that it could be
extended to automatically mixing metaclasses, but he (Guido) didn't
want to do that in Python until more experience was had with the new
MRO scheme in general.  And I think we have enough experience with
that *now*, to be able to take a step forward, by providing a
stdlib-blessed metaclass mixer.  It not only makes the prototype,
PyPI-based version of PEP 487 more usable immediately, it will also
encourage people to develop metaclasses as *mixins* rather than
one-size-fits-all monoliths.

For example, there's no reason that both of PEP 487''s features need
to live in the *same* metaclass, if you could trivially mix
metaclasses at the point where you inherit from bases with different
metaclasses.  (And eventually, a future version of Python could do the
mixing automatically, without the `noconflict` function.  The theory
was well-understood for other languages, after all, before Python 2.2
even came out.)


> No, you can't do it currently without risking a backwards
> incompatibility through the introduction of a custom metaclass.

Right...  which is precisely why I'm suggesting  the `noconflict()`
metaclass factory function as a *general* solution for providing
useful metaclasses, and why I think that PEP 487 should break the
namespacing and subclass init features into separate metaclasses, and
add that noconflict feature.  It will then become a good example for
people moving forward writing metaclasses.

Basically, as long as you don't have the pointless conflict errors,
you can write co-operative metaclass mixins as easily as you can write
regular co-operative mixins.  I was missing this point myself because
I've been too steeped in Python 2's complexities: writing a usable
version of `noconflict()` is a lot more complex and its invocation far
more obscure.  In Python 2, there's classic classes, class- and
module-level __metaclass__, ExtensionClass, and all sorts of other
headaches for automatic mixing.  In Python 3, though, all that stuff
goes out the window, and even my 90-line version that's almost half
comments is probably still overengineered compared to what's actually
needed to do the mixing.


>> Further, if the claim is that metaclass conflict potential makes PEP
>> 487 worthy of a language change, then by the same logic method
>> decorators are just as worthy of a language change, since any mixin
>> required to use a method decorator would be *just as susceptible* to
>> metaclass conflicts as SubclassInit.
>
> There wouldn't be a custom metaclass involved in the native
> implementation of PEP 487, only in the backport.

Right...  and if there were a native implementation of PEP 422, that
would also be the case for PEP 422.  The point is that if the PEP 487
can justify a *language* change to avoid needing a metaclass, then
arguably PEP 422 has an even *better* justification, because its need
to avoid needing a metaclass is at least as strong.  Indeed, you said
the same yourself as recently as 2013:

    https://mail.python.org/pipermail/python-dev/2013-February/123925.html


> PEP 422 has never been approved

I took this post from Guido (and his other comments in the same thread
where you asked for Pronouncement on it) as meaning it was basically a
done deal, approved in all but some final edits:

   https://mail.python.org/pipermail/python-dev/2013-February/123957.html


> Given my change of heart, I believe that at this point, if you were
> willing to champion a revived PEP 422 that implemented the behaviour
> you're after, that would be ideal, with monkeypatching the desired
> behaviour in as a fallback plan if the PEP is still ultimately
> rejected. Alternatively, you could go the monkeypatching path first,
> and then potentially seek standardisation later after you've had some
> practical experience with it - I now consider it an orthogonal
> capability to the feature in PEP 487, so the acceptance of the latter
> wouldn't necessarily preclude acceptance of a hook for class
> postprocessing injection.

A lot of things have changed since the original discussion, mostly in
the direction of me having even *less* time for Python work than
previously, so it's unlikely that me championing a PEP is a realistic
possibility.  Frankly, I'm immensely fatigued at the discussion
*already*, and the need to go over the same questions a *third* time
seems like not something I'm going to want to put energy into.

However it sounds like there *is* some growing consensus towards the
idea of simply notifying interested class members of their class
membership, so if there ends up being a consensus to standardize
*that* protocol and what part of the class-building process it gets
invoked in, then I will implement a backport (or use such a backport
if someone else implements it), when I actually start porting my
libraries to Python 3.  But that would make my timeline somewhat
dependent on how much of a consensus there is, and how much clarity I
could get before going forward.

Even if PEP 422 never was officially tagged with "Approved" status in
the PEP itself, our 2013 conversation with Guido made it sound like it
was totally a done deal; if there was something *other* than PEP 487
that threw it off that track, I never saw it.  So I'm understandably a
little bit reluctant to start off implementing a new protocol that
then two or three years from *now* will suddenly not be a done deal
any more, with whatever I did being retroactively declared the wrong
thing to do again.

I suppose, though, that that my best option all in all is just to do
whatever the heck seems best for porting, and worry about
standardization later.  If a member-notification protocol is
standardized, I can always change DecoratorTools to use it *later*,
after all, as long as the actual implementation mechanism inside
DecoratorTools is opaque to its consumers.  (i.e., if I don't actually
expose the member-notification protocol directly)

(And, in retrospect, I could have, and probably should have, taken
this approach from the get-go in 2012.  It just seemed really, *really*
important to you back then that I *not* do it.)