[Python-Dev] PEP 487 vs 422 (dynamic class decoration)

[Nick Coghlan]

On 4 April 2015 at 06:36, PJ Eby <pje at telecommunity.com> wrote:
> On Fri, Apr 3, 2015 at 4:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> No, you can't do it currently without risking a backwards
>> incompatibility through the introduction of a custom metaclass.
>
> Right...  which is precisely why I'm suggesting  the `noconflict()`
> metaclass factory function as a *general* solution for providing
> useful metaclasses, and why I think that PEP 487 should break the
> namespacing and subclass init features into separate metaclasses, and
> add that noconflict feature.  It will then become a good example for
> people moving forward writing metaclasses.
>
> Basically, as long as you don't have the pointless conflict errors,
> you can write co-operative metaclass mixins as easily as you can write
> regular co-operative mixins.  I was missing this point myself because
> I've been too steeped in Python 2's complexities: writing a usable
> version of `noconflict()` is a lot more complex and its invocation far
> more obscure.  In Python 2, there's classic classes, class- and
> module-level __metaclass__, ExtensionClass, and all sorts of other
> headaches for automatic mixing.  In Python 3, though, all that stuff
> goes out the window, and even my 90-line version that's almost half
> comments is probably still overengineered compared to what's actually
> needed to do the mixing.

D'oh, I had the same problem you did - I'd been assuming this was
entirely infeasible because of all the complexities it involved back
in Python 2, and had never adequately reconsidered the question in a
PEP 3115 based world :(

So actually reading https://gist.github.com/pjeby/75ca26f8d2a7a0c68e30
properly, you're starting to convince me that a "noconflict" metaclass
resolver would be a valuable and viable addition to the Python 3 type
system machinery.

The future possible language level enhancement would then be to make
that automatic resolution of metaclass conflicts part of the *default*
metaclass determination process. I realise you've been trying to
explain that to me for a few days now, I'm just writing it out
explicitly to make it clear I finally get it :)

>> Given my change of heart, I believe that at this point, if you were
>> willing to champion a revived PEP 422 that implemented the behaviour
>> you're after, that would be ideal, with monkeypatching the desired
>> behaviour in as a fallback plan if the PEP is still ultimately
>> rejected. Alternatively, you could go the monkeypatching path first,
>> and then potentially seek standardisation later after you've had some
>> practical experience with it - I now consider it an orthogonal
>> capability to the feature in PEP 487, so the acceptance of the latter
>> wouldn't necessarily preclude acceptance of a hook for class
>> postprocessing injection.
>
> A lot of things have changed since the original discussion, mostly in
> the direction of me having even *less* time for Python work than
> previously, so it's unlikely that me championing a PEP is a realistic
> possibility.  Frankly, I'm immensely fatigued at the discussion
> *already*, and the need to go over the same questions a *third* time
> seems like not something I'm going to want to put energy into.

Heh, one of the main reasons PEP 422 ended up languishing for so long
is that I started putting more time into other projects (PyPA, the
PSF, the import system, Python 3 advocacy, etc), so with both you & me
occupied elsewhere, we didn't really have anyone driving the
discussion forward on the metaclass machinery side of things. Martin's
very pertinent challenges to some of the unnecessary design complexity
in PEP 422 has noticeably changed that dynamic for the better :)

> However it sounds like there *is* some growing consensus towards the
> idea of simply notifying interested class members of their class
> membership, so if there ends up being a consensus to standardize
> *that* protocol and what part of the class-building process it gets
> invoked in, then I will implement a backport (or use such a backport
> if someone else implements it), when I actually start porting my
> libraries to Python 3.  But that would make my timeline somewhat
> dependent on how much of a consensus there is, and how much clarity I
> could get before going forward.

In a separate RFE, Martin convinced me that we really want to kick as
much of this off from type.__init__ as we can, and that the problem
with zero-argument super() currently not working when called from
metaclass __init__ methods should be treated as a bug in the way
zero-argument super() is currently implemented.

That position makes a lot of sense to me (especially since it was
backed up with a patch to fix the bug using a modifed implementation
that's inspired by the way that setting __qualname__ works), and is
what makes it possible to assume we can just use the metaclass system
to deal with this, rather than having to rely on modifications to
__build_class__.

> Even if PEP 422 never was officially tagged with "Approved" status in
> the PEP itself, our 2013 conversation with Guido made it sound like it
> was totally a done deal; if there was something *other* than PEP 487
> that threw it off that track, I never saw it.  So I'm understandably a
> little bit reluctant to start off implementing a new protocol that
> then two or three years from *now* will suddenly not be a done deal
> any more, with whatever I did being retroactively declared the wrong
> thing to do again.

That's entirely fair. The main thing that threw PEP 422 off track was
that I received a lot of good requests for clarification in the
previous round of PEP 422 discussions, and I wasn't really happy with
the answers I was coming up with when I contemplated redrafting it, so
it was easy to rationalise postponing further work on it. Martin's
recent feedback then served to finally crystallise those doubts into
"this isn't the right answer after all".

> I suppose, though, that that my best option all in all is just to do
> whatever the heck seems best for porting, and worry about
> standardization later.  If a member-notification protocol is
> standardized, I can always change DecoratorTools to use it *later*,
> after all, as long as the actual implementation mechanism inside
> DecoratorTools is opaque to its consumers.  (i.e., if I don't actually
> expose the member-notification protocol directly)
>
> (And, in retrospect, I could have, and probably should have, taken
> this approach from the get-go in 2012.  It just seemed really, *really*
> important to you back then that I *not* do it.)

My apologies for that - while I don't actually recall what I was
thinking when I said it, I suspect I was all fired up that PEP 422 was
definitely the right answer, and hence thought I'd have an official
solution in place for you in fairly short order. I should have let you
know explicitly when I started having doubts about it, so you could
reassess your porting options.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia