[Python-3000] PEP for Metaclasses in Python 3000

[Guido van Rossum]

On 3/9/07, Talin <talin at acm.org> wrote:
> I had a conversation with Guido last night at the Python user's group
> meeting, and we hashed out some of the details of how metaclasses should
> work. I've gone ahead and written up a PEP, which I present for your review.

Executive summary: I'm defending the PEP and the metaclass syntax it
proposes, and in fact I want the "clas arguments" to have the same
syntax as a call, including *args and **kwds. I'm only suggesting to
find a new name instead of __metacreate__.

While I'm responding to the first message in the thread, I've actually
read the entire thread, and I'm responding in tit-for-tat mode to much
of it. (Yes, you'll find that there are actually a few messages in the
thread so far to which I *didn't* respond, and even some by Greg
Ewing. :-)

>      There were some objections in the discussion to the 'two-phase'
>      creation process, where the metaclass is invoked twice, once to
>      create the class dictionary and once to 'finish' the class. Some
>      people felt that these two phases should be completely separate, in
>      that there ought to be separate syntax for specifying the custom
>      dict as for specifying the metaclass. However, in most cases, the
>      two will be intimately tied together, and the metaclass will most
>      likely have an intimate knowledge of the internal details of the
>      class dict. Requiring the programmer to insure that the correct dict
>      type and the correct metaclass type are used together creates an
>      additional and unneeded burden on the programmer.

Moreover, I expect that in most cases the metaclass will be implicitly
provided by one of the base classes. It would be insane to require the
*user* of the metaclass to remember to also specify the custom dict
creator if the metaclass needs it.

>      It would be possible to leave the existing __metaclass__ syntax in
>      place. Alternatively, it would not be too difficult to modify the
>      syntax rules of the Py3K translation tool to convert from the old to
>      the new syntax.

I'm in favor of ripping it out. Also the module level __metaclass__,
which was mostly intended as a way to make all baseless classes in a
given module new-style by default.

On 3/9/07, Steven Bethard <steven.bethard at gmail.com> wrote:
> <bikeshed>
> I'd rather __metacreate__ be called something like __createdict__,
> __creationdict__, __metadict__, __getmetadict__, etc. where it's
> clearer that the purpose is to create a dict object.
> </bikeshed>

Agreed; there's no need that the attribute name contains the word
'meta' again, that's already implied in it being an attribute of the
metaclass.

On 3/9/07, Brett Cannon <brett at python.org> wrote:
> Do the keywords have to follow the metaclass keyword, or is order
> irrelevant?  While order makes sense, it would be a new precedent for
> keyword arguments to have an important order.

I'd like the syntax between () to be identical to that for a function
call, i.e. including *args and **kwds. That way the cognitive load for
the user is the smallest. Sorting out the semantics of the keywords is
purely a runtime activity anywaty.

> Does the language spec guarantee that the body of a class will be
> executed in definition order?  Or is that considered implicit by the
> fact that the class body is executed as code?

Yes, and yes.

On 3/9/07, Jack Diederich <jackdied at jackdied.com> wrote:
> I am a very big fan of ordered dicts in classes.

Would you mind showing us some of your use cases / examples?

> One possibility is that
> suites in classes always store their order in a special dict that keeps a
> side list of key order.  A final invisible class decorator around
> every class would then toss out the order and leave only a regular dict.

This was Talin's initial suggestion (in private conversation with me).
I dissuaded him against it because: (a) it would force us to provide a
specific ordered dict implementation and anything it didn't handle
would remain impossible; and (b) I'm sure there are other use cases
such as trapping duplicate method definitions (either to issue errors
or to assign special semantics, as may be useful for certain generic
function packages).

In general I find hardcoding a specific mechanism hard to defend at
such a meta-level; I'm much more in favor of providing policy and
letting libraries pick their own mechanism. This also lets us provide
an ordered-dict-like implementation that only provides just enough
information needed for a typical implementation (e.g. something that
just keeps a log of all __setitem__ calls) without limiting future
uses.

> An advantage is that the order of the keys would be available to both
> metaclasses and class decorators.  It would also require no changes in
> signature for either.  As far as I can tell ordered dicts are the only
> demonstrated use case (and common! looking at any SQL/ORM library).

I think that whether the order of the keys is made available to the
class decorator should be up to the metaclass. After all it is in
charge of what it returns -- it may nor return something resembling a
class at all!

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> >        class Foo(base1, base2, metaclass=mymeta):
> >          ...
>
> -1 on this syntax, I think it's ugly.

Like beauty, ugliness is in the eye of the beholder.

> Alternative proposals:
>
>    class Foo(base1, base2) as MyMeta:
>      ...

-1000; the 'as' keyword in Python suggests assignment or renaming;
here you seem to be using the VB meaning of 'as'.

> or
[snip]
-1 on all of these; '=' is just as arbitrary here as 'as' or '<<' or '@'.

Anyway the syntax debate is mostly bikeshed color discussion.

> The existing syntax actually has some advantages, e.g.
> you can do things like
>
>    class Foo:
>      class __metaclass__:
>        # Real class methods defined here
>
> which I think is rather elegant, so I'm not sure I'd
> like the new syntax to completely replace the old one.

Again, tastes differ. :-) Personally, I find code that does this
completely unreadable and I would never accept it in a code review.

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> In any case, I've thought of a way to reduce the performance
> penalty to near-zero. Consider that the compiler knows
> all the names that will be assigned in the class dict
> and what order they are in. So all it needs to do is
> compile
>
>    class Foo:
>      a = 5
>      def b(self):
>        ...
>      def c(self, x):
>        ...
>
> as though it were
>
>    class Foo:
>      __names__ = ('a', 'b', 'c')
>      a = 5
>      def b(self):
>        ...
>      def c(self, x):
>        ...
>
> The tuple can be a constant of the code object, so the
> only overhead is one extra item added to the dict at
> class creation time, and none at all for subsequent
> access to the class dict. And you don't even need a
> special kind of dict.

-1; I like Python's compiler to remain stupid. It would also easily be
thwarted by use of locals(), exec(), etc. (And yes, there's code using
exec in a class def in the stdlib!) Also, Josiah's example of
conditional code branches, etc.

On 3/10/07, Talin <talin at acm.org> wrote:
> I didn't want to call it __metadict__, as it was in the earlier
> proposal, because in the current version it's doing more than just
> creating the class dict, it's also processing any additional keywords
> that are passed in to the class base list.

How about __prepare__?

On 3/10/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Generally, about the only need I forsee for arbitrary keyword arguments
> in the class 'foo(...)' signature is to pass arguments to the 'metadict'
> factory (method, function, etc.).  Otherwise it seems more like a syntax
> looking for a purpose than a purpose looking for a syntax.

Or to the metaclass later on. I see keyword args as a way to pass
parameters to a metaclass that would otherwise have to be assigned to
class variables, in a namespace that's much more crowded. Imagine
being able to write this:

  class C(implements=(I1, I2)):
    ...

> Newdict needs to be callable, and if you want to pass certain
> semantic-altering arguments, make newdict a factory can call it directly...
>
>     class foo(...):
>         __metaclass__ = mytype, newdict(no_dupes=True)
>
> One wouldn't even need any of the default arguments only stuff to make
> the two 'newdict' examples above use the same callable.

As I said before, this seems to break abstraction to me; if a
metaclass requires an ordered dict, the user shouldn't have to
remember to pass the appropriate constructor each time that metaclass
is used. And most of the time the metaclass is implied by the base
classes, not explicitly provided using __metaclass__ -- the latter is
mostly an escape hook to resolve metaclass conflicts when combining
bases with different metaclasses.

On 3/10/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:40 PM 3/10/2007 -0800, Josiah Carlson wrote:
> >Great!  If there is no need to mix and match metaclasses and dicts
> >arbitrarily, then there is no need to pass alternate parameters to the
> >dictionary construction method,
>
> Er, wha?  Those two things are utterly unrelated.

Also, I'm in utter disagreement with anyone who believes that
assignment to __metaclass__ in the class body is somehow better or
more elegant than a keyword argument in the class "parameter list"
(formerly known as bases :-).

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Okay, then make it
>
>    class Foo(base1, base2) isa MyMeta:
>       ...

And why is that better than a metaclass keyword? 'isa' isn't even an
English word. Bah.

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Josiah Carlson wrote:
> > As such, the compiler, etc., needs to be taught to pull out the
> > __metaclass__ definition before executing the class body, but that's
> > better than mucking with the syntax of class foo(...).
>
> I'd be happy with that.

And I think "execution-out-of-order-because-we-don't-want-new-syntax"
is a poor excuse for evolutionary language design.

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I still think that extra arguments are unnecessary in
> any case, because anything you could do by passing args
> to the dict creation function could be done by using a
> function instead of a class for the "metaclass".
>
> Does anyone have a counter-example?

Not that you couldn't write it your way, but I find this rather cool looking:

  class C(implements=(I1, I2)): ...

On 3/10/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Note that you can already pass information to the
> metaclass itself using special attributes in the class
> namespace. This would only be need if you absolutely
> had to pass information when creating the *dict*.
> Things like your 'sealed' option don't seem to fall
> into that category.

I don't know about sealed, but using class attributes topassing
parameter (other than the namespace itself) to the metaclass seems a
pretty lousy mechanism, and keyword arguments in the classdef are a
much cleaner solution for this need. For example, this solves the
problem that those attributes remain in the class dict  but aren't
necessarily candidates for inheritance (example: __slots__, whose
inheritance story is, um, complicated; in general C.__slots__ does not
necessarily tell you all the slots that the class has).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)