class "type" and metaclasses (was Re: Style for overwritten methods of abstract classes)

[Alex Martelli]

Stefan Schwarzer wrote:
        ...
> [Interesting code example of metaclass snipped]
> 
> To be honest, I haven't understood everything of this but it inspired
> my curiosity regarding the "type" class. I didn't find anything about
> it in the Python 2.2 docs (though I won't say it's not there ;-) ).

I don't think the type type is yet covered in the docs -- I see it still 
mentioned there as a built-in function (which it isn't any more; try:

[alex at lancelot cb4ok]$ python
Python 2.2 (#1, Dec 23 2001, 20:09:01)
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> type(type)
<type 'type'>
>>>

> to be the tutorial "Unifying types and classes in Python 2.2" at
> http://www.python.org/2.2/descrintro.html . If I haven't overlooked it
> that also doesn't mention the type class, does it?

http://www.python.org/2.2/descrintro.html#metaclasses
"The built-in 'type' is the most common metaclass; it is the metaclass of 
all built-in types."

> - Where can I find more information on metaclass programming in Python?

http://www.python.org/doc/essays/metaclasses/

but it's not really recommended -- in 2.2, things are simpler, as explained 
in the above-referenced paragraph of the descrintro.html.


> - (How) are the built-in function type and the class named "type" (that
>   didn't exist prior to Python 2.2?) related?

Many entities that were built-in functions (acting as factories) up to 2.1 
have become types in 2.2.  type, int, str ...


> - Should I be very concerned with metaclass programming? I can't really
>   estimate how useful they are. On the other hand, I suppose that would
>   not have been added to Python if it weren't useful (contrasted to beeing
>   merely "nice"). Are metaclasses rather for experts?

Yes, they're still intended for experts, although they've become much 
easier to use with 2.2 wrt 2.1.  I've never felt the need for actual use of 
metaclasses in production code, myself... yet.  Most of what you can 
accomplish with a metaclass, you can also do with clever metaprogramming in 
__getattr__ and friends.  However, doing it in a metaclass has potential 
advantages: the magic happens at class-construction time, rather than on 
each attribute access -- so, errors may be diagnosed earlier (making them 
easier to ferret out in tests, and easier to correlate with what's actually 
wrong with the code) and performance overhead may be less.

However, rather than thinking of them as something that's been added to 
Python, I see them as part of Python's general stance that *the underlying 
mechanics are exposed* -- you don't HAVE to mess with them, but, if you 
want, you can... they're very instructive to study and experiment with, 
although using them in production code may not be appropriate.

Back when I was a motorcycle enthusiast, I carefully studied every detail 
of my machine's "low-level" behavior.  I never actually needed to do any 
serious maintenance myself (excellent motorcycle mechanics are not scarce 
in the hometown of Ducati &c:-), but I felt the understanding and 
familiarity with the "underlying mechanisms" made me a better driver 
anyway, besides getting me ready should an emergency ever call for detailed 
knowledge.  To be honest, these days I drive a Honda Civic (or, more often, 
just ride a bus:-) and I have very vague ideas about what exactly keeps it 
ticking in fine detail, just like most drivers (and bus passengers).  But 
should I ever be bitten by the motorcycle bug again, I think that once 
again I will appreciate having all underlying mechanisms detailed and 
exposed... it's just a different mindset.  Not all bikers share it, mind 
you -- it's optional.   You CAN approach a motorbike, just like you 
approach a car, as a "black box" -- it's a very convenient and practical 
vehicle even then, after all.  But, you do have the option of going deeper.


> No, I don't like B&D programming. If I would, I probably wouldn't use
> Python but rather C++ or Java. ;-) My question aimed more at a simple
> or elegant approach. (Please don't be upset if you consider your
> suggestion simple and/or elegant. To be more specifically: the usage
> of the so defined class may be simple but its implementation not so.
> ;-) ).

I make no claims about the quality of the solution I proposed (actually 
hacked together somewhat fast) and wouldn't be surprised if it could be 
substantially improved.

The fundamental idea however does seem quite simple to me.  When a class 
statement executes, the metaclass is instantiated, so the metaclass's 
__init__ executes.  At that time, I want this metaclass-instance (this 
class) to become callable (with the normal semantics, that of producing 
class-instances) if and only if no abstract methods are left; otherwise, 
calling the class must raise an exception with a suitable error-message 
about what methods have not been overridden (have remained abstract).  All 
I do in that metaclass is try to enforce this idea, in order to parallel or 
mimic the "abstract classes" (non-instantiable classes) of certain other 
languages.

The whole setup seems reasonably elegant to me within the specified mindset 
of "must override" and "may override".  How else is a class going to be 
able to constrain its subclasses regarding what may or may not, must or 
must not, be overridden?  I think the mindset itself is at fault, and the 
normal Pythonic approach (let anybody override whatever, diagnose problems 
at runtime only if conflicts arise) far superior.  But the ability to 
constrain the 'may' and 'must' was the very essence of the thread, no?


One case where I think a similar use of metaclasses might be _productive_ 
is to emulate Haskell's typeclass construct.  In a Haskell typeclass, 
'abstract' methods may be specified in apparently mutually-recursive terms, 
for example (in Python syntax):

class OneOrMany:
    def doOne(self, one):
        self.doMany([one])
    def doMany(self, many):
        for one in many:
            self.doOne(one)

To implement a typeclass into an instantiable class, you have to override 
some subset of its methods so that all dependency cycles are broken.  Here, 
you might override either or both of doOne and doMany, otherwise the class 
would not be implementable.  Now THIS is what I consider elegant: it 
clearly specifies the intended semantic dependencies between the methods, 
and doesn't arbitrarily define one as more fundamental than others, when 
they're actually all on the same plane.  To translate this into a Python 
metaclass, you'd have to automatically or explicitly identify the 
dependencies (doOne->doMany, doMany->doOne), notice which dependencies are 
broken by subclassing (overriding some subsets of the methods), and check 
if the resulting dependency graph is acyclic -- otherwise, keep the class 
non-instantiable until by further subclassing the graph does become acyclic.


Alex