Are decorators really that different from metaclasses...

[Bengt Richter]

On Sat, 28 Aug 2004 08:45:20 -0400, Paul Morrow <pm_mon at yahoo.com> wrote:
[...]
>
>Cool!  Thanks!  Now I need to ponder why a function's docstring needs a 
>different implementation than a class's docstring (i.e. why not just 
>make it a straigtforward attribute of the function object).
>
Disclaimer: I am just speculating from appearances accessible interactively,
so don't take this as developer's documention of internals. I'm just reading
between the lines, trying to recognize the underlying ideas ;-)

So, depending on your idea of 'straightforward' ISTM docstrings do look like
'straightforward' attributes in the way they behave:

 >>> def func():
 ...     'func docstring'
 ...
 >>> class CNew(object):
 ...     'CNew docstring'
 ...
 >>> class COld:
 ...     'COld docstring'
 ...
 >>> func.__doc__, CNew.__doc__, COld.__doc__
 ('func docstring', 'CNew docstring', 'COld docstring')

But you were wondering about different _implementation_ of
class and function docstrings. But implementation is a layered
thing that evolves, even while surface behavior stays the same.

So your question comes down to how implementation differs for
getting the three __doc__ attributes in the last tuple above.

As far as the language is concerned, writing func.__doc__ is
no different than x.y -- we don't know what func is bound to,
and we don't know whether __doc__ is an object attribute or
a method in the object's class or a method somewhere in the
chain of base classes, or whether func is a class and __doc__
is a class variable, etc. So we have to look into how and when
those differences are discovered and have their effect.

<aside>
Implementation is layered. High level functionality is composed
of primitives. A really nice thing about Python is that so much of
the dynamic primitive infrastructure is accessible and expressible
in python source. Much work has obviously gone into making primitives
that are actually implemented in C behave and be inspectable "as-if"
they were implemented in python. IOW, there is a kind of primitive
abstractions layer that is platform independent and source-representable.

But whatever the representation, it is (IMO) the world of abstract ideas
that is the most important, the world where sets and numbers and mappings
and maybe deities live. That's where we find the real beauty that our
graven source code sometimes distracts from when we forget what's what ;-)

C implementation can make optimized shortcuts to define as-if
behaviour that looks identical to general behavior with various
limitations that when encountered act as-if they were exceptions
of general behaviour.
</aside>

Ok, focusing on x.y, what do we need to find out?
It looks like the __getattribute__ method (or its C behavioural equivalent)
is the key to getting x's y attribute. So where do we find __getattribute__?
Well, it's a method, and methods would be found in x's class dict, and if not there,
it would be looked for in the chain of base classes. But how do we find x's class dict?
x.__class__.__dict__ doesn't work if x happens to be an old style class.
So type(x) serves now as the logical equivalent of x.__class__ :

 >>> for t in map(type, [func, CNew, CNew(), COld, COld()]): print t
 ...
 <type 'function'>
 <type 'type'>
 <class '__main__.CNew'>
 <type 'classobj'>
 <type 'instance'>
 >>> for t in map(lambda o: type(type(o).__dict__), [func, CNew, CNew(), COld, COld()]): print t
 ...
 <type 'dictproxy'>
 <type 'dictproxy'>
 <type 'dictproxy'>
 <type 'dictproxy'>
 <type 'dictproxy'>

In every case we have a dict proxy, that we can now look for attributes in,
so we can look for __getattribute__:

 >>> for t in map(lambda o:type(o).__dict__.get('__getattribute__'),[func,CNew,CNew(),COld,COld()]): print t
 ...
 <slot wrapper '__getattribute__' of 'function' objects>
 <slot wrapper '__getattribute__' of 'type' objects>
 None
 <slot wrapper '__getattribute__' of 'classobj' objects>
 <slot wrapper '__getattribute__' of 'instance' objects>

Kind of interesting. Now if we let the the attribute chase continue for CNew vs CNew():

 >>> type(CNew).__getattribute__
 <slot wrapper '__getattribute__' of 'type' objects>
 >>> type(CNew()).__getattribute__
 <slot wrapper '__getattribute__' of 'object' objects>

IOW, it looks like the None got replaced by object.__getattribute__
found in the base class of class CNew(object) for CNew() (since there
was no override defined in CNew) but for CNew itself, the search started
in type(CNew), so found type.__getattribute__

If we look in a thing's __dict__ we are apparently avoiding the attribute chase,
but if we look via thing.__dict__ we are using the __dict__ attribute to get
the __dict__, so to allow us to see what __dict__ is without attribute processing,
we have to do thing.__dict__['__dict__'] or thing.__dict__.get('__dict__').
We can then look at what kind of thing that is:

 >>> for ob in [func,CNew,CNew(),COld,COld()]:
 ...     print type(ob).__dict__.get('__dict__')
 ...
 <attribute '__dict__' of 'function' objects>
 <attribute '__dict__' of 'type' objects>
 <attribute '__dict__' of 'CNew' objects>
 None
 None

What kinds of things are these?

 >>> for ob in [func,CNew,CNew(),COld,COld()]:
 ...     print type(type(ob).__dict__.get('__dict__'))
 ...
 <type 'getset_descriptor'>
 <type 'getset_descriptor'>
 <type 'getset_descriptor'>
 <type 'NoneType'>
 <type 'NoneType'>

Looks to me like a mechanism to unify access to attributes at a particular
infrastructure level, being used in this case to deal with a bunch of things
that can act like dictionaries, but which might have very different implementations
and limitations. E.g., IWT you could make dictionaries lazily implemented for objects
whose .__dict__'s are almost always empty, and just create them on first need.
Even if they start with standard name content, one could imagine creating a full dict
only on update. But those are optimization games.

Bottom line, comparing implementation of __doc__ in functions and new style classes
to answer your question, they don't seem that different. I.e., both seem to be
implemented via descriptor/property-like things:

 >>> type(func).__dict__['__doc__']
 <member '__doc__' of 'function' objects>
 >>> type(func).__dict__['__doc__'].__get__
 <method-wrapper object at 0x00918890>
 >>> type(func).__dict__['__doc__'].__get__(func)
 'func docstring'

 >>> type(CNew).__dict__['__doc__']
 <attribute '__doc__' of 'type' objects>
 >>> type(CNew).__dict__['__doc__'].__get__
 <method-wrapper object at 0x00918890>
 >>> type(CNew).__dict__['__doc__'].__get__(CNew)
 'CNew docstring'

Note that old style classes and objects don't use a __doc__ descriptor:

 >>> type(COld)
 <type 'classobj'>
 >>> type(type(COld).__dict__['__doc__'])
 <type 'str'>

I.e., the above is the actual doc string of classobj, not a descriptor

 >>> type(COld)
 <type 'classobj'>
 >>> type(COld).__dict__['__doc__']
 'classobj(name, bases, dict)\n\nCreate a class object.  The name must be a string; the second ar
 gument\na tuple of classes, and the third a dictionary.'

(Of course, normally it is not retrieved this way).
vs.

 >>> type(CNew)
 <type 'type'>
 >>> type(CNew).__dict__['__doc__']
 <attribute '__doc__' of 'type' objects>
 >>> type(type(CNew).__dict__['__doc__'])
 <type 'getset_descriptor'>

So there is a difference between func and old-style-class docstring implementations.
Were you using old style classes for your question?

I don't know why I did this...

Regards,
Bengt Richter