dunder-docs (was Python is DOOMED! Again!)

[Steven D'Aprano]

Rustom Mody wrote:

> The other day I was taking a class in which I was showing
> - introspection for discovering -- help, type, dir etc at the repl
> - mapping of surface syntax to internals eg. a + b ←→ a.__add__(b)
> 
> And a student asked me the diff between
> dir([])
> and
> [].__dir__()
> 
> I didnt know what to say...

Surely the right answer would have been "I don't know, let's check the
interactive interpreter first, and the docs second."

Checking the REPL first would have revealed that [].__dir__ raises
AttributeError. In other words, lists don't have a __dir__ method.

Checking the docs would then have revealed that __dir__ is only required
when a class wishes to customise the output of dir(). Or at least, that's
what I recall them saying. I'm too lazy to look it up :-)

Oh very well, because it's you...

https://docs.python.org/2/library/functions.html#dir

Note that there are two inaccuracies in the documentation for __dir__:

    [quote]
    If the object has a method named __dir__(), this method will 
    be called and must return the list of attributes.
    [end quote]

The first inaccuracy is that like all (nearly all?) dunder methods, Python
only looks for __dir__ on the class, not the instance itself. Hence:


py> class K(object):
...     def __dir__(self):
...             return ["eggs", "spam"]
...
py> k = K()
py> from types import MethodType
py> k.__dir__ = MethodType(lambda self: ["cheese", "spam"], k)
py> dir(k)
['eggs', 'spam']

Well, actually that's only true for new-style classes. For old-style classic
classes (Python 2 only), it will work on the instance as well. That is, in
Python 2 only, if I had left out the "object" base class, dir(k) would have
returned ["cheese", "spam"].

Now that I have confused your students, we shall never mention classic
classes again (until next time).

The second inaccuracy is that method should not return the attributes
themselves, but their names.


> Now surely the amount of python I dont know is significantly larger than
> what I know Still it would be nice to have surface-syntax ←→ dunder-magic
> more systematically documented

Each function or operator is unique. However, there are certain general
behaviours that typically hold.

Dunder methods are only looked up on the class, not the instance. (Except
for classic classes, if there are any exceptions to this rule, it is
probably a bug.) So the correct analog of surface syntax `obj + spam` is
*not* `obj.__add__(spam)` but actually:

    type(obj).__add__(spam)

Similar for other operators and functions. In the case of dir, the
comparison should be:

    dir([]) versus type([]).__dir__()

Except of course dir doesn't require there to be a __dir__ method.

len tries to call __len__ if it exists, and if not, it tries walking the
iterable counting items.

bool tries calling __bool__ (Python 3) or __nonzero__ (Python 2), if it
exists. If not, it tries returning len(obj) != 0. If that fails, objects
are true by default.

In the case of operators, operator syntax `x $ y` for some operator $ will
generally do something like this:


    X = type(x)
    Y = type(y)
    if issubclass(Y, X):
        # Try the reflected version first, if it exists.
        dunder = getattr(Y, "__rdunder__", None)
        if dunder:
            result = dunder(y, x)
            if result is not NotImplemented:
                return result
        # Or the non-reflected version second.
        dunder = getattr(X, "__dunder__", None)
        if dunder:
            result = dunder(x, y)
            if result is not NotImplemented:
                return result
    else:
         # Like the above, except we check the non-reflected 
         # version first, and the reflected version second.
         ...
    raise TypeError


Some methods are their own reflection, e.g. __eq__. In the special case of
__eq__ and __ne__, if the class defines one method but not the other,
recent versions of Python will automatically call the other method and
return its boolean not. Unary operators obviously have no concept of a
reflected version.

Comparison operators prefer to call the rich comparison methods but may fall
back on the older __cmp__ dunder method (Python 2 only).


-- 
Steven