Can __iter__ be used as a classmethod?

[Alex Martelli]

Giovanni Bajo wrote:

> "Alex Martelli" <aleax at aleax.it> ha scritto nel messaggio
> news:n519a.3620$zo2.111194 at news2.tin.it...
> 
>> E.g., if X is a class, X.__iter__ affects iterations on INSTANCES
>> of X, *NOT* iterations on X itself;
>> the latter are instead affected
>> by type(X).__iter__, if any.  type(X) is X's metaclass -- unless
>> it's a custom one, type(X) will be the built-in named 'type', which
>> has no special method __iter__.
> 
> Let's talk about new-style classes only (otherwise I could blow a fuse
> before I understand something ;)

Sure, let's -- no "classic classes" in the following (and, feel free
to switch the discussion to it.comp.lang == with [Python] at the start
of the subject == if discussing this in Italian will make it easier for
you... otherwise, keeping it on c.l.py does seem preferable).


> Is it possible to add (class) methods to class objects by defining them in
> a custom metaclass, or is the metaclass used only to lookup special
> methods (__iter__, etc.) for operations executed on class objects? Since I

Just as any instance "inherits" attributes from its class, so does any class
"inherit" attributes from its metaclass -- there is no difference between
the two cases, nor between callable and non-callable attributes.  Therefore,
for example:

[alex at lancelot bin]$ python
Python 2.3a2 (#1, Feb 21 2003, 10:22:48)
[GCC 3.2 (Mandrake Linux 9.0 3.2-1mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class meta(type):
...   def amethod(*args): print 'a method', args
...
>>> class aclass:
...   __metaclass__ = meta
...
>>> aclass.amethod()
a method (<class '__main__.aclass'>,)


However, the "instance-from-type inheritance" is not transitive:

>>> aninstance = aclass()
>>> aninstance.amethod()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'aclass' object has no attribute 'amethod'
>>>

See at the very end of the post a paragraph explaining (I hope) why
it's a very good thing that such inheritance is not transitive AND
that operations look up special-named attributes on the types of their
arguments, not on the arguments themselves -- the case of __call__.
I hope that such motivations/rationales make things easier to
understand and therefore to retain and to use in practice.

> define methods of the instances within the class (object's type)
> definition, I would think that I might define methods of the class objects
> within the metaclass (class' type) definition.

Sure, you can do that if you wish.  You MUST do that for special
methods that you want to be automatically invoked for operations
on the class object, you CAN do that for methods that you want to
be invokable on the class-object but not on its instance-objects.


> Given:
> 
> def A(object):

I assume you rather mean:

class A(object):

and will proceed based on this assumption.

>     def __iter__(cls):
>         pass
>     __iter__ = classmethod(__iter__)
> 
> how do you classify (word joke not intended) the (rebound) __iter__? Is it
> a class method of class A? Is it still used for iterations on instances of
> A? How does the classmethod() affected the type and the semantic of
> __iter__?

iter(A()) will do its best, but it will get a None and therefore fail with
a TypeError: iteration over non-sequence.  But if the A.__iter__ call did
return an iterator, as it should, then there would be no problem using it
for iterations over instances of A:

>>> class A(object):
...   def __iter__(cls):
...     return iter((1,2,3))
...   __iter__ = classmethod(__iter__)
...
>>> for xx in A(): print xx
...
1
2
3
>>>

The problem is not with A.__iter__ being a classmethod (and thus getting
as its automatic first argument the class object rather than an instance
of the class), it's with A.__iter__ returning None, which is not a valid
iterator object (None has no method named 'next' callable without arguments
etc etc).


>>Attributes that you look up on a specific object are not the
>>same thing as special-methods that are looked up (on the TYPE
>>of the object -- except for classic classes) by operations
>>that you perform on the object.  Normal method calls use normal
>>attribute lookup.
> 
> Probably I'm confused because to me:
> 
> class B(object):
>     def f(self):
>         pass
>     def __iter__(self):
>         pass
> 
> f and __iter__ seems to be of the same kind (type). But you say that

They are!  And both return None (which makes B instances not iterable
on, but that's another issue).  It's easy to check that they are
indeed the same type, and both equally well callable (and bereft of
any effect) as long as you use identical syntax to call both:

>>> class B(object):
...   def f(self): pass
...   def __iter__(self): pass
...
>>> type(B.f)
<type 'instancemethod'>
>>> type(B.__iter__)
<type 'instancemethod'>
>>> B().f()
>>> B().__iter__()
>>>


> B().f() is doing a lookup on the specific instance (and the lookup
> resolves to B.f), while iter(B()) is looked up on the TYPE of the object,

Sure!  iter(B()) is NOT the same thing as B().__iter__() -- that's
all there is to it.

Many built-ins work similarly.  abs(B()) looks up type(B).__abs__ --
it's NOT the same thing as B().__abs__, which looks up __abs__ in
the INSTANCE object.  Check this, maybe it's easier to see with
abs than with iter perhaps?

>>> class Z(object):
...   def __init__(self):
...     self.__abs__ = 23
...   def __abs__(self):
...     return 45
...
>>> print Z.__abs__
<unbound method Z.__abs__>
>>> print Z().__abs__
23
>>> print abs(Z())
45
>>>


> but it resolves to B.__iter__ as well. So, if the lookup are performed on
> different entities, why do they resolve in the same way? Is it just

Because every instance "inherits" attributes from its class, unless
the instance has re-bound the name specifically as per-instance attribute
names; thus when you look up any attribute in X() you will often get the
same thing as by looking it up in X, though there are many exceptions:

-- if the instance X() has specifically bound that attribute name in
   itself,
-- for methods bound in X (or inherited by X from base classes),
   where you get a different type of object depending on where you
   look it up (because of descriptor mechanics...),
-- for other names bound to special descriptors in X (e.g., property),
-- for attributes that X itself inherits from type(X) -- inheritance
   is not transitive,

and the like.


> because they are not rebound after the instance is created? Would that

That's one condition (rebinding an attribute name on an instance
specifically does affect lookups for that name on that instance).

> mean that rebinding B().__iter__ has no effect since B.__iter__ is always
> looked up when iterating over instances of B(), while B().f() can be
> freely rebound?

It's not true that rebinding name '__iter__' on an instance has NO
effect -- it does affect calls DIRECTLY to thatinstance.__iter__, AND
iterations on INSTANCES of that instance (if, of course, that instance
is usable and gets used as a type. AKA class).  But it has no effect
on ITERATIONS on that instance, or calls to iter(...) with that
instance as their argument.

The freedom you have to rebind name '__iter__' on an instance is
exactly the same as you have to rebind name 'f' on that instance.
The effects are the same *when you fetch those two attributes from
the instance in the same, identical way* -- say the instance is
bound to name 'x', then:

-- IF you have bound names 'f' and/or '__iter__' directly on x,
   then by fetching x.f and/or x.__iter__ you get exactly the
   object you had bound to that name on x;
-- otherwise, if in type(x) you had bound 'f' and/or 'iter' to
   objects of type 'function', then by fetching x.f and/or
   x.__iter__ you get a bound-method whose im_self attribute
   is x and whose im_func attribute is the function object you
   had bound to that name in type(x)

This doesn't affect what happen when you call iter(x) or have
it implicitly called for you by looping e.g. with "for a in x:".

In THIS case, name 'f' does not enter the picture, however you
may have bound, re-bound, or failed to bind it.  Name '__iter__'
DOES enter the picture, and it's looked up in type(x).

It's just the same for builtin such as abs, hash, len, ... and
the correlated special-methods __abs__, __hash__, __len__, ....

The built-in MAY sometimes do a bit more work and/or checks on
your behalf -- some built-ins (and operations expressed in
other ways than by built-ins) fall back to some alternative
solution if appropriate (e.g. hash(x) falls back to id(x)
if type(x) has none of '__hash__', '__eq__', '__cmp__' as
attribute names) -- but apart from such potential extras, the
normal operation of these built-ins is to look up the special
method on the type (aka class) of their argument [always
excepting arguments that are instances of classic-classes,
as we said we would right at the start of this post!].


Besides built-ins, there are quite a few Python operations
that (as a part or whole of their job) look up attributes with
special names on the types of their arguments.  Consider for
example calling an object:
    x()
this is affected by type(x).__call__.  So, for example, when
you call a class x, this executes the __call__ special method
(if any) of type(x), x's metaclass, quite independently of
whether class x itself defines a __call__ special method (THAT
one would be used when INSTANCES of x are called).  You would
not expect that calling a class (to instantiate it) is affected
by whether the INSTANCES of that class are callable, right? This
may help explain why inheritance is not transitive AND why the
operations and built-ins access special attributes on the TYPES
of their arguments, not on their arguments themselves.


Alex