generator object, next method

[Duncan Booth]

Paul Rubin wrote:

> Duncan Booth <duncan.booth at invalid.invalid> writes:
>> 1) Every time you access gen.next you create a new method-wrapper
>> object. 
> 
> Why is that?  I thought gen.next is a callable and gen.next() actually
> advances the iterator.  Why shouldn't gen.next always be the same
> object? 

It is a consequence of allowing methods to be first class objects, so 
instead of just calling them you can also save the bound method in a 
variable and call it with the 'self' context remembered in the method.

It is easier to see what is happening if you look first at ordinary Python 
classes and instances:

>>> class C:
	def next(self): pass

	
>>> c = C()
>>> c.next
<bound method C.next of <__main__.C instance at 0x00B4D6E8>>
>>> C.next
<unbound method C.next>

Here, 'next' is a method of the class C. You can call the unbound method, 
but then you have to explicitly pass the 'self' argument.

Whenever you access the method through an instance it creates a new 'bound 
method' object which stores references to both the original function, and 
the value to be passed in as the first parameter. Usually this object is 
simply called and discarded, but you can also save it for later use.

Python could perhaps bypass the creation of bound method objects when 
calling a function directly, but it would still need them for cases where 
the method isn't called immediately (and it isn't obvious it would be an 
improvement if it tried to optimise this case).

It would be possible for a language such as Python to try to either 
generate these bound method objects in advance (which would be horribly 
inefficient if you created lots of objects each of which had hundreds of 
methods which were never called), or to cache bound method objects so as to 
reuse them (which would be inefficient if you have lots of methods called 
only once on each object). Python chooses to accept the hit of creating 
lots of small objects, but tries to make the overhead of this as low as 
possible (which is one reason the memory gets reused immediately).

The next method in a generator works in the same way as bound methods, 
although the actual types involved are C coded. You can still access both 
the bound and unbound forms of the next method. The bound form carries the 
information about the first parameter, and the unbound form has to be given 
that information:

>>> gen = iterator()
>>> gen.next
<method-wrapper object at 0x00B43E30>
>>> type(gen).next
<slot wrapper 'next' of 'generator' objects>
>>> gen.next()
'hi'
>>> type(gen).next(gen)
'hi'