The Magick of __call__ (Or, Digging Deeper Than I Ought To)

[Terry Reedy]

On 4/1/2011 11:07 AM, Corey Richardson wrote:
> All callables (things you can foo(bar)) are really just objects that
> implement the __call__ method, as far as I understand.
 > Well then, that would appear to make methods themselves callable,

Method are just function objects that are class attributes.
They are sometimes wrapped as method objects.
Details are different between Py2 and Py3.

>>>> class Foo(object):
> ...     def __call__(self, foo):
> ...             self.name(foo)

A __call__ instance method makes instances of the class callable.

> ...     def name(self, bar):
> ...             print "Name: {0}".format(bar)
> ...
>>>> foo = Foo()
>>>> foo("Me!")
> Name: Me!
>
> Ok, nothing out of the ordinary. But what happens if....
>
>>>> foo.name.__call__("Corey")
> Name: Corey
>>>> eval("foo.name" + (".__call__" * 9001) + "('Corey')")
> Name: Corey
>>>> foo.name.__call__.__call__.__call__.__call__.__call__("Corey")
> Name: Corey
>>>> eval("foo.name" + (".__call__" * 100000000) + "('Corey')")

Yes, trying to create a string of 900 million chars may cause a problem, 
especially when you try to concatenate anything on the end, which 
requires another 900 million char string to be created before the 
temporary is freed. How much memory do you have? To find out if the 
strings creation, or compilation, every finished

s = "foo.name" + (".__call__" * 100000000) + "('Corey')"
print 's done'
c = compile(s,'test','eval')
print 'c done'
eval(s)

If s is created, the compiled bytecode will be 200 mil bytes, I believe. 
If eval ever starts, it will first create 100 million wrappers, as Chris 
showed, before calling the last. Each is 32 bytes in Py3.2.

> looked!). Would looking at something such as PyPy's version of it be
> good for me / does anyone else have insights?

As to why method wrappers are not reused (explaining in Py3 terms):
accessing a method through the class returns the function/method itself;
accessing a method through an instance returns some type of bound method 
wrapper. There are several types of wrapper for the various types of 
callablesdef , but each has attributes that enable access to both the 
method and instance. The __call__ method for a particular type of 
wrapper knows how to call instances of that type of bound method. In 
other words, method attributes of instances are computed attributes, 
much like data properties. Functions are non-data descriptors, as 
explained in the descriptor HOW-TO.

Every time a method is accessed through an instance, a new wrapper is 
created. Why? 1. If you want to reuse a bound methods, just bind it to a 
name or something and reuse it. 2. To automatically keep it for possible 
reuse, which normally is not too common, it wold have to be kept in some 
hidden dict which would grow indefinitely unless pruned occasionally.

-- 
Terry Jan Reedy