why is this so slow?

[Andrew Dalke]

Lowell Kirsh wrote:

> I created the following class (I know it's a dirty hack) so I could do 
> foo.bar instead of using a dictionary and having to type foo['bar'] :
> 
>     class DefaultAttr(object):
>         def __getattribute__(self, attr):
>             if not hasattr(self, attr):
>                 return ''
>             return object.__getattribute__(self,attr)
> 
> but its use is totally slowing down my program. Why is it so slow and is 
> there a better way?

When you try foo.bar the __getattribute__ ends up
being called with self = foo and attr = "bar".

The hasattr(self, attr) is implemented something like

def hasattr(obj, attr):
   try:
     getattr(obj, attr)
     return 1
   except:
     return 0

That is, hasattr will do the same thing that getattr
does.  Which is exactly what foo.bar does.  So you
have a recursive call here.  To see that I'll
instrument the code you presented

count = 0

class DefaultAttr(object):
     def __getattribute__(self, attr):
         global count
         if not hasattr(self, attr):
             count += 1
             return ''
         return object.__getattribute__(self,attr)

After I defined the above I can test it like this

 >>> x = DefaultAttr()
 >>> print count
0
 >>> print x.y

 >>> print count
500
 >>>

See that count got set to 500?  What happened was
the Python stack hit it's limit

 >>> import sys
 >>> sys.getrecursionlimit()
1000

Looks like there's one Python stack frame for
the hasattr and one for the __getattribute__.

You don't see the exception because the hasattr
assumes that *any* exception means that the
attribute doesn't exist.  The actual code is in
dist/src/Python/bltinmodule.c in builtin_hasattr

         v = PyObject_GetAttr(v, name);
         if (v == NULL) {
                 PyErr_Clear();
                 Py_INCREF(Py_False);
                 return Py_False;
         }
         Py_DECREF(v);
         Py_INCREF(Py_True);
         return Py_True;

(Don't we tell people that a bare except is a no-no?
Eg, what if I hit ^C during the GetAttr test?  Will
it be ignored?  Or what if the process runs out of
memory?  Perhaps this is related to "Thar the Windows
stack blows!" commentary in the python-dev summary
for 2004-08-01 to 08-15?)

In other words, all the time is spent in hitting
the stack limit.

You might try it the other way around and return the
underlying attribute then only if that fails do you
return the default, like this

count = 0
class DefaultAttr(object):
     def __getattribute__(self, attr):
         global count
         count += 1
         try:
             return super(DefaultAttr, self).__getattribute__(attr)
         except AttributeError:
             return ""

x = DefaultAttr()
print "Start with", count
print "y is", repr(x.y)
print "And now", count
x.z = 5
print "Checking ...", count
print "z is", repr(x.z)
print "count after z", count
del x.z
print "z is now", repr(x.z)
print "count after 2nd z", count

When I run that I get

Start with 0
y is ''
And now 1
Checking ... 1
z is 5
count after z 2
z is now ''
count after 2nd z 3



				Andrew
				dalke at dalkescientific.com