Huge Dicts and perfomance
Brian Kelley
bkelley at wi.mit.edu
Thu Dec 20 13:44:30 EST 2001
I don't know if this affects you but:
There is a relatively easy way to increase dictionary performance.
I use dictionaries to store graph formats with edges and vertices. I
have found that using an instance as the dictionary lookup is quite
slow. Instead, I use integer handles as the lookup.
Attached is a piece of code that hashes on instances and also hashes on
a generated integer handle. Each node object has an attribute
node.handle which is unique for every node.
Note that using __hash__ to return the handle doesn't speed things up!
Python has to do a *lot* of machinations to see if two instances are the
same thing. For an order of magnitude speed up for using objects as
dictionary keys, use handles instead. You just have to make sure that
the handles are unique :)
In the test code included, I have noticed that using integer handles
instead of instances speeds lookup by a factor of 20.
See the attached code for a better idea of what I mean.
I hope this helps.
Brian
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test_handles.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20011220/9634208c/attachment.ksh>
More information about the Python-list
mailing list