map del efficiency python2.2 and 2.1

Tim Peters tim.one at comcast.net
Tue Jul 16 17:12:56 EDT 2002


[John Hunter]
> I have a file formatted like
>
> word1 int1
> word2 int2
>
> with 800,000+ lines like that.  I want to load that file into a map
>
>   m[word] = int
>
> and my original implementation was quite slow.

...

> In the process of profiling, however, I found something surprising.
> Most of the total execution time of the code is spent clearing the map
> from memory, after all the lines in my script have executed (ie, the
> equivalent of calling __del__ on a dictionary type.

...

Before trying anything else, rebuild with pymalloc enabled.  We've seen many
cases where the platform free() ends up in seemingly quadratic-time behavior
when deleting gazillions of objects, presumably while trying to coalesce
them into larger free blocks.  pymalloc usually fixes that.  Would also be
interesting to try this under current CVS Python (where pymalloc is enabled
by default, and handles strings too).






More information about the Python-list mailing list