slowdown with massive memory usage

[Hallvard B Furuseth]

Andrew MacIntyre wrote:
> On Sat, 30 Jul 2004, Hallvard B Furuseth wrote:
>> I have a program which starts by reading a lot of data into various
>> dicts.
>>
>> When I moved a function to create one such dict from near the beginning
>> of the program to a later time, that function slowed down by a factor
>> of 8-14:
> (...)
>
> Python 2.2 didn't use PyMalloc by default.  This leaves Python at the
> mercy of the platform malloc()/realloc()/free(), and Python has found
> rough spots with nearly every platform's implementation of these - which
> is why PyMalloc was written.
> 
> While it isn't certain that this is your problem, if you can rebuild your
> Python interpreter to include PyMalloc (--with-pymalloc I think), you can
> find out.

Thanks.  I'll check that when I get time.  Until then, malloc gets the
blame until proven innocent, since profiling and test output turned out
nothing else that was different.  (See my reply to Istvan.)

> Be warned that there were some bugs in PyMalloc that were fixed before
> Python 2.3 was released (when PyMalloc became a default option); as far as
> I recall, these bugfixes were never backported to 2.2x.  So I wouldn't
> recommend running a 2.2.3 PyMalloc enabled interpreter in production
> without seriously testing all your production code.

If PyMalloc helps, I'll push for an upgrade to 2.3.  Thanks again.

-- 
Hallvard