Slow Ref cleanup in 2.2.1 with 100K+ objects on Linux. - or SAP DBAPI problem?

[Tim Peters]

[Brad Clements]
> ...
> So .. 230K+ records have been copied, and during this process it
> probably pulled in 200K+ other related parent records and is keeping
> all of those in RAM (duh, bad design).
>
> But interestingly, the copy subroutine has finished, and is
> "returning" to it's parent, however the process seems hung. I thought
> I had read somewhere about a problem cleaning up ref's (There aren't
> any cycles in this case), but I can't recall exactly.
>
> This is python 2.2.1 on RH 9
> ...
> Anyone seen a condition like this before?

Reports of C's free() taking time quadratic in the # of small objects
allocated have been seen before on Linux.  pymalloc in 2.3 cures it.  As an
experiment, you could trying rebuilding 2.2.1 with pymalloc enabled and see
if it makes things better.  pymalloc can't be recommended in 2.2 because of
a security hole, and also because it wasn't properly tuned for the changes
in Python's typical memory usage between the time pymalloc was first written
and when 2.2 came out.  Fixing all this for 2.3 was a miserable amount of
work, and won't get backported to the 2.2 line (well, not unless a
masochistic volunteer appears).  Trying your program under 2.3b1 would be an
even more interesting experiment (pymalloc is on by default in 2.3, the
security hole was plugged, and it's properly tuned for 2.3's typical memory
usage patterns).