[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

["Martin v. Löwis"]

>> Allocation of a new pool would have to do a linear search in these
>> pointers (finding the arena with the least number of pools);
> 
> You mean the least number of free pools, right?

Correct.

> IIUC, the heuristic is to favour
> a small number of busy arenas rather than a lot of sparse ones.

Correct. Or, more precisely, the hope is indeed to make most arenas
sparse, so that they eventually see all their pools freed.

> And, by linear search in these pointers, do you mean just probe the 64 lists for
> the first non-NULL list head?

Correct.

> If so, then it's likely fast enough for a rather infrequent operation.

I would hope so, yes. However, the same hope applied to the current
code (how much time can it take to sink an arena in a linear list?),
so if we have the prospect of using larger arenas some day, this might
change.

> Now, we should find a way to benchmark this without having to steal Mike's
> machine and wait 30 minutes every time.

I think this can be simulated by using just arena objects, with no
associated arenas, and just adjusting pool counters. Allocate 100,000
arena objects, and start out with them all being completely allocated.
Then randomly chose one arena to deallocate a pool from; from time to
time, also allocate a new pool. Unfortunately, this will require some
hacking of the code to take the measurements.

Alternatively, make the arena size 4k, and the pool size 32 bytes, and
then come with a pattern to allocate and deallocate 8 byte blocks.
Not sure whether the code works for these parameters, though (but
it might be useful to fix it for non-standard sizes). This would require
only 400MiB of memory to run the test.

I think obmalloc is fairly independent from the rest of Python,
so it should be possible to link it with a separate main() function,
and nothing else of Python.

Regards,
Martin