[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[Ivan Krstić]

On Dec 22, 2008, at 1:13 PM, Mike Coleman wrote:
> On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> BTW: Rather than using a huge in-memory dict, I'd suggest to either
>> use an on-disk dictionary such as the ones found in mxBeeBase or
>> a database.
>
> I really want this to work in-memory.  I have 64G RAM, and I'm only
> trying to use 45G of it ("only" 45G :-), and I don't need the results
> to persist after the program finishes.

It's still not clear to me, from reading the whole thread, precisely  
what you're seeing. A self-contained test case, preferably with  
generated random data, would be great, and save everyone a lot of  
investigation time. In the meantime, can you 1) turn off all swap  
files and partitions, and 2) confirm positively that your CPU cycles  
are burning up in userland?

(In general, unless you know exactly why your workload needs swap, and  
have written your program to take swapping into account, having _any_  
swap on a machine with 64GB RAM is lunacy. The machine will grind to a  
complete standstill long before filling up gigabytes of swap.)

--
Ivan Krstić <krstic at solarsail.hcs.harvard.edu> | http://radian.org