Memory limit to dict?

[Olivier Langlois]

Just out of curiosity. What is the OS that you are using? I am asking
because I do not know about XP 64bits edition but if you are using
Windows XP 32 bits version, no matter how much RAM you have on your
machine, the OS has an inherent 4GB address space limit per process.

Olivier Langlois
http://www.quazal.com

> -----Original Message----->
> I was wondering whether certain data structures in Python, e.g. dict,
> might have limits as to the amount of memory they're allowed to take
up.
> Is there any documentation on that?
> 
> Why am I asking? I'm reading 3.6 GB worth of BLAST output files into a
> nested dictionary structure (dict within dict ...). Looks something
like
> this:
> 
> { GenomeID:
>   { ProteinID:
>     { GenomeID:
>       { ProteinID, Score, PercentValue, EValue } } } }
> 
> Now, the thing is: Even on a machine with 16 GB RAM, the program
> terminates with a MemoryError, obviously long before the machine's RAM
> is used up.
> 
> I've no idea how far the Windows task manager's resource monitor can
be
> trusted -- probably not as far as I could throw a heavy-set St Bernard
> --, but it seems to stop roughly when that monitor records a swap file
> size of 2.2 GB.
> 
> Barring any revamping of the code itself, which I will have to do
> anyway, is there anything so far that would indicate a problem
inherent
> to Python?