Memory limit to dict?

[Olivier Langlois]

With 32 bits address space, you have exactly 4GB. Of course out of that
4GB, the OS reserves some of that space for itself. So I think that 2GB
of available memory for a user process is good approximation. However,
on top of that, you need to account for space overhead that Python and
the OS heap management will generate. I have written an article
discussing memory allocation overhead:

http://www3.sympatico.ca/olanglois/CFixAlloc.html

It is for C++ but I believe that concepts still apply for Python users
and could give the original poster a better understanding about the
problem he is facing.

I have kept thinking about the original problem and I now believe that
the only solution if he wants to store 3.6GB of data in a Python script
is to recompile Python in 64 bits. I do not know if this is something
that someone has already done successfully...

Olivier Langlois
http://www.quazal.com
 

> -----Original Message-----
> From: python-list-bounces+olanglois=quazal.com at python.org
[mailto:python-
> list-bounces+olanglois=quazal.com at python.org] On Behalf Of Burton
Samograd
> Sent: April 11, 2006 2:10 PM
> To: python-list at python.org
> Subject: Re: Memory limit to dict?
> 
> Peter Beattie <Peter.Beattie at web.de> writes:
> 
> <snip>
> > I've no idea how far the Windows task manager's resource monitor can
be
> > trusted -- probably not as far as I could throw a heavy-set St
Bernard
> > --, but it seems to stop roughly when that monitor records a swap
file
> > size of 2.2 GB.
> 
> Not being a windows expert at all, but I would assume with 32 bit
> windows each process in the system can have an address space of ~2
> gigs.  In linux the process address space is split in half, bottom 2
> gigs for OS mappings, top for the process, so it looks like you might
> just be hitting the maximum allowed address space mapping.
> 
> You should partition your data into hierarchial modules and let python
> do the swapping for you...although you have 16 gigs (I have to put a
> holy crap after that!) you will always run into process limits, at
> least until true 64 bit os's are in vouge.
> 
> --
> burton samograd					kruhft .at.
gmail
> kruhft.blogspot.com	www.myspace.com/kruhft	metashell.blogspot.com
> --
> http://mail.python.org/mailman/listinfo/python-list