32-bit python memory limits?

[Grant Edwards]

On 2008-06-24, Gary Robinson <garyrob at mac.com> wrote:

> I'm running a Python job on OS X 10.5.3 and the Python 2.5.2
> that's available as a binary download at python.org for OS X.
>
> I ran a python program tonight that ended up using much more
> memory than anticipated. It just kept on using more and more
> memory. Instead of killing it, I just watched it, using
> Activity Monitor. I assumed that when it had 2GB allocated it
> would blow up, because I thought 32-bit python could only
> address 2GB.

You assumed incorrectly. :)

> But Activity Monitor reported that it had allocated 3.99GB of
> virtual memory before it finally blew up with malloc errors.
> Was my understanding of a 2GB limit wrong? I guess so! But I'm
> pretty sure I saw it max out at 2GB on linux...

You may have been using a Linux kernel with a 2GB user VM
limit.

> Anybody have an explanation, or is it just that my
> understanding of a 2GB limit was wrong? Or was it perhaps
> right for earlier versions, or on linux...??

In the past, it was pretty common for Linux kernels to be
configured so that a user-space program had 2GB of virtual
address space.  For reasons that aren't really important here,
doing so made it simpler to manage various kernel vs. user
memory issues.  Back when nobody could afford more than a few
hundred MB of RAM, that made sense.  These days, the Linux
kernel can be built for 2GB, 3GB or 4GB of VM space. There's a
tiny bit of extra overhead if you pick 3GB or 4GB, so users
with 2GB or less of physical RAM still often configure kernels
for a 2GB limit.

http://kerneltrap.org/node/2450

-- 
Grant Edwards                   grante             Yow!  Can you MAIL a BEAN
                                  at               CAKE?
                               visi.com