[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

[David Cournapeau]

On Wed, 2008-06-04 at 21:38 -0400, Dan Yamins wrote:

> 
> Anne, thanks so much for your help.  I still a little confused.   If
> your scenario about the the memory allocation is working is right,
> does that mean that even if I put a lot of ram on the machine, e.g. >
> 16GB, I still can't request it in blocks larger than the limit imposed
> by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?

Definitely. For 32 bits, it is actually smaller than 4 Gb, because the
whole address range is used by the processus and the OS, and the OS
generally needs at least 1 Gb (I don't know the default for mac os X,
but both linux and windows default to 2 Gb: so a processus cannot use
more than 2 Gb of memory).

The whole point of having a 64 bits CPU in 64 bits mode is to have more
than 32 bits for the (virtual) address space. If it is not in 64 bits
mode, the pointer is 32 bits (void* is 4 bytes, as returned by malloc),
and you would have a hard time accessing anything above 32 bits. It is
possible to address more than 32 bits on 32 bits OS using some weird
extensions, with a "semi segmented" address (you address the memory with
an offset + an address, like in the old times), but I have never used
it.

IOW: with more than 4 Gb of "real" memory, the hardware can address it,
in 32 bits or 64 bits, but *you* cannot address  it in 32 bits mode
because your pointers are still 4 bytes. To address more than 4 Gb, you
need 8 bytes pointers. 

So the real question is how to do it on mac os X, and unfortunately, I
cannot help you. I know that mac os X supports executing both 32 bits
and 64 bits ("super fat" binaries with the ability to put up to 4
differents kind of binaries: ppc and x86, 32 bits and 64 bits in both
cases), but I don't know which subsystem you can use, and how to compile
it (you will certainly have to compile your application differently). I
remember having seen mentioned several times that for Leopard, you
cannot build a 64 bits binary using GUI, because no GUI subsystem is 64
bits compatible yet.

>     What I really want is to be able to have my ability to request
> memory limited just by the amount of memory on the machine, and not
> have it depend on something about paging/memory mapping limits.  Is
> this a stupid/naive thing? 

It is not stupid, I think everyone whished it was possible.
Unfortunately, in mac os X case at least, you cannot do it so simply
(yet).

David