[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

[Anne Archibald]

2008/6/4 Dan Yamins <dyamins at gmail.com>:

> Anne, thanks so much for your help.  I still a little confused.   If your
> scenario about the the memory allocation is working is right, does that mean
> that even if I put a lot of ram on the machine, e.g. > 16GB, I still can't
> request it in blocks larger than the limit imposed by the processor
> architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?    What I really want is
> to be able to have my ability to request memory limited just by the amount
> of memory on the machine, and not have it depend on something about
> paging/memory mapping limits.  Is this a stupid/naive thing?

No, that's a perfectly reasonable thing to want, and it *should* be
possible with your hardware. Maybe I'm barking up the wrong tree and
the problem is something else. But if you'll bear with me:

For the last ten years or so, normal computers have used 32 bits to
address memory. This means  that any given process can only get at
2**32 different bytes of memory, which is 4 GB. The magic of virtual
memory means you can possibly have a number of different processes
addressing different 4 GB chunks, some of which may temporarily be
stored on disk at any given time. If you want to get at a chunk of
memory bigger than 4 GB, you need all of these things:

* Your CPU must be 64-bit, or else it has no hope of being able to
access more than 4 GB (in fact more than 3 GB for complicated reasons)
of physical memory.
* Your operating system must be 64-bit (64-bit PC CPUs have a 32-bit
compatibility mode they often find themselves running in) so that it
can know about all the real RAM you have installed and so that it can
support 64-bit programs.
* Your program must be 64-bit, so that it can access huge chunks of RAM.

For compatibility reasons, many 64-bit machines have most of their
software compiled in 32-bit mode. This isn't necessarily a problem -
the programs probably even run faster - but they can't access more
than 4 GB each. So it's worth finding out whether your python
interpreter is compiled as a 64-bit program (and whether that's even
possible under your version of OSX). Unfortunately I  don't know how
to find that out in your case.

Anne