[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

[Charles R Harris]

On Wed, Jun 4, 2008 at 9:07 PM, Anne Archibald <peridot.faceted at gmail.com>
wrote:

> 2008/6/4 Dan Yamins <dyamins at gmail.com>:
> >
> >
> >
> >>
> >> Try
> >>
> >> In [3]: numpy.dtype(numpy.uintp).itemsize
> >> Out[3]: 4
> >>
> >> which is the size in bytes of the integer needed to hold a pointer. The
> >> output above is for 32 bit python/numpy.
> >>
> >> Chuck
> >
> > Check, the answer is 4, as you got for the 32-bit.   What would the
> answer
> > be on a 64-bit architecture?  Why is this diagnostic?
>
> In a 64-bit setting, a pointer needs to be 64 bits long, that is,
> eight bytes, not four.
>
> What Charles pointed out was that while the inner product is very big,
> it seems to fit into memory on his 32-bit Linux machine; is it
> possible that OSX is preventing your python process from using even
> the meager 2-3 GB that a 32-bit process ought to get? In particular,
> try running Charles' script in a fresh python interpreter and see if
> it works; it may be that other arrays you had allocated are taking up
> some of the space that this one could.
>
> You will probably still want a 64-bit python, though, in order to have
> a little elbow room.
>

I think the difference is that Linux takes up 1 GiB, leaving 3 GiB to the
process, while I suspect OS X is taking up 2 GiB. I don't have a Mac, so I
don't really know. When I ran the the problem it took about 1.8 GiB, making
it a close run thing if the Mac only gives 32 bit proccesses 2 GiB.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/3b24b3bd/attachment.html>