[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

[Charles R Harris]

On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins <dyamins at gmail.com> wrote:

> I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor.
> In
> this, setting, I'm working with large arrays of binary data.  E.g, I want
> to
> make calls like:
>                Z = numpy.inner(a,b)
> where and b are fairly large  -- e.g. 20000 rows by 100 columns.
>
> However, when such a call is made, I get a memory error that I don't
> understand.
> Specifically:
>
> >>> s = numpy.random.binomial(1,.5,(20000,100))   #creates 20000x100 bin.
> array
> >>> r = numpy.inner(s,s)
> Python(1714) malloc: *** mmap(size=1600000000) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> Naively, the numpy.inner call should be fine on my system, since my
> computer has
> enough memory. (And, when it's run, I have checked to see that at least 5
> GB of
> RAM is free.)  The error message thus suggests there's some problem to do
> with
> memory mapping going on here: that somehow, numpy.inner is calling the mmap
> modeul, and that the address space is being exceeded.  And that all my
> extra RAM
> isn't being used here at all.
>
> So, I have three questions about this:
>     1) Why is mmap being called in the first place?  I've written to Travis
> Oliphant, and he's explained that numpy.inner does NOT directly do any
> memory
> mapping and shouldn't call mmap.  Instead, it should just operate with
> things in
> memory -- in which case my 8 GB should allow the computation to go through
> just
> fine.  What's going on?
>
>     2) How can I stop this from happening?  I want to be able to leverage
> large
> amounts of ram on my machine to scale up my computations and not be
> dependent on
> the limitations of the address space size.  If the mmap is somehow being
> called
> by the OS, is there some option I can set that will make it do things in
> regular
> memory instead?  (Sorry if this is a stupid question.)
>
>     3) Even if I had to use memory mapping, why is the 1.6 GB requirement
> failing?  I'm using a recent enough version of python, and I have a 64-bit
> processor with sufficient amount of memory.  I should be able to allocate
> at
> least 4 GB of address space, right?  But the system seems to be balking at
> the
> 1.6 GB request. (Again, sorry if this is stupid.)
>

Are both python and your version of OS X fully 64 bits?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/93c0238c/attachment.html>