[Numpy-discussion] find machine maximum numpy array size

[Robert Kern]

On Mon, Jan 17, 2011 at 06:35, Tom Holderness
<tom.holderness at newcastle.ac.uk> wrote:
> Hi,
>
> How do I find the maximum possible array size for a given data type on a given architecture?
> For example if I do the following on a 32-bit Windows machine:
>
> matrix = np.zeros((8873,9400),np.dtype('f8'))
>
> I get,
> Traceback (most recent call last):
>  File "<pyshell#115>", line 1, in <module>
>    matrix = np.zeros((8873,9400),np.dtype('f8'))
> MemoryError
>
> If I reduce the matrix size then it works.
> However, if I run the original command on an equivalent 32-bit Linux machine this works fine (presumably some limit of memory allocation in the Windows kernel? I tested increasing the available RAM and it doesn't solve the problem).
>
> Is there a way I can find this limit? When distributing software to users (who all run different architectures) it would be great if we could check this before running the process and catch the error before the user hits "run".

No, there is no way to know a priori. It's not just dependent on the
system, but also on where memory is currently allocated. You could in
principle determine the maximum size of the address space from the
system. However, if you have, say, 3 Gb of address space for your
process, and you allocate two blocks of 1 Gb each, those allocations
may not be right next to each other. If the blocks start on 0.5 Gb and
2.0 Gb respectively, there is 1 Gb of free address space, but broken
up into two 0.5 Gb blocks. You could not allocate a third 1 Gb block
because there is nowhere to put it.

You can detect this problem early by doing an np.empty() of the right
size early in the program.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco