Max Size of Array when using Numeric.py

[Ionel Simionescu]

Paul F. Dubois <dubois1 at llnl.gov> wrote in message
news:nygX3.2040$V4.64389 at news1.frmt1.sfba.home.com...
| There is no intrinsic limit on the size of NumPy arrays. You just ran out
of
| memory, I guess.

"just run out of memory" will happen when your memory manager cannot
allocate that much space of *contiguous* memory. This can have puzzling
effects when your heap is badly fragmented.

| You could write a C extension that created the array and did the reading
of
| each element and placing it into the array so that you only had the one
| copy.

While I see that arrays defined by extension modules can be guaranteed to
remain unique, the same can be also assured with Numeric, because arrays can
be written "in place".

However, when you need to change the type of an array, you must duplicate it
anyway, even if your arrays are defined in extensions.


Helka FOLCH <helka.folch at edf.fr> wrote in message
news:38286054.C8612EB5 at edf.fr...
> I've tried to create an array object whose shape is (5500,200) but I get
> a memory error. Is there a maximum size for array objects?
>
> Maybe there's a better way of creating the array than what I did :
> My program reads the array which is in another format from a file. It
> transforms the array to the right format and holds it in a variable(mv)
> as a string. Then I just do:
>
> m = array(eval(mv),Float)


If you afford to use an <eval> to get the actual values, then your string,
<mv>, should be really large, and it may be it the one which eats up your
memory, rather than the arrays.

Ways to really improve the speed of reading & converting the data can rely
on the struct standard module.

However, those are really useful when you must tackle heterogeneous data
series (i.e. data structures).

For homogeneous data series you can use Numeric's own methods:

a = fromstring(data, typecode)
and
data = a.tostring()

, where data is a binary string you can read from/write into a file, or,
even better, store in a database.

ionel