[Numpy-discussion] Numeric -> numarray gives MemoryError

[christopheranderson at shaw.ca]

Hi,

I've started to move some of my Numeric code to numarray 0.5 and have run into a problem. This is a relatively large optimization problem that requires finite difference estimation of gradients.

The implementation uses either Numeric/LinearAlgebra/RandomArray or numarray/LinearAlgebra2/RandomArray2; these are the only extensions used. The code itself is fairly straightforward and ports from Numeric to numarray with no modifications. Typical run times are on the order of hours with Numeric.

When using Numeric, I am able to repeat multiple optimization runs (> 10) with no problems. When the same code is run with numarray instead, I get this error part of the way through the first run:

Traceback (most recent call last):
  File "runFD2.py", line 81, in ?
    W1, W2 = estimFD2(inputs, outputs, Nh, Ne, alpha)
  File "optimFD2.py", line 171, in estimFD2
    gradW1, gradW2 = dEdW(inputs, outputs, MSE, alpha, W1, W2, f1, f2)

  File "optimFD2.py", line 83, in dEdW
    y_plus = evalFD2(inputs, W1_plus, W2, f1, f2)
  File "optimFD2.py", line 63, in evalFD2
    h1 = dot(inputs, W1)
  File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 940, in dot
    return innerproduct(a, swapaxes(inputarray(b), -1, -2))
  File "F:\Python22\Lib\site-packages\numarray\ufunc.py", line 1866, in innerproduct
    a = a.astype(rtype)
  File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 478, in astype

    return self.copy()
  File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 553, in copy
    c = ndarray.NDArray.copy(self)
  File "F:\Python22\Lib\site-packages\numarray\ndarray.py", line 571, in copy
    arr._data = memory.new_memory(arr._itemsize * arr.nelements())
MemoryError
>>>

This is a little frustrating, because numarray is clearly much faster. Checking the results along the way shows that numarray gives the same outputs as Numeric, so it doesn't appear to be a porting issue. The datasets involved are large (~10000x20). Initial memory usage with Numeric and numarray is similar, and is well below the machine's limit. The code is running on a Windows 2000 machine using the binary release of numarray.

What might be causing this problem? Should numarray be less tolerant than Numeric in a situation like this? 

Thanks.

Chris