[Numpy-discussion] Numeric -> numarray gives MemoryError
christopheranderson at shaw.ca
christopheranderson at shaw.ca
Tue May 27 08:55:15 EDT 2003
Hi,
I've started to move some of my Numeric code to numarray 0.5 and have run into a problem. This is a relatively large optimization problem that requires finite difference estimation of gradients.
The implementation uses either Numeric/LinearAlgebra/RandomArray or numarray/LinearAlgebra2/RandomArray2; these are the only extensions used. The code itself is fairly straightforward and ports from Numeric to numarray with no modifications. Typical run times are on the order of hours with Numeric.
When using Numeric, I am able to repeat multiple optimization runs (> 10) with no problems. When the same code is run with numarray instead, I get this error part of the way through the first run:
Traceback (most recent call last):
File "runFD2.py", line 81, in ?
W1, W2 = estimFD2(inputs, outputs, Nh, Ne, alpha)
File "optimFD2.py", line 171, in estimFD2
gradW1, gradW2 = dEdW(inputs, outputs, MSE, alpha, W1, W2, f1, f2)
File "optimFD2.py", line 83, in dEdW
y_plus = evalFD2(inputs, W1_plus, W2, f1, f2)
File "optimFD2.py", line 63, in evalFD2
h1 = dot(inputs, W1)
File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 940, in dot
return innerproduct(a, swapaxes(inputarray(b), -1, -2))
File "F:\Python22\Lib\site-packages\numarray\ufunc.py", line 1866, in innerproduct
a = a.astype(rtype)
File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 478, in astype
return self.copy()
File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 553, in copy
c = ndarray.NDArray.copy(self)
File "F:\Python22\Lib\site-packages\numarray\ndarray.py", line 571, in copy
arr._data = memory.new_memory(arr._itemsize * arr.nelements())
MemoryError
>>>
This is a little frustrating, because numarray is clearly much faster. Checking the results along the way shows that numarray gives the same outputs as Numeric, so it doesn't appear to be a porting issue. The datasets involved are large (~10000x20). Initial memory usage with Numeric and numarray is similar, and is well below the machine's limit. The code is running on a Windows 2000 machine using the binary release of numarray.
What might be causing this problem? Should numarray be less tolerant than Numeric in a situation like this?
Thanks.
Chris
More information about the NumPy-Discussion
mailing list