[Numpy-discussion] Numeric -> numarray gives MemoryError

[Todd Miller]

On Tue, 2003-05-27 at 11:52, christopheranderson at shaw.ca wrote:
> Hi,
> 
> I've started to move some of my Numeric code to numarray 0.5 and have run into a problem. This is a relatively large optimization problem that requires finite difference estimation of gradients.
> 
> The implementation uses either Numeric/LinearAlgebra/RandomArray or numarray/LinearAlgebra2/RandomArray2; these are the only extensions used. The code itself is fairly straightforward and ports from Numeric to numarray with no modifications. Typical run times are on the order of hours with Numeric.
> 
> When using Numeric, I am able to repeat multiple optimization runs (> 10) with no problems. When the same code is run with numarray instead, I get this error part of the way through the first run:
> 
> Traceback (most recent call last):
>   File "runFD2.py", line 81, in ?
>     W1, W2 = estimFD2(inputs, outputs, Nh, Ne, alpha)
>   File "optimFD2.py", line 171, in estimFD2
>     gradW1, gradW2 = dEdW(inputs, outputs, MSE, alpha, W1, W2, f1, f2)
> 
>   File "optimFD2.py", line 83, in dEdW
>     y_plus = evalFD2(inputs, W1_plus, W2, f1, f2)
>   File "optimFD2.py", line 63, in evalFD2
>     h1 = dot(inputs, W1)
>   File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 940, in dot
>     return innerproduct(a, swapaxes(inputarray(b), -1, -2))
>   File "F:\Python22\Lib\site-packages\numarray\ufunc.py", line 1866, in innerproduct
>     a = a.astype(rtype)
>   File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 478, in astype
> 
>     return self.copy()
>   File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 553, in copy
>     c = ndarray.NDArray.copy(self)
>   File "F:\Python22\Lib\site-packages\numarray\ndarray.py", line 571, in copy
>     arr._data = memory.new_memory(arr._itemsize * arr.nelements())
> MemoryError
> >>>
> 
> This is a little frustrating, because numarray is clearly much faster. Checking the results along the way shows that numarray gives the same outputs as Numeric, so it doesn't appear to be a porting issue. The datasets involved are large (~10000x20). Initial memory usage with Numeric and numarray is similar, and is well below the machine's limit. The code is running on a Windows 2000 machine using the binary release of numarray.
> 

Well... the faster part is good. :)

> What might be causing this problem?

It sounds to me like a memory leak (reference counting error) in
numarray.  

Did you run the same code with older versions of numarray,  or did you
start with numarray-0.5?

How long does it take to fail?

Can you mail me your code (or better, the smallest derivative of it
which reproduces the problem)?

>  Should numarray be less tolerant than Numeric in a situation like
this?

No,  it's just newer and less mature.
 
> 
> Thanks.

You're welcome!

Todd
> 
> Chris
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: ObjectStore.
> If flattening out C++ or Java code to make your application fit in a
> relational database is painful, don't do it! Check out ObjectStore.
> Now part of Progress Software. http://www.objectstore.net/sourceforge
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller 			jmiller at stsci.edu
STSCI / ESS / SSB