[SciPy-User] scipy.linalg.solve()'s overwrite option does not work

Joe Kington jkington at wisc.edu
Sat Nov 6 18:28:16 EDT 2010 

On Sat, Nov 6, 2010 at 4:46 PM, braingateway <braingateway at gmail.com> wrote:

> Joe Kington :
> >
> > On Sat, Nov 6, 2010 at 12:13 PM, braingateway <braingateway at gmail.com
> > <mailto:braingateway at gmail.com>> wrote:
> >
> >     David Warde-Farley:
> >     > On 2010-11-05, at 9:21 PM, braingateway wrote:
> >     >
> >     >
> >     >> Hi everyone,
> >     >> I believe the overwrite option is used for reduce memory usage.
> >     But I
> >     >> did following test, and find out it does not work at all. Maybe I
> >     >> misunderstood the purpose of overwrite option. If anybody could
> >     explain
> >     >> this, I shall highly appreciate your help.
> >     >>
> >     >
> >     > First of all, this is a SciPy issue, so please don't crosspost
> >     to NumPy-discussion.
> >     >
> >     >
> >     >>>>> a=npy.random.randn(20,20)
> >     >>>>> x=npy.random.randn(20,4)
> >     >>>>> a=npy.matrix(a)
> >     >>>>> x=npy.matrix(x)
> >     >>>>> b=a*x
> >     >>>>> import scipy.linalg as sla
> >     >>>>> a0=npy.matrix(a)
> >     >>>>> a is a0
> >     >>>>>
> >     >> False
> >     >>
> >     >>>>> b0=npy.matrix(b)
> >     >>>>> b is b0
> >     >>>>>
> >     >> False
> >     >>
> >     >
> >     > You shouldn't use 'is' to compare arrays unless you mean to
> >     compare them by object identity. Use all(b == b0) to compare by
> value.
> >     >
> >     > David
> >     >
> >     >
> >     Thanks for reply, but I have to say u did not understand my post
> >     at all.
> >     I did this 'is' comparison on purpose, because I wanna know if the
> >     overwrite flag is work or not.
> >     See following example:
> >      >>> a=numpy.matrix([0,0,1])
> >      >>> a
> >     matrix([[0, 0, 1]])
> >      >>> a0=a
> >      >>> a0 is a
> >     True
> >
> >
> > Just because two ndarray objects aren't the same doesn't mean that
> > they don't share the same memory...
> >
> > Consider this:
> > import numpy as np
> > x = np.arange(10)
> > y = x.T
> > x is y # --> Yields False
> > Nonetheless, x and y share the same data, and storing y doesn't double
> > the amount of memory used, as it's effectively just a pointer to the
> > same memory as x
> >
> > Instead of using "is", you should use "numpy.may_share_memory(x, y)"
> Thanks a lot for pointing this out! I were struggling to figure out
> whether the different objects share memory or not. And good to know
> a0=numpy.matrix(a) actually did not share the memory.
>  >>> print 'a0 shares memory with a?', npy.may_share_memory(a,a0)
> a0 shares memory with a? False
>  >>> print 'b0 shares memory with b?', npy.may_share_memory(b,b0)
> b0 shares memory with b? False
> I also heard that even may_share_memory is 'True', does not necessarily
> mean they share any element. Maybe, is 'a0.base is a' usually more
> suitable for this purpose?
>

Not to take this on too much of a tangent, but since you asked:

"x.base is y.base" usually doesn't work, even when x.base and y.base point
to the same memory...

Again, consider the transpose of an array combined with a bit of indexing:
import numpy as np
x = np.arange(10)
y = x[:10].T
x.base is y.base # <-- yields False
y.base is x # <-- yields False
np.may_share_memory(x,y) # <-- correctly yields True

The moral of this story is don't use object identity of any sort to
determine if ndarrays share memory.

I also heard that even may_share_memory is 'True', does not necessarily
> mean they share any element.


The reason why "may_share_memory" is not a guarantee that the arrays
actually do is due to situations like this:
import numpy as np
x = np.arange(10)
a = x[::2]
b = x[1::2]
np.may_share_memory(a, b) # <-- yields True

However, we could change every element in "a" without affecting "b", so they
don't _actually_ share memory.   The reason why "may_share_memory" yields
True is essentially due to the fact that "a" and "b" are both views into
overlapping regions of the same array.

As far as I know, "may_share_memory" is really the only test available
without dropping down to C to determine if two ndarray objects share chunks
of memory. It just errs on the side of caution, if you will.  If it returns
True, then the two ndarrays refer to overlapping regions of memory.

Of course, checking to see if the contents of the array are exactly the same
is a _completely_ different test, and "(x == y.ravel()).all()" is a fine way
to do that.  Just keep in mind that checking if the elements are the same is
very different than checking if the arrays share the same memory.  For
example:
x = np.arange(10)
y = x.copy()
(x == y).all() # <-- correctly yields True
np.may_share_memory(x, y) # <-- correctly yields False
They're two different tests, entirely.  Of course, for what you're doing
here, testing to see if the elements are the same is entirely reasonable.



>
> Back to the original question: is there anyone actually saw the
> overwrite_a or overwrite_b really showed its effect?
>

I'm afraid I'm not much help on the original question...



> If you could show me a repeatable example, not only for
> scipy.linalg.solve(), it can also be other functions, who provide this
> option, such as eig(). If it does not show any advantage in memory
> usage, I might still using numpy.linalg.
> >
> >     This means a0 and a is actually point to a same object. Then a0 act
> >     similar to the C pointer of a.
> >     I compared a0/b0 and a/b by 'is' first to show I did create a new
> >     object
> >     from the original matrix, so the following (a0==a).all()
> >     comparison can
> >     actually prove the values inside the a and b were not overwritten.
> >
> >     Sincerely,
> >     LittleBigBrain
> >     > _______________________________________________
> >     > SciPy-User mailing list
> >     > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     > http://mail.scipy.org/mailman/listinfo/scipy-user
> >     >
> >
> >     _______________________________________________
> >     SciPy-User mailing list
> >     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20101106/965f0726/attachment.html>

More information about the SciPy-User mailing list