From cjw at sympatico.ca  Wed May  5 13:28:02 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Wed May  5 13:28:02 2004
Subject: [Numpy-discussion] numarraycore
Message-ID: <40994E36.2040203@sympatico.ca>

It would help to have some documentation on the purpose and usage of the 
class UsesOpPriority and its variable op_priority.






From jmiller at stsci.edu  Wed May  5 13:52:03 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May  5 13:52:03 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <40994E36.2040203@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
Message-ID: <1083790275.28609.17.camel@halloween.stsci.edu>

On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> It would help to have some documentation on the purpose and usage of the 
> class UsesOpPriority and its variable op_priority.

The basic idea was that NumArray subclasses which want to use numarray
operators would:

1.  subclass from UsesOpPriority 

2.  set a class level op_priority > 0. 


NumArrays have op_priority 0,  higher priorities are given "precedence".


Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
A.op_priority==0 and B.op_priority==1, then:  
A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
the type(A+B) could be NumArraySubclass rather than NumArray.  Different
subclasses could use higher or lower op_priorities to perform the same
kind of operator resolution among themselves.


Todd
-- 
Todd Miller <jmiller at stsci.edu>




From cjw at sympatico.ca  Wed May  5 16:01:01 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Wed May  5 16:01:01 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <1083790275.28609.17.camel@halloween.stsci.edu>
References: <40994E36.2040203@sympatico.ca> <1083790275.28609.17.camel@halloween.stsci.edu>
Message-ID: <409971DF.4070908@sympatico.ca>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040505/a655a6ed/attachment.html>

From travis at enthought.com  Thu May  6 05:30:02 2004
From: travis at enthought.com (Travis N. Vaught)
Date: Thu May  6 05:30:02 2004
Subject: [Numpy-discussion] ANN: SciPy 2004 Conference - Python for Scientific Computing
Message-ID: <409A2F1C.7050801@enthought.com>

Greetings,

The 1st annual *SciPy Conference* will be held this year at Caltech, 
September 2-3, 2004.  As some of you may know, we've experienced great 
participation in two SciPy "Workshops" (with ~70 attendees in both 2002 
and 2003) and this year we're graduating to a "conference."  With the 
prestige of a conference comes the responsibility of a keynote address.  
This year, Jim Hugunin has answered the call and will be speaking to 
kickoff the meeting on Thursday September 2nd.  Jim is the creator of 
Numeric Python, Jython, and co-designer of AspectJ. Jim is currently 
working on IronPython--a fast implementation of Python for .NET and Mono.

Registration is now open.  More information can be found here:

http://www.scipy.org/wikis/scipy04

You may register early online for $100.00.  Registration includes 
breakfast and lunch Thursday & Friday and a very nice dinner Thursday 
night.  After July 16, registration will cost $150.00.

Call for Presenters:
If you are interested in presenting at the conference, you may submit an 
abstract in Plain Text, PDF or MS Word formats to abstracts at scipy.org -- 
the deadline for abstract submission is July 1, 2004.  Papers and/or 
presentation slides are acceptable and are due by August 20, 2004.

We're also planning three days of informal "Coding Sprints" prior to the 
conference -- August 30 to September 1, 2004.  Conference registration 
is not required to participate in the sprints.  Please email the list, 
however, if you plan to attend.  Topics for these sprints will be 
determined via the mailing lists as well, so please submit any 
suggestions for topics to the scipy-user list:

list signup: http://www.scipy.org/mailinglists/
list address: scipy-user at scipy.org

Please forward this announcement to anyone/list that might be interested.

I look forward to seeing you at the conference.

Best Regards,

Travis N. Vaught




From jmiller at stsci.edu  Thu May  6 11:09:07 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May  6 11:09:07 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <409971DF.4070908@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
	 <1083790275.28609.17.camel@halloween.stsci.edu>
	 <409971DF.4070908@sympatico.ca>
Message-ID: <1083866875.2626.315.camel@halloween.stsci.edu>

On Wed, 2004-05-05 at 18:59, Colin J. Williams wrote:
> 
> Todd Miller wrote:
> > On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> >   
> > > It would help to have some documentation on the purpose and usage of the 
> > > class UsesOpPriority and its variable op_priority.
> > >     
> > 
> Todd,
> 
> Thanks for this, perhaps it could be added to the pdf .  I like the
> basic idea.

OK.  What is it you want to do?

> > The basic idea was that NumArray subclasses which want to use numarray
> > operators would:
> > 
> > 1.  subclass from UsesOpPriority 
> UsesOpPriority has no methods and thus, to access the methods of
> NumArray, it would appear to be necessary to subclass from NumArray.

For limited but important uses (e.g. masked array) subclassing from
NumArray is not necessary;  I'd summarize these as the cases where a
class wants to take control of operators from NumArray.

> > 2.  set a class level op_priority > 0. 
> > 
> > 
> > NumArrays have op_priority 0,  higher priorities are given "precedence".
> > 
> > 
> > Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
> > A.op_priority==0 and B.op_priority==1, then:  
> > A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
> > the type(A+B) could be NumArraySubclass rather than NumArray.  Different
> > subclasses could use higher or lower op_priorities to perform the same
> > kind of operator resolution among themselves.
> >   
> I wonder about the desirability of having different levels of priority
> for the same level of subclassing and am puzzled that a float variable
> is used for the priority.

Different levels of priority for the same level of sub-classing makes
sense to me:  it let's you answer the question "What's the result type
of a PyMatrix instance + a SomeOtherSubclass instance?"

A float enables insertions of new members into the middle of an existing
order.

> 
> It would nice if this depth could be determined automatically. 
> Perhaps something like the script below could be used.

I don't think we want something automatic here.   The choices involved
are arbitrary so we just need to be able to define what we want.

Other than masked arrays, the problem we were trying to solve with
UsesOpPriority is more like:

class A(NumArray):              
	op_priority = 1

class B(NumArray):              
	op_priority = 2

whenever A op B is encountered, the result should be a B.  

That said, the solution we currently have is better described by:
whenever A op B is encountered, B.__rop__(A) is called in preference to
A.__op__(B).  With the example above,  even though B.__rop__ is called,
the result is still an A.  :-(   

I fixed this in CVS today to work more like the original description.

Regards,
Todd

> 
> Colin W.
> > Todd
> >   
> # tClassing.py
> 
> class A:
>   def __init__(self):
>     self.p= 0
>   def prior(self):
>     self.p+= 1
> class B(A):
>   def __init__(self):
>     A.__init__(self)
>     self.prior()
> class C(B):
>   def __init__(self):
>     B.__init__(self)
>     self.prior()
> 
> print A().p
> print B().p
> c= C()
> print 'Priority for C methods:', c.p
> ------------------------------------------------------- This SF.Net
> email is sponsored by Sleepycat Software Learn developer strategies
> Cisco, Motorola, Ericsson & Lucent use to deliver higher performing
> products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________ Numpy-discussion
> mailing list Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller <jmiller at stsci.edu>




From cjw at sympatico.ca  Thu May  6 16:17:01 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu May  6 16:17:01 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <1083866875.2626.315.camel@halloween.stsci.edu>
References: <40994E36.2040203@sympatico.ca>	 <1083790275.28609.17.camel@halloween.stsci.edu>	 <409971DF.4070908@sympatico.ca> <1083866875.2626.315.camel@halloween.stsci.edu>
Message-ID: <409AC74E.2080609@sympatico.ca>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040506/7d85270a/attachment.html>

From karthik at james.hut.fi  Fri May  7 01:23:05 2004
From: karthik at james.hut.fi (Karthikesh Raju)
Date: Fri May  7 01:23:05 2004
Subject: [Numpy-discussion] covariance and other linear algebra functions
Message-ID: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>

Hi,

Is am searching for a covariance function and other linear algebra
functions like eigen value decomposition. i am interested in both
scipy (havent managed to get scipy installed on the main computation
systems) and also numarray solution.

With warm regards
karthik


-----------------------------------------------------------------------
Karthikesh Raju,		    email: karthik at james.hut.fi
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
P.O Box 5400, FIN 02015 HUT,
Espoo, FINLAND
-----------------------------------------------------------------------



From jsaenz at wm.lc.ehu.es  Fri May  7 01:43:03 2004
From: jsaenz at wm.lc.ehu.es (Jon Saenz)
Date: Fri May  7 01:43:03 2004
Subject: [Numpy-discussion] covariance and other linear algebra functions
In-Reply-To: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>
References: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>
Message-ID: <20040507103548.I17596@lcdx00.wm.lc.ehu.es>

With NumPy:
http://www.pyclimate.org



Jon Saenz.				| Tfno: +34 946012445
Depto. Fisica Aplicada II               | Fax:  +34 946013500
Facultad de Ciencias.   \\ Universidad del Pais Vasco \\
Apdo. 644   \\ 48080 - Bilbao  \\ SPAIN

On Fri, 7 May 2004, Karthikesh Raju wrote:

> Hi,
>
> Is am searching for a covariance function and other linear algebra
> functions like eigen value decomposition. i am interested in both
> scipy (havent managed to get scipy installed on the main computation
> systems) and also numarray solution.
>
> With warm regards
> karthik
>
>
> -----------------------------------------------------------------------
> Karthikesh Raju,		    email: karthik at james.hut.fi
> Researcher,			    http://www.cis.hut.fi/karthik
> Helsinki University of Technology,  Tel: +358-9-451 5389
> Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
> Department of Computer Sc.,
> P.O Box 5400, FIN 02015 HUT,
> Espoo, FINLAND
> -----------------------------------------------------------------------
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by Sleepycat Software
> Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to deliver
> higher performing products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>



From jmiller at stsci.edu  Fri May  7 07:24:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri May  7 07:24:02 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <409AC74E.2080609@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
	 <1083790275.28609.17.camel@halloween.stsci.edu>
	 <409971DF.4070908@sympatico.ca>
	 <1083866875.2626.315.camel@halloween.stsci.edu>
	 <409AC74E.2080609@sympatico.ca>
Message-ID: <1083939769.3737.179.camel@localhost.localdomain>

On Thu, 2004-05-06 at 19:16, Colin J. Williams wrote:
> 
> Todd Miller wrote:
> > On Wed, 2004-05-05 at 18:59, Colin J. Williams wrote:
> >   
> > > Todd Miller wrote:
> > >     
> > > > On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> > > >   
> > > >       
> > > > > It would help to have some documentation on the purpose and usage of the 
> > > > > class UsesOpPriority and its variable op_priority.
> > > > >     
> > > > >         
> > > Todd,
> > > 
> > > Thanks for this, perhaps it could be added to the pdf .  I like the
> > > basic idea.
> > >     
> > OK.  What is it you want to do?
> Re: Above.
>    Document whatever you decide to do with UsesOpPriority

I put more comments in numarraycore.   If a lot of people want to
subclass NumArray, it will probably require more discussion because
UsesOpPriority is a limited scheme.  For now,  MaskedArrays do mostly
what I want and it should work for PyMatrix as well.

> Re: Below.
>   Access the methods of NumArray in a Pythonic manner.
>   Subclassing from UsesOpPriority doesn't appear to permit that.

No, it doesn't.  

> >   
> > > > The basic idea was that NumArray subclasses which want to use numarray
> > > > operators would:
> > > > 
> > > > 1.  subclass from UsesOpPriority 
> > > >       
> > > UsesOpPriority has no methods and thus, to access the methods of
> > > NumArray, it would appear to be necessary to subclass from NumArray.
> > >     
> > For limited but important uses (e.g. masked array) subclassing from
> > NumArray is not necessary;  I'd summarize these as the cases where a
> > class wants to take control of operators from NumArray.
> >   
> Thus, if masked array require use of the UsesOpPrioity, then direct
> subclassing from there would appear to make sense.  
> 
> However, where one wishes to use some of the NumArray methods, then,
> contrary to 1, above, subclassing from NumArray should, I suggest, be
> the way to go.

That's true... I overlooked the fact that UsesOpPriority is inherited
automatically by NumArray subclasses.  Sorry for the confusion.

> >   
> > > > 2.  set a class level op_priority > 0. 
> > > > 
> > > > 
> > > > NumArrays have op_priority 0,  higher priorities are given "precedence".
> > > > 
> > > > 
> > > > Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
> > > > A.op_priority==0 and B.op_priority==1, then:  
> > > > A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
> > > > the type(A+B) could be NumArraySubclass rather than NumArray.  Different
> > > > subclasses could use higher or lower op_priorities to perform the same
> > > > kind of operator resolution among themselves.
> > > >   
> > > >       
> > > I wonder about the desirability of having different levels of priority
> > > for the same level of subclassing and am puzzled that a float variable
> > > is used for the priority.
> > >     
> > Different levels of priority for the same level of sub-classing makes
> > sense to me:  it let's you answer the question "What's the result type
> > of a PyMatrix instance + a SomeOtherSubclass instance?"
> >   
> I agree, there is the possibility that one would wish to evaluate such
> expressions right to left and 
> the priority provides a way of doing this.
> > A float enables insertions of new members into the middle of an existing
> > order.
> >   
> True.  I hope that there will be a database somewhere of all the
> various values for op_priority

Here's the database:  :-)

class                  op_priority

NumArray               0
MaskedArray            1


Regards,
Todd







From gazzar at email.com  Sun May  9 21:14:00 2004
From: gazzar at email.com (Gary Ruben)
Date: Sun May  9 21:14:00 2004
Subject: [Numpy-discussion] Announce: ErrorVal.py 1.0 for error-bounds calculations
Message-ID: <20040510041258.DA8523CE181@ws3-4.us4.outblaze.com>

I finally got around to putting my errorbar calculation module on my website.
<http://users.bigpond.net.au/gazzar/python.html>
This should be of interest to Numeric Python users who deal with calculation of error-bounds on experimental data.

Synopsis:

A module providing a Python number type class and helper functions to ease
the task of computing error bounds on experimental data values.

This module defines an abstract data type, Err, which carries a central/prime
value and upper and lower error bounds.
Helper functions are provided to allow Python Numeric rank-1 arrays of Err objects
to be constructed with ease and to apply Numeric Ufuncs.
Written under Python 2.3.3 and Numeric 23.0

Example of usage:

upperPressure = ArrayOfErr([909., 802., 677., 585., 560., 548.], 1.0)
lowerPressure = ArrayOfErr([144., 246., 378., 469., 493., 505.], 1.0)
pressureDiff = upperPressure - lowerPressure
V_RStandard = ArrayOfErr([2.016, 2.016, 2.020, 2.017, 2.021, 2.019], 0.001)
R = 100.2                                   # standard resistor value [Ohm]
I = V_RStandard / R                         # current [A]
V_RAB = ArrayOfErr([6.167, 6.168, 6.170, 6.160, (6.153, 0.02), (5.894, 0.01)], 0.002)
R_AB = V_RAB / I
logR_AB = ApplyUfuncToErr(R_AB, log10)      # This means log10(R_AB)
print PrimeVals(logR_AB)
print MinVals(logR_AB)
print MaxVals(logR_AB)


Enjoy,
Gary Ruben

-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm




From faheem at email.unc.edu  Mon May 10 16:32:00 2004
From: faheem at email.unc.edu (Faheem Mitha)
Date: Mon May 10 16:32:00 2004
Subject: [Numpy-discussion] add.reduce for character arrays
In-Reply-To: <1082480003.3843.330.camel@localhost.localdomain>
References: <slrnc83e38.1gd.faheem@Chrestomanci.home.earth> 
 <1082383540.19596.72.camel@halloween.stsci.edu>  <Pine.LNX.4.58.0404192315010.25609@Chrestomanci>
 <1082480003.3843.330.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0405101842420.17884@Chrestomanci>


On Tue, 20 Apr 2004, Todd Miller wrote:

> On Mon, 2004-04-19 at 23:20, Faheem Mitha wrote:
> >
> > Yes, that works. Thanks.
> >
> > In [13]: numarray.objects.add.reduce(s,dim=1)
> > Out[13]: ObjectArray(['abc', 'def', 'ghi'])
> >
> > BTW, that makes me wonder what the rules are for the order in which this
> > reduction proceeds for arrays bigger than dimension two. Since there is no
> > total order in this case, the answer is not obvious.
>
> (I think) the answer is that for numarray, reductions ultimately occur
> along the innermost axis.  Off-axis reductions are performed by a
> swapaxis between the off-axis and the innermost axis, followed by the
> reduction, followed by a swapaxis back between the new innermost axis
> and the off-axis.  I'm not sure there's an especially good reason for
> the swap back (since the original swapped axis no longer exists),  but
> it seems to be what Numeric does and not doing it in numarray broke
> extension (linear_algebra?) self-tests.
>
> > For string
> > concatentation (since it is not commutative) at least this matters. I
> > didn't see this documented in the manual but I may have missed it.
>
> Reductions always occur in order of increasing indexes with the
> "reduction-so-far" as the left operand and the next index as the right
> operand.  Reductions occur in a sort of depth-first manner with the
> outermost dimensions varying most slowly.

Sorry about the slow reply. This message got lost in the mess in my inbox,
which I have recently being cleaning out.

I just wanted to say that it would be useful to have the above documented
in some appropriate place in the numarray manual. 99% of the time the
exact procedure for a reduction won't matter, but when someone is using a
non-commutative operator it will. If this is documented in the manual I
could not find it.

Thanks.

                                                               Faheem.



From karthik at james.hut.fi  Wed May 12 01:04:04 2004
From: karthik at james.hut.fi (Karthikesh Raju)
Date: Wed May 12 01:04:04 2004
Subject: [Numpy-discussion] Question about complex variables
Message-ID: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>

Hi All,

i found something strange about complex matrices/arrays. Is it true that
transposing them does not yeald the mathematically correct transpose (the
signs of the imaginary part is reversed)?  This leads to wrong results
while calculating covariances etc ..

Actually i find that things go a bit wild with complex number, is it true?

Warm regards

karthik

-----------------------------------------------------------------------
Karthikesh Raju,		    email: karthik at james.hut.fi
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
P.O Box 5400, FIN 02015 HUT,
Espoo, FINLAND
-----------------------------------------------------------------------



From rsilva at ime.usp.br  Wed May 12 03:46:05 2004
From: rsilva at ime.usp.br (Paulo J. S. Silva)
Date: Wed May 12 03:46:05 2004
Subject: [Numpy-discussion] Question about complex variables
In-Reply-To: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>
References: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>
Message-ID: <1084358724.21434.15.camel@catirina>

Karthik,

You are making a common mistake. In Mathematics, the "transpose" of a
matrix is just the matrix with the axis transposed like numarray/numeric
does. What you called transpose is actually the "adjoint" or "conjugate
transpose" of a matrix. Actually, this last operation (the adjoint) is
the most interesting one in Linear Algebra/Functional Analysis. For more
details take a look at:

http://en.wikipedia.org/wiki/Conjugate_transpose
http://en.wikipedia.org/wiki/Transpose

In the reals both operations coincide, thus the confusion. This is the
reason why matlab can use the same symbol for both operations.

Now, enough Mathematics, let's go to numarray. To get the adjoint of a
matrix A in numarray, you can simply do:

conjugate(transpose(A))

If you want, you may define a function named adjoint to perform the
above operations. 

Note that this creates a new matrix, not only a new view of the original
matrix like "transpose" function in numarray. 

Hope it helps,

Paulo
-- 
Paulo Jos? da Silva e Silva 
Professor Assistente do Dep. de Ci?ncia da Computa??o
(Assistant Professor of the Computer Science Dept.)
Universidade de S?o Paulo - Brazil

e-mail: rsilva at ime.usp.br          Web: http://www.ime.usp.br/~rsilva

Teoria ? o que n?o entendemos o    (Theory is something we don't)
suficiente para chamar de pr?tica. (understand well enough to call) 
                                   (practice)




From alvaro at antalia.com  Wed May 12 12:45:07 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Wed May 12 12:45:07 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
Message-ID: <1084391465.2145.8.camel@siff>

Hello,

I am new to numarray. I have been searching the documentation, and see
no evident vectorial and concise way to get the indexes for the minimum
of an array times.min().

Currently my two alternatives are: (times is NxN array)

husband, wife = nonzero(equal(times.min(),times)) 
#gives tuple of 1x1 arrays, each containing one index

and (even uglier)

husband = compress(times.min()==times,indices([N,N])[0])
wife    = compress(times.min()==times,indices([N,N])[1])

These are weird ways to get something as simple. I am surely missing
something, but I have tried several slicing strategies before without
success.

For getting the minimum times in each row I use:

choose(argmin(times),transpose(times))

What are the idioms in numpy for these tasks?


Thank you very much in advance, ?.

I would 

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From owen at astro.washington.edu  Wed May 12 12:56:10 2004
From: owen at astro.washington.edu (Russell E Owen)
Date: Wed May 12 12:56:10 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084391465.2145.8.camel@siff>
References: <1084391465.2145.8.camel@siff>
Message-ID: <p06100501bcc82fdc3eca@[140.142.183.23]>

At 9:51 PM +0200 5/12/04, ?lvaro Tejero Cantero wrote:
>I am new to numarray. I have been searching the documentation, and see
>no evident vectorial and concise way to get the indexes for the minimum
>of an array times.min().

To find the index of ONE minimum value is easy, 
though it is buried in the nd_image sub-package 
(where many users might miss it):
numarray.nd_image.minimum_position

I do not know a clean way to find all locations 
of the minimum value. I hope somebody else does.

-- Russell



From tim.hochberg at cox.net  Wed May 12 13:14:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed May 12 13:14:05 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084391465.2145.8.camel@siff>
References: <1084391465.2145.8.camel@siff>
Message-ID: <40A28558.1090008@cox.net>

?lvaro Tejero Cantero wrote:

>Hello,
>
>I am new to numarray. I have been searching the documentation, and see
>no evident vectorial and concise way to get the indexes for the minimum
>of an array times.min().
>  
>
If I understand your question correctly, you want argmin:

 >>> import numarray as na
 >>> from numarray import random_array
 >>> times = random_array.randint(0, 16, [4,4])
 >>> times
array([[15,  3, 11, 10],
       [ 1,  1, 15,  7],
       [ 4,  3,  5,  6],
       [10, 15, 12,  3]])
 >>> na.argmin(times)    # Takes min along axis 0
array([1, 1, 2, 3])
 >>> na.argmin(times, 1) # Take min along axis 1
array([1, 1, 1, 3])

-tim

>Currently my two alternatives are: (times is NxN array)
>
>husband, wife = nonzero(equal(times.min(),times)) 
>#gives tuple of 1x1 arrays, each containing one index
>
>and (even uglier)
>
>husband = compress(times.min()==times,indices([N,N])[0])
>wife    = compress(times.min()==times,indices([N,N])[1])
>
>These are weird ways to get something as simple. I am surely missing
>something, but I have tried several slicing strategies before without
>success.
>
>For getting the minimum times in each row I use:
>
>choose(argmin(times),transpose(times))
>
>What are the idioms in numpy for these tasks?
>
>
>Thank you very much in advance, ?.
>
>I would 
>
>  
>





From alvaro at antalia.com  Wed May 12 14:23:06 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Wed May 12 14:23:06 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100501bcc82fdc3eca@[140.142.183.23]>
References: <1084391465.2145.8.camel@siff>
	 <p06100501bcc82fdc3eca@[140.142.183.23]>
Message-ID: <1084397325.2354.16.camel@siff>

Hello,

> To find the index of ONE minimum value is easy, 
> though it is buried in the nd_image sub-package 
> (where many users might miss it):
> numarray.nd_image.minimum_position

It works great... but what about efficiency? If I do times.min() and
then numarray.nd_image.minimum_positioan(times) I am running twice
essentially the same extremum-finding routine, which is prohibitibe for
large N..., am I right?

Which makes me thing of a more general question: I know that some of the
array functions are coded in C for speed, but what about the classical
python-for loop, as in (r Nx3 array of particle positions)

[ [r[i]-r[j] for i in arange(N)] for j in arange(N)]

is this handled to C code?


> I do not know a clean way to find all locations 
> of the minimum value. I hope somebody else does.

Yes... although for the problem at hand that motivated my query, my
times matrix is symmetric... I don't really need all the minima, but
does numarray have any special datatype for symmetric matrixes, that
prevents storage of unneded (e.g. supradiagonal) elements?.


Thank you very much, I'm on my way to get some beautiful code out of old
fortranisms 

?.
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From tim.hochberg at cox.net  Wed May 12 14:34:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed May 12 14:34:05 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084397325.2354.16.camel@siff>
References: <1084391465.2145.8.camel@siff>	 <p06100501bcc82fdc3eca@[140.142.183.23]> <1084397325.2354.16.camel@siff>
Message-ID: <40A2980F.2060607@cox.net>

?lvaro Tejero Cantero wrote:

>Hello,
>
>  
>
>>To find the index of ONE minimum value is easy, 
>>though it is buried in the nd_image sub-package 
>>(where many users might miss it):
>>numarray.nd_image.minimum_position
>>    
>>
>
>It works great... but what about efficiency? If I do times.min() and
>then numarray.nd_image.minimum_positioan(times) I am running twice
>essentially the same extremum-finding routine, which is prohibitibe for
>large N..., am I right?
>
>Which makes me thing of a more general question: I know that some of the
>array functions are coded in C for speed, but what about the classical
>python-for loop, as in (r Nx3 array of particle positions)
>
>[ [r[i]-r[j] for i in arange(N)] for j in arange(N)]
>
>is this handled to C code?
>  
>


Try this:

 >>> import numarray as na
 >>> r = na.arange(5)
 >>> na.subtract.outer(r, r)
array([[ 0, -1, -2, -3, -4],
       [ 1,  0, -1, -2, -3],
       [ 2,  1,  0, -1, -2],
       [ 3,  2,  1,  0, -1],
       [ 4,  3,  2,  1,  0]])

Look up the special methods on ufuncs (outer, reduce, etc) for more details.

>>I do not know a clean way to find all locations 
>>of the minimum value. I hope somebody else does.
>>    
>>
>
>Yes... although for the problem at hand that motivated my query, my
>times matrix is symmetric... I don't really need all the minima, but
>does numarray have any special datatype for symmetric matrixes, that
>prevents storage of unneded (e.g. supradiagonal) elements?.
>  
>
Not that I know of.

-tim



>
>Thank you very much, I'm on my way to get some beautiful code out of old
>fortranisms 
>
>?.
>  
>





From perry at stsci.edu  Wed May 12 15:11:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 12 15:11:02 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084397325.2354.16.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> It works great... but what about efficiency? If I do times.min() and
> then numarray.nd_image.minimum_positioan(times) I am running twice
> essentially the same extremum-finding routine, which is prohibitibe for
> large N..., am I right?
>
Well, yes. But when you ask to find all the things that equal
the minimum, you pretty much must look twice (if you want to know
where they all are if more than one). Once to determine the
minimum, the next time to locate all of them.
You didn't really say whether you needed just the first minimum or
all minima (if I recall correctly).

> Which makes me thing of a more general question: I know that some of the
> array functions are coded in C for speed, but what about the classical
> python-for loop, as in (r Nx3 array of particle positions)
>
> [ [r[i]-r[j] for i in arange(N)] for j in arange(N)]
>
> is this handled to C code?
>
As Tim mentions, yes this can be done efficiently. But there
is no general answer this sort of open question. It depends
on what you are doing. Sometimes there are functions or
tricks to avoid loops in Python, sometimes not.
>
> Yes... although for the problem at hand that motivated my query, my
> times matrix is symmetric... I don't really need all the minima, but
> does numarray have any special datatype for symmetric matrixes, that
> prevents storage of unneded (e.g. supradiagonal) elements?.
>
Not for special cases like this. One could probably write a special
subclass to do this, but for a savings of a factor of 2 in memory,
it usually would not be worth the trouble (unlike sparse matrices)

Perry




From owen at astro.washington.edu  Wed May 12 16:59:07 2004
From: owen at astro.washington.edu (Russell E Owen)
Date: Wed May 12 16:59:07 2004
Subject: [Numpy-discussion] puzzling compiler warning for numeric extension
Message-ID: <p06100501bcc864cfb9df@[140.142.183.76]>

I modified an existing numeric c extension today, adding a function 
that returns a new array. This is the first time I've used 
NA_NewArray. The code seems to run fine, but I get a disturbing 
compiler warning:

[rowen:Shared Files/Guider/PyGuide] rowen% python setup.py build
...
src/radProfModule.c: In function `Py_radSqByRadInd':
src/radProfModule.c:378: warning: return from incompatible pointer type

The associated code can be summarized as follows (and it is simple 
enough that I've also appended a copy). The error is thrown on the 
last line:

static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
	long nElt;
	Long *radSqByRadInd;

	if (!PyArg_ParseTuple(args, "l", &nElt))
		return NULL;
...
	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
}

So...am I actually doing something wrong, or is this warning normal?

-- Russell

P.S. the full code:

static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
	long nElt;
	Long radInd;
	char ModName[] = "radSqByRadInd";
	Long *radSqByRadInd;

	if (!PyArg_ParseTuple(args, "l", &nElt))
		return NULL;

	if (nElt < 0) {
		PyErr_Format(PyExc_ValueError, "%s: nPts < 0", ModName);
		return NULL;
	}

	// allocate max(3, nElt) elements (simplifies the computation
	// and avoids calling calloc on 0 elements)
	radSqByRadInd = calloc(MAX(nElt, 3), sizeof *radSqByRadInd);
	if (radSqByRadInd == NULL) {
		PyErr_Format(PyExc_MemoryError, "%s: insufficient 
memory", ModName);
		return NULL;
	}

	for (radInd=0; radInd<3; radInd++) {
		radSqByRadInd[radInd] = radInd;
	}
	for (radInd=3; radInd<nElt; radInd++) {
		radSqByRadInd[radInd] = (radInd - 1) * (radInd - 1);
	}

	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
}



From rkern at ucsd.edu  Wed May 12 17:29:32 2004
From: rkern at ucsd.edu (Robert Kern)
Date: Wed May 12 17:29:32 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>
Message-ID: <40A2C0EE.703@ucsd.edu>

Perry Greenfield wrote:

> ?lvaro Tejero Cantero wrote:
> 
>>It works great... but what about efficiency? If I do times.min() and
>>then numarray.nd_image.minimum_positioan(times) I am running twice
>>essentially the same extremum-finding routine, which is prohibitibe for
>>large N..., am I right?
>>
> 
> Well, yes. But when you ask to find all the things that equal
> the minimum, you pretty much must look twice (if you want to know
> where they all are if more than one). Once to determine the
> minimum, the next time to locate all of them.

Nah, you can accumulate indices corresponding to the current minimum 
value as you go. Discard the list of indices and start again if you get 
a new minimum value.

minval = data[0]  # let's suppose data is a vector for demonstration
minindices = [0]

for i in xrange(1, len(data)):
     x = data[i]
     if x < minval:
         minindices = [i]
         minval = x
     elif x == minval:
         minindices.append(i)

Whether or not this is faster (when implemented in C) than going over it 
twice using numarray functions is another question. My guess: not enough.

[snip]

>>Yes... although for the problem at hand that motivated my query, my
>>times matrix is symmetric... I don't really need all the minima, but
>>does numarray have any special datatype for symmetric matrixes, that
>>prevents storage of unneded (e.g. supradiagonal) elements?.
>>
> 
> Not for special cases like this. One could probably write a special
> subclass to do this, but for a savings of a factor of 2 in memory,
> it usually would not be worth the trouble (unlike sparse matrices)

OTOH, having subclasses that follow LAPACK's symmetric packed storage 
scheme would be very useful not because of the space factor but the time 
saved by being able to use the symmetric algorithms in LAPACK. I think.

> Perry

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From alvaro at antalia.com  Thu May 13 03:32:02 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 03:32:02 2004
Subject: outer idiom (was Re: [Numpy-discussion] Getting the indexes of the
	myarray.min())
In-Reply-To: <40A2980F.2060607@cox.net>
References: <1084391465.2145.8.camel@siff>
	 <p06100501bcc82fdc3eca@[140.142.183.23]> <1084397325.2354.16.camel@siff>
	 <40A2980F.2060607@cox.net>
Message-ID: <1084444629.800.12.camel@siff>

Hello,

(for background, I am trying to get an array of interparticle distances,
in which calculation of supradiagonal elements is unneeded because of
symmetry. I am not doing anything about this now, though).

>>> import numarray.random_array as rdn
>>> N, D = 1000, 3 #number of particles and dimensions of space
>>> r = rnd.random([N,D])  # r[i] gives the D coordinates of particle i
                           # r[:,0] gives the all the x coordinates
What I was doing:

>>> r_rel = [[r[i]-r[j] for i in arange(N)] for j in arange N]

now Tim says:

> Try this:
> 
>  >>> import numarray as na
>  >>> r = na.arange(5)
>  >>> na.subtract.outer(r, r)
> array([[ 0, -1, -2, -3, -4],
>        [ 1,  0, -1, -2, -3],
>        [ 2,  1,  0, -1, -2],
>        [ 3,  2,  1,  0, -1],
>        [ 4,  3,  2,  1,  0]])
> 

but this gives
>>> subtract.outer(r,r).shape
(10, 3, 10, 3)

that is, subtracts y coordinates to x coordinates which is not intended.
AFAIK the outer solution is MUCH faster than the nested for loops, so
what I do now is

>>> r_rel = transpose(array([subtract.outer(r[:,0],r[:,0]),
                             subtract.outer(r[:,1],r[:,1]), 
                             subtract.outer(r[:,2],r[:,2])]))
>>> r_rel.shape #as with the double loop 
(10,10,3) 


My question is then if there is any more elegant way to do this,
especially giving as a result independence of the number of dimensions).

Maybe an "axis"  (=0 in this case?) keyword for the outer function would
be useful in this context?



Thanks for the helpful welcome to the list!,
 ?.

PS. the problem with the min() function regarded a matrix of collision
times between particles, which is symmetrical. The elements are reals,
so I only expect to find one minimum.

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 04:07:10 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 04:07:10 2004
Subject: [Numpy-discussion] Error in choose (bug?)
Message-ID: <1084446790.917.9.camel@siff>

I have a matrix of particle collision times: times[i,j] gives the time
for particle "i" to collide with particle "j".

If I do, in order to get the first expected collision time for each
particle, the following (random array for testing purposes)	:

>>> N=30
>>> times = rnd.random([N,N])
>>> choose(argmin(times,transpose(times))
Segmentation fault (of the python interactive shell!!!)

With N=100 I get a more informative traceback, and rest within the
shell:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
    return _choose(selector, population, outarr, clipmode)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
    result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
    blockingparameters)
ValueError: _operator_compute: too many inputs + outputs


For N=10,N=15 I get the expected output, but for N=20 I get again the
brutal segfault...


regards, 
?.

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 04:50:01 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 04:50:01 2004
Subject: [Numpy-discussion] Unexpected warnings in where
Message-ID: <1084449282.909.21.camel@siff>

Hi,

I cannot understand why I do receive these warnings:

Warning: Encountered invalid numeric result(s)  in sqrt
Warning: Encountered invalid numeric result(s)  in divide

after

>>> newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV, far_future)


RV, discr, RR, VV are NxN arrays of reals (Float64), while
far_future=1e20 (scalar).

1. sqrt(<0) should be impossible, since it's explicitly ruled out that
discr<0 in the where condition.

2. /0 should also be impossible, since VV==0 is True only for the
diagonal...

I've done some testing and where(logical_and(RV<0, discr>0), discr, 666)
doesn't show any negative numbers, neither does
where(logical_and(RV<0,discr>0),VV,666) show any zeroes...


	
Please.. what am I doing wrong here?


-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From jmiller at stsci.edu  Thu May 13 05:43:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 05:43:12 2004
Subject: [Numpy-discussion] puzzling compiler warning for numeric
	extension
In-Reply-To: <p06100501bcc864cfb9df@[140.142.183.76]>
References: <p06100501bcc864cfb9df@[140.142.183.76]>
Message-ID: <1084452101.483.4.camel@halloween.stsci.edu>

On Wed, 2004-05-12 at 19:54, Russell E Owen wrote:
> I modified an existing numeric c extension today, adding a function 
> that returns a new array. This is the first time I've used 
> NA_NewArray. The code seems to run fine, but I get a disturbing 
> compiler warning:
> 
> [rowen:Shared Files/Guider/PyGuide] rowen% python setup.py build
> ...
> src/radProfModule.c: In function `Py_radSqByRadInd':
> src/radProfModule.c:378: warning: return from incompatible pointer type
> 
> The associated code can be summarized as follows (and it is simple 
> enough that I've also appended a copy). The error is thrown on the 
> last line:
> 
> static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
> 	long nElt;
> 	Long *radSqByRadInd;
> 
> 	if (!PyArg_ParseTuple(args, "l", &nElt))
> 		return NULL;
> ...
> 	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
> }
> 
> So...am I actually doing something wrong, or is this warning normal?

The problem is that your function returns a PyObject* but NA_NewArray
returns a PyArrayObject*.   This particular "error" is generally
harmless since all that is really in conflict is the compiler's view of
the same memory address.  You can eliminate the warning by sticking in a
cast to PyObject* like this:

	return (PyObject *) NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);

Regards,
Todd

> 

> 
> -- Russell
> 
> P.S. the full code:
> 
> static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
> 	long nElt;
> 	Long radInd;
> 	char ModName[] = "radSqByRadInd";
> 	Long *radSqByRadInd;
> 
> 	if (!PyArg_ParseTuple(args, "l", &nElt))
> 		return NULL;
> 
> 	if (nElt < 0) {
> 		PyErr_Format(PyExc_ValueError, "%s: nPts < 0", ModName);
> 		return NULL;
> 	}
> 
> 	// allocate max(3, nElt) elements (simplifies the computation
> 	// and avoids calling calloc on 0 elements)
> 	radSqByRadInd = calloc(MAX(nElt, 3), sizeof *radSqByRadInd);
> 	if (radSqByRadInd == NULL) {
> 		PyErr_Format(PyExc_MemoryError, "%s: insufficient 
> memory", ModName);
> 		return NULL;
> 	}
> 
> 	for (radInd=0; radInd<3; radInd++) {
> 		radSqByRadInd[radInd] = radInd;
> 	}
> 	for (radInd=3; radInd<nElt; radInd++) {
> 		radSqByRadInd[radInd] = (radInd - 1) * (radInd - 1);
> 	}
> 
> 	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
> }
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller <jmiller at stsci.edu>




From jmiller at stsci.edu  Thu May 13 06:06:00 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 06:06:00 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084449282.909.21.camel@siff>
References: <1084449282.909.21.camel@siff>
Message-ID: <1084453439.483.16.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 07:54, ?lvaro Tejero Cantero wrote:
> Hi,
> 
> I cannot understand why I do receive these warnings:
> 
> Warning: Encountered invalid numeric result(s)  in sqrt
> Warning: Encountered invalid numeric result(s)  in divide
> 
> after
> 
> >>> newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV, far_future)
> 
> 
> RV, discr, RR, VV are NxN arrays of reals (Float64), while
> far_future=1e20 (scalar).
> 
> 1. sqrt(<0) should be impossible, since it's explicitly ruled out that
> discr<0 in the where condition.
> 
> 2. /0 should also be impossible, since VV==0 is True only for the
> diagonal...
> 
> I've done some testing and where(logical_and(RV<0, discr>0), discr, 666)
> doesn't show any negative numbers, neither does
> where(logical_and(RV<0,discr>0),VV,666) show any zeroes...
> 
> 
> 	
> Please.. what am I doing wrong here?

I think what you're seeing is this:

where(condition, expression1, expression2)

Even though condition is selecting parts of expression1 which are valid,
expression1 is still fully evaluated.

Regards,
Todd

-- 
Todd Miller <jmiller at stsci.edu>




From perry at stsci.edu  Thu May 13 06:29:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 06:29:02 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <40A2C0EE.703@ucsd.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>

Robert Kern wrote:

> > Well, yes. But when you ask to find all the things that equal
> > the minimum, you pretty much must look twice (if you want to know
> > where they all are if more than one). Once to determine the
> > minimum, the next time to locate all of them.
> 
> Nah, you can accumulate indices corresponding to the current minimum 
> value as you go. Discard the list of indices and start again if you get 
> a new minimum value.
> 
True enough (though I suspect some special cases may not work any 
faster, e.g., an array of all zeros with the last element equal to
-1; you spend all the time copying indices for nearly the whole damn
thing for no purpose). There are a bunch of things that can be done
to eliminate multiple passes but they tend to lead to many different
specialized functions. One has to trade off the number of such functions
against the speed savings. Another example is getting max and min values
for an array. I've long thought that this is so often done they could
be done in one pass. There isn't a function that does this yet though.

> >>Yes... although for the problem at hand that motivated my query, my
> >>times matrix is symmetric... I don't really need all the minima, but
> >>does numarray have any special datatype for symmetric matrixes, that
> >>prevents storage of unneded (e.g. supradiagonal) elements?.
> >>
> > 
> > Not for special cases like this. One could probably write a special
> > subclass to do this, but for a savings of a factor of 2 in memory,
> > it usually would not be worth the trouble (unlike sparse matrices)
> 
> OTOH, having subclasses that follow LAPACK's symmetric packed storage 
> scheme would be very useful not because of the space factor but the time 
> saved by being able to use the symmetric algorithms in LAPACK. I think.
> 
I'd agree that that would be a much stronger motivating factor.

Perry 



From alvaro at antalia.com  Thu May 13 09:28:35 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 09:28:35 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084453439.483.16.camel@halloween.stsci.edu>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
Message-ID: <1084466049.1084.20.camel@siff>

> I think what you're seeing is this:
> 
> where(condition, expression1, expression2)
> 
> Even though condition is selecting parts of expression1 which are valid,
> expression1 is still fully evaluated.

Is this behaviour what is intended, or do you consider it a shortcoming
of the implementation?. In theory avoiding unneeded evaluations of
expression1 is very desirable...

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 09:31:10 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 09:31:10 2004
Subject: [Numpy-discussion] Example code
Message-ID: <1084466214.1070.25.camel@siff>

Hello,

Do you know of publicly available example code with numarray?. Short of
the tutorial and the page by Magnus Hetland, and the intro by David
Mertz, I haven't found much code written in numarray through google.

It would be helpful for me to have more code to look at. I am explicitly
discarding numpy code in order to keep away from subtle differences, but
perhaps this is overcautious.

Thanks!
?.
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From rays at blue-cove.com  Thu May 13 09:50:04 2004
From: rays at blue-cove.com (Ray Schumacher)
Date: Thu May 13 09:50:04 2004
Subject: [Numpy-discussion] Example code
In-Reply-To: <1084466214.1070.25.camel@siff>
Message-ID: <5.2.0.4.2.20040513094730.0609a1e8@blue-cove.com>

At 06:37 PM 5/13/2004 +0200, ?lvaro Tejero Cantero wrote:
>Hello,
>
>Do you know of publicly available example code with numarray?. Short of
>the tutorial and the page by Magnus Hetland, and the intro by David
>Mertz, I haven't found much code written in numarray through google.

Try:
http://rjs.org/astro/1004x/Python/CAP7X.PY

I might have a few others around I can share...


Ray Schumacher
Blue Cove Interactive
7220 Trade Street, Suite 101
San Diego, CA 92121
858.695.8801
http://Blue-Cove.com




From jmiller at stsci.edu  Thu May 13 09:58:07 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 09:58:07 2004
Subject: [Numpy-discussion] Error in choose (bug?)
In-Reply-To: <1084446790.917.9.camel@siff>
References: <1084446790.917.9.camel@siff>
Message-ID: <1084467425.483.274.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 07:13, ?lvaro Tejero Cantero wrote:
> I have a matrix of particle collision times: times[i,j] gives the time
> for particle "i" to collide with particle "j".
> 
> If I do, in order to get the first expected collision time for each
> particle, the following (random array for testing purposes)	:
> 
> >>> N=30
> >>> times = rnd.random([N,N])
> >>> choose(argmin(times,transpose(times))
> Segmentation fault (of the python interactive shell!!!)
> 
> With N=100 I get a more informative traceback, and rest within the
> shell:
> 
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
>     return _choose(selector, population, outarr, clipmode)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
>     result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
>     blockingparameters)
> ValueError: _operator_compute: too many inputs + outputs
> 
> 
> For N=10,N=15 I get the expected output, but for N=20 I get again the
> brutal segfault...
> 
> 
> regards, 
> ?.

I was able to reproduce this bug, logged it on Source Forge, and will
get to it as soon as possible... probably tomorrow or Monday.

Regards,
Todd






From perry at stsci.edu  Thu May 13 10:00:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 10:00:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084466049.1084.20.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFOEHICDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> > I think what you're seeing is this:
> >
> > where(condition, expression1, expression2)
> >
> > Even though condition is selecting parts of expression1 which are valid,
> > expression1 is still fully evaluated.
>
> Is this behaviour what is intended, or do you consider it a shortcoming
> of the implementation?. In theory avoiding unneeded evaluations of
> expression1 is very desirable...
>
It's about the only way it could work given how an array-based
library works (unless there were a mechanism to postpone
evaluation of expressions until their context can be
determined). The expression is evaluated before it is passed to the
where function.

Perry Greenfield




From jmiller at stsci.edu  Thu May 13 10:02:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 10:02:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084466049.1084.20.camel@siff>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
	 <1084466049.1084.20.camel@siff>
Message-ID: <1084467685.483.281.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 12:34, ?lvaro Tejero Cantero wrote:
> > I think what you're seeing is this:
> > 
> > where(condition, expression1, expression2)
> > 
> > Even though condition is selecting parts of expression1 which are valid,
> > expression1 is still fully evaluated.
> 
> Is this behaviour what is intended, or do you consider it a shortcoming
> of the implementation?. In theory avoiding unneeded evaluations of
> expression1 is very desirable...

This just falls out of Python function call semantics: parameters are
evaluated before a function is called.   It is a property of Python so I
think we're stuck with it.

Two things come to mind to get rid of the warnings:

1. Stuff the input arrays with values which avoid the warnings.
condition = logical_and(RV<0,discr>0)
discr[condition] = 1
VV[condition] = 1
newtimes = where(condition, (-RV-sqrt(discr))/VV, far_future)

2. Use the Error system to suppress the warnings:

Error.pushMode(dividebyzero="ignore", invalid="ignore")
newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV,
far_future)
Error.popMode()

Regards,
Todd





From lilliarasconton at hotmail.com  Thu May 13 11:11:04 2004
From: lilliarasconton at hotmail.com (Clare Booker)
Date: Thu May 13 11:11:04 2004
Subject: [Numpy-discussion] 500 DOLLARS For You, Just For Opening This Notice, Thu, 13 May 2004 14:09:55 -0500
Message-ID: <MLCSOJEEEEYATGGMAAKTHX@yahoo.com>

Thu, 13 May 2004 14:09:55 -0500

CASH TODAY will help you get back on your feet.   http://www.lifetimedealz.com/cash23/ 

If late payments and unexpected bills have set you back financially CASH TODAY can help. We specialize in helping people pay their bills on time without any expensive fees or hassles.

http://www.lifetimedealz.com/cash23/ 

It only takes a minute to apply for a CASH TODAY personal loan for up to 500 dollars. Once you're approved we deposit money directly into your account within 1/2 hour of Approval.

http://www.lifetimedealz.com/cash23/ 


No more ads: http://www.lifetimedealz.com/away.html

tale

chlorophyll hebephrenic activation natchez nonsensical bilge liturgy atreus dunlap optoacoustic blonde wilson winter edwards carbonaceous 

behead flinty embower divination thyronine priori proceed chablis ma ferromagnetic camber retrospect tricky ahem simplicial villein brandt bombast graff petroleum anachronistic lufthansa coincident chanson hunt confucius gogo fixture trilogy deviate erlenmeyer alterate canberra doorknob dryad neve customary consul implore alarm 

From alvaro at antalia.com  Thu May 13 11:51:02 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 11:51:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <40A3C119.40403@pfdubois.com>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
	 <1084466049.1084.20.camel@siff>
	 <1084467685.483.281.camel@halloween.stsci.edu>
	 <40A3C119.40403@pfdubois.com>
Message-ID: <1084474601.1932.1.camel@siff>

On Thu, 2004-05-13 at 20:40, Paul Dubois wrote:
> Use MA, it is designed to solve this problem.

Is it as fast as using numarray arrays?
Are you aware of example code floating around, apart from the docs?

Thank you!
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From rowen at u.washington.edu  Thu May 13 12:23:04 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Thu May 13 12:23:04 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
Message-ID: <p06100509bcc97a82aef9@[128.95.99.44]>

At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
>... One has to trade off the number of such functions
>against the speed savings. Another example is getting max and min values
>for an array. I've long thought that this is so often done they could
>be done in one pass. There isn't a function that does this yet though.

Statistics is another area where multiple return values could be of 
interest -- one may want the mean and std dev, and making two passes 
is wasteful (since some of the same info needs to be computed both 
times).

A do-all function that computes min, min location, max, max location, 
mean and std dev all at once would be nice (especially if the 
returned values were accessed by name, rather than just being a tuple 
of values, so they could be referenced safely and readably).

-- Russell



From perry at stsci.edu  Thu May 13 12:43:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 12:43:01 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <CEELJPECNGEGKFNDLHFFEEHJCDAA.perry@stsci.edu>

> Russell E Owen wrote:
> 
> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
> >... One has to trade off the number of such functions
> >against the speed savings. Another example is getting max and min values
> >for an array. I've long thought that this is so often done they could
> >be done in one pass. There isn't a function that does this yet though.
> 
> Statistics is another area where multiple return values could be of 
> interest -- one may want the mean and std dev, and making two passes 
> is wasteful (since some of the same info needs to be computed both 
> times).
> 
> A do-all function that computes min, min location, max, max location, 
> mean and std dev all at once would be nice (especially if the 
> returned values were accessed by name, rather than just being a tuple 
> of values, so they could be referenced safely and readably).
> 
> -- Russell
> 
We will definitely add something like this for 1.0 or 1.1.
(but probably for min and max location, it will just be
for the first encountered).

Perry



From focke at slac.stanford.edu  Thu May 13 14:10:07 2004
From: focke at slac.stanford.edu (Warren Focke)
Date: Thu May 13 14:10:07 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
 <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <Pine.LNX.4.58.0405131310400.27825@auriga.slac.stanford.edu>


On Thu, 13 May 2004, Russell E Owen wrote:

> Statistics is another area where multiple return values could be of
> interest -- one may want the mean and std dev, and making two passes
> is wasteful (since some of the same info needs to be computed both
> times).

Single pass std deviations don't work very well if you've got a lot of
data points and the std deviation is small compared to the average.
I'm not arguing aginst including them, but maybe the documentation for
such a function should include a caveat.

Warren Focke




From southey at uiuc.edu  Thu May 13 14:22:02 2004
From: southey at uiuc.edu (Bruce Southey)
Date: Thu May 13 14:22:02 2004
Subject: [Numpy-discussion] Getting the indexes
 of the myarray.min()
Message-ID: <ee2ef33e.400618f5.81a0e00@expms6.cites.uiuc.edu>

Hi,
Raymond D. Hettinger is writing a general statistics module 'statistics.py  A
collection of functions for summarizing data' that is somewhere in a Python
CVS (I can not find the exact reference but it appeared in a fairly recent
Python thread). He uses a one-pass algorithm from Knuth for the variance that
has good numerical stability.

Below is a rather rough version modified from my situation (masked arrays) which
uses Knuth's algorithm for the variance. It lacks features like checking
dimensions (assumes variance can be computed) and documentation.

Regards
Bruce Southey

import numarray

def SummaryStats(Matrix):
        mshape=Matrix.getshape()
        nrows=mshape[0]
        ncols=mshape[1]
        #print nrows, ncols
        # Create matrices to hold statistics
        N_obs =numarray.zeros(ncols, type='Float64')
        Sum   =numarray.zeros(ncols, type='Float64')
        Var   =numarray.zeros(ncols, type='Float64')
        Min   =numarray.zeros(ncols, type='Float64')
        Max   =numarray.zeros(ncols, type='Float64')
        Mean  =numarray.zeros(ncols, type='Float64')
        AdjM  =numarray.zeros(ncols, type='Float64')
        NewM  =numarray.zeros(ncols, type='Float64')
        DifM  =numarray.zeros(ncols, type='Float64')

        for row in range(nrows):
                for col in range(ncols):
                        t_value=Matrix[row,col]
                        N_obs[col] = N_obs[col] + 1
                        Sum[col] = Sum[col] + t_value
                        if t_value > Max[col]:
                                Max[col]=t_value
                        if t_value < Min[col]:
                                Min[col]=t_value
                        if N_obs[col]==1:
                                Mean[col]=t_value
                        AdjM[col]=(t_value-Mean[col])/(N_obs[col])-DifM[col]
                        NewM[col]=Mean[col]+AdjM[col]
                        DifM[col]=(NewM[col]-Mean[col])-AdjM[col]
                        Var[col] = Var[col] +
(t_value-Mean[col])*(t_value-NewM[col])
                        Mean[col] = NewM[col]
        print 'N_obs\n', N_obs
        print 'Sum\n', Sum
        print 'Mean\n', Mean
        print 'Var\n', Var/(nrows-1)

if __name__ == '__main__':
        MValues=numarray.array([[1,2,1],[3,2,2],[5,1,1],[4,3,2]])
        SummaryStats(MValues)



---- Original message ----
>Date: Thu, 13 May 2004 15:42:30 -0400
>From: "Perry Greenfield" <perry at stsci.edu>  
>Subject: RE: [Numpy-discussion] Getting the indexes of the myarray.min()  
>To: "Russell E Owen" <rowen at u.washington.edu>, "numarray"
<numpy-discussion at lists.sourceforge.net>
>
>> Russell E Owen wrote:
>> 
>> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
>> >... One has to trade off the number of such functions
>> >against the speed savings. Another example is getting max and min values
>> >for an array. I've long thought that this is so often done they could
>> >be done in one pass. There isn't a function that does this yet though.
>> 
>> Statistics is another area where multiple return values could be of 
>> interest -- one may want the mean and std dev, and making two passes 
>> is wasteful (since some of the same info needs to be computed both 
>> times).
>> 
>> A do-all function that computes min, min location, max, max location, 
>> mean and std dev all at once would be nice (especially if the 
>> returned values were accessed by name, rather than just being a tuple 
>> of values, so they could be referenced safely and readably).
>> 
>> -- Russell
>> 
>We will definitely add something like this for 1.0 or 1.1.
>(but probably for min and max location, it will just be
>for the first encountered).
>
>Perry
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: SourceForge.net Broadband
>Sign-up now for SourceForge Broadband and get the fastest
>6.0/768 connection for only $19.95/mo for the first 3 months!
>http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion



From perry at stsci.edu  Thu May 13 14:31:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 14:31:01 2004
Subject: outer idiom (was Re: [Numpy-discussion] Getting the indexes of themyarray.min())
In-Reply-To: <1084444629.800.12.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFAEHKCDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> but this gives
> >>> subtract.outer(r,r).shape
> (10, 3, 10, 3)
>
> that is, subtracts y coordinates to x coordinates which is not intended.
> AFAIK the outer solution is MUCH faster than the nested for loops, so
> what I do now is
>
> >>> r_rel = transpose(array([subtract.outer(r[:,0],r[:,0]),
>                              subtract.outer(r[:,1],r[:,1]),
>                              subtract.outer(r[:,2],r[:,2])]))
> >>> r_rel.shape #as with the double loop
> (10,10,3)
>
>
> My question is then if there is any more elegant way to do this,
> especially giving as a result independence of the number of dimensions).
>
Not that I can think of at the moment.

> Maybe an "axis"  (=0 in this case?) keyword for the outer function would
> be useful in this context?
>
Perhaps. Right now these ufunc methods are pretty complicated so
it may not be easy to do, but I understand that there is certainly
utility in being able to do that. We'll look into it (but not
right away so don't hold your breath).

Perry




From perry at stsci.edu  Thu May 13 14:41:00 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 14:41:00 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFEEHJCDAA.perry@stsci.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFIEHKCDAA.perry@stsci.edu>

> > A do-all function that computes min, min location, max, max location, 
> > mean and std dev all at once would be nice (especially if the 
> > returned values were accessed by name, rather than just being a tuple 
> > of values, so they could be referenced safely and readably).
> > 
> > -- Russell
> > 
> We will definitely add something like this for 1.0 or 1.1.
> (but probably for min and max location, it will just be
> for the first encountered).
> 
> Perry
> 
To elaborate on this a bit, I'm thinking that the minmax
capability probably should be separated from statistics
(it's very common to want to do min max without wanting
to do mean or stdev, plus as others have noted, the statistics
can be a bit more involved).

There is one other aspect that needs user input.
How to handle ieee special values for things like minmax.
Right now the ufunc reductions have some odd behavior that
is a result of underlying C library behavior. A NaN isn't 
consistently handled in such comparisons (it appears to 
depend on the order when comparing a NaN to a regular float,
whichever appears second appears to be accepted in pairwise
comparisons).

I was thinking that one could add an exclude keyword
to the functions to indicate that ieee special values should
not be included (well, I suppose Inf should be for min max)
otherwise they would be. Any thoughts on this?

Perry



From jsaenz at wm.lc.ehu.es  Fri May 14 03:04:10 2004
From: jsaenz at wm.lc.ehu.es (Jon Saenz)
Date: Fri May 14 03:04:10 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
 <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <20040514115430.G48469@lcdx00.wm.lc.ehu.es>

What about an object (TimeSeries) which can be "explored" (optionally)
during its init method and, if so, it creates the dictionary with those
values? If it is not explored, the dictionary would be assigned to None
and, if requested, the "exploratory" statistics would be computed then.
This could be the basis for other computations on time series.

Just my two cents.


Jon Saenz.				| Tfno: +34 946012445
Depto. Fisica Aplicada II               | Fax:  +34 946013500
Facultad de Ciencias.   \\ Universidad del Pais Vasco \\
Apdo. 644   \\ 48080 - Bilbao  \\ SPAIN

On Thu, 13 May 2004, Russell E Owen wrote:

> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
> >... One has to trade off the number of such functions
> >against the speed savings. Another example is getting max and min values
> >for an array. I've long thought that this is so often done they could
> >be done in one pass. There isn't a function that does this yet though.
>
> Statistics is another area where multiple return values could be of
> interest -- one may want the mean and std dev, and making two passes
> is wasteful (since some of the same info needs to be computed both
> times).
>
> A do-all function that computes min, min location, max, max location,
> mean and std dev all at once would be nice (especially if the
> returned values were accessed by name, rather than just being a tuple
> of values, so they could be referenced safely and readably).
>
> -- Russell
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>



From eric at enthought.com  Sun May 16 21:36:02 2004
From: eric at enthought.com (eric jones)
Date: Sun May 16 21:36:02 2004
Subject: [Numpy-discussion] ANN: Job Openings at Enthought
Message-ID: <40A840F3.2050608@enthought.com>

Hello,

This is a bit of a status report I guess for SciPy (www.scipy.org) and
Enthought (www.enthought.com), but, really, it is a long winded job
posting. 

Among other things, Enthought develops scientific software
for clients. Python is central to to our strategy in this arena.  We have
long supported SciPy and believe strongly in Open Source Software. 
The purpose is to give people a feeling about what we do, the technical
challenges we face, and the skills needed to meet them.

SciPy
-----
There has been a lot of work on SciPy lately by the Travis Oliphant, Travis
Vaught, Pearu Peterson, Joe Cooper, and on the website by Jon-Eric 
Steinbomer
and Janet Swisher.  The site is now upgraded to Plone and we have a 
SciPy 0.3
package out.  If you're interested in seeing the activity level at 
scipy.org,
please visit.

http://www.scipy.org/map?rmurl=http%3A//scipy.net/scipyusage/

It looks like we're averaging about a 100 downloads a day if you add up all
source and binary distributions.  When SciPy 0.1 was released a couple a 
years
ago, I think we averaged about 10.  Its extremely difficult to 
extrapolate a user
base from such information, the obvious growth is good news.

Other Stuff
-----------
In addition to SciPy we have multiple projects going on right now that 
either
already plug into or will be plugins to a Python-based scientific 
application
framework called Envisage (sorta like an IDE for science apps) that we're
working on.  The idea is:

(1) To lower the bar as far as work required to get a GUI put
   on the front end of a scientific algorithm.
(2) Develop scientific plugins that can play well together so
   that, for instance, an electromagnetics simulator built by one
   person can be used in conjunction with an optimization plugin
   built by another person and results can be visualized with a
   3D plugin.

This is a hard, long path that requires much work and though.  We have
started the process and actually have built an app on a very early 
version of
the Envisage framework.  The app works great, but it's usage of the 
framework is a
little mixed at this point.  There were a lot of compromises we ended up 
making
on the framework side in order to ship on time.

Also, I am not sure whether "easy-to-program" and "flexible architecture for
compatible plugins" are not mutually exclusive.  I always have in my 
mind that I
want a smart scientist to be able to build the GUI in a short amount of time
after the algorithm (which is the "hard part") is finished.  I still 
harbor this
wish, but the lessons I've had over the last couple of years suggest 
that the GUI is
actually the most expensive part of development for large apps.  This is 
partially
due to the fact that commercial apps are rarely built around "research" 
algorithms
-- meaning that the algorithm code usually already exists in some form.  
Still,
UIs take tons of time.  Testing them is hard.  Flexibility requires more
complexity (factories, adapters, etc.) than seems necessary at first.  
Further,
building *good* UI's is really hard.  Scientist rarely have the patience or
perspective for it -- and yet it is very often difficult to build a good 
UI for
science apps without a scientist's understanding of the underlying problem.

We have a number of tools along with SciPy that we're building that are 
pieces
of this puzzle.

  1. Traits -- They at the heart of everything we do.  They provide
     some explicit typing, default GUI's, an observer event model for
     anything that uses them.  They can also require some getting used
     to.
  2. Kiva/Enable -- This is the generic drawing system for Python.
  3. Chaco -- Interactive 2D Plotting
  4. PyFace -- MVC layer on top of wxPython.  Supports trees, menus,
     and a few other things right now.  This will grow.
  5. Envisage -- Plugin based GUI framework for scientific applications.
     Beyond the basic GUI that comes from Envisage, we already have a couple
     plugins for profiling code using hotshot and also searching for memory
     leaks.  David Morrill wrote these and they are called gotcha and 
enroll.
     Martin Chilvers wrote a simple scintilla based plugin for editing 
scripts,
     and a PyCrust shell is available by default in the app (not
     a plugin at the moment).
    
     We would love to see a number of other plugins:
      a. Debugger.
      b. IPython-like PyCrust going.
      c. A more full featured script editing plugin.
         (there is a huge list of features here).
      d. Mayavi
      e. etc.

These are all used actively in commercial products that we deliver.  
However,
they are also in various stages of development and will continue to evolve
and grow.  Some are still pretty green.  Portions of these are openly 
available
now.  All five of the listed tool sets will be open at some point this 
summer
when they are cleaned up a bit.

Jobs
----
All of this leads to the fact that we are hiring and looking for smart,
pleasant, communicative people with integrity that are excited about 
building
this sort of stuff.  There is a lot of software architecture to be done that
needs talented designers.  There is UI design/development that needs 
scientists
that care about UIs or UI developers/designers that are really quick at 
grasping
scientific problems.  We also need really strong scientists/scientific
developers to do some of the backend coding for commercial products and also
SciPy development.  If you have a background in geophysics, electromagnetics
(especially multi-pole methods) that is a big plus. For this, a PhD is 
helpful, but
not required.  Parallel computing skills wouldn't hurt.  3D 
visualization a la
VTK/Python is a major need.  So is knowledge about 2D rendering to help 
finish
out Kiva and the Enable toolset. Strong Python skills, Python extension 
writing
knowledge and strong C/C++ skills are a big benefit.  Design/development 
with or
of Java NetBeans or Eclipse-like architecture is great -- or any other 
solid GUI
architecture.  Dedication to writing clean/clear code meant for other 
humans
to read is a requirement.

We have 3 or 4 scientific/python positions to fill over the coming 
months, and
we're looking for candidates that have the best mix of the above 
skills.  You'll
join our existing team of scientist/engineers, computer scientists, 
technical
writers, and an HCI specialist.  We like this mix and feel it is the 
best way to
build this sort of software. If you are interested in working at Enthought,
please send us your resume at jobs at enthought.com.  If not, please send this
posting to the smartest people you know that fit some part of the above 
description
(Python experience not explicitly required).  Salaries are competitive.  
Candidates must
be willing to relocate to Austin, Tx.
     
thanks,
eric jones

PS: There are additional positions listed at 
http://www.enthought.com/careers.htm
for those interested in business application development (not 
necessarily with Python).





From alvaro at antalia.com  Mon May 17 09:25:05 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Mon May 17 09:25:05 2004
Subject: [Numpy-discussion] MaskedArray problem?
Message-ID: <1084811467.3230.5.camel@siff>

Hello,

I'm trying to rewrite my simulation using masked arrays. For that I
define masks for triangular_inferior NxN and
diagonal_triangular_superior NxN arrays.

I'm unsuccesfully trying to know what happens with the program when I
operate over a MaskedArray: it always says

Traceback (most recent call last):
  File "<stdin>", line 31, in ?
  File "<stdin>", line 26, in run
  File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 990,
in __pow__
    return power(self, other, third)
  File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 1585,
in power
    return masked_array(Numeric.power(fa, fb), m)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 877,
in _cache_miss2
    mode, win1, win2, wout, cfunc, ufargs = \
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 921,
in _setup
    intypes = (in1._type.name, in2._type.name)
AttributeError: 'ObjectArray' object has no attribute '_type'


For reference, I copy a simplified version of the program. The statement
that causes the AttributeError is the last from run(N,D,S):

---------------------------

import numarray.ma as mka
import numarray.random_array as rnd
import numarray as nmr

def relative(x, msk):
    (N,D) = x.shape
    return mka.array([nmr.subtract.outer(x[:,i],x[:,i]) for i in nmr.arange(D)],
                     mask = nmr.array([msk for i in nmr.arange(D)],typecode=mka.MaskType))

def scalar(x,y):
    return mka.add.reduce(x*y,axis=0)

def run(D,N,S):
    trig_inf = nmr.array(([[(0,1)[i<j] for i in nmr.arange(N)] for j in nmr.arange(N)]), typecode=mka.MaskType)
    diag_trig_sup = 1 - trig_inf #print mka.is_mask(diag_trig_sup) gives "1"

    # positions and velocities
    r = rnd.random([N,D]) #should be overlap-checked
    v = rnd.random([N,D])

    # relative positions and velocities
    r_rel = relative(r, diag_trig_sup)
    v_rel = relative(v, diag_trig_sup)

    RV, VV, RR = scalar(r_rel,v_rel), scalar(v_rel,v_rel), scalar(r_rel,r_rel)
    discr = RV**2 - VV*(RR-S**2)      #<<<<----------PROBLEM HERE----------------



if __name__ == '__main__':
    run(D=3,N=4,S=1e-8)


-------------------------

Any other advice that you may give me is very wellcome. My deadline is
approaching and I'm really reluctant to rewrite all this in C++ but I
don't see the light.

Thank you in advance,

 ?.


-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From jmiller at stsci.edu  Mon May 17 12:04:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon May 17 12:04:02 2004
Subject: [Numpy-discussion] Error in choose (bug?)
In-Reply-To: <1084467425.483.274.camel@halloween.stsci.edu>
References: <1084446790.917.9.camel@siff>
	 <1084467425.483.274.camel@halloween.stsci.edu>
Message-ID: <1084820558.12475.363.camel@localhost.localdomain>

On Thu, 2004-05-13 at 12:57, Todd Miller wrote:
> On Thu, 2004-05-13 at 07:13, ?lvaro Tejero Cantero wrote:
> > I have a matrix of particle collision times: times[i,j] gives the time
> > for particle "i" to collide with particle "j".
> > 
> > If I do, in order to get the first expected collision time for each
> > particle, the following (random array for testing purposes)	:
> > 
> > >>> N=30
> > >>> times = rnd.random([N,N])
> > >>> choose(argmin(times,transpose(times))
> > Segmentation fault (of the python interactive shell!!!)
> > 
> > With N=100 I get a more informative traceback, and rest within the
> > shell:
> > 
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
> >     return _choose(selector, population, outarr, clipmode)
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
> >     result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
> >     blockingparameters)
> > ValueError: _operator_compute: too many inputs + outputs
> > 
> > 
> > For N=10,N=15 I get the expected output, but for N=20 I get again the
> > brutal segfault...
> > 
> > 
> > regards, 
> > ?.

I tracked this down today and understand the problem:  there's a bug in
the ufunc error checking and a ridiculous limit in choose.   I'm still
working out the "real" solution but a quick workaround is to edit
numarray.h and set MAXARGS to a larger number, say N+10.

Regards,
Todd
-- 
Todd Miller <jmiller at stsci.edu>




From jmiller at stsci.edu  Mon May 17 13:48:09 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon May 17 13:48:09 2004
Subject: [Numpy-discussion] MaskedArray problem?
In-Reply-To: <1084811467.3230.5.camel@siff>
References: <1084811467.3230.5.camel@siff>
Message-ID: <1084826827.12476.476.camel@localhost.localdomain>

On Mon, 2004-05-17 at 12:31, ?lvaro Tejero Cantero wrote:
> Hello,
> 
> I'm trying to rewrite my simulation using masked arrays. For that I
> define masks for triangular_inferior NxN and
> diagonal_triangular_superior NxN arrays.
> 
> I'm unsuccesfully trying to know what happens with the program when I
> operate over a MaskedArray: it always says
> 
> Traceback (most recent call last):
>   File "<stdin>", line 31, in ?
>   File "<stdin>", line 26, in run
>   File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 990,
> in __pow__
>     return power(self, other, third)
>   File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 1585,
> in power
>     return masked_array(Numeric.power(fa, fb), m)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 877,
> in _cache_miss2
>     mode, win1, win2, wout, cfunc, ufargs = \
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 921,
> in _setup
>     intypes = (in1._type.name, in2._type.name)
> AttributeError: 'ObjectArray' object has no attribute '_type'
> 
> 
> For reference, I copy a simplified version of the program. The statement
> that causes the AttributeError is the last from run(N,D,S):
> 
> ---------------------------
> 
> import numarray.ma as mka
> import numarray.random_array as rnd
> import numarray as nmr
> 
> def relative(x, msk):
>     (N,D) = x.shape
>     return mka.array([nmr.subtract.outer(x[:,i],x[:,i]) for i in nmr.arange(D)],
>                      mask = nmr.array([msk for i in nmr.arange(D)],typecode=mka.MaskType))
> 
> def scalar(x,y):
>     return mka.add.reduce(x*y,axis=0)
> 
> def run(D,N,S):
>     trig_inf = nmr.array(([[(0,1)[i<j] for i in nmr.arange(N)] for j in nmr.arange(N)]), typecode=mka.MaskType)
>     diag_trig_sup = 1 - trig_inf #print mka.is_mask(diag_trig_sup) gives "1"
> 
>     # positions and velocities
>     r = rnd.random([N,D]) #should be overlap-checked
>     v = rnd.random([N,D])
> 
>     # relative positions and velocities
>     r_rel = relative(r, diag_trig_sup)
>     v_rel = relative(v, diag_trig_sup)
> 
>     RV, VV, RR = scalar(r_rel,v_rel), scalar(v_rel,v_rel), scalar(r_rel,r_rel)
>     discr = RV**2 - VV*(RR-S**2)      #<<<<----------PROBLEM HERE----------------
> 
> 
> 
> if __name__ == '__main__':
>     run(D=3,N=4,S=1e-8)
> 
> 
> -------------------------
> 
> Any other advice that you may give me is very wellcome. My deadline is
> approaching and I'm really reluctant to rewrite all this in C++ but I
> don't see the light.
> 
> Thank you in advance,
> 
>  ?.

Here is a work around I put in numarray's MA.py which is now in CVS:

cvs diff: Diffing Packages/MA/Lib
Index: Packages/MA/Lib/MA.py
===================================================================
RCS file: /cvsroot/numpy/numarray/Packages/MA/Lib/MA.py,v
retrieving revision 1.7
retrieving revision 1.9
diff -c -r1.7 -r1.9
*** Packages/MA/Lib/MA.py       1 Oct 2003 15:44:35 -0000       1.7
--- Packages/MA/Lib/MA.py       17 May 2004 20:21:58 -0000      1.9
***************
*** 1263,1270 ****
--- 1263,1274 ----
                  result = value
              result.shape = d.shape
          else:
+             if not m.is_c_array():
+                 m = m.copy()
              try:
                  result = Numeric.array(d, typecode=d.typecode(),
copy=1)
+                 if not result.is_c_array():
+                     result = result.copy()
                  Numeric.putmask(result, m, value)
              except:
                  result = obj.choose(m, (d, value))

You can either get the new MA from Source Forge CVS (project numpy, down
at the moment) or use this diff by editing your own MA.py and adding the
lines shown with +'s.  The root problem is limitations in numarray's
putmask() function which are avoided by making contiguous copies of some
of the arrays in MA's filled() function.

Regards,
Todd

-- 
Todd Miller <jmiller at stsci.edu>




From news at lumbersolutions.com  Mon May 17 16:43:02 2004
From: news at lumbersolutions.com (Lumber Wholesale)
Date: Mon May 17 16:43:02 2004
Subject: [Numpy-discussion] IPE Cumaru Massaranduba Cedar
Message-ID: <hxvshy.wl4n9v@www.lumbersolutions.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040517/7a465e51/attachment.html>

From news at lumbersolutions.com  Mon May 17 16:43:03 2004
From: news at lumbersolutions.com (Lumber Wholesale)
Date: Mon May 17 16:43:03 2004
Subject: [Numpy-discussion] IPE Cumaru Massaranduba Cedar
Message-ID: <hxvshy.827kx1@www.lumbersolutions.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040517/35bb4db5/attachment.html>

From crasmussen at lanl.gov  Wed May 19 08:00:19 2004
From: crasmussen at lanl.gov (Craig Rasmussen)
Date: Wed May 19 08:00:19 2004
Subject: [Numpy-discussion] Linear solvers
In-Reply-To: <1084467425.483.274.camel@halloween.stsci.edu>
References: <1084446790.917.9.camel@siff> <1084467425.483.274.camel@halloween.stsci.edu>
Message-ID: <2999D4AB-A9A5-11D8-B1FD-000A957CA856@lanl.gov>

I need to solve a matrix equation of the form

      A u = q

for u, where u and q are N-vectors and A is a 
symmetric-positive-definite matrix.
For now, at least, A is a 2-D matrix.

I would like to use Python and numarray.  Are there Python modules that
I could use to obtain a solution for u?

Thanks,
Craig




From nadavh at visionsense.com  Wed May 19 09:44:09 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed May 19 09:44:09 2004
Subject: [Numpy-discussion] Linear solvers
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED10@exchange2k.envision.co.il>

The simplest solution is to use "solve_linear_equations" functin from numarray.linear_algebra. For more sophisticate solution you may try scipy
(www.scipy.org)

  Nadav

-----Original Message-----
From:	Craig Rasmussen [mailto:crasmussen at lanl.gov]
Sent:	Wed 19-May-04 17:59
To:	numarray
Cc:	
Subject:	[Numpy-discussion] Linear solvers
I need to solve a matrix equation of the form

      A u = q

for u, where u and q are N-vectors and A is a 
symmetric-positive-definite matrix.
For now, at least, A is a 2-D matrix.

I would like to use Python and numarray.  Are there Python modules that
I could use to obtain a solution for u?

Thanks,
Craig



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From haase at msg.ucsf.edu  Wed May 19 11:26:07 2004
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed May 19 11:26:07 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
Message-ID: <200405191125.30848.haase@msg.ucsf.edu>

Hi,
the random_array poisson functions returns negative values if mean=0:
>>>from numarray import random_array as ra
>>> ra.seed(x=1, y=1)
>>> ra.poisson(0)
5
>>> ra.poisson(0)
-2

My "math book" tells me that it should be always zero.
This seems to be a constructed case, but I'm using this to put
"quantum statistic" into a simulated image:
obj = na.array( something )
imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
The object array might have lots of zeros surrounding the "actual object". 
Thinking of a fluorescent object sending out photons it makes sense to not get 
any photons at all from 'empty' regions.
I'm using numarray 0.8;

Thanks for numarray,
Sebastian Haase



From jmiller at stsci.edu  Wed May 19 12:05:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 19 12:05:04 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
In-Reply-To: <200405191125.30848.haase@msg.ucsf.edu>
References: <200405191125.30848.haase@msg.ucsf.edu>
Message-ID: <1084993432.3736.16.camel@localhost.localdomain>

On Wed, 2004-05-19 at 14:25, Sebastian Haase wrote:
> Hi,
> the random_array poisson functions returns negative values if mean=0:
> >>>from numarray import random_array as ra
> >>> ra.seed(x=1, y=1)
> >>> ra.poisson(0)
> 5
> >>> ra.poisson(0)
> -2
> 
> My "math book" tells me that it should be always zero.
> This seems to be a constructed case, but I'm using this to put
> "quantum statistic" into a simulated image:
> obj = na.array( something )
> imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
> The object array might have lots of zeros surrounding the "actual object". 
> Thinking of a fluorescent object sending out photons it makes sense to not get 
> any photons at all from 'empty' regions.
> I'm using numarray 0.8;

I tried this on Fedora-1 i386 with Python-2.3.3 and it returned zero
consistently.  What platform are you on?

Regards,
Todd








From haase at msg.ucsf.edu  Wed May 19 12:18:01 2004
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed May 19 12:18:01 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
In-Reply-To: <1084993432.3736.16.camel@localhost.localdomain>
References: <200405191125.30848.haase@msg.ucsf.edu> <1084993432.3736.16.camel@localhost.localdomain>
Message-ID: <200405191217.02792.haase@msg.ucsf.edu>

On Wednesday 19 May 2004 12:03 pm, you wrote:
> On Wed, 2004-05-19 at 14:25, Sebastian Haase wrote:
> > Hi,
> >
> > the random_array poisson functions returns negative values if mean=0:
> > >>>from numarray import random_array as ra
> > >>> ra.seed(x=1, y=1)
> > >>> ra.poisson(0)
> >
> > 5
> >
> > >>> ra.poisson(0)
> >
> > -2
> >
> > My "math book" tells me that it should be always zero.
> > This seems to be a constructed case, but I'm using this to put
> > "quantum statistic" into a simulated image:
> > obj = na.array( something )
> > imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
> > The object array might have lots of zeros surrounding the "actual
> > object". Thinking of a fluorescent object sending out photons it makes
> > sense to not get any photons at all from 'empty' regions.
> > I'm using numarray 0.8;
>
> I tried this on Fedora-1 i386 with Python-2.3.3 and it returned zero
> consistently.  What platform are you on?
>
> Regards,
> Todd

I running debian (Woody)
$ uname -a
Linux baboon 2.4.18 #1 Tue Dec 16 14:11:01 PST 2003 i686 unknown
$python -v
<snip>
Python 2.2.1 (#1, Feb 28 2004, 00:52:10)
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2

and I get this:
>>> ra.poisson(0, 100000).min()
-4
>>> ra.poisson(0, 100000).min()
-4
>>> ra.poisson(0, 100000).mean()
1.9383
>>> ra.poisson(0, 100000).mean()
1.9607
>>> ra.poisson(0, 100000).max()
29
>>> ra.poisson(0, 100000).max()
28

Thanks for checking,
Sebastian



From falted at pytables.org  Thu May 20 01:31:02 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 20 01:31:02 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
Message-ID: <200405201030.39979.falted@pytables.org>

Hi,

I'm willing to use a lot the searchsorted function in numarray, but I'm a
bit surprised about the poor performance of it. Look at that:

>>> from time import time
>>> import numarray
>>> import Numeric
>>> na=numarray.arange(1000*1000)
>>> nu=Numeric.arange(1000*1000)
>>> t1=time();numarray.searchsorted(na,200*1000);time()-t1
200000
0.00055098533630371094
>>> t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
200000
7.7962875366210938e-05

It may seem that Numeric is better optimised, but the standard python module
bisect is even faster than numarray.searchsorted:

>>> import bisect
>>> t1=time();bisect.bisect_left(na,200*1000);time()-t1
200000
8.8930130004882812e-05
>>> t1=time();bisect.bisect_left(nu,200*1000);time()-t1
200000
8.6069107055664062e-05

So, bisect performance is similar to that of Numeric searchsorted and both
are almost an order of magnitude better than numarray searchsorted. This a
little bit surprising as the bisect_left module is written in plain python.
From the python 2.3.3 sources:

def bisect_left(a, x, lo=0, hi=None):
    if hi is None:
        hi = len(a)
    while lo < hi:
        mid = (lo+hi)//2
        if a[mid] < x: lo = mid+1
        else: hi = mid
    return lo

I'm using python2.3.3, numarray 0.9.1 (latest CVS) and Debian Linux (sid).

Cheers,

-- 
Francesc Alted




From rkern at ucsd.edu  Thu May 20 02:27:09 2004
From: rkern at ucsd.edu (Robert Kern)
Date: Thu May 20 02:27:09 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
In-Reply-To: <200405201030.39979.falted@pytables.org>
References: <200405201030.39979.falted@pytables.org>
Message-ID: <40AC79DD.1000905@ucsd.edu>

Francesc Alted wrote:
> Hi,
> 
> I'm willing to use a lot the searchsorted function in numarray, but I'm a
> bit surprised about the poor performance of it. Look at that:
> 
> 
>>>>from time import time
>>>>import numarray
>>>>import Numeric
>>>>na=numarray.arange(1000*1000)
>>>>nu=Numeric.arange(1000*1000)
>>>>t1=time();numarray.searchsorted(na,200*1000);time()-t1
> 
> 200000
> 0.00055098533630371094
> 
>>>>t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
> 
> 200000
> 7.7962875366210938e-05
> 
> It may seem that Numeric is better optimised, but the standard python module
> bisect is even faster than numarray.searchsorted:
> 
> 
>>>>import bisect
>>>>t1=time();bisect.bisect_left(na,200*1000);time()-t1
> 
> 200000
> 8.8930130004882812e-05
> 
>>>>t1=time();bisect.bisect_left(nu,200*1000);time()-t1
> 
> 200000
> 8.6069107055664062e-05

A better timing (IMHO), but with similar conclusions:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import timeit
 >>> t1 = timeit.Timer("Numeric.searchsorted(a,200000)",
"import Numeric;a=Numeric.arange(1000000)")
 >>> t2 = timeit.Timer("numarray.searchsorted(a,200000)",
"import numarray;a=numarray.arange(1000000)")
 >>> t3 = timeit.Timer("bisect.bisect_left(a,200000)",
"import Numeric;import bisect;a=Numeric.arange(1000000)")
 >>> t4 = timeit.Timer("bisect.bisect_left(a,200000)",
"import numarray;import bisect;a=numarray.arange(1000000)")
 >>> t1.repeat(3,10000)
[0.15758609771728516, 0.17469501495361328, 0.15456986427307129]
 >>> t2.repeat(3,10000)
[6.7581729888916016, 6.9644770622253418, 6.6776731014251709]
 >>> t3.repeat(3,10000)
[0.41335701942443848, 0.45698308944702148, 0.39665889739990234]
 >>> t4.repeat(3,10000)
[0.49930000305175781, 0.48063492774963379, 0.52067780494689941]

[Apologies for the linewraps.]

I also get similar results with double arrays. Weird.

Python 2.3 on Mac OS X 10.3.sumthin', latest CVS checkout of numarray, 
Numeric 23.1.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From jmiller at stsci.edu  Thu May 20 10:14:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 20 10:14:06 2004
Subject: [Numpy-discussion] Performance issues with
	numarray.searchsorted()
In-Reply-To: <40AC79DD.1000905@ucsd.edu>
References: <200405201030.39979.falted@pytables.org>
	 <40AC79DD.1000905@ucsd.edu>
Message-ID: <1085073211.7525.2043.camel@halloween.stsci.edu>

On Thu, 2004-05-20 at 05:26, Robert Kern wrote:
> Francesc Alted wrote:
> > Hi,
> > 
> > I'm willing to use a lot the searchsorted function in numarray, but I'm a
> > bit surprised about the poor performance of it. Look at that:
> > 
> > 
> >>>>from time import time
> >>>>import numarray
> >>>>import Numeric
> >>>>na=numarray.arange(1000*1000)
> >>>>nu=Numeric.arange(1000*1000)
> >>>>t1=time();numarray.searchsorted(na,200*1000);time()-t1
> > 
> > 200000
> > 0.00055098533630371094
> > 
> >>>>t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
> > 
> > 200000
> > 7.7962875366210938e-05
> > 
> > It may seem that Numeric is better optimised, but the standard python module
> > bisect is even faster than numarray.searchsorted:
> > 
> > 
> >>>>import bisect
> >>>>t1=time();bisect.bisect_left(na,200*1000);time()-t1
> > 
> > 200000
> > 8.8930130004882812e-05
> > 
> >>>>t1=time();bisect.bisect_left(nu,200*1000);time()-t1
> > 
> > 200000
> > 8.6069107055664062e-05
> 
> A better timing (IMHO), but with similar conclusions:
> 
> Python 2.3 (#1, Sep 13 2003, 00:49:11)
> [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import timeit
>  >>> t1 = timeit.Timer("Numeric.searchsorted(a,200000)",
> "import Numeric;a=Numeric.arange(1000000)")
>  >>> t2 = timeit.Timer("numarray.searchsorted(a,200000)",
> "import numarray;a=numarray.arange(1000000)")
>  >>> t3 = timeit.Timer("bisect.bisect_left(a,200000)",
> "import Numeric;import bisect;a=Numeric.arange(1000000)")
>  >>> t4 = timeit.Timer("bisect.bisect_left(a,200000)",
> "import numarray;import bisect;a=numarray.arange(1000000)")
>  >>> t1.repeat(3,10000)
> [0.15758609771728516, 0.17469501495361328, 0.15456986427307129]
>  >>> t2.repeat(3,10000)
> [6.7581729888916016, 6.9644770622253418, 6.6776731014251709]
>  >>> t3.repeat(3,10000)
> [0.41335701942443848, 0.45698308944702148, 0.39665889739990234]
>  >>> t4.repeat(3,10000)
> [0.49930000305175781, 0.48063492774963379, 0.52067780494689941]
> 
> [Apologies for the linewraps.]
> 
> I also get similar results with double arrays. Weird.
> 
> Python 2.3 on Mac OS X 10.3.sumthin', latest CVS checkout of numarray, 
> Numeric 23.1.
Here's what I got with the numarray benchmarks after adding an extra
case:

benchmark                    i    numarray (usec)    Numeric (usec)  numarray:Numeric

searchsorted(a,p/5)        0.0               446                12              36.7
                           1.0               438                11              36.8
                           2.0               450                12              35.0
                           3.0               459                14              32.6
                           4.0               511                25              19.7
                           5.0               636                56              11.2
                           6.0               653                64              10.1

searchsorted(a,a)          0.0               285                 5              48.0
                           1.0               283                 7              39.6
                           2.0               291                29               9.8
                           3.0               368               308               1.2
                           4.0              1771              4120               0.4
                           5.0             17335             52127               0.3
                           6.0            201277            605787               0.3

p = 10**i
a = arange(p)

The first case agrees with your results of ~10x difference in favor of
Numeric at 10**6 elements.  The last case shows a ~3x numarray advantage
given large numbers of values.

My analysis is this:  since searchsorted runs in O(log2(N)) time,  even
with 10**6 elements there are only 20 iterations or so.  This is just
not enough to overcome numarray's Python ufunc overhead.  I'll see what
I can do,  but I think we're up against the standard numarray
performance wall for small arrays.	

Regards,
Todd






From falted at pytables.org  Thu May 20 10:42:05 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 20 10:42:05 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
In-Reply-To: <1085073211.7525.2043.camel@halloween.stsci.edu>
References: <200405201030.39979.falted@pytables.org> <40AC79DD.1000905@ucsd.edu> <1085073211.7525.2043.camel@halloween.stsci.edu>
Message-ID: <200405201941.08841.falted@pytables.org>

A Dijous 20 Maig 2004 19:13, vareu escriure:
> The first case agrees with your results of ~10x difference in favor of
> Numeric at 10**6 elements.  The last case shows a ~3x numarray advantage
> given large numbers of values.
> 
> My analysis is this:  since searchsorted runs in O(log2(N)) time,  even
> with 10**6 elements there are only 20 iterations or so.  This is just
> not enough to overcome numarray's Python ufunc overhead.  I'll see what
> I can do,  but I think we're up against the standard numarray
> performance wall for small arrays.	

I see. What about creating a special case for when the item to be searched
is an scalar (and not an array)? In such a case, it would be theoretically
possible to get at least the bisect.bisect() performance.

Besides, if this special case would be implemented, users would be able to
call repeatedly this scalar version of searchsorted() in order to deal with
very small "values" arrays (len()<5) and still having a gain.

Just a thought,

-- 
Francesc Alted




From jdhunter at ace.bsd.uchicago.edu  Fri May 21 12:24:02 2004
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Fri May 21 12:24:02 2004
Subject: [Numpy-discussion] image correlations
Message-ID: <m2k6z51u6d.fsf@mother.paradise.lost>

I have a series of luminance images that I want to do some correlation
analyses on.  Each image is an MxN frame of a movie.  I want to
compute the correlation between a given pixel i,j, and every other
pixel in the image over each frame.  That is, if I assume
xij is a numFrames length time series, and xkl is another numFrames
length time series (the pixel intensities at two points in the image
over time), I want to compute the 

corrcoeff(xij, xkl) for every kl with ij fixed.

I know I could do this by looping over the pixels of the image, but
I'm hoping for something a bit faster.

Any suggestions?

John Hunter



From jmiller at stsci.edu  Fri May 21 12:50:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri May 21 12:50:04 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <m2k6z51u6d.fsf@mother.paradise.lost>
References: <m2k6z51u6d.fsf@mother.paradise.lost>
Message-ID: <1085168950.15016.66.camel@halloween.stsci.edu>

On Fri, 2004-05-21 at 15:01, John Hunter wrote:
> I have a series of luminance images that I want to do some correlation
> analyses on.  Each image is an MxN frame of a movie.  I want to
> compute the correlation between a given pixel i,j, and every other
> pixel in the image over each frame.  That is, if I assume
> xij is a numFrames length time series, and xkl is another numFrames
> length time series (the pixel intensities at two points in the image
> over time), I want to compute the 
> 
> corrcoeff(xij, xkl) for every kl with ij fixed.
> 
> I know I could do this by looping over the pixels of the image, but
> I'm hoping for something a bit faster.
> 
> Any suggestions?

For numarray try numarray.convolve.correlate2d and set fft=1.

Todd




From sdhyok at email.unc.edu  Fri May 21 20:07:37 2004
From: sdhyok at email.unc.edu (Shin, Daehyok)
Date: Fri May 21 20:07:37 2004
Subject: [Numpy-discussion] Read only array?
Message-ID: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>

I am wondering if there is any way to set an array as read-only,
so that any trial to modify values of elements in the array raises some
warning.
Is it a cool feature to prevent unexpected side effect?

Daehyok Shin (Peter)




From jmiller at stsci.edu  Sat May 22 04:02:01 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Sat May 22 04:02:01 2004
Subject: [Numpy-discussion] Read only array?
In-Reply-To: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>
References: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>
Message-ID: <1085223644.3737.58.camel@localhost.localdomain>

On Fri, 2004-05-21 at 23:05, Shin, Daehyok wrote:
> I am wondering if there is any way to set an array as read-only,
> so that any trial to modify values of elements in the array raises some
> warning.
> Is it a cool feature to prevent unexpected side effect?
> 
> Daehyok Shin (Peter)
> 

In numarray you can do this:

>>> from numarray import *
>>> def make_readonly(a):
..     v = a.view()
..     s = a.tostring()
..     v._data = s
..     return v
..
>>> a = arange(100)
>>> b = make_readonly(a)
>>> b[0] = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: NA_setFromPythonScalar: assigment to readonly array buffer

This works because of the buffer protocol and the fact that a string is
a read-only buffer.  Using the buffer protocol is a numarray feature so
I'm pretty sure Numeric can't do this.  Also note that in C, this
read-only array is actually writable.

Regards,
Todd




From zingale at ucolick.org  Sun May 23 14:06:00 2004
From: zingale at ucolick.org (Mike Zingale)
Date: Sun May 23 14:06:00 2004
Subject: [Numpy-discussion] numarray equivalent to sign() function
Message-ID: <Pine.GSO.4.53.0405231401270.29915@mambo.ucolick.org>

Hi, I am in the process of converting some code from Numeric to numarray,
and it seems that numarray no longer has the sign() function -- is that
so?

ex:

sign(-30.0) = -1
sign(0) = 0
sign(1000) = 1

Is there a replacement?

Thanks,

Mike



From jdhunter at ace.bsd.uchicago.edu  Tue May 25 10:24:01 2004
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Tue May 25 10:24:01 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <1085168950.15016.66.camel@halloween.stsci.edu> (Todd Miller's
 message of "21 May 2004 15:49:10 -0400")
References: <m2k6z51u6d.fsf@mother.paradise.lost>
	<1085168950.15016.66.camel@halloween.stsci.edu>
Message-ID: <m2vfikl9vq.fsf@mother.paradise.lost>

>>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:

    >> I have a series of luminance images that I want to do some
    >> correlation analyses on.  Each image is an MxN frame of a
    >> movie.  I want to compute the correlation between a given pixel
    >> i,j, and every other pixel in the image over each frame.  That
    >> is, if I assume xij is a numFrames length time series, and xkl
    >> is another numFrames length time series (the pixel intensities
    >> at two points in the image over time), I want to compute the
    
    >> corrcoeff(xij, xkl) for every kl with ij fixed.
    
    >> I know I could do this by looping over the pixels of the image,
    >> but I'm hoping for something a bit faster.

    Todd> For numarray try numarray.convolve.correlate2d and set
    Todd> fft=1.

I've looked at this and don't know if I'm missing something or if this
isn't the right function for me.  I want to correlate a given pixel
intensity with the intensity at all other pixels over a series of
images.   Is this possible with correlate2d?

JDH





From perry at stsci.edu  Tue May 25 11:52:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue May 25 11:52:01 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <m2vfikl9vq.fsf@mother.paradise.lost>
Message-ID: <CEELJPECNGEGKFNDLHFFEEIOCDAA.perry@stsci.edu>

John Hunter writes:
> >>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:
> 
>     >> I have a series of luminance images that I want to do some
>     >> correlation analyses on.  Each image is an MxN frame of a
>     >> movie.  I want to compute the correlation between a given pixel
>     >> i,j, and every other pixel in the image over each frame.  That
>     >> is, if I assume xij is a numFrames length time series, and xkl
>     >> is another numFrames length time series (the pixel intensities
>     >> at two points in the image over time), I want to compute the
>     
>     >> corrcoeff(xij, xkl) for every kl with ij fixed.
>     
>     >> I know I could do this by looping over the pixels of the image,
>     >> but I'm hoping for something a bit faster.
> 
>     Todd> For numarray try numarray.convolve.correlate2d and set
>     Todd> fft=1.
> 
Something that Todd and I have talked about is making general
functions (like fft, convolve, correlate) broadcastable to
other dimensions. Not much has been done to do that but I think 
a general mechanism could be developed to do just that. In that
way one could do a 1-d correlation over a 2 or 3-d array without
looping over the extra dimensions (all that would be done implicitly
in C) with the option of specifying which dimension the function
should be applied to while looping over all the other dimensions.

In the meantime, it would seemt that one possible way of dealing
with this is to simply recast your array as a 1-d array to use
the 1-d correlation. This means some memory copying and explicit
padding (how do you want the correlation to handle points beyond
the ends?). Supposing you want it to wraparound (in effect a
circular correlation) you could do something like this (untested
and probably has some mistakes :-)

if imagecube is a T x M x N array where there are T time samples.

data = concatenate((imagecube, imagecube, imagecube))
    # pad the time series on both ends with identical copies
reference = imagecube[:, i, j]
flatdata = ravel(transpose(data))
flatresult = correlate(flatdata, reference)
result = transpose(reshape(flatresult, (N,M,T)))[T:2*T,:,:]

This is admittedly a bit clumsy, but should be much, much faster
than iterating over all pixels.

It would be much nicer just to have

result = correlate(imagecube, imagecube[:,i,j], axis=0)

which the broadcasting facility would permit.



Perry



From as8ca at virginia.edu  Wed May 26 07:49:40 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 07:49:40 2004
Subject: [Numpy-discussion] numarray.where confusion
Message-ID: <20040526144841.GA8683@virginia.edu>

Hi,

I am having trouble understanding how exactly "where" works in
numarray.

What I am trying to do:

I am preparing a two-level mask in an array and then assign values to
the array where both masks are true:

>>> from numarray import *
>>> a = arange(10)
>>> # First mask
>>> m1 = where(a > 5)
>>> a[m1]
array([6, 7, 8, 9])
>>> # Second mask
>>> m2 = where(a[m1] < 8)
>>> a[m1][m2]
array([6, 7])
>>> # So far so good
>>> # Now change some values
>>> a[m1][m2] = array([10, 20])
>>> a[m1][m2]
array([6, 7])
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> # Didn't  work
>>> # Let's try a temporary variable
>>> t = a[m1]
>>> t[m2]
array([6, 7])
>>> t[m2] = array([10, 20])
>>> t[m2], t
(array([10, 20]), array([10, 20,  8,  9]))
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

So, my assignment to a[m1][m2] seems to work (no messages), but it
doesn't produce the effect I want it to.

I have read the documentation but I couldn't find something that would
explain this behavior.

So my questions:

- did I miss something important in the documentation,
- I am expecting something I shouldn't, or
- there is a bug in numarray?

Thanks,
Alok

-- 
Alok Singhal (as8ca at virginia.edu)       __
Graduate Student, dept. of Astronomy   /  _
University of Virginia                 \_O \
http://www.astro.virginia.edu/~as8ca/    __/



From perry at stsci.edu  Wed May 26 08:25:14 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 26 08:25:14 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526144841.GA8683@virginia.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>

Alok Singhal wrote:
> Hi,
> 
> I am having trouble understanding how exactly "where" works in
> numarray.
> 
> What I am trying to do:
> 
> I am preparing a two-level mask in an array and then assign values to
> the array where both masks are true:
> 
> >>> from numarray import *
> >>> a = arange(10)
> >>> # First mask
> >>> m1 = where(a > 5)
> >>> a[m1]
> array([6, 7, 8, 9])
> >>> # Second mask
> >>> m2 = where(a[m1] < 8)
> >>> a[m1][m2]
> array([6, 7])
> >>> # So far so good
> >>> # Now change some values
> >>> a[m1][m2] = array([10, 20])
> >>> a[m1][m2]
> array([6, 7])
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
> >>> # Didn't  work
> >>> # Let's try a temporary variable
> >>> t = a[m1]
> >>> t[m2]
> array([6, 7])
> >>> t[m2] = array([10, 20])
> >>> t[m2], t
> (array([10, 20]), array([10, 20,  8,  9]))
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
> 
> So, my assignment to a[m1][m2] seems to work (no messages), but it
> doesn't produce the effect I want it to.
> 
> I have read the documentation but I couldn't find something that would
> explain this behavior.
> 
> So my questions:
> 
> - did I miss something important in the documentation,
> - I am expecting something I shouldn't, or
> - there is a bug in numarray?
> 
(due to confusions with "a" in text I'll use x in place of "a")
I believe the problem you are seeing (I'm not 100% certain yet)
is that although it is possible to assign to an array-indexed
array, that doing that twice over doesn't work since Python is,
in effect, treating x[m1] as an expression even though it is
on the left side. That expression results in a new array that the
second indexing updates, but then is thrown away since it is not
assigned to anything else.

Your second try creates a temporary t which is also not a view into
a so when you update t, a is not updated.

try 

x[m1[0][m2]] = array([10,20])

instead. The intent here is to provide x with the net index array
by indexing m1 first rather than indexing x first.
(note the odd use of m1[0]; this is necessary since where() will
return a tuple of index arrays (to allow use in multidimensional 
cases as indices, so the m1[0] extracts the array from the tuple;
Since m1 is a tuple, indexing it with another index array (well, 
tuple containing an index array) doesn't work).

Perry Greenfield



From jmiller at stsci.edu  Wed May 26 08:42:16 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 26 08:42:16 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526144841.GA8683@virginia.edu>
References: <20040526144841.GA8683@virginia.edu>
Message-ID: <1085586113.31969.123.camel@halloween.stsci.edu>

On Wed, 2004-05-26 at 10:48, Alok Singhal wrote:
> Hi,
> 
> I am having trouble understanding how exactly "where" works in
> numarray.
> 
> What I am trying to do:
> 
> I am preparing a two-level mask in an array and then assign values to
> the array where both masks are true:
> 
> >>> from numarray import *
> >>> a = arange(10)
> >>> # First mask
> >>> m1 = where(a > 5)
> >>> a[m1]
> array([6, 7, 8, 9])
> >>> # Second mask
> >>> m2 = where(a[m1] < 8)
> >>> a[m1][m2]

a[m1] is a new array here.

> array([6, 7])
> >>> # So far so good
> >>> # Now change some values
> >>> a[m1][m2] = array([10, 20])

And here too.  This does a write into what is effectively a temporary
variable returned by the expression a[m1].  Although the write occurs,
it is lost.

> >>> a[m1][m2]
> array([6, 7])
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Here's how I did it (there was an easier way I overlooked):

a = arange(10)
m1 = where(a > 5, 1, 0).astype('Bool')
m2 = where(a < 8, 1, 0).astype('Bool')
a[m1 & m2] = array([10, 20])

The principle here is to keep the masks as "full sized" boolean arrays
rather than index arrays so they can be combined using the bitwise and
operator.   The resulting mask can be used to index just once
eliminating the temporary.

Regards,
Todd




From falted at pytables.org  Wed May 26 09:07:10 2004
From: falted at pytables.org (Francesc Alted)
Date: Wed May 26 09:07:10 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <1085586113.31969.123.camel@halloween.stsci.edu>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu>
Message-ID: <200405261806.37029.falted@pytables.org>

A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
> Here's how I did it (there was an easier way I overlooked):
> 
> a = arange(10)
> m1 = where(a > 5, 1, 0).astype('Bool')
> m2 = where(a < 8, 1, 0).astype('Bool')
> a[m1 & m2] = array([10, 20])

Perhaps the easier way looks like this?

>>> a = arange(10)
>>> a[(a>5) & (a<8)] = array([10, 20])
>>> a
array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])

Indexing is a very powerful (and fun) thing, indeed :)

-- 
Francesc Alted




From jmiller at stsci.edu  Wed May 26 09:29:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 26 09:29:06 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405261806.37029.falted@pytables.org>
References: <20040526144841.GA8683@virginia.edu>
	 <1085586113.31969.123.camel@halloween.stsci.edu>
	 <200405261806.37029.falted@pytables.org>
Message-ID: <1085588893.31969.125.camel@halloween.stsci.edu>

On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
> A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
> > Here's how I did it (there was an easier way I overlooked):
> > 
> > a = arange(10)
> > m1 = where(a > 5, 1, 0).astype('Bool')
> > m2 = where(a < 8, 1, 0).astype('Bool')
> > a[m1 & m2] = array([10, 20])
> 
> Perhaps the easier way looks like this?
> 
> >>> a = arange(10)
> >>> a[(a>5) & (a<8)] = array([10, 20])

Much, much better.  Thanks!

Todd

> >>> a
> array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])
> 
> Indexing is a very powerful (and fun) thing, indeed :)
-- 
Todd Miller <jmiller at stsci.edu>




From as8ca at virginia.edu  Wed May 26 10:19:09 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 10:19:09 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405261806.37029.falted@pytables.org> <1085586113.31969.123.camel@halloween.stsci.edu> <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <200405261806.37029.falted@pytables.org> <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <20040526144841.GA8683@virginia.edu> <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>
Message-ID: <20040526171845.GA18671@virginia.edu>

On 26/05/04: 11:24, Perry Greenfield wrote:
> (due to confusions with "a" in text I'll use x in place of "a")
> I believe the problem you are seeing (I'm not 100% certain yet)
> is that although it is possible to assign to an array-indexed
> array, that doing that twice over doesn't work since Python is,
> in effect, treating x[m1] as an expression even though it is
> on the left side. That expression results in a new array that the
> second indexing updates, but then is thrown away since it is not
> assigned to anything else.
> 
> Your second try creates a temporary t which is also not a view into
> a so when you update t, a is not updated.

Thanks or this info.  It makes sense now.  I suspected earlier that t
was not a view but a copy, but didn't realise that the same thing was
happening with x[m1][m2].

> try 
> 
> x[m1[0][m2]] = array([10,20])
> 
> instead. The intent here is to provide x with the net index array
> by indexing m1 first rather than indexing x first.
> (note the odd use of m1[0]; this is necessary since where() will
> return a tuple of index arrays (to allow use in multidimensional 
> cases as indices, so the m1[0] extracts the array from the tuple;
> Since m1 is a tuple, indexing it with another index array (well, 
> tuple containing an index array) doesn't work).

This works, but for the fact that in my real code I *am* dealing with
multidimensional arrays.  But this is a nice trick to remember.

(So, the following "does not work":

    x = arange(9)
    x.shape=(3,3)
    m1 = where(x > 4)
    m2 = where(x[m1] < 7)
    x[m1[0][m2]]
)

On 26/05/04: 11:41, Todd Miller wrote:
> Here's how I did it (there was an easier way I overlooked):
> 
> a = arange(10)
> m1 = where(a > 5, 1, 0).astype('Bool')
> m2 = where(a < 8, 1, 0).astype('Bool')
> a[m1 & m2] = array([10, 20])

Ah.  This works!  Even for multidimensional arrays.

On 26/05/04: 18:06, Francesc Alted wrote:
> Perhaps the easier way looks like this?
> 
> >>> a = arange(10)
> >>> a[(a>5) & (a<8)] = array([10, 20])
> >>> a
> array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])
> 
> Indexing is a very powerful (and fun) thing, indeed :)

I like this too.  Thank you all for the help!

Alok

-- 
Alok Singhal (as8ca at virginia.edu)       __
Graduate Student, dept. of Astronomy   /  _
University of Virginia                 \_O \
http://www.astro.virginia.edu/~as8ca/    __/



From strawman at astraw.com  Wed May 26 10:44:05 2004
From: strawman at astraw.com (Andrew Straw)
Date: Wed May 26 10:44:05 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <1085588893.31969.125.camel@halloween.stsci.edu>
References: <20040526144841.GA8683@virginia.edu>	 <1085586113.31969.123.camel@halloween.stsci.edu>	 <200405261806.37029.falted@pytables.org> <1085588893.31969.125.camel@halloween.stsci.edu>
Message-ID: <40B474C3.4070903@astraw.com>

Todd Miller wrote:

>On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
>  
>
>>A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
>>    
>>
>>>Here's how I did it (there was an easier way I overlooked):
>>>
>>>a = arange(10)
>>>m1 = where(a > 5, 1, 0).astype('Bool')
>>>m2 = where(a < 8, 1, 0).astype('Bool')
>>>a[m1 & m2] = array([10, 20])
>>>      
>>>
>>Perhaps the easier way looks like this?
>>
>>    
>>
>>>>>a = arange(10)
>>>>>a[(a>5) & (a<8)] = array([10, 20])
>>>>>          
>>>>>
Is there an equivalently slick way to accomplish to what I'm trying 
below?  (the the values in c[:,1] get changed based on the same-row 
values in c[:,0]?)

from numarray import *
a=arange(10)
b=arange(10)+20
c=concatenate((a[:,NewAxis],b[:,NewAxis]),axis=1)
c[c[:,0]>7][:,1] = 0 # doesn't work because it makes a copy and 
therefore doesn't modify c

Cheers!
Andrew



From as8ca at virginia.edu  Wed May 26 11:04:13 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 11:04:13 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <40B474C3.4070903@astraw.com>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <200405261806.37029.falted@pytables.org> <1085588893.31969.125.camel@halloween.stsci.edu> <40B474C3.4070903@astraw.com>
Message-ID: <20040526180339.GB20143@virginia.edu>

On 26/05/04: 10:43, Andrew Straw wrote:
> Todd Miller wrote:
> >On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
> >
> >>>>>a = arange(10)
> >>>>>a[(a>5) & (a<8)] = array([10, 20])
> >>>>>
> Is there an equivalently slick way to accomplish to what I'm trying 
> below?  (the the values in c[:,1] get changed based on the same-row 
> values in c[:,0]?)
> 
> from numarray import *
> a=arange(10)
> b=arange(10)+20
> c=concatenate((a[:,NewAxis],b[:,NewAxis]),axis=1)
> c[c[:,0]>7][:,1] = 0 # doesn't work because it makes a copy and 
> therefore doesn't modify c

Well, for your case, the following works:

>>> print c
[[ 0 20]
 [ 1 21]
 [ 2 22]
 [ 3 23]
 [ 4 24]
 [ 5 25]
 [ 6 26]
 [ 7 27]
 [ 8 28]
 [ 9 29]]
>>> t0 = c[:, 0]
>>> t1 = c[:, 1]
>>> t1[t0 > 7] = 0
>>> print c
[[ 0 20]
 [ 1 21]
 [ 2 22]
 [ 3 23]
 [ 4 24]
 [ 5 25]
 [ 6 26]
 [ 7 27]
 [ 8  0]
 [ 9  0]]

Not sure this helps in your real code though.

Alok

-- 
Alok Singhal (as8ca at virginia.edu)          *   *          
Graduate Student, dept. of Astronomy   *           *     *
University of Virginia                                    
http://www.astro.virginia.edu/~as8ca/              *    * 



From perry at stsci.edu  Wed May 26 12:03:06 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 26 12:03:06 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526171845.GA18671@virginia.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>

> > try 
> > 
> > x[m1[0][m2]] = array([10,20])
> > 
> > instead. The intent here is to provide x with the net index array
> > by indexing m1 first rather than indexing x first.
> > (note the odd use of m1[0]; this is necessary since where() will
> > return a tuple of index arrays (to allow use in multidimensional 
> > cases as indices, so the m1[0] extracts the array from the tuple;
> > Since m1 is a tuple, indexing it with another index array (well, 
> > tuple containing an index array) doesn't work).
> 
> This works, but for the fact that in my real code I *am* dealing with
> multidimensional arrays.  But this is a nice trick to remember.
> 
> (So, the following "does not work":
> 
>     x = arange(9)
>     x.shape=(3,3)
>     m1 = where(x > 4)
>     m2 = where(x[m1] < 7)
>     x[m1[0][m2]]
> )
 
correct. You'd have to break apart the m1 tuple and
index all the components, e.g.,

m11, m12 = m1
x[m11[m2],m12[m2]] = ...

This gets clumsier with the more dimensions that must
be handled, but you still can do it. It would be most
useful if the indexed array is very large, the number
of items selected is relatively small and one
doesn't want to incur the memory overhead of all the
mask arrays of the admittedly much nicer notational
approach that Francesc illustrated.

Perry
 



From falted at pytables.org  Thu May 27 00:47:04 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 27 00:47:04 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>
Message-ID: <200405270946.06228.falted@pytables.org>

A Dimecres 26 Maig 2004 21:01, Perry Greenfield va escriure:
> correct. You'd have to break apart the m1 tuple and
> index all the components, e.g.,
> 
> m11, m12 = m1
> x[m11[m2],m12[m2]] = ...
> 
> This gets clumsier with the more dimensions that must
> be handled, but you still can do it. It would be most
> useful if the indexed array is very large, the number
> of items selected is relatively small and one
> doesn't want to incur the memory overhead of all the
> mask arrays of the admittedly much nicer notational
> approach that Francesc illustrated.

Well, boolean arrays have the property that they use very little memory
(only 1 byte / element), and normally perform quite well doing indexing.
Some timings:

>>> import timeit
>>> t1 = timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2],m12[m2]]","from numarray import arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t2 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t1.repeat(3,1000)
[3.1320240497589111, 3.1235389709472656, 3.1198310852050781]
>>> t2.repeat(3,1000)
[1.1218469142913818, 1.117638111114502, 1.1156759262084961]

i.e. using boolean arrays for indexing is roughly 3 times faster.

For larger arrays this difference is even more noticeable:

>>> t3 = timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2],m12[m2]]","from numarray import arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t4 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t3.repeat(3,10)
[3.1818649768829346, 3.20477294921875, 3.190640926361084]
>>> t4.repeat(3,10)
[0.42328095436096191, 0.42140507698059082, 0.41979002952575684]

as you see, now the difference is almost an order of magnitude (!).

So, perhaps assuming the small memory overhead, in most of cases it is
better to use boolean selections. However, it would be nice to know the
ultimate reason of why this happens, because the Perry approach seems
intuitively faster.

-- 
Francesc Alted




From perry at stsci.edu  Thu May 27 10:48:00 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 27 10:48:00 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405270946.06228.falted@pytables.org>
Message-ID: <CEELJPECNGEGKFNDLHFFKEJHCDAA.perry@stsci.edu>

Francesc Alted va escriure:

> A Dimecres 26 Maig 2004 21:01, Perry Greenfield va escriure:
> > correct. You'd have to break apart the m1 tuple and
> > index all the components, e.g.,
> > 
> > m11, m12 = m1
> > x[m11[m2],m12[m2]] = ...
> > 
> > This gets clumsier with the more dimensions that must
> > be handled, but you still can do it. It would be most
> > useful if the indexed array is very large, the number
> > of items selected is relatively small and one
> > doesn't want to incur the memory overhead of all the
> > mask arrays of the admittedly much nicer notational
> > approach that Francesc illustrated.
> 
> Well, boolean arrays have the property that they use very little memory
> (only 1 byte / element), and normally perform quite well doing indexing.
> Some timings:
> 
> >>> import timeit
> >>> t1 = 
> timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2]
> ,m12[m2]]","from numarray import 
> arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t2 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import 
> arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t1.repeat(3,1000)
> [3.1320240497589111, 3.1235389709472656, 3.1198310852050781]
> >>> t2.repeat(3,1000)
> [1.1218469142913818, 1.117638111114502, 1.1156759262084961]
> 
> i.e. using boolean arrays for indexing is roughly 3 times faster.
> 
> For larger arrays this difference is even more noticeable:
> 
> >>> t3 = 
> timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2]
> ,m12[m2]]","from numarray import 
> arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t4 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import 
> arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t3.repeat(3,10)
> [3.1818649768829346, 3.20477294921875, 3.190640926361084]
> >>> t4.repeat(3,10)
> [0.42328095436096191, 0.42140507698059082, 0.41979002952575684]
> 
> as you see, now the difference is almost an order of magnitude (!).
> 
> So, perhaps assuming the small memory overhead, in most of cases it is
> better to use boolean selections. However, it would be nice to know the
> ultimate reason of why this happens, because the Perry approach seems
> intuitively faster.
>
Yes I agree. It was good of you to post these timings. I don't 
think we had actually compared the two approaches though the
results don't surprise me (though I suspect the results may change
if the first mask has a very small percentage of elements; the
large timing test has nearly all elements selected for the first
mask).

Perry



From kck93jom at inland.net  Mon May 31 23:18:01 2004
From: kck93jom at inland.net (Cheryle Rosalyn)
Date: Mon May 31 23:18:01 2004
Subject: [Numpy-discussion] Little Pen-i-sMake U Shy To Screw?? Hehe where
Message-ID: <gfo4pmqf8p4hpcmxxq@creaturewill>

mountain season laughed perfect done justice now 
tie maybe excite my loose experience listen conscience across likely slipped step ache thought 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040531/4d320c2e/attachment.html>

From cjw at sympatico.ca  Wed May  5 13:28:02 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Wed May  5 13:28:02 2004
Subject: [Numpy-discussion] numarraycore
Message-ID: <40994E36.2040203@sympatico.ca>

It would help to have some documentation on the purpose and usage of the 
class UsesOpPriority and its variable op_priority.






From jmiller at stsci.edu  Wed May  5 13:52:03 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May  5 13:52:03 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <40994E36.2040203@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
Message-ID: <1083790275.28609.17.camel@halloween.stsci.edu>

On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> It would help to have some documentation on the purpose and usage of the 
> class UsesOpPriority and its variable op_priority.

The basic idea was that NumArray subclasses which want to use numarray
operators would:

1.  subclass from UsesOpPriority 

2.  set a class level op_priority > 0. 


NumArrays have op_priority 0,  higher priorities are given "precedence".


Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
A.op_priority==0 and B.op_priority==1, then:  
A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
the type(A+B) could be NumArraySubclass rather than NumArray.  Different
subclasses could use higher or lower op_priorities to perform the same
kind of operator resolution among themselves.


Todd
-- 
Todd Miller <jmiller at stsci.edu>




From cjw at sympatico.ca  Wed May  5 16:01:01 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Wed May  5 16:01:01 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <1083790275.28609.17.camel@halloween.stsci.edu>
References: <40994E36.2040203@sympatico.ca> <1083790275.28609.17.camel@halloween.stsci.edu>
Message-ID: <409971DF.4070908@sympatico.ca>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040505/a655a6ed/attachment-0001.html>

From travis at enthought.com  Thu May  6 05:30:02 2004
From: travis at enthought.com (Travis N. Vaught)
Date: Thu May  6 05:30:02 2004
Subject: [Numpy-discussion] ANN: SciPy 2004 Conference - Python for Scientific Computing
Message-ID: <409A2F1C.7050801@enthought.com>

Greetings,

The 1st annual *SciPy Conference* will be held this year at Caltech, 
September 2-3, 2004.  As some of you may know, we've experienced great 
participation in two SciPy "Workshops" (with ~70 attendees in both 2002 
and 2003) and this year we're graduating to a "conference."  With the 
prestige of a conference comes the responsibility of a keynote address.  
This year, Jim Hugunin has answered the call and will be speaking to 
kickoff the meeting on Thursday September 2nd.  Jim is the creator of 
Numeric Python, Jython, and co-designer of AspectJ. Jim is currently 
working on IronPython--a fast implementation of Python for .NET and Mono.

Registration is now open.  More information can be found here:

http://www.scipy.org/wikis/scipy04

You may register early online for $100.00.  Registration includes 
breakfast and lunch Thursday & Friday and a very nice dinner Thursday 
night.  After July 16, registration will cost $150.00.

Call for Presenters:
If you are interested in presenting at the conference, you may submit an 
abstract in Plain Text, PDF or MS Word formats to abstracts at scipy.org -- 
the deadline for abstract submission is July 1, 2004.  Papers and/or 
presentation slides are acceptable and are due by August 20, 2004.

We're also planning three days of informal "Coding Sprints" prior to the 
conference -- August 30 to September 1, 2004.  Conference registration 
is not required to participate in the sprints.  Please email the list, 
however, if you plan to attend.  Topics for these sprints will be 
determined via the mailing lists as well, so please submit any 
suggestions for topics to the scipy-user list:

list signup: http://www.scipy.org/mailinglists/
list address: scipy-user at scipy.org

Please forward this announcement to anyone/list that might be interested.

I look forward to seeing you at the conference.

Best Regards,

Travis N. Vaught




From jmiller at stsci.edu  Thu May  6 11:09:07 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May  6 11:09:07 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <409971DF.4070908@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
	 <1083790275.28609.17.camel@halloween.stsci.edu>
	 <409971DF.4070908@sympatico.ca>
Message-ID: <1083866875.2626.315.camel@halloween.stsci.edu>

On Wed, 2004-05-05 at 18:59, Colin J. Williams wrote:
> 
> Todd Miller wrote:
> > On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> >   
> > > It would help to have some documentation on the purpose and usage of the 
> > > class UsesOpPriority and its variable op_priority.
> > >     
> > 
> Todd,
> 
> Thanks for this, perhaps it could be added to the pdf .  I like the
> basic idea.

OK.  What is it you want to do?

> > The basic idea was that NumArray subclasses which want to use numarray
> > operators would:
> > 
> > 1.  subclass from UsesOpPriority 
> UsesOpPriority has no methods and thus, to access the methods of
> NumArray, it would appear to be necessary to subclass from NumArray.

For limited but important uses (e.g. masked array) subclassing from
NumArray is not necessary;  I'd summarize these as the cases where a
class wants to take control of operators from NumArray.

> > 2.  set a class level op_priority > 0. 
> > 
> > 
> > NumArrays have op_priority 0,  higher priorities are given "precedence".
> > 
> > 
> > Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
> > A.op_priority==0 and B.op_priority==1, then:  
> > A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
> > the type(A+B) could be NumArraySubclass rather than NumArray.  Different
> > subclasses could use higher or lower op_priorities to perform the same
> > kind of operator resolution among themselves.
> >   
> I wonder about the desirability of having different levels of priority
> for the same level of subclassing and am puzzled that a float variable
> is used for the priority.

Different levels of priority for the same level of sub-classing makes
sense to me:  it let's you answer the question "What's the result type
of a PyMatrix instance + a SomeOtherSubclass instance?"

A float enables insertions of new members into the middle of an existing
order.

> 
> It would nice if this depth could be determined automatically. 
> Perhaps something like the script below could be used.

I don't think we want something automatic here.   The choices involved
are arbitrary so we just need to be able to define what we want.

Other than masked arrays, the problem we were trying to solve with
UsesOpPriority is more like:

class A(NumArray):              
	op_priority = 1

class B(NumArray):              
	op_priority = 2

whenever A op B is encountered, the result should be a B.  

That said, the solution we currently have is better described by:
whenever A op B is encountered, B.__rop__(A) is called in preference to
A.__op__(B).  With the example above,  even though B.__rop__ is called,
the result is still an A.  :-(   

I fixed this in CVS today to work more like the original description.

Regards,
Todd

> 
> Colin W.
> > Todd
> >   
> # tClassing.py
> 
> class A:
>   def __init__(self):
>     self.p= 0
>   def prior(self):
>     self.p+= 1
> class B(A):
>   def __init__(self):
>     A.__init__(self)
>     self.prior()
> class C(B):
>   def __init__(self):
>     B.__init__(self)
>     self.prior()
> 
> print A().p
> print B().p
> c= C()
> print 'Priority for C methods:', c.p
> ------------------------------------------------------- This SF.Net
> email is sponsored by Sleepycat Software Learn developer strategies
> Cisco, Motorola, Ericsson & Lucent use to deliver higher performing
> products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________ Numpy-discussion
> mailing list Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller <jmiller at stsci.edu>




From cjw at sympatico.ca  Thu May  6 16:17:01 2004
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu May  6 16:17:01 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <1083866875.2626.315.camel@halloween.stsci.edu>
References: <40994E36.2040203@sympatico.ca>	 <1083790275.28609.17.camel@halloween.stsci.edu>	 <409971DF.4070908@sympatico.ca> <1083866875.2626.315.camel@halloween.stsci.edu>
Message-ID: <409AC74E.2080609@sympatico.ca>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040506/7d85270a/attachment-0001.html>

From karthik at james.hut.fi  Fri May  7 01:23:05 2004
From: karthik at james.hut.fi (Karthikesh Raju)
Date: Fri May  7 01:23:05 2004
Subject: [Numpy-discussion] covariance and other linear algebra functions
Message-ID: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>

Hi,

Is am searching for a covariance function and other linear algebra
functions like eigen value decomposition. i am interested in both
scipy (havent managed to get scipy installed on the main computation
systems) and also numarray solution.

With warm regards
karthik


-----------------------------------------------------------------------
Karthikesh Raju,		    email: karthik at james.hut.fi
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
P.O Box 5400, FIN 02015 HUT,
Espoo, FINLAND
-----------------------------------------------------------------------



From jsaenz at wm.lc.ehu.es  Fri May  7 01:43:03 2004
From: jsaenz at wm.lc.ehu.es (Jon Saenz)
Date: Fri May  7 01:43:03 2004
Subject: [Numpy-discussion] covariance and other linear algebra functions
In-Reply-To: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>
References: <Pine.SGI.4.58.0405071118120.17460@james.hut.fi>
Message-ID: <20040507103548.I17596@lcdx00.wm.lc.ehu.es>

With NumPy:
http://www.pyclimate.org



Jon Saenz.				| Tfno: +34 946012445
Depto. Fisica Aplicada II               | Fax:  +34 946013500
Facultad de Ciencias.   \\ Universidad del Pais Vasco \\
Apdo. 644   \\ 48080 - Bilbao  \\ SPAIN

On Fri, 7 May 2004, Karthikesh Raju wrote:

> Hi,
>
> Is am searching for a covariance function and other linear algebra
> functions like eigen value decomposition. i am interested in both
> scipy (havent managed to get scipy installed on the main computation
> systems) and also numarray solution.
>
> With warm regards
> karthik
>
>
> -----------------------------------------------------------------------
> Karthikesh Raju,		    email: karthik at james.hut.fi
> Researcher,			    http://www.cis.hut.fi/karthik
> Helsinki University of Technology,  Tel: +358-9-451 5389
> Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
> Department of Computer Sc.,
> P.O Box 5400, FIN 02015 HUT,
> Espoo, FINLAND
> -----------------------------------------------------------------------
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by Sleepycat Software
> Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to deliver
> higher performing products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>



From jmiller at stsci.edu  Fri May  7 07:24:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri May  7 07:24:02 2004
Subject: [Numpy-discussion] numarraycore
In-Reply-To: <409AC74E.2080609@sympatico.ca>
References: <40994E36.2040203@sympatico.ca>
	 <1083790275.28609.17.camel@halloween.stsci.edu>
	 <409971DF.4070908@sympatico.ca>
	 <1083866875.2626.315.camel@halloween.stsci.edu>
	 <409AC74E.2080609@sympatico.ca>
Message-ID: <1083939769.3737.179.camel@localhost.localdomain>

On Thu, 2004-05-06 at 19:16, Colin J. Williams wrote:
> 
> Todd Miller wrote:
> > On Wed, 2004-05-05 at 18:59, Colin J. Williams wrote:
> >   
> > > Todd Miller wrote:
> > >     
> > > > On Wed, 2004-05-05 at 16:27, Colin J. Williams wrote:
> > > >   
> > > >       
> > > > > It would help to have some documentation on the purpose and usage of the 
> > > > > class UsesOpPriority and its variable op_priority.
> > > > >     
> > > > >         
> > > Todd,
> > > 
> > > Thanks for this, perhaps it could be added to the pdf .  I like the
> > > basic idea.
> > >     
> > OK.  What is it you want to do?
> Re: Above.
>    Document whatever you decide to do with UsesOpPriority

I put more comments in numarraycore.   If a lot of people want to
subclass NumArray, it will probably require more discussion because
UsesOpPriority is a limited scheme.  For now,  MaskedArrays do mostly
what I want and it should work for PyMatrix as well.

> Re: Below.
>   Access the methods of NumArray in a Pythonic manner.
>   Subclassing from UsesOpPriority doesn't appear to permit that.

No, it doesn't.  

> >   
> > > > The basic idea was that NumArray subclasses which want to use numarray
> > > > operators would:
> > > > 
> > > > 1.  subclass from UsesOpPriority 
> > > >       
> > > UsesOpPriority has no methods and thus, to access the methods of
> > > NumArray, it would appear to be necessary to subclass from NumArray.
> > >     
> > For limited but important uses (e.g. masked array) subclassing from
> > NumArray is not necessary;  I'd summarize these as the cases where a
> > class wants to take control of operators from NumArray.
> >   
> Thus, if masked array require use of the UsesOpPrioity, then direct
> subclassing from there would appear to make sense.  
> 
> However, where one wishes to use some of the NumArray methods, then,
> contrary to 1, above, subclassing from NumArray should, I suggest, be
> the way to go.

That's true... I overlooked the fact that UsesOpPriority is inherited
automatically by NumArray subclasses.  Sorry for the confusion.

> >   
> > > > 2.  set a class level op_priority > 0. 
> > > > 
> > > > 
> > > > NumArrays have op_priority 0,  higher priorities are given "precedence".
> > > > 
> > > > 
> > > > Thus, given A=NumArray(...) and B=NumArraySubclass(...), and
> > > > A.op_priority==0 and B.op_priority==1, then:  
> > > > A+B would execute as B.__radd__(A) rather than A.__add__(B) and hence
> > > > the type(A+B) could be NumArraySubclass rather than NumArray.  Different
> > > > subclasses could use higher or lower op_priorities to perform the same
> > > > kind of operator resolution among themselves.
> > > >   
> > > >       
> > > I wonder about the desirability of having different levels of priority
> > > for the same level of subclassing and am puzzled that a float variable
> > > is used for the priority.
> > >     
> > Different levels of priority for the same level of sub-classing makes
> > sense to me:  it let's you answer the question "What's the result type
> > of a PyMatrix instance + a SomeOtherSubclass instance?"
> >   
> I agree, there is the possibility that one would wish to evaluate such
> expressions right to left and 
> the priority provides a way of doing this.
> > A float enables insertions of new members into the middle of an existing
> > order.
> >   
> True.  I hope that there will be a database somewhere of all the
> various values for op_priority

Here's the database:  :-)

class                  op_priority

NumArray               0
MaskedArray            1


Regards,
Todd







From gazzar at email.com  Sun May  9 21:14:00 2004
From: gazzar at email.com (Gary Ruben)
Date: Sun May  9 21:14:00 2004
Subject: [Numpy-discussion] Announce: ErrorVal.py 1.0 for error-bounds calculations
Message-ID: <20040510041258.DA8523CE181@ws3-4.us4.outblaze.com>

I finally got around to putting my errorbar calculation module on my website.
<http://users.bigpond.net.au/gazzar/python.html>
This should be of interest to Numeric Python users who deal with calculation of error-bounds on experimental data.

Synopsis:

A module providing a Python number type class and helper functions to ease
the task of computing error bounds on experimental data values.

This module defines an abstract data type, Err, which carries a central/prime
value and upper and lower error bounds.
Helper functions are provided to allow Python Numeric rank-1 arrays of Err objects
to be constructed with ease and to apply Numeric Ufuncs.
Written under Python 2.3.3 and Numeric 23.0

Example of usage:

upperPressure = ArrayOfErr([909., 802., 677., 585., 560., 548.], 1.0)
lowerPressure = ArrayOfErr([144., 246., 378., 469., 493., 505.], 1.0)
pressureDiff = upperPressure - lowerPressure
V_RStandard = ArrayOfErr([2.016, 2.016, 2.020, 2.017, 2.021, 2.019], 0.001)
R = 100.2                                   # standard resistor value [Ohm]
I = V_RStandard / R                         # current [A]
V_RAB = ArrayOfErr([6.167, 6.168, 6.170, 6.160, (6.153, 0.02), (5.894, 0.01)], 0.002)
R_AB = V_RAB / I
logR_AB = ApplyUfuncToErr(R_AB, log10)      # This means log10(R_AB)
print PrimeVals(logR_AB)
print MinVals(logR_AB)
print MaxVals(logR_AB)


Enjoy,
Gary Ruben

-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm




From faheem at email.unc.edu  Mon May 10 16:32:00 2004
From: faheem at email.unc.edu (Faheem Mitha)
Date: Mon May 10 16:32:00 2004
Subject: [Numpy-discussion] add.reduce for character arrays
In-Reply-To: <1082480003.3843.330.camel@localhost.localdomain>
References: <slrnc83e38.1gd.faheem@Chrestomanci.home.earth> 
 <1082383540.19596.72.camel@halloween.stsci.edu>  <Pine.LNX.4.58.0404192315010.25609@Chrestomanci>
 <1082480003.3843.330.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.58.0405101842420.17884@Chrestomanci>


On Tue, 20 Apr 2004, Todd Miller wrote:

> On Mon, 2004-04-19 at 23:20, Faheem Mitha wrote:
> >
> > Yes, that works. Thanks.
> >
> > In [13]: numarray.objects.add.reduce(s,dim=1)
> > Out[13]: ObjectArray(['abc', 'def', 'ghi'])
> >
> > BTW, that makes me wonder what the rules are for the order in which this
> > reduction proceeds for arrays bigger than dimension two. Since there is no
> > total order in this case, the answer is not obvious.
>
> (I think) the answer is that for numarray, reductions ultimately occur
> along the innermost axis.  Off-axis reductions are performed by a
> swapaxis between the off-axis and the innermost axis, followed by the
> reduction, followed by a swapaxis back between the new innermost axis
> and the off-axis.  I'm not sure there's an especially good reason for
> the swap back (since the original swapped axis no longer exists),  but
> it seems to be what Numeric does and not doing it in numarray broke
> extension (linear_algebra?) self-tests.
>
> > For string
> > concatentation (since it is not commutative) at least this matters. I
> > didn't see this documented in the manual but I may have missed it.
>
> Reductions always occur in order of increasing indexes with the
> "reduction-so-far" as the left operand and the next index as the right
> operand.  Reductions occur in a sort of depth-first manner with the
> outermost dimensions varying most slowly.

Sorry about the slow reply. This message got lost in the mess in my inbox,
which I have recently being cleaning out.

I just wanted to say that it would be useful to have the above documented
in some appropriate place in the numarray manual. 99% of the time the
exact procedure for a reduction won't matter, but when someone is using a
non-commutative operator it will. If this is documented in the manual I
could not find it.

Thanks.

                                                               Faheem.



From karthik at james.hut.fi  Wed May 12 01:04:04 2004
From: karthik at james.hut.fi (Karthikesh Raju)
Date: Wed May 12 01:04:04 2004
Subject: [Numpy-discussion] Question about complex variables
Message-ID: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>

Hi All,

i found something strange about complex matrices/arrays. Is it true that
transposing them does not yeald the mathematically correct transpose (the
signs of the imaginary part is reversed)?  This leads to wrong results
while calculating covariances etc ..

Actually i find that things go a bit wild with complex number, is it true?

Warm regards

karthik

-----------------------------------------------------------------------
Karthikesh Raju,		    email: karthik at james.hut.fi
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
P.O Box 5400, FIN 02015 HUT,
Espoo, FINLAND
-----------------------------------------------------------------------



From rsilva at ime.usp.br  Wed May 12 03:46:05 2004
From: rsilva at ime.usp.br (Paulo J. S. Silva)
Date: Wed May 12 03:46:05 2004
Subject: [Numpy-discussion] Question about complex variables
In-Reply-To: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>
References: <Pine.SGI.4.58.0405121059280.96956@james.hut.fi>
Message-ID: <1084358724.21434.15.camel@catirina>

Karthik,

You are making a common mistake. In Mathematics, the "transpose" of a
matrix is just the matrix with the axis transposed like numarray/numeric
does. What you called transpose is actually the "adjoint" or "conjugate
transpose" of a matrix. Actually, this last operation (the adjoint) is
the most interesting one in Linear Algebra/Functional Analysis. For more
details take a look at:

http://en.wikipedia.org/wiki/Conjugate_transpose
http://en.wikipedia.org/wiki/Transpose

In the reals both operations coincide, thus the confusion. This is the
reason why matlab can use the same symbol for both operations.

Now, enough Mathematics, let's go to numarray. To get the adjoint of a
matrix A in numarray, you can simply do:

conjugate(transpose(A))

If you want, you may define a function named adjoint to perform the
above operations. 

Note that this creates a new matrix, not only a new view of the original
matrix like "transpose" function in numarray. 

Hope it helps,

Paulo
-- 
Paulo Jos? da Silva e Silva 
Professor Assistente do Dep. de Ci?ncia da Computa??o
(Assistant Professor of the Computer Science Dept.)
Universidade de S?o Paulo - Brazil

e-mail: rsilva at ime.usp.br          Web: http://www.ime.usp.br/~rsilva

Teoria ? o que n?o entendemos o    (Theory is something we don't)
suficiente para chamar de pr?tica. (understand well enough to call) 
                                   (practice)




From alvaro at antalia.com  Wed May 12 12:45:07 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Wed May 12 12:45:07 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
Message-ID: <1084391465.2145.8.camel@siff>

Hello,

I am new to numarray. I have been searching the documentation, and see
no evident vectorial and concise way to get the indexes for the minimum
of an array times.min().

Currently my two alternatives are: (times is NxN array)

husband, wife = nonzero(equal(times.min(),times)) 
#gives tuple of 1x1 arrays, each containing one index

and (even uglier)

husband = compress(times.min()==times,indices([N,N])[0])
wife    = compress(times.min()==times,indices([N,N])[1])

These are weird ways to get something as simple. I am surely missing
something, but I have tried several slicing strategies before without
success.

For getting the minimum times in each row I use:

choose(argmin(times),transpose(times))

What are the idioms in numpy for these tasks?


Thank you very much in advance, ?.

I would 

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From owen at astro.washington.edu  Wed May 12 12:56:10 2004
From: owen at astro.washington.edu (Russell E Owen)
Date: Wed May 12 12:56:10 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084391465.2145.8.camel@siff>
References: <1084391465.2145.8.camel@siff>
Message-ID: <p06100501bcc82fdc3eca@[140.142.183.23]>

At 9:51 PM +0200 5/12/04, ?lvaro Tejero Cantero wrote:
>I am new to numarray. I have been searching the documentation, and see
>no evident vectorial and concise way to get the indexes for the minimum
>of an array times.min().

To find the index of ONE minimum value is easy, 
though it is buried in the nd_image sub-package 
(where many users might miss it):
numarray.nd_image.minimum_position

I do not know a clean way to find all locations 
of the minimum value. I hope somebody else does.

-- Russell



From tim.hochberg at cox.net  Wed May 12 13:14:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed May 12 13:14:05 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084391465.2145.8.camel@siff>
References: <1084391465.2145.8.camel@siff>
Message-ID: <40A28558.1090008@cox.net>

?lvaro Tejero Cantero wrote:

>Hello,
>
>I am new to numarray. I have been searching the documentation, and see
>no evident vectorial and concise way to get the indexes for the minimum
>of an array times.min().
>  
>
If I understand your question correctly, you want argmin:

 >>> import numarray as na
 >>> from numarray import random_array
 >>> times = random_array.randint(0, 16, [4,4])
 >>> times
array([[15,  3, 11, 10],
       [ 1,  1, 15,  7],
       [ 4,  3,  5,  6],
       [10, 15, 12,  3]])
 >>> na.argmin(times)    # Takes min along axis 0
array([1, 1, 2, 3])
 >>> na.argmin(times, 1) # Take min along axis 1
array([1, 1, 1, 3])

-tim

>Currently my two alternatives are: (times is NxN array)
>
>husband, wife = nonzero(equal(times.min(),times)) 
>#gives tuple of 1x1 arrays, each containing one index
>
>and (even uglier)
>
>husband = compress(times.min()==times,indices([N,N])[0])
>wife    = compress(times.min()==times,indices([N,N])[1])
>
>These are weird ways to get something as simple. I am surely missing
>something, but I have tried several slicing strategies before without
>success.
>
>For getting the minimum times in each row I use:
>
>choose(argmin(times),transpose(times))
>
>What are the idioms in numpy for these tasks?
>
>
>Thank you very much in advance, ?.
>
>I would 
>
>  
>





From alvaro at antalia.com  Wed May 12 14:23:06 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Wed May 12 14:23:06 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100501bcc82fdc3eca@[140.142.183.23]>
References: <1084391465.2145.8.camel@siff>
	 <p06100501bcc82fdc3eca@[140.142.183.23]>
Message-ID: <1084397325.2354.16.camel@siff>

Hello,

> To find the index of ONE minimum value is easy, 
> though it is buried in the nd_image sub-package 
> (where many users might miss it):
> numarray.nd_image.minimum_position

It works great... but what about efficiency? If I do times.min() and
then numarray.nd_image.minimum_positioan(times) I am running twice
essentially the same extremum-finding routine, which is prohibitibe for
large N..., am I right?

Which makes me thing of a more general question: I know that some of the
array functions are coded in C for speed, but what about the classical
python-for loop, as in (r Nx3 array of particle positions)

[ [r[i]-r[j] for i in arange(N)] for j in arange(N)]

is this handled to C code?


> I do not know a clean way to find all locations 
> of the minimum value. I hope somebody else does.

Yes... although for the problem at hand that motivated my query, my
times matrix is symmetric... I don't really need all the minima, but
does numarray have any special datatype for symmetric matrixes, that
prevents storage of unneded (e.g. supradiagonal) elements?.


Thank you very much, I'm on my way to get some beautiful code out of old
fortranisms 

?.
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From tim.hochberg at cox.net  Wed May 12 14:34:05 2004
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Wed May 12 14:34:05 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084397325.2354.16.camel@siff>
References: <1084391465.2145.8.camel@siff>	 <p06100501bcc82fdc3eca@[140.142.183.23]> <1084397325.2354.16.camel@siff>
Message-ID: <40A2980F.2060607@cox.net>

?lvaro Tejero Cantero wrote:

>Hello,
>
>  
>
>>To find the index of ONE minimum value is easy, 
>>though it is buried in the nd_image sub-package 
>>(where many users might miss it):
>>numarray.nd_image.minimum_position
>>    
>>
>
>It works great... but what about efficiency? If I do times.min() and
>then numarray.nd_image.minimum_positioan(times) I am running twice
>essentially the same extremum-finding routine, which is prohibitibe for
>large N..., am I right?
>
>Which makes me thing of a more general question: I know that some of the
>array functions are coded in C for speed, but what about the classical
>python-for loop, as in (r Nx3 array of particle positions)
>
>[ [r[i]-r[j] for i in arange(N)] for j in arange(N)]
>
>is this handled to C code?
>  
>


Try this:

 >>> import numarray as na
 >>> r = na.arange(5)
 >>> na.subtract.outer(r, r)
array([[ 0, -1, -2, -3, -4],
       [ 1,  0, -1, -2, -3],
       [ 2,  1,  0, -1, -2],
       [ 3,  2,  1,  0, -1],
       [ 4,  3,  2,  1,  0]])

Look up the special methods on ufuncs (outer, reduce, etc) for more details.

>>I do not know a clean way to find all locations 
>>of the minimum value. I hope somebody else does.
>>    
>>
>
>Yes... although for the problem at hand that motivated my query, my
>times matrix is symmetric... I don't really need all the minima, but
>does numarray have any special datatype for symmetric matrixes, that
>prevents storage of unneded (e.g. supradiagonal) elements?.
>  
>
Not that I know of.

-tim



>
>Thank you very much, I'm on my way to get some beautiful code out of old
>fortranisms 
>
>?.
>  
>





From perry at stsci.edu  Wed May 12 15:11:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 12 15:11:02 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <1084397325.2354.16.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> It works great... but what about efficiency? If I do times.min() and
> then numarray.nd_image.minimum_positioan(times) I am running twice
> essentially the same extremum-finding routine, which is prohibitibe for
> large N..., am I right?
>
Well, yes. But when you ask to find all the things that equal
the minimum, you pretty much must look twice (if you want to know
where they all are if more than one). Once to determine the
minimum, the next time to locate all of them.
You didn't really say whether you needed just the first minimum or
all minima (if I recall correctly).

> Which makes me thing of a more general question: I know that some of the
> array functions are coded in C for speed, but what about the classical
> python-for loop, as in (r Nx3 array of particle positions)
>
> [ [r[i]-r[j] for i in arange(N)] for j in arange(N)]
>
> is this handled to C code?
>
As Tim mentions, yes this can be done efficiently. But there
is no general answer this sort of open question. It depends
on what you are doing. Sometimes there are functions or
tricks to avoid loops in Python, sometimes not.
>
> Yes... although for the problem at hand that motivated my query, my
> times matrix is symmetric... I don't really need all the minima, but
> does numarray have any special datatype for symmetric matrixes, that
> prevents storage of unneded (e.g. supradiagonal) elements?.
>
Not for special cases like this. One could probably write a special
subclass to do this, but for a savings of a factor of 2 in memory,
it usually would not be worth the trouble (unlike sparse matrices)

Perry




From owen at astro.washington.edu  Wed May 12 16:59:07 2004
From: owen at astro.washington.edu (Russell E Owen)
Date: Wed May 12 16:59:07 2004
Subject: [Numpy-discussion] puzzling compiler warning for numeric extension
Message-ID: <p06100501bcc864cfb9df@[140.142.183.76]>

I modified an existing numeric c extension today, adding a function 
that returns a new array. This is the first time I've used 
NA_NewArray. The code seems to run fine, but I get a disturbing 
compiler warning:

[rowen:Shared Files/Guider/PyGuide] rowen% python setup.py build
...
src/radProfModule.c: In function `Py_radSqByRadInd':
src/radProfModule.c:378: warning: return from incompatible pointer type

The associated code can be summarized as follows (and it is simple 
enough that I've also appended a copy). The error is thrown on the 
last line:

static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
	long nElt;
	Long *radSqByRadInd;

	if (!PyArg_ParseTuple(args, "l", &nElt))
		return NULL;
...
	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
}

So...am I actually doing something wrong, or is this warning normal?

-- Russell

P.S. the full code:

static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
	long nElt;
	Long radInd;
	char ModName[] = "radSqByRadInd";
	Long *radSqByRadInd;

	if (!PyArg_ParseTuple(args, "l", &nElt))
		return NULL;

	if (nElt < 0) {
		PyErr_Format(PyExc_ValueError, "%s: nPts < 0", ModName);
		return NULL;
	}

	// allocate max(3, nElt) elements (simplifies the computation
	// and avoids calling calloc on 0 elements)
	radSqByRadInd = calloc(MAX(nElt, 3), sizeof *radSqByRadInd);
	if (radSqByRadInd == NULL) {
		PyErr_Format(PyExc_MemoryError, "%s: insufficient 
memory", ModName);
		return NULL;
	}

	for (radInd=0; radInd<3; radInd++) {
		radSqByRadInd[radInd] = radInd;
	}
	for (radInd=3; radInd<nElt; radInd++) {
		radSqByRadInd[radInd] = (radInd - 1) * (radInd - 1);
	}

	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
}



From rkern at ucsd.edu  Wed May 12 17:29:32 2004
From: rkern at ucsd.edu (Robert Kern)
Date: Wed May 12 17:29:32 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFGEHGCDAA.perry@stsci.edu>
Message-ID: <40A2C0EE.703@ucsd.edu>

Perry Greenfield wrote:

> ?lvaro Tejero Cantero wrote:
> 
>>It works great... but what about efficiency? If I do times.min() and
>>then numarray.nd_image.minimum_positioan(times) I am running twice
>>essentially the same extremum-finding routine, which is prohibitibe for
>>large N..., am I right?
>>
> 
> Well, yes. But when you ask to find all the things that equal
> the minimum, you pretty much must look twice (if you want to know
> where they all are if more than one). Once to determine the
> minimum, the next time to locate all of them.

Nah, you can accumulate indices corresponding to the current minimum 
value as you go. Discard the list of indices and start again if you get 
a new minimum value.

minval = data[0]  # let's suppose data is a vector for demonstration
minindices = [0]

for i in xrange(1, len(data)):
     x = data[i]
     if x < minval:
         minindices = [i]
         minval = x
     elif x == minval:
         minindices.append(i)

Whether or not this is faster (when implemented in C) than going over it 
twice using numarray functions is another question. My guess: not enough.

[snip]

>>Yes... although for the problem at hand that motivated my query, my
>>times matrix is symmetric... I don't really need all the minima, but
>>does numarray have any special datatype for symmetric matrixes, that
>>prevents storage of unneded (e.g. supradiagonal) elements?.
>>
> 
> Not for special cases like this. One could probably write a special
> subclass to do this, but for a savings of a factor of 2 in memory,
> it usually would not be worth the trouble (unlike sparse matrices)

OTOH, having subclasses that follow LAPACK's symmetric packed storage 
scheme would be very useful not because of the space factor but the time 
saved by being able to use the symmetric algorithms in LAPACK. I think.

> Perry

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From alvaro at antalia.com  Thu May 13 03:32:02 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 03:32:02 2004
Subject: outer idiom (was Re: [Numpy-discussion] Getting the indexes of the
	myarray.min())
In-Reply-To: <40A2980F.2060607@cox.net>
References: <1084391465.2145.8.camel@siff>
	 <p06100501bcc82fdc3eca@[140.142.183.23]> <1084397325.2354.16.camel@siff>
	 <40A2980F.2060607@cox.net>
Message-ID: <1084444629.800.12.camel@siff>

Hello,

(for background, I am trying to get an array of interparticle distances,
in which calculation of supradiagonal elements is unneeded because of
symmetry. I am not doing anything about this now, though).

>>> import numarray.random_array as rdn
>>> N, D = 1000, 3 #number of particles and dimensions of space
>>> r = rnd.random([N,D])  # r[i] gives the D coordinates of particle i
                           # r[:,0] gives the all the x coordinates
What I was doing:

>>> r_rel = [[r[i]-r[j] for i in arange(N)] for j in arange N]

now Tim says:

> Try this:
> 
>  >>> import numarray as na
>  >>> r = na.arange(5)
>  >>> na.subtract.outer(r, r)
> array([[ 0, -1, -2, -3, -4],
>        [ 1,  0, -1, -2, -3],
>        [ 2,  1,  0, -1, -2],
>        [ 3,  2,  1,  0, -1],
>        [ 4,  3,  2,  1,  0]])
> 

but this gives
>>> subtract.outer(r,r).shape
(10, 3, 10, 3)

that is, subtracts y coordinates to x coordinates which is not intended.
AFAIK the outer solution is MUCH faster than the nested for loops, so
what I do now is

>>> r_rel = transpose(array([subtract.outer(r[:,0],r[:,0]),
                             subtract.outer(r[:,1],r[:,1]), 
                             subtract.outer(r[:,2],r[:,2])]))
>>> r_rel.shape #as with the double loop 
(10,10,3) 


My question is then if there is any more elegant way to do this,
especially giving as a result independence of the number of dimensions).

Maybe an "axis"  (=0 in this case?) keyword for the outer function would
be useful in this context?



Thanks for the helpful welcome to the list!,
 ?.

PS. the problem with the min() function regarded a matrix of collision
times between particles, which is symmetrical. The elements are reals,
so I only expect to find one minimum.

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 04:07:10 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 04:07:10 2004
Subject: [Numpy-discussion] Error in choose (bug?)
Message-ID: <1084446790.917.9.camel@siff>

I have a matrix of particle collision times: times[i,j] gives the time
for particle "i" to collide with particle "j".

If I do, in order to get the first expected collision time for each
particle, the following (random array for testing purposes)	:

>>> N=30
>>> times = rnd.random([N,N])
>>> choose(argmin(times,transpose(times))
Segmentation fault (of the python interactive shell!!!)

With N=100 I get a more informative traceback, and rest within the
shell:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
    return _choose(selector, population, outarr, clipmode)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
    result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
    blockingparameters)
ValueError: _operator_compute: too many inputs + outputs


For N=10,N=15 I get the expected output, but for N=20 I get again the
brutal segfault...


regards, 
?.

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 04:50:01 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 04:50:01 2004
Subject: [Numpy-discussion] Unexpected warnings in where
Message-ID: <1084449282.909.21.camel@siff>

Hi,

I cannot understand why I do receive these warnings:

Warning: Encountered invalid numeric result(s)  in sqrt
Warning: Encountered invalid numeric result(s)  in divide

after

>>> newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV, far_future)


RV, discr, RR, VV are NxN arrays of reals (Float64), while
far_future=1e20 (scalar).

1. sqrt(<0) should be impossible, since it's explicitly ruled out that
discr<0 in the where condition.

2. /0 should also be impossible, since VV==0 is True only for the
diagonal...

I've done some testing and where(logical_and(RV<0, discr>0), discr, 666)
doesn't show any negative numbers, neither does
where(logical_and(RV<0,discr>0),VV,666) show any zeroes...


	
Please.. what am I doing wrong here?


-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From jmiller at stsci.edu  Thu May 13 05:43:12 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 05:43:12 2004
Subject: [Numpy-discussion] puzzling compiler warning for numeric
	extension
In-Reply-To: <p06100501bcc864cfb9df@[140.142.183.76]>
References: <p06100501bcc864cfb9df@[140.142.183.76]>
Message-ID: <1084452101.483.4.camel@halloween.stsci.edu>

On Wed, 2004-05-12 at 19:54, Russell E Owen wrote:
> I modified an existing numeric c extension today, adding a function 
> that returns a new array. This is the first time I've used 
> NA_NewArray. The code seems to run fine, but I get a disturbing 
> compiler warning:
> 
> [rowen:Shared Files/Guider/PyGuide] rowen% python setup.py build
> ...
> src/radProfModule.c: In function `Py_radSqByRadInd':
> src/radProfModule.c:378: warning: return from incompatible pointer type
> 
> The associated code can be summarized as follows (and it is simple 
> enough that I've also appended a copy). The error is thrown on the 
> last line:
> 
> static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
> 	long nElt;
> 	Long *radSqByRadInd;
> 
> 	if (!PyArg_ParseTuple(args, "l", &nElt))
> 		return NULL;
> ...
> 	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
> }
> 
> So...am I actually doing something wrong, or is this warning normal?

The problem is that your function returns a PyObject* but NA_NewArray
returns a PyArrayObject*.   This particular "error" is generally
harmless since all that is really in conflict is the compiler's view of
the same memory address.  You can eliminate the warning by sticking in a
cast to PyObject* like this:

	return (PyObject *) NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);

Regards,
Todd

> 

> 
> -- Russell
> 
> P.S. the full code:
> 
> static PyObject *Py_radSqByRadInd(PyObject *self, PyObject *args) {
> 	long nElt;
> 	Long radInd;
> 	char ModName[] = "radSqByRadInd";
> 	Long *radSqByRadInd;
> 
> 	if (!PyArg_ParseTuple(args, "l", &nElt))
> 		return NULL;
> 
> 	if (nElt < 0) {
> 		PyErr_Format(PyExc_ValueError, "%s: nPts < 0", ModName);
> 		return NULL;
> 	}
> 
> 	// allocate max(3, nElt) elements (simplifies the computation
> 	// and avoids calling calloc on 0 elements)
> 	radSqByRadInd = calloc(MAX(nElt, 3), sizeof *radSqByRadInd);
> 	if (radSqByRadInd == NULL) {
> 		PyErr_Format(PyExc_MemoryError, "%s: insufficient 
> memory", ModName);
> 		return NULL;
> 	}
> 
> 	for (radInd=0; radInd<3; radInd++) {
> 		radSqByRadInd[radInd] = radInd;
> 	}
> 	for (radInd=3; radInd<nElt; radInd++) {
> 		radSqByRadInd[radInd] = (radInd - 1) * (radInd - 1);
> 	}
> 
> 	return NA_NewArray((void *)radSqByRadInd, tLong, 1, nElt);
> }
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller <jmiller at stsci.edu>




From jmiller at stsci.edu  Thu May 13 06:06:00 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 06:06:00 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084449282.909.21.camel@siff>
References: <1084449282.909.21.camel@siff>
Message-ID: <1084453439.483.16.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 07:54, ?lvaro Tejero Cantero wrote:
> Hi,
> 
> I cannot understand why I do receive these warnings:
> 
> Warning: Encountered invalid numeric result(s)  in sqrt
> Warning: Encountered invalid numeric result(s)  in divide
> 
> after
> 
> >>> newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV, far_future)
> 
> 
> RV, discr, RR, VV are NxN arrays of reals (Float64), while
> far_future=1e20 (scalar).
> 
> 1. sqrt(<0) should be impossible, since it's explicitly ruled out that
> discr<0 in the where condition.
> 
> 2. /0 should also be impossible, since VV==0 is True only for the
> diagonal...
> 
> I've done some testing and where(logical_and(RV<0, discr>0), discr, 666)
> doesn't show any negative numbers, neither does
> where(logical_and(RV<0,discr>0),VV,666) show any zeroes...
> 
> 
> 	
> Please.. what am I doing wrong here?

I think what you're seeing is this:

where(condition, expression1, expression2)

Even though condition is selecting parts of expression1 which are valid,
expression1 is still fully evaluated.

Regards,
Todd

-- 
Todd Miller <jmiller at stsci.edu>




From perry at stsci.edu  Thu May 13 06:29:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 06:29:02 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <40A2C0EE.703@ucsd.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>

Robert Kern wrote:

> > Well, yes. But when you ask to find all the things that equal
> > the minimum, you pretty much must look twice (if you want to know
> > where they all are if more than one). Once to determine the
> > minimum, the next time to locate all of them.
> 
> Nah, you can accumulate indices corresponding to the current minimum 
> value as you go. Discard the list of indices and start again if you get 
> a new minimum value.
> 
True enough (though I suspect some special cases may not work any 
faster, e.g., an array of all zeros with the last element equal to
-1; you spend all the time copying indices for nearly the whole damn
thing for no purpose). There are a bunch of things that can be done
to eliminate multiple passes but they tend to lead to many different
specialized functions. One has to trade off the number of such functions
against the speed savings. Another example is getting max and min values
for an array. I've long thought that this is so often done they could
be done in one pass. There isn't a function that does this yet though.

> >>Yes... although for the problem at hand that motivated my query, my
> >>times matrix is symmetric... I don't really need all the minima, but
> >>does numarray have any special datatype for symmetric matrixes, that
> >>prevents storage of unneded (e.g. supradiagonal) elements?.
> >>
> > 
> > Not for special cases like this. One could probably write a special
> > subclass to do this, but for a savings of a factor of 2 in memory,
> > it usually would not be worth the trouble (unlike sparse matrices)
> 
> OTOH, having subclasses that follow LAPACK's symmetric packed storage 
> scheme would be very useful not because of the space factor but the time 
> saved by being able to use the symmetric algorithms in LAPACK. I think.
> 
I'd agree that that would be a much stronger motivating factor.

Perry 



From alvaro at antalia.com  Thu May 13 09:28:35 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 09:28:35 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084453439.483.16.camel@halloween.stsci.edu>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
Message-ID: <1084466049.1084.20.camel@siff>

> I think what you're seeing is this:
> 
> where(condition, expression1, expression2)
> 
> Even though condition is selecting parts of expression1 which are valid,
> expression1 is still fully evaluated.

Is this behaviour what is intended, or do you consider it a shortcoming
of the implementation?. In theory avoiding unneeded evaluations of
expression1 is very desirable...

-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From alvaro at antalia.com  Thu May 13 09:31:10 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 09:31:10 2004
Subject: [Numpy-discussion] Example code
Message-ID: <1084466214.1070.25.camel@siff>

Hello,

Do you know of publicly available example code with numarray?. Short of
the tutorial and the page by Magnus Hetland, and the intro by David
Mertz, I haven't found much code written in numarray through google.

It would be helpful for me to have more code to look at. I am explicitly
discarding numpy code in order to keep away from subtle differences, but
perhaps this is overcautious.

Thanks!
?.
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From rays at blue-cove.com  Thu May 13 09:50:04 2004
From: rays at blue-cove.com (Ray Schumacher)
Date: Thu May 13 09:50:04 2004
Subject: [Numpy-discussion] Example code
In-Reply-To: <1084466214.1070.25.camel@siff>
Message-ID: <5.2.0.4.2.20040513094730.0609a1e8@blue-cove.com>

At 06:37 PM 5/13/2004 +0200, ?lvaro Tejero Cantero wrote:
>Hello,
>
>Do you know of publicly available example code with numarray?. Short of
>the tutorial and the page by Magnus Hetland, and the intro by David
>Mertz, I haven't found much code written in numarray through google.

Try:
http://rjs.org/astro/1004x/Python/CAP7X.PY

I might have a few others around I can share...


Ray Schumacher
Blue Cove Interactive
7220 Trade Street, Suite 101
San Diego, CA 92121
858.695.8801
http://Blue-Cove.com




From jmiller at stsci.edu  Thu May 13 09:58:07 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 09:58:07 2004
Subject: [Numpy-discussion] Error in choose (bug?)
In-Reply-To: <1084446790.917.9.camel@siff>
References: <1084446790.917.9.camel@siff>
Message-ID: <1084467425.483.274.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 07:13, ?lvaro Tejero Cantero wrote:
> I have a matrix of particle collision times: times[i,j] gives the time
> for particle "i" to collide with particle "j".
> 
> If I do, in order to get the first expected collision time for each
> particle, the following (random array for testing purposes)	:
> 
> >>> N=30
> >>> times = rnd.random([N,N])
> >>> choose(argmin(times,transpose(times))
> Segmentation fault (of the python interactive shell!!!)
> 
> With N=100 I get a more informative traceback, and rest within the
> shell:
> 
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
>     return _choose(selector, population, outarr, clipmode)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
>     result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
>     blockingparameters)
> ValueError: _operator_compute: too many inputs + outputs
> 
> 
> For N=10,N=15 I get the expected output, but for N=20 I get again the
> brutal segfault...
> 
> 
> regards, 
> ?.

I was able to reproduce this bug, logged it on Source Forge, and will
get to it as soon as possible... probably tomorrow or Monday.

Regards,
Todd






From perry at stsci.edu  Thu May 13 10:00:02 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 10:00:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084466049.1084.20.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFOEHICDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> > I think what you're seeing is this:
> >
> > where(condition, expression1, expression2)
> >
> > Even though condition is selecting parts of expression1 which are valid,
> > expression1 is still fully evaluated.
>
> Is this behaviour what is intended, or do you consider it a shortcoming
> of the implementation?. In theory avoiding unneeded evaluations of
> expression1 is very desirable...
>
It's about the only way it could work given how an array-based
library works (unless there were a mechanism to postpone
evaluation of expressions until their context can be
determined). The expression is evaluated before it is passed to the
where function.

Perry Greenfield




From jmiller at stsci.edu  Thu May 13 10:02:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 13 10:02:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <1084466049.1084.20.camel@siff>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
	 <1084466049.1084.20.camel@siff>
Message-ID: <1084467685.483.281.camel@halloween.stsci.edu>

On Thu, 2004-05-13 at 12:34, ?lvaro Tejero Cantero wrote:
> > I think what you're seeing is this:
> > 
> > where(condition, expression1, expression2)
> > 
> > Even though condition is selecting parts of expression1 which are valid,
> > expression1 is still fully evaluated.
> 
> Is this behaviour what is intended, or do you consider it a shortcoming
> of the implementation?. In theory avoiding unneeded evaluations of
> expression1 is very desirable...

This just falls out of Python function call semantics: parameters are
evaluated before a function is called.   It is a property of Python so I
think we're stuck with it.

Two things come to mind to get rid of the warnings:

1. Stuff the input arrays with values which avoid the warnings.
condition = logical_and(RV<0,discr>0)
discr[condition] = 1
VV[condition] = 1
newtimes = where(condition, (-RV-sqrt(discr))/VV, far_future)

2. Use the Error system to suppress the warnings:

Error.pushMode(dividebyzero="ignore", invalid="ignore")
newtimes = where(logical_and(RV<0,discr>0), (-RV-sqrt(discr))/VV,
far_future)
Error.popMode()

Regards,
Todd





From lilliarasconton at hotmail.com  Thu May 13 11:11:04 2004
From: lilliarasconton at hotmail.com (Clare Booker)
Date: Thu May 13 11:11:04 2004
Subject: [Numpy-discussion] 500 DOLLARS For You, Just For Opening This Notice, Thu, 13 May 2004 14:09:55 -0500
Message-ID: <MLCSOJEEEEYATGGMAAKTHX@yahoo.com>

Thu, 13 May 2004 14:09:55 -0500

CASH TODAY will help you get back on your feet.   http://www.lifetimedealz.com/cash23/ 

If late payments and unexpected bills have set you back financially CASH TODAY can help. We specialize in helping people pay their bills on time without any expensive fees or hassles.

http://www.lifetimedealz.com/cash23/ 

It only takes a minute to apply for a CASH TODAY personal loan for up to 500 dollars. Once you're approved we deposit money directly into your account within 1/2 hour of Approval.

http://www.lifetimedealz.com/cash23/ 


No more ads: http://www.lifetimedealz.com/away.html

tale

chlorophyll hebephrenic activation natchez nonsensical bilge liturgy atreus dunlap optoacoustic blonde wilson winter edwards carbonaceous 

behead flinty embower divination thyronine priori proceed chablis ma ferromagnetic camber retrospect tricky ahem simplicial villein brandt bombast graff petroleum anachronistic lufthansa coincident chanson hunt confucius gogo fixture trilogy deviate erlenmeyer alterate canberra doorknob dryad neve customary consul implore alarm 

From alvaro at antalia.com  Thu May 13 11:51:02 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Thu May 13 11:51:02 2004
Subject: [Numpy-discussion] Unexpected warnings in where
In-Reply-To: <40A3C119.40403@pfdubois.com>
References: <1084449282.909.21.camel@siff>
	 <1084453439.483.16.camel@halloween.stsci.edu>
	 <1084466049.1084.20.camel@siff>
	 <1084467685.483.281.camel@halloween.stsci.edu>
	 <40A3C119.40403@pfdubois.com>
Message-ID: <1084474601.1932.1.camel@siff>

On Thu, 2004-05-13 at 20:40, Paul Dubois wrote:
> Use MA, it is designed to solve this problem.

Is it as fast as using numarray arrays?
Are you aware of example code floating around, apart from the docs?

Thank you!
-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From rowen at u.washington.edu  Thu May 13 12:23:04 2004
From: rowen at u.washington.edu (Russell E Owen)
Date: Thu May 13 12:23:04 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
Message-ID: <p06100509bcc97a82aef9@[128.95.99.44]>

At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
>... One has to trade off the number of such functions
>against the speed savings. Another example is getting max and min values
>for an array. I've long thought that this is so often done they could
>be done in one pass. There isn't a function that does this yet though.

Statistics is another area where multiple return values could be of 
interest -- one may want the mean and std dev, and making two passes 
is wasteful (since some of the same info needs to be computed both 
times).

A do-all function that computes min, min location, max, max location, 
mean and std dev all at once would be nice (especially if the 
returned values were accessed by name, rather than just being a tuple 
of values, so they could be referenced safely and readably).

-- Russell



From perry at stsci.edu  Thu May 13 12:43:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 12:43:01 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <CEELJPECNGEGKFNDLHFFEEHJCDAA.perry@stsci.edu>

> Russell E Owen wrote:
> 
> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
> >... One has to trade off the number of such functions
> >against the speed savings. Another example is getting max and min values
> >for an array. I've long thought that this is so often done they could
> >be done in one pass. There isn't a function that does this yet though.
> 
> Statistics is another area where multiple return values could be of 
> interest -- one may want the mean and std dev, and making two passes 
> is wasteful (since some of the same info needs to be computed both 
> times).
> 
> A do-all function that computes min, min location, max, max location, 
> mean and std dev all at once would be nice (especially if the 
> returned values were accessed by name, rather than just being a tuple 
> of values, so they could be referenced safely and readably).
> 
> -- Russell
> 
We will definitely add something like this for 1.0 or 1.1.
(but probably for min and max location, it will just be
for the first encountered).

Perry



From focke at slac.stanford.edu  Thu May 13 14:10:07 2004
From: focke at slac.stanford.edu (Warren Focke)
Date: Thu May 13 14:10:07 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
 <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <Pine.LNX.4.58.0405131310400.27825@auriga.slac.stanford.edu>


On Thu, 13 May 2004, Russell E Owen wrote:

> Statistics is another area where multiple return values could be of
> interest -- one may want the mean and std dev, and making two passes
> is wasteful (since some of the same info needs to be computed both
> times).

Single pass std deviations don't work very well if you've got a lot of
data points and the std deviation is small compared to the average.
I'm not arguing aginst including them, but maybe the documentation for
such a function should include a caveat.

Warren Focke




From southey at uiuc.edu  Thu May 13 14:22:02 2004
From: southey at uiuc.edu (Bruce Southey)
Date: Thu May 13 14:22:02 2004
Subject: [Numpy-discussion] Getting the indexes
 of the myarray.min()
Message-ID: <ee2ef33e.400618f5.81a0e00@expms6.cites.uiuc.edu>

Hi,
Raymond D. Hettinger is writing a general statistics module 'statistics.py  A
collection of functions for summarizing data' that is somewhere in a Python
CVS (I can not find the exact reference but it appeared in a fairly recent
Python thread). He uses a one-pass algorithm from Knuth for the variance that
has good numerical stability.

Below is a rather rough version modified from my situation (masked arrays) which
uses Knuth's algorithm for the variance. It lacks features like checking
dimensions (assumes variance can be computed) and documentation.

Regards
Bruce Southey

import numarray

def SummaryStats(Matrix):
        mshape=Matrix.getshape()
        nrows=mshape[0]
        ncols=mshape[1]
        #print nrows, ncols
        # Create matrices to hold statistics
        N_obs =numarray.zeros(ncols, type='Float64')
        Sum   =numarray.zeros(ncols, type='Float64')
        Var   =numarray.zeros(ncols, type='Float64')
        Min   =numarray.zeros(ncols, type='Float64')
        Max   =numarray.zeros(ncols, type='Float64')
        Mean  =numarray.zeros(ncols, type='Float64')
        AdjM  =numarray.zeros(ncols, type='Float64')
        NewM  =numarray.zeros(ncols, type='Float64')
        DifM  =numarray.zeros(ncols, type='Float64')

        for row in range(nrows):
                for col in range(ncols):
                        t_value=Matrix[row,col]
                        N_obs[col] = N_obs[col] + 1
                        Sum[col] = Sum[col] + t_value
                        if t_value > Max[col]:
                                Max[col]=t_value
                        if t_value < Min[col]:
                                Min[col]=t_value
                        if N_obs[col]==1:
                                Mean[col]=t_value
                        AdjM[col]=(t_value-Mean[col])/(N_obs[col])-DifM[col]
                        NewM[col]=Mean[col]+AdjM[col]
                        DifM[col]=(NewM[col]-Mean[col])-AdjM[col]
                        Var[col] = Var[col] +
(t_value-Mean[col])*(t_value-NewM[col])
                        Mean[col] = NewM[col]
        print 'N_obs\n', N_obs
        print 'Sum\n', Sum
        print 'Mean\n', Mean
        print 'Var\n', Var/(nrows-1)

if __name__ == '__main__':
        MValues=numarray.array([[1,2,1],[3,2,2],[5,1,1],[4,3,2]])
        SummaryStats(MValues)



---- Original message ----
>Date: Thu, 13 May 2004 15:42:30 -0400
>From: "Perry Greenfield" <perry at stsci.edu>  
>Subject: RE: [Numpy-discussion] Getting the indexes of the myarray.min()  
>To: "Russell E Owen" <rowen at u.washington.edu>, "numarray"
<numpy-discussion at lists.sourceforge.net>
>
>> Russell E Owen wrote:
>> 
>> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
>> >... One has to trade off the number of such functions
>> >against the speed savings. Another example is getting max and min values
>> >for an array. I've long thought that this is so often done they could
>> >be done in one pass. There isn't a function that does this yet though.
>> 
>> Statistics is another area where multiple return values could be of 
>> interest -- one may want the mean and std dev, and making two passes 
>> is wasteful (since some of the same info needs to be computed both 
>> times).
>> 
>> A do-all function that computes min, min location, max, max location, 
>> mean and std dev all at once would be nice (especially if the 
>> returned values were accessed by name, rather than just being a tuple 
>> of values, so they could be referenced safely and readably).
>> 
>> -- Russell
>> 
>We will definitely add something like this for 1.0 or 1.1.
>(but probably for min and max location, it will just be
>for the first encountered).
>
>Perry
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: SourceForge.net Broadband
>Sign-up now for SourceForge Broadband and get the fastest
>6.0/768 connection for only $19.95/mo for the first 3 months!
>http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion



From perry at stsci.edu  Thu May 13 14:31:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 14:31:01 2004
Subject: outer idiom (was Re: [Numpy-discussion] Getting the indexes of themyarray.min())
In-Reply-To: <1084444629.800.12.camel@siff>
Message-ID: <CEELJPECNGEGKFNDLHFFAEHKCDAA.perry@stsci.edu>

?lvaro Tejero Cantero wrote:
>
> but this gives
> >>> subtract.outer(r,r).shape
> (10, 3, 10, 3)
>
> that is, subtracts y coordinates to x coordinates which is not intended.
> AFAIK the outer solution is MUCH faster than the nested for loops, so
> what I do now is
>
> >>> r_rel = transpose(array([subtract.outer(r[:,0],r[:,0]),
>                              subtract.outer(r[:,1],r[:,1]),
>                              subtract.outer(r[:,2],r[:,2])]))
> >>> r_rel.shape #as with the double loop
> (10,10,3)
>
>
> My question is then if there is any more elegant way to do this,
> especially giving as a result independence of the number of dimensions).
>
Not that I can think of at the moment.

> Maybe an "axis"  (=0 in this case?) keyword for the outer function would
> be useful in this context?
>
Perhaps. Right now these ufunc methods are pretty complicated so
it may not be easy to do, but I understand that there is certainly
utility in being able to do that. We'll look into it (but not
right away so don't hold your breath).

Perry




From perry at stsci.edu  Thu May 13 14:41:00 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 13 14:41:00 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <CEELJPECNGEGKFNDLHFFEEHJCDAA.perry@stsci.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFIEHKCDAA.perry@stsci.edu>

> > A do-all function that computes min, min location, max, max location, 
> > mean and std dev all at once would be nice (especially if the 
> > returned values were accessed by name, rather than just being a tuple 
> > of values, so they could be referenced safely and readably).
> > 
> > -- Russell
> > 
> We will definitely add something like this for 1.0 or 1.1.
> (but probably for min and max location, it will just be
> for the first encountered).
> 
> Perry
> 
To elaborate on this a bit, I'm thinking that the minmax
capability probably should be separated from statistics
(it's very common to want to do min max without wanting
to do mean or stdev, plus as others have noted, the statistics
can be a bit more involved).

There is one other aspect that needs user input.
How to handle ieee special values for things like minmax.
Right now the ufunc reductions have some odd behavior that
is a result of underlying C library behavior. A NaN isn't 
consistently handled in such comparisons (it appears to 
depend on the order when comparing a NaN to a regular float,
whichever appears second appears to be accepted in pairwise
comparisons).

I was thinking that one could add an exclude keyword
to the functions to indicate that ieee special values should
not be included (well, I suppose Inf should be for min max)
otherwise they would be. Any thoughts on this?

Perry



From jsaenz at wm.lc.ehu.es  Fri May 14 03:04:10 2004
From: jsaenz at wm.lc.ehu.es (Jon Saenz)
Date: Fri May 14 03:04:10 2004
Subject: [Numpy-discussion] Getting the indexes of the myarray.min()
In-Reply-To: <p06100509bcc97a82aef9@[128.95.99.44]>
References: <CEELJPECNGEGKFNDLHFFOEHHCDAA.perry@stsci.edu>
 <p06100509bcc97a82aef9@[128.95.99.44]>
Message-ID: <20040514115430.G48469@lcdx00.wm.lc.ehu.es>

What about an object (TimeSeries) which can be "explored" (optionally)
during its init method and, if so, it creates the dictionary with those
values? If it is not explored, the dictionary would be assigned to None
and, if requested, the "exploratory" statistics would be computed then.
This could be the basis for other computations on time series.

Just my two cents.


Jon Saenz.				| Tfno: +34 946012445
Depto. Fisica Aplicada II               | Fax:  +34 946013500
Facultad de Ciencias.   \\ Universidad del Pais Vasco \\
Apdo. 644   \\ 48080 - Bilbao  \\ SPAIN

On Thu, 13 May 2004, Russell E Owen wrote:

> At 9:27 AM -0400 2004-05-13, Perry Greenfield wrote:
> >... One has to trade off the number of such functions
> >against the speed savings. Another example is getting max and min values
> >for an array. I've long thought that this is so often done they could
> >be done in one pass. There isn't a function that does this yet though.
>
> Statistics is another area where multiple return values could be of
> interest -- one may want the mean and std dev, and making two passes
> is wasteful (since some of the same info needs to be computed both
> times).
>
> A do-all function that computes min, min location, max, max location,
> mean and std dev all at once would be nice (especially if the
> returned values were accessed by name, rather than just being a tuple
> of values, so they could be referenced safely and readably).
>
> -- Russell
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>



From eric at enthought.com  Sun May 16 21:36:02 2004
From: eric at enthought.com (eric jones)
Date: Sun May 16 21:36:02 2004
Subject: [Numpy-discussion] ANN: Job Openings at Enthought
Message-ID: <40A840F3.2050608@enthought.com>

Hello,

This is a bit of a status report I guess for SciPy (www.scipy.org) and
Enthought (www.enthought.com), but, really, it is a long winded job
posting. 

Among other things, Enthought develops scientific software
for clients. Python is central to to our strategy in this arena.  We have
long supported SciPy and believe strongly in Open Source Software. 
The purpose is to give people a feeling about what we do, the technical
challenges we face, and the skills needed to meet them.

SciPy
-----
There has been a lot of work on SciPy lately by the Travis Oliphant, Travis
Vaught, Pearu Peterson, Joe Cooper, and on the website by Jon-Eric 
Steinbomer
and Janet Swisher.  The site is now upgraded to Plone and we have a 
SciPy 0.3
package out.  If you're interested in seeing the activity level at 
scipy.org,
please visit.

http://www.scipy.org/map?rmurl=http%3A//scipy.net/scipyusage/

It looks like we're averaging about a 100 downloads a day if you add up all
source and binary distributions.  When SciPy 0.1 was released a couple a 
years
ago, I think we averaged about 10.  Its extremely difficult to 
extrapolate a user
base from such information, the obvious growth is good news.

Other Stuff
-----------
In addition to SciPy we have multiple projects going on right now that 
either
already plug into or will be plugins to a Python-based scientific 
application
framework called Envisage (sorta like an IDE for science apps) that we're
working on.  The idea is:

(1) To lower the bar as far as work required to get a GUI put
   on the front end of a scientific algorithm.
(2) Develop scientific plugins that can play well together so
   that, for instance, an electromagnetics simulator built by one
   person can be used in conjunction with an optimization plugin
   built by another person and results can be visualized with a
   3D plugin.

This is a hard, long path that requires much work and though.  We have
started the process and actually have built an app on a very early 
version of
the Envisage framework.  The app works great, but it's usage of the 
framework is a
little mixed at this point.  There were a lot of compromises we ended up 
making
on the framework side in order to ship on time.

Also, I am not sure whether "easy-to-program" and "flexible architecture for
compatible plugins" are not mutually exclusive.  I always have in my 
mind that I
want a smart scientist to be able to build the GUI in a short amount of time
after the algorithm (which is the "hard part") is finished.  I still 
harbor this
wish, but the lessons I've had over the last couple of years suggest 
that the GUI is
actually the most expensive part of development for large apps.  This is 
partially
due to the fact that commercial apps are rarely built around "research" 
algorithms
-- meaning that the algorithm code usually already exists in some form.  
Still,
UIs take tons of time.  Testing them is hard.  Flexibility requires more
complexity (factories, adapters, etc.) than seems necessary at first.  
Further,
building *good* UI's is really hard.  Scientist rarely have the patience or
perspective for it -- and yet it is very often difficult to build a good 
UI for
science apps without a scientist's understanding of the underlying problem.

We have a number of tools along with SciPy that we're building that are 
pieces
of this puzzle.

  1. Traits -- They at the heart of everything we do.  They provide
     some explicit typing, default GUI's, an observer event model for
     anything that uses them.  They can also require some getting used
     to.
  2. Kiva/Enable -- This is the generic drawing system for Python.
  3. Chaco -- Interactive 2D Plotting
  4. PyFace -- MVC layer on top of wxPython.  Supports trees, menus,
     and a few other things right now.  This will grow.
  5. Envisage -- Plugin based GUI framework for scientific applications.
     Beyond the basic GUI that comes from Envisage, we already have a couple
     plugins for profiling code using hotshot and also searching for memory
     leaks.  David Morrill wrote these and they are called gotcha and 
enroll.
     Martin Chilvers wrote a simple scintilla based plugin for editing 
scripts,
     and a PyCrust shell is available by default in the app (not
     a plugin at the moment).
    
     We would love to see a number of other plugins:
      a. Debugger.
      b. IPython-like PyCrust going.
      c. A more full featured script editing plugin.
         (there is a huge list of features here).
      d. Mayavi
      e. etc.

These are all used actively in commercial products that we deliver.  
However,
they are also in various stages of development and will continue to evolve
and grow.  Some are still pretty green.  Portions of these are openly 
available
now.  All five of the listed tool sets will be open at some point this 
summer
when they are cleaned up a bit.

Jobs
----
All of this leads to the fact that we are hiring and looking for smart,
pleasant, communicative people with integrity that are excited about 
building
this sort of stuff.  There is a lot of software architecture to be done that
needs talented designers.  There is UI design/development that needs 
scientists
that care about UIs or UI developers/designers that are really quick at 
grasping
scientific problems.  We also need really strong scientists/scientific
developers to do some of the backend coding for commercial products and also
SciPy development.  If you have a background in geophysics, electromagnetics
(especially multi-pole methods) that is a big plus. For this, a PhD is 
helpful, but
not required.  Parallel computing skills wouldn't hurt.  3D 
visualization a la
VTK/Python is a major need.  So is knowledge about 2D rendering to help 
finish
out Kiva and the Enable toolset. Strong Python skills, Python extension 
writing
knowledge and strong C/C++ skills are a big benefit.  Design/development 
with or
of Java NetBeans or Eclipse-like architecture is great -- or any other 
solid GUI
architecture.  Dedication to writing clean/clear code meant for other 
humans
to read is a requirement.

We have 3 or 4 scientific/python positions to fill over the coming 
months, and
we're looking for candidates that have the best mix of the above 
skills.  You'll
join our existing team of scientist/engineers, computer scientists, 
technical
writers, and an HCI specialist.  We like this mix and feel it is the 
best way to
build this sort of software. If you are interested in working at Enthought,
please send us your resume at jobs at enthought.com.  If not, please send this
posting to the smartest people you know that fit some part of the above 
description
(Python experience not explicitly required).  Salaries are competitive.  
Candidates must
be willing to relocate to Austin, Tx.
     
thanks,
eric jones

PS: There are additional positions listed at 
http://www.enthought.com/careers.htm
for those interested in business application development (not 
necessarily with Python).





From alvaro at antalia.com  Mon May 17 09:25:05 2004
From: alvaro at antalia.com (=?ISO-8859-1?Q?=C1lvaro?= Tejero Cantero)
Date: Mon May 17 09:25:05 2004
Subject: [Numpy-discussion] MaskedArray problem?
Message-ID: <1084811467.3230.5.camel@siff>

Hello,

I'm trying to rewrite my simulation using masked arrays. For that I
define masks for triangular_inferior NxN and
diagonal_triangular_superior NxN arrays.

I'm unsuccesfully trying to know what happens with the program when I
operate over a MaskedArray: it always says

Traceback (most recent call last):
  File "<stdin>", line 31, in ?
  File "<stdin>", line 26, in run
  File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 990,
in __pow__
    return power(self, other, third)
  File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 1585,
in power
    return masked_array(Numeric.power(fa, fb), m)
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 877,
in _cache_miss2
    mode, win1, win2, wout, cfunc, ufargs = \
  File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 921,
in _setup
    intypes = (in1._type.name, in2._type.name)
AttributeError: 'ObjectArray' object has no attribute '_type'


For reference, I copy a simplified version of the program. The statement
that causes the AttributeError is the last from run(N,D,S):

---------------------------

import numarray.ma as mka
import numarray.random_array as rnd
import numarray as nmr

def relative(x, msk):
    (N,D) = x.shape
    return mka.array([nmr.subtract.outer(x[:,i],x[:,i]) for i in nmr.arange(D)],
                     mask = nmr.array([msk for i in nmr.arange(D)],typecode=mka.MaskType))

def scalar(x,y):
    return mka.add.reduce(x*y,axis=0)

def run(D,N,S):
    trig_inf = nmr.array(([[(0,1)[i<j] for i in nmr.arange(N)] for j in nmr.arange(N)]), typecode=mka.MaskType)
    diag_trig_sup = 1 - trig_inf #print mka.is_mask(diag_trig_sup) gives "1"

    # positions and velocities
    r = rnd.random([N,D]) #should be overlap-checked
    v = rnd.random([N,D])

    # relative positions and velocities
    r_rel = relative(r, diag_trig_sup)
    v_rel = relative(v, diag_trig_sup)

    RV, VV, RR = scalar(r_rel,v_rel), scalar(v_rel,v_rel), scalar(r_rel,r_rel)
    discr = RV**2 - VV*(RR-S**2)      #<<<<----------PROBLEM HERE----------------



if __name__ == '__main__':
    run(D=3,N=4,S=1e-8)


-------------------------

Any other advice that you may give me is very wellcome. My deadline is
approaching and I'm really reluctant to rewrite all this in C++ but I
don't see the light.

Thank you in advance,

 ?.


-- 
?lvaro Tejero Cantero

http://alqua.org    --	documentos libres
			free documents




From jmiller at stsci.edu  Mon May 17 12:04:02 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon May 17 12:04:02 2004
Subject: [Numpy-discussion] Error in choose (bug?)
In-Reply-To: <1084467425.483.274.camel@halloween.stsci.edu>
References: <1084446790.917.9.camel@siff>
	 <1084467425.483.274.camel@halloween.stsci.edu>
Message-ID: <1084820558.12475.363.camel@localhost.localdomain>

On Thu, 2004-05-13 at 12:57, Todd Miller wrote:
> On Thu, 2004-05-13 at 07:13, ?lvaro Tejero Cantero wrote:
> > I have a matrix of particle collision times: times[i,j] gives the time
> > for particle "i" to collide with particle "j".
> > 
> > If I do, in order to get the first expected collision time for each
> > particle, the following (random array for testing purposes)	:
> > 
> > >>> N=30
> > >>> times = rnd.random([N,N])
> > >>> choose(argmin(times,transpose(times))
> > Segmentation fault (of the python interactive shell!!!)
> > 
> > With N=100 I get a more informative traceback, and rest within the
> > shell:
> > 
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1670, in choose
> >     return _choose(selector, population, outarr, clipmode)
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__
> >     result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0)
> >   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1564, in _doit
> >     blockingparameters)
> > ValueError: _operator_compute: too many inputs + outputs
> > 
> > 
> > For N=10,N=15 I get the expected output, but for N=20 I get again the
> > brutal segfault...
> > 
> > 
> > regards, 
> > ?.

I tracked this down today and understand the problem:  there's a bug in
the ufunc error checking and a ridiculous limit in choose.   I'm still
working out the "real" solution but a quick workaround is to edit
numarray.h and set MAXARGS to a larger number, say N+10.

Regards,
Todd
-- 
Todd Miller <jmiller at stsci.edu>




From jmiller at stsci.edu  Mon May 17 13:48:09 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Mon May 17 13:48:09 2004
Subject: [Numpy-discussion] MaskedArray problem?
In-Reply-To: <1084811467.3230.5.camel@siff>
References: <1084811467.3230.5.camel@siff>
Message-ID: <1084826827.12476.476.camel@localhost.localdomain>

On Mon, 2004-05-17 at 12:31, ?lvaro Tejero Cantero wrote:
> Hello,
> 
> I'm trying to rewrite my simulation using masked arrays. For that I
> define masks for triangular_inferior NxN and
> diagonal_triangular_superior NxN arrays.
> 
> I'm unsuccesfully trying to know what happens with the program when I
> operate over a MaskedArray: it always says
> 
> Traceback (most recent call last):
>   File "<stdin>", line 31, in ?
>   File "<stdin>", line 26, in run
>   File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 990,
> in __pow__
>     return power(self, other, third)
>   File "/usr/lib/python2.3/site-packages/numarray/ma/MA.py", line 1585,
> in power
>     return masked_array(Numeric.power(fa, fb), m)
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 877,
> in _cache_miss2
>     mode, win1, win2, wout, cfunc, ufargs = \
>   File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 921,
> in _setup
>     intypes = (in1._type.name, in2._type.name)
> AttributeError: 'ObjectArray' object has no attribute '_type'
> 
> 
> For reference, I copy a simplified version of the program. The statement
> that causes the AttributeError is the last from run(N,D,S):
> 
> ---------------------------
> 
> import numarray.ma as mka
> import numarray.random_array as rnd
> import numarray as nmr
> 
> def relative(x, msk):
>     (N,D) = x.shape
>     return mka.array([nmr.subtract.outer(x[:,i],x[:,i]) for i in nmr.arange(D)],
>                      mask = nmr.array([msk for i in nmr.arange(D)],typecode=mka.MaskType))
> 
> def scalar(x,y):
>     return mka.add.reduce(x*y,axis=0)
> 
> def run(D,N,S):
>     trig_inf = nmr.array(([[(0,1)[i<j] for i in nmr.arange(N)] for j in nmr.arange(N)]), typecode=mka.MaskType)
>     diag_trig_sup = 1 - trig_inf #print mka.is_mask(diag_trig_sup) gives "1"
> 
>     # positions and velocities
>     r = rnd.random([N,D]) #should be overlap-checked
>     v = rnd.random([N,D])
> 
>     # relative positions and velocities
>     r_rel = relative(r, diag_trig_sup)
>     v_rel = relative(v, diag_trig_sup)
> 
>     RV, VV, RR = scalar(r_rel,v_rel), scalar(v_rel,v_rel), scalar(r_rel,r_rel)
>     discr = RV**2 - VV*(RR-S**2)      #<<<<----------PROBLEM HERE----------------
> 
> 
> 
> if __name__ == '__main__':
>     run(D=3,N=4,S=1e-8)
> 
> 
> -------------------------
> 
> Any other advice that you may give me is very wellcome. My deadline is
> approaching and I'm really reluctant to rewrite all this in C++ but I
> don't see the light.
> 
> Thank you in advance,
> 
>  ?.

Here is a work around I put in numarray's MA.py which is now in CVS:

cvs diff: Diffing Packages/MA/Lib
Index: Packages/MA/Lib/MA.py
===================================================================
RCS file: /cvsroot/numpy/numarray/Packages/MA/Lib/MA.py,v
retrieving revision 1.7
retrieving revision 1.9
diff -c -r1.7 -r1.9
*** Packages/MA/Lib/MA.py       1 Oct 2003 15:44:35 -0000       1.7
--- Packages/MA/Lib/MA.py       17 May 2004 20:21:58 -0000      1.9
***************
*** 1263,1270 ****
--- 1263,1274 ----
                  result = value
              result.shape = d.shape
          else:
+             if not m.is_c_array():
+                 m = m.copy()
              try:
                  result = Numeric.array(d, typecode=d.typecode(),
copy=1)
+                 if not result.is_c_array():
+                     result = result.copy()
                  Numeric.putmask(result, m, value)
              except:
                  result = obj.choose(m, (d, value))

You can either get the new MA from Source Forge CVS (project numpy, down
at the moment) or use this diff by editing your own MA.py and adding the
lines shown with +'s.  The root problem is limitations in numarray's
putmask() function which are avoided by making contiguous copies of some
of the arrays in MA's filled() function.

Regards,
Todd

-- 
Todd Miller <jmiller at stsci.edu>




From news at lumbersolutions.com  Mon May 17 16:43:02 2004
From: news at lumbersolutions.com (Lumber Wholesale)
Date: Mon May 17 16:43:02 2004
Subject: [Numpy-discussion] IPE Cumaru Massaranduba Cedar
Message-ID: <hxvshy.wl4n9v@www.lumbersolutions.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040517/7a465e51/attachment-0001.html>

From news at lumbersolutions.com  Mon May 17 16:43:03 2004
From: news at lumbersolutions.com (Lumber Wholesale)
Date: Mon May 17 16:43:03 2004
Subject: [Numpy-discussion] IPE Cumaru Massaranduba Cedar
Message-ID: <hxvshy.827kx1@www.lumbersolutions.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040517/35bb4db5/attachment-0001.html>

From crasmussen at lanl.gov  Wed May 19 08:00:19 2004
From: crasmussen at lanl.gov (Craig Rasmussen)
Date: Wed May 19 08:00:19 2004
Subject: [Numpy-discussion] Linear solvers
In-Reply-To: <1084467425.483.274.camel@halloween.stsci.edu>
References: <1084446790.917.9.camel@siff> <1084467425.483.274.camel@halloween.stsci.edu>
Message-ID: <2999D4AB-A9A5-11D8-B1FD-000A957CA856@lanl.gov>

I need to solve a matrix equation of the form

      A u = q

for u, where u and q are N-vectors and A is a 
symmetric-positive-definite matrix.
For now, at least, A is a 2-D matrix.

I would like to use Python and numarray.  Are there Python modules that
I could use to obtain a solution for u?

Thanks,
Craig




From nadavh at visionsense.com  Wed May 19 09:44:09 2004
From: nadavh at visionsense.com (Nadav Horesh)
Date: Wed May 19 09:44:09 2004
Subject: [Numpy-discussion] Linear solvers
Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED10@exchange2k.envision.co.il>

The simplest solution is to use "solve_linear_equations" functin from numarray.linear_algebra. For more sophisticate solution you may try scipy
(www.scipy.org)

  Nadav

-----Original Message-----
From:	Craig Rasmussen [mailto:crasmussen at lanl.gov]
Sent:	Wed 19-May-04 17:59
To:	numarray
Cc:	
Subject:	[Numpy-discussion] Linear solvers
I need to solve a matrix equation of the form

      A u = q

for u, where u and q are N-vectors and A is a 
symmetric-positive-definite matrix.
For now, at least, A is a 2-D matrix.

I would like to use Python and numarray.  Are there Python modules that
I could use to obtain a solution for u?

Thanks,
Craig



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion






From haase at msg.ucsf.edu  Wed May 19 11:26:07 2004
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed May 19 11:26:07 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
Message-ID: <200405191125.30848.haase@msg.ucsf.edu>

Hi,
the random_array poisson functions returns negative values if mean=0:
>>>from numarray import random_array as ra
>>> ra.seed(x=1, y=1)
>>> ra.poisson(0)
5
>>> ra.poisson(0)
-2

My "math book" tells me that it should be always zero.
This seems to be a constructed case, but I'm using this to put
"quantum statistic" into a simulated image:
obj = na.array( something )
imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
The object array might have lots of zeros surrounding the "actual object". 
Thinking of a fluorescent object sending out photons it makes sense to not get 
any photons at all from 'empty' regions.
I'm using numarray 0.8;

Thanks for numarray,
Sebastian Haase



From jmiller at stsci.edu  Wed May 19 12:05:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 19 12:05:04 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
In-Reply-To: <200405191125.30848.haase@msg.ucsf.edu>
References: <200405191125.30848.haase@msg.ucsf.edu>
Message-ID: <1084993432.3736.16.camel@localhost.localdomain>

On Wed, 2004-05-19 at 14:25, Sebastian Haase wrote:
> Hi,
> the random_array poisson functions returns negative values if mean=0:
> >>>from numarray import random_array as ra
> >>> ra.seed(x=1, y=1)
> >>> ra.poisson(0)
> 5
> >>> ra.poisson(0)
> -2
> 
> My "math book" tells me that it should be always zero.
> This seems to be a constructed case, but I'm using this to put
> "quantum statistic" into a simulated image:
> obj = na.array( something )
> imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
> The object array might have lots of zeros surrounding the "actual object". 
> Thinking of a fluorescent object sending out photons it makes sense to not get 
> any photons at all from 'empty' regions.
> I'm using numarray 0.8;

I tried this on Fedora-1 i386 with Python-2.3.3 and it returned zero
consistently.  What platform are you on?

Regards,
Todd








From haase at msg.ucsf.edu  Wed May 19 12:18:01 2004
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed May 19 12:18:01 2004
Subject: [Numpy-discussion] random_array.poisson(0) gives -1 and -2
In-Reply-To: <1084993432.3736.16.camel@localhost.localdomain>
References: <200405191125.30848.haase@msg.ucsf.edu> <1084993432.3736.16.camel@localhost.localdomain>
Message-ID: <200405191217.02792.haase@msg.ucsf.edu>

On Wednesday 19 May 2004 12:03 pm, you wrote:
> On Wed, 2004-05-19 at 14:25, Sebastian Haase wrote:
> > Hi,
> >
> > the random_array poisson functions returns negative values if mean=0:
> > >>>from numarray import random_array as ra
> > >>> ra.seed(x=1, y=1)
> > >>> ra.poisson(0)
> >
> > 5
> >
> > >>> ra.poisson(0)
> >
> > -2
> >
> > My "math book" tells me that it should be always zero.
> > This seems to be a constructed case, but I'm using this to put
> > "quantum statistic" into a simulated image:
> > obj = na.array( something )
> > imageFromDetector = ra.poisson( obj ) + gaussianNoiseArray
> > The object array might have lots of zeros surrounding the "actual
> > object". Thinking of a fluorescent object sending out photons it makes
> > sense to not get any photons at all from 'empty' regions.
> > I'm using numarray 0.8;
>
> I tried this on Fedora-1 i386 with Python-2.3.3 and it returned zero
> consistently.  What platform are you on?
>
> Regards,
> Todd

I running debian (Woody)
$ uname -a
Linux baboon 2.4.18 #1 Tue Dec 16 14:11:01 PST 2003 i686 unknown
$python -v
<snip>
Python 2.2.1 (#1, Feb 28 2004, 00:52:10)
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2

and I get this:
>>> ra.poisson(0, 100000).min()
-4
>>> ra.poisson(0, 100000).min()
-4
>>> ra.poisson(0, 100000).mean()
1.9383
>>> ra.poisson(0, 100000).mean()
1.9607
>>> ra.poisson(0, 100000).max()
29
>>> ra.poisson(0, 100000).max()
28

Thanks for checking,
Sebastian



From falted at pytables.org  Thu May 20 01:31:02 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 20 01:31:02 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
Message-ID: <200405201030.39979.falted@pytables.org>

Hi,

I'm willing to use a lot the searchsorted function in numarray, but I'm a
bit surprised about the poor performance of it. Look at that:

>>> from time import time
>>> import numarray
>>> import Numeric
>>> na=numarray.arange(1000*1000)
>>> nu=Numeric.arange(1000*1000)
>>> t1=time();numarray.searchsorted(na,200*1000);time()-t1
200000
0.00055098533630371094
>>> t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
200000
7.7962875366210938e-05

It may seem that Numeric is better optimised, but the standard python module
bisect is even faster than numarray.searchsorted:

>>> import bisect
>>> t1=time();bisect.bisect_left(na,200*1000);time()-t1
200000
8.8930130004882812e-05
>>> t1=time();bisect.bisect_left(nu,200*1000);time()-t1
200000
8.6069107055664062e-05

So, bisect performance is similar to that of Numeric searchsorted and both
are almost an order of magnitude better than numarray searchsorted. This a
little bit surprising as the bisect_left module is written in plain python.
From the python 2.3.3 sources:

def bisect_left(a, x, lo=0, hi=None):
    if hi is None:
        hi = len(a)
    while lo < hi:
        mid = (lo+hi)//2
        if a[mid] < x: lo = mid+1
        else: hi = mid
    return lo

I'm using python2.3.3, numarray 0.9.1 (latest CVS) and Debian Linux (sid).

Cheers,

-- 
Francesc Alted




From rkern at ucsd.edu  Thu May 20 02:27:09 2004
From: rkern at ucsd.edu (Robert Kern)
Date: Thu May 20 02:27:09 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
In-Reply-To: <200405201030.39979.falted@pytables.org>
References: <200405201030.39979.falted@pytables.org>
Message-ID: <40AC79DD.1000905@ucsd.edu>

Francesc Alted wrote:
> Hi,
> 
> I'm willing to use a lot the searchsorted function in numarray, but I'm a
> bit surprised about the poor performance of it. Look at that:
> 
> 
>>>>from time import time
>>>>import numarray
>>>>import Numeric
>>>>na=numarray.arange(1000*1000)
>>>>nu=Numeric.arange(1000*1000)
>>>>t1=time();numarray.searchsorted(na,200*1000);time()-t1
> 
> 200000
> 0.00055098533630371094
> 
>>>>t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
> 
> 200000
> 7.7962875366210938e-05
> 
> It may seem that Numeric is better optimised, but the standard python module
> bisect is even faster than numarray.searchsorted:
> 
> 
>>>>import bisect
>>>>t1=time();bisect.bisect_left(na,200*1000);time()-t1
> 
> 200000
> 8.8930130004882812e-05
> 
>>>>t1=time();bisect.bisect_left(nu,200*1000);time()-t1
> 
> 200000
> 8.6069107055664062e-05

A better timing (IMHO), but with similar conclusions:

Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import timeit
 >>> t1 = timeit.Timer("Numeric.searchsorted(a,200000)",
"import Numeric;a=Numeric.arange(1000000)")
 >>> t2 = timeit.Timer("numarray.searchsorted(a,200000)",
"import numarray;a=numarray.arange(1000000)")
 >>> t3 = timeit.Timer("bisect.bisect_left(a,200000)",
"import Numeric;import bisect;a=Numeric.arange(1000000)")
 >>> t4 = timeit.Timer("bisect.bisect_left(a,200000)",
"import numarray;import bisect;a=numarray.arange(1000000)")
 >>> t1.repeat(3,10000)
[0.15758609771728516, 0.17469501495361328, 0.15456986427307129]
 >>> t2.repeat(3,10000)
[6.7581729888916016, 6.9644770622253418, 6.6776731014251709]
 >>> t3.repeat(3,10000)
[0.41335701942443848, 0.45698308944702148, 0.39665889739990234]
 >>> t4.repeat(3,10000)
[0.49930000305175781, 0.48063492774963379, 0.52067780494689941]

[Apologies for the linewraps.]

I also get similar results with double arrays. Weird.

Python 2.3 on Mac OS X 10.3.sumthin', latest CVS checkout of numarray, 
Numeric 23.1.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From jmiller at stsci.edu  Thu May 20 10:14:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Thu May 20 10:14:06 2004
Subject: [Numpy-discussion] Performance issues with
	numarray.searchsorted()
In-Reply-To: <40AC79DD.1000905@ucsd.edu>
References: <200405201030.39979.falted@pytables.org>
	 <40AC79DD.1000905@ucsd.edu>
Message-ID: <1085073211.7525.2043.camel@halloween.stsci.edu>

On Thu, 2004-05-20 at 05:26, Robert Kern wrote:
> Francesc Alted wrote:
> > Hi,
> > 
> > I'm willing to use a lot the searchsorted function in numarray, but I'm a
> > bit surprised about the poor performance of it. Look at that:
> > 
> > 
> >>>>from time import time
> >>>>import numarray
> >>>>import Numeric
> >>>>na=numarray.arange(1000*1000)
> >>>>nu=Numeric.arange(1000*1000)
> >>>>t1=time();numarray.searchsorted(na,200*1000);time()-t1
> > 
> > 200000
> > 0.00055098533630371094
> > 
> >>>>t1=time();Numeric.searchsorted(nu,200*1000);time()-t1
> > 
> > 200000
> > 7.7962875366210938e-05
> > 
> > It may seem that Numeric is better optimised, but the standard python module
> > bisect is even faster than numarray.searchsorted:
> > 
> > 
> >>>>import bisect
> >>>>t1=time();bisect.bisect_left(na,200*1000);time()-t1
> > 
> > 200000
> > 8.8930130004882812e-05
> > 
> >>>>t1=time();bisect.bisect_left(nu,200*1000);time()-t1
> > 
> > 200000
> > 8.6069107055664062e-05
> 
> A better timing (IMHO), but with similar conclusions:
> 
> Python 2.3 (#1, Sep 13 2003, 00:49:11)
> [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import timeit
>  >>> t1 = timeit.Timer("Numeric.searchsorted(a,200000)",
> "import Numeric;a=Numeric.arange(1000000)")
>  >>> t2 = timeit.Timer("numarray.searchsorted(a,200000)",
> "import numarray;a=numarray.arange(1000000)")
>  >>> t3 = timeit.Timer("bisect.bisect_left(a,200000)",
> "import Numeric;import bisect;a=Numeric.arange(1000000)")
>  >>> t4 = timeit.Timer("bisect.bisect_left(a,200000)",
> "import numarray;import bisect;a=numarray.arange(1000000)")
>  >>> t1.repeat(3,10000)
> [0.15758609771728516, 0.17469501495361328, 0.15456986427307129]
>  >>> t2.repeat(3,10000)
> [6.7581729888916016, 6.9644770622253418, 6.6776731014251709]
>  >>> t3.repeat(3,10000)
> [0.41335701942443848, 0.45698308944702148, 0.39665889739990234]
>  >>> t4.repeat(3,10000)
> [0.49930000305175781, 0.48063492774963379, 0.52067780494689941]
> 
> [Apologies for the linewraps.]
> 
> I also get similar results with double arrays. Weird.
> 
> Python 2.3 on Mac OS X 10.3.sumthin', latest CVS checkout of numarray, 
> Numeric 23.1.
Here's what I got with the numarray benchmarks after adding an extra
case:

benchmark                    i    numarray (usec)    Numeric (usec)  numarray:Numeric

searchsorted(a,p/5)        0.0               446                12              36.7
                           1.0               438                11              36.8
                           2.0               450                12              35.0
                           3.0               459                14              32.6
                           4.0               511                25              19.7
                           5.0               636                56              11.2
                           6.0               653                64              10.1

searchsorted(a,a)          0.0               285                 5              48.0
                           1.0               283                 7              39.6
                           2.0               291                29               9.8
                           3.0               368               308               1.2
                           4.0              1771              4120               0.4
                           5.0             17335             52127               0.3
                           6.0            201277            605787               0.3

p = 10**i
a = arange(p)

The first case agrees with your results of ~10x difference in favor of
Numeric at 10**6 elements.  The last case shows a ~3x numarray advantage
given large numbers of values.

My analysis is this:  since searchsorted runs in O(log2(N)) time,  even
with 10**6 elements there are only 20 iterations or so.  This is just
not enough to overcome numarray's Python ufunc overhead.  I'll see what
I can do,  but I think we're up against the standard numarray
performance wall for small arrays.	

Regards,
Todd






From falted at pytables.org  Thu May 20 10:42:05 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 20 10:42:05 2004
Subject: [Numpy-discussion] Performance issues with numarray.searchsorted()
In-Reply-To: <1085073211.7525.2043.camel@halloween.stsci.edu>
References: <200405201030.39979.falted@pytables.org> <40AC79DD.1000905@ucsd.edu> <1085073211.7525.2043.camel@halloween.stsci.edu>
Message-ID: <200405201941.08841.falted@pytables.org>

A Dijous 20 Maig 2004 19:13, vareu escriure:
> The first case agrees with your results of ~10x difference in favor of
> Numeric at 10**6 elements.  The last case shows a ~3x numarray advantage
> given large numbers of values.
> 
> My analysis is this:  since searchsorted runs in O(log2(N)) time,  even
> with 10**6 elements there are only 20 iterations or so.  This is just
> not enough to overcome numarray's Python ufunc overhead.  I'll see what
> I can do,  but I think we're up against the standard numarray
> performance wall for small arrays.	

I see. What about creating a special case for when the item to be searched
is an scalar (and not an array)? In such a case, it would be theoretically
possible to get at least the bisect.bisect() performance.

Besides, if this special case would be implemented, users would be able to
call repeatedly this scalar version of searchsorted() in order to deal with
very small "values" arrays (len()<5) and still having a gain.

Just a thought,

-- 
Francesc Alted




From jdhunter at ace.bsd.uchicago.edu  Fri May 21 12:24:02 2004
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Fri May 21 12:24:02 2004
Subject: [Numpy-discussion] image correlations
Message-ID: <m2k6z51u6d.fsf@mother.paradise.lost>

I have a series of luminance images that I want to do some correlation
analyses on.  Each image is an MxN frame of a movie.  I want to
compute the correlation between a given pixel i,j, and every other
pixel in the image over each frame.  That is, if I assume
xij is a numFrames length time series, and xkl is another numFrames
length time series (the pixel intensities at two points in the image
over time), I want to compute the 

corrcoeff(xij, xkl) for every kl with ij fixed.

I know I could do this by looping over the pixels of the image, but
I'm hoping for something a bit faster.

Any suggestions?

John Hunter



From jmiller at stsci.edu  Fri May 21 12:50:04 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Fri May 21 12:50:04 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <m2k6z51u6d.fsf@mother.paradise.lost>
References: <m2k6z51u6d.fsf@mother.paradise.lost>
Message-ID: <1085168950.15016.66.camel@halloween.stsci.edu>

On Fri, 2004-05-21 at 15:01, John Hunter wrote:
> I have a series of luminance images that I want to do some correlation
> analyses on.  Each image is an MxN frame of a movie.  I want to
> compute the correlation between a given pixel i,j, and every other
> pixel in the image over each frame.  That is, if I assume
> xij is a numFrames length time series, and xkl is another numFrames
> length time series (the pixel intensities at two points in the image
> over time), I want to compute the 
> 
> corrcoeff(xij, xkl) for every kl with ij fixed.
> 
> I know I could do this by looping over the pixels of the image, but
> I'm hoping for something a bit faster.
> 
> Any suggestions?

For numarray try numarray.convolve.correlate2d and set fft=1.

Todd




From sdhyok at email.unc.edu  Fri May 21 20:07:37 2004
From: sdhyok at email.unc.edu (Shin, Daehyok)
Date: Fri May 21 20:07:37 2004
Subject: [Numpy-discussion] Read only array?
Message-ID: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>

I am wondering if there is any way to set an array as read-only,
so that any trial to modify values of elements in the array raises some
warning.
Is it a cool feature to prevent unexpected side effect?

Daehyok Shin (Peter)




From jmiller at stsci.edu  Sat May 22 04:02:01 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Sat May 22 04:02:01 2004
Subject: [Numpy-discussion] Read only array?
In-Reply-To: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>
References: <OAEOKPIGCLDDHAEMCAKIGEDMCNAA.sdhyok@email.unc.edu>
Message-ID: <1085223644.3737.58.camel@localhost.localdomain>

On Fri, 2004-05-21 at 23:05, Shin, Daehyok wrote:
> I am wondering if there is any way to set an array as read-only,
> so that any trial to modify values of elements in the array raises some
> warning.
> Is it a cool feature to prevent unexpected side effect?
> 
> Daehyok Shin (Peter)
> 

In numarray you can do this:

>>> from numarray import *
>>> def make_readonly(a):
..     v = a.view()
..     s = a.tostring()
..     v._data = s
..     return v
..
>>> a = arange(100)
>>> b = make_readonly(a)
>>> b[0] = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: NA_setFromPythonScalar: assigment to readonly array buffer

This works because of the buffer protocol and the fact that a string is
a read-only buffer.  Using the buffer protocol is a numarray feature so
I'm pretty sure Numeric can't do this.  Also note that in C, this
read-only array is actually writable.

Regards,
Todd




From zingale at ucolick.org  Sun May 23 14:06:00 2004
From: zingale at ucolick.org (Mike Zingale)
Date: Sun May 23 14:06:00 2004
Subject: [Numpy-discussion] numarray equivalent to sign() function
Message-ID: <Pine.GSO.4.53.0405231401270.29915@mambo.ucolick.org>

Hi, I am in the process of converting some code from Numeric to numarray,
and it seems that numarray no longer has the sign() function -- is that
so?

ex:

sign(-30.0) = -1
sign(0) = 0
sign(1000) = 1

Is there a replacement?

Thanks,

Mike



From jdhunter at ace.bsd.uchicago.edu  Tue May 25 10:24:01 2004
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Tue May 25 10:24:01 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <1085168950.15016.66.camel@halloween.stsci.edu> (Todd Miller's
 message of "21 May 2004 15:49:10 -0400")
References: <m2k6z51u6d.fsf@mother.paradise.lost>
	<1085168950.15016.66.camel@halloween.stsci.edu>
Message-ID: <m2vfikl9vq.fsf@mother.paradise.lost>

>>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:

    >> I have a series of luminance images that I want to do some
    >> correlation analyses on.  Each image is an MxN frame of a
    >> movie.  I want to compute the correlation between a given pixel
    >> i,j, and every other pixel in the image over each frame.  That
    >> is, if I assume xij is a numFrames length time series, and xkl
    >> is another numFrames length time series (the pixel intensities
    >> at two points in the image over time), I want to compute the
    
    >> corrcoeff(xij, xkl) for every kl with ij fixed.
    
    >> I know I could do this by looping over the pixels of the image,
    >> but I'm hoping for something a bit faster.

    Todd> For numarray try numarray.convolve.correlate2d and set
    Todd> fft=1.

I've looked at this and don't know if I'm missing something or if this
isn't the right function for me.  I want to correlate a given pixel
intensity with the intensity at all other pixels over a series of
images.   Is this possible with correlate2d?

JDH





From perry at stsci.edu  Tue May 25 11:52:01 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Tue May 25 11:52:01 2004
Subject: [Numpy-discussion] image correlations
In-Reply-To: <m2vfikl9vq.fsf@mother.paradise.lost>
Message-ID: <CEELJPECNGEGKFNDLHFFEEIOCDAA.perry@stsci.edu>

John Hunter writes:
> >>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:
> 
>     >> I have a series of luminance images that I want to do some
>     >> correlation analyses on.  Each image is an MxN frame of a
>     >> movie.  I want to compute the correlation between a given pixel
>     >> i,j, and every other pixel in the image over each frame.  That
>     >> is, if I assume xij is a numFrames length time series, and xkl
>     >> is another numFrames length time series (the pixel intensities
>     >> at two points in the image over time), I want to compute the
>     
>     >> corrcoeff(xij, xkl) for every kl with ij fixed.
>     
>     >> I know I could do this by looping over the pixels of the image,
>     >> but I'm hoping for something a bit faster.
> 
>     Todd> For numarray try numarray.convolve.correlate2d and set
>     Todd> fft=1.
> 
Something that Todd and I have talked about is making general
functions (like fft, convolve, correlate) broadcastable to
other dimensions. Not much has been done to do that but I think 
a general mechanism could be developed to do just that. In that
way one could do a 1-d correlation over a 2 or 3-d array without
looping over the extra dimensions (all that would be done implicitly
in C) with the option of specifying which dimension the function
should be applied to while looping over all the other dimensions.

In the meantime, it would seemt that one possible way of dealing
with this is to simply recast your array as a 1-d array to use
the 1-d correlation. This means some memory copying and explicit
padding (how do you want the correlation to handle points beyond
the ends?). Supposing you want it to wraparound (in effect a
circular correlation) you could do something like this (untested
and probably has some mistakes :-)

if imagecube is a T x M x N array where there are T time samples.

data = concatenate((imagecube, imagecube, imagecube))
    # pad the time series on both ends with identical copies
reference = imagecube[:, i, j]
flatdata = ravel(transpose(data))
flatresult = correlate(flatdata, reference)
result = transpose(reshape(flatresult, (N,M,T)))[T:2*T,:,:]

This is admittedly a bit clumsy, but should be much, much faster
than iterating over all pixels.

It would be much nicer just to have

result = correlate(imagecube, imagecube[:,i,j], axis=0)

which the broadcasting facility would permit.



Perry



From as8ca at virginia.edu  Wed May 26 07:49:40 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 07:49:40 2004
Subject: [Numpy-discussion] numarray.where confusion
Message-ID: <20040526144841.GA8683@virginia.edu>

Hi,

I am having trouble understanding how exactly "where" works in
numarray.

What I am trying to do:

I am preparing a two-level mask in an array and then assign values to
the array where both masks are true:

>>> from numarray import *
>>> a = arange(10)
>>> # First mask
>>> m1 = where(a > 5)
>>> a[m1]
array([6, 7, 8, 9])
>>> # Second mask
>>> m2 = where(a[m1] < 8)
>>> a[m1][m2]
array([6, 7])
>>> # So far so good
>>> # Now change some values
>>> a[m1][m2] = array([10, 20])
>>> a[m1][m2]
array([6, 7])
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> # Didn't  work
>>> # Let's try a temporary variable
>>> t = a[m1]
>>> t[m2]
array([6, 7])
>>> t[m2] = array([10, 20])
>>> t[m2], t
(array([10, 20]), array([10, 20,  8,  9]))
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

So, my assignment to a[m1][m2] seems to work (no messages), but it
doesn't produce the effect I want it to.

I have read the documentation but I couldn't find something that would
explain this behavior.

So my questions:

- did I miss something important in the documentation,
- I am expecting something I shouldn't, or
- there is a bug in numarray?

Thanks,
Alok

-- 
Alok Singhal (as8ca at virginia.edu)       __
Graduate Student, dept. of Astronomy   /  _
University of Virginia                 \_O \
http://www.astro.virginia.edu/~as8ca/    __/



From perry at stsci.edu  Wed May 26 08:25:14 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 26 08:25:14 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526144841.GA8683@virginia.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>

Alok Singhal wrote:
> Hi,
> 
> I am having trouble understanding how exactly "where" works in
> numarray.
> 
> What I am trying to do:
> 
> I am preparing a two-level mask in an array and then assign values to
> the array where both masks are true:
> 
> >>> from numarray import *
> >>> a = arange(10)
> >>> # First mask
> >>> m1 = where(a > 5)
> >>> a[m1]
> array([6, 7, 8, 9])
> >>> # Second mask
> >>> m2 = where(a[m1] < 8)
> >>> a[m1][m2]
> array([6, 7])
> >>> # So far so good
> >>> # Now change some values
> >>> a[m1][m2] = array([10, 20])
> >>> a[m1][m2]
> array([6, 7])
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
> >>> # Didn't  work
> >>> # Let's try a temporary variable
> >>> t = a[m1]
> >>> t[m2]
> array([6, 7])
> >>> t[m2] = array([10, 20])
> >>> t[m2], t
> (array([10, 20]), array([10, 20,  8,  9]))
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
> 
> So, my assignment to a[m1][m2] seems to work (no messages), but it
> doesn't produce the effect I want it to.
> 
> I have read the documentation but I couldn't find something that would
> explain this behavior.
> 
> So my questions:
> 
> - did I miss something important in the documentation,
> - I am expecting something I shouldn't, or
> - there is a bug in numarray?
> 
(due to confusions with "a" in text I'll use x in place of "a")
I believe the problem you are seeing (I'm not 100% certain yet)
is that although it is possible to assign to an array-indexed
array, that doing that twice over doesn't work since Python is,
in effect, treating x[m1] as an expression even though it is
on the left side. That expression results in a new array that the
second indexing updates, but then is thrown away since it is not
assigned to anything else.

Your second try creates a temporary t which is also not a view into
a so when you update t, a is not updated.

try 

x[m1[0][m2]] = array([10,20])

instead. The intent here is to provide x with the net index array
by indexing m1 first rather than indexing x first.
(note the odd use of m1[0]; this is necessary since where() will
return a tuple of index arrays (to allow use in multidimensional 
cases as indices, so the m1[0] extracts the array from the tuple;
Since m1 is a tuple, indexing it with another index array (well, 
tuple containing an index array) doesn't work).

Perry Greenfield



From jmiller at stsci.edu  Wed May 26 08:42:16 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 26 08:42:16 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526144841.GA8683@virginia.edu>
References: <20040526144841.GA8683@virginia.edu>
Message-ID: <1085586113.31969.123.camel@halloween.stsci.edu>

On Wed, 2004-05-26 at 10:48, Alok Singhal wrote:
> Hi,
> 
> I am having trouble understanding how exactly "where" works in
> numarray.
> 
> What I am trying to do:
> 
> I am preparing a two-level mask in an array and then assign values to
> the array where both masks are true:
> 
> >>> from numarray import *
> >>> a = arange(10)
> >>> # First mask
> >>> m1 = where(a > 5)
> >>> a[m1]
> array([6, 7, 8, 9])
> >>> # Second mask
> >>> m2 = where(a[m1] < 8)
> >>> a[m1][m2]

a[m1] is a new array here.

> array([6, 7])
> >>> # So far so good
> >>> # Now change some values
> >>> a[m1][m2] = array([10, 20])

And here too.  This does a write into what is effectively a temporary
variable returned by the expression a[m1].  Although the write occurs,
it is lost.

> >>> a[m1][m2]
> array([6, 7])
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Here's how I did it (there was an easier way I overlooked):

a = arange(10)
m1 = where(a > 5, 1, 0).astype('Bool')
m2 = where(a < 8, 1, 0).astype('Bool')
a[m1 & m2] = array([10, 20])

The principle here is to keep the masks as "full sized" boolean arrays
rather than index arrays so they can be combined using the bitwise and
operator.   The resulting mask can be used to index just once
eliminating the temporary.

Regards,
Todd




From falted at pytables.org  Wed May 26 09:07:10 2004
From: falted at pytables.org (Francesc Alted)
Date: Wed May 26 09:07:10 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <1085586113.31969.123.camel@halloween.stsci.edu>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu>
Message-ID: <200405261806.37029.falted@pytables.org>

A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
> Here's how I did it (there was an easier way I overlooked):
> 
> a = arange(10)
> m1 = where(a > 5, 1, 0).astype('Bool')
> m2 = where(a < 8, 1, 0).astype('Bool')
> a[m1 & m2] = array([10, 20])

Perhaps the easier way looks like this?

>>> a = arange(10)
>>> a[(a>5) & (a<8)] = array([10, 20])
>>> a
array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])

Indexing is a very powerful (and fun) thing, indeed :)

-- 
Francesc Alted




From jmiller at stsci.edu  Wed May 26 09:29:06 2004
From: jmiller at stsci.edu (Todd Miller)
Date: Wed May 26 09:29:06 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405261806.37029.falted@pytables.org>
References: <20040526144841.GA8683@virginia.edu>
	 <1085586113.31969.123.camel@halloween.stsci.edu>
	 <200405261806.37029.falted@pytables.org>
Message-ID: <1085588893.31969.125.camel@halloween.stsci.edu>

On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
> A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
> > Here's how I did it (there was an easier way I overlooked):
> > 
> > a = arange(10)
> > m1 = where(a > 5, 1, 0).astype('Bool')
> > m2 = where(a < 8, 1, 0).astype('Bool')
> > a[m1 & m2] = array([10, 20])
> 
> Perhaps the easier way looks like this?
> 
> >>> a = arange(10)
> >>> a[(a>5) & (a<8)] = array([10, 20])

Much, much better.  Thanks!

Todd

> >>> a
> array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])
> 
> Indexing is a very powerful (and fun) thing, indeed :)
-- 
Todd Miller <jmiller at stsci.edu>




From as8ca at virginia.edu  Wed May 26 10:19:09 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 10:19:09 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405261806.37029.falted@pytables.org> <1085586113.31969.123.camel@halloween.stsci.edu> <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <200405261806.37029.falted@pytables.org> <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <20040526144841.GA8683@virginia.edu> <CEELJPECNGEGKFNDLHFFKEJBCDAA.perry@stsci.edu>
Message-ID: <20040526171845.GA18671@virginia.edu>

On 26/05/04: 11:24, Perry Greenfield wrote:
> (due to confusions with "a" in text I'll use x in place of "a")
> I believe the problem you are seeing (I'm not 100% certain yet)
> is that although it is possible to assign to an array-indexed
> array, that doing that twice over doesn't work since Python is,
> in effect, treating x[m1] as an expression even though it is
> on the left side. That expression results in a new array that the
> second indexing updates, but then is thrown away since it is not
> assigned to anything else.
> 
> Your second try creates a temporary t which is also not a view into
> a so when you update t, a is not updated.

Thanks or this info.  It makes sense now.  I suspected earlier that t
was not a view but a copy, but didn't realise that the same thing was
happening with x[m1][m2].

> try 
> 
> x[m1[0][m2]] = array([10,20])
> 
> instead. The intent here is to provide x with the net index array
> by indexing m1 first rather than indexing x first.
> (note the odd use of m1[0]; this is necessary since where() will
> return a tuple of index arrays (to allow use in multidimensional 
> cases as indices, so the m1[0] extracts the array from the tuple;
> Since m1 is a tuple, indexing it with another index array (well, 
> tuple containing an index array) doesn't work).

This works, but for the fact that in my real code I *am* dealing with
multidimensional arrays.  But this is a nice trick to remember.

(So, the following "does not work":

    x = arange(9)
    x.shape=(3,3)
    m1 = where(x > 4)
    m2 = where(x[m1] < 7)
    x[m1[0][m2]]
)

On 26/05/04: 11:41, Todd Miller wrote:
> Here's how I did it (there was an easier way I overlooked):
> 
> a = arange(10)
> m1 = where(a > 5, 1, 0).astype('Bool')
> m2 = where(a < 8, 1, 0).astype('Bool')
> a[m1 & m2] = array([10, 20])

Ah.  This works!  Even for multidimensional arrays.

On 26/05/04: 18:06, Francesc Alted wrote:
> Perhaps the easier way looks like this?
> 
> >>> a = arange(10)
> >>> a[(a>5) & (a<8)] = array([10, 20])
> >>> a
> array([ 0,  1,  2,  3,  4,  5, 10, 20,  8,  9])
> 
> Indexing is a very powerful (and fun) thing, indeed :)

I like this too.  Thank you all for the help!

Alok

-- 
Alok Singhal (as8ca at virginia.edu)       __
Graduate Student, dept. of Astronomy   /  _
University of Virginia                 \_O \
http://www.astro.virginia.edu/~as8ca/    __/



From strawman at astraw.com  Wed May 26 10:44:05 2004
From: strawman at astraw.com (Andrew Straw)
Date: Wed May 26 10:44:05 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <1085588893.31969.125.camel@halloween.stsci.edu>
References: <20040526144841.GA8683@virginia.edu>	 <1085586113.31969.123.camel@halloween.stsci.edu>	 <200405261806.37029.falted@pytables.org> <1085588893.31969.125.camel@halloween.stsci.edu>
Message-ID: <40B474C3.4070903@astraw.com>

Todd Miller wrote:

>On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
>  
>
>>A Dimecres 26 Maig 2004 17:41, Todd Miller va escriure:
>>    
>>
>>>Here's how I did it (there was an easier way I overlooked):
>>>
>>>a = arange(10)
>>>m1 = where(a > 5, 1, 0).astype('Bool')
>>>m2 = where(a < 8, 1, 0).astype('Bool')
>>>a[m1 & m2] = array([10, 20])
>>>      
>>>
>>Perhaps the easier way looks like this?
>>
>>    
>>
>>>>>a = arange(10)
>>>>>a[(a>5) & (a<8)] = array([10, 20])
>>>>>          
>>>>>
Is there an equivalently slick way to accomplish to what I'm trying 
below?  (the the values in c[:,1] get changed based on the same-row 
values in c[:,0]?)

from numarray import *
a=arange(10)
b=arange(10)+20
c=concatenate((a[:,NewAxis],b[:,NewAxis]),axis=1)
c[c[:,0]>7][:,1] = 0 # doesn't work because it makes a copy and 
therefore doesn't modify c

Cheers!
Andrew



From as8ca at virginia.edu  Wed May 26 11:04:13 2004
From: as8ca at virginia.edu (Alok Singhal)
Date: Wed May 26 11:04:13 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <40B474C3.4070903@astraw.com>
References: <20040526144841.GA8683@virginia.edu> <1085586113.31969.123.camel@halloween.stsci.edu> <200405261806.37029.falted@pytables.org> <1085588893.31969.125.camel@halloween.stsci.edu> <40B474C3.4070903@astraw.com>
Message-ID: <20040526180339.GB20143@virginia.edu>

On 26/05/04: 10:43, Andrew Straw wrote:
> Todd Miller wrote:
> >On Wed, 2004-05-26 at 12:06, Francesc Alted wrote:
> >
> >>>>>a = arange(10)
> >>>>>a[(a>5) & (a<8)] = array([10, 20])
> >>>>>
> Is there an equivalently slick way to accomplish to what I'm trying 
> below?  (the the values in c[:,1] get changed based on the same-row 
> values in c[:,0]?)
> 
> from numarray import *
> a=arange(10)
> b=arange(10)+20
> c=concatenate((a[:,NewAxis],b[:,NewAxis]),axis=1)
> c[c[:,0]>7][:,1] = 0 # doesn't work because it makes a copy and 
> therefore doesn't modify c

Well, for your case, the following works:

>>> print c
[[ 0 20]
 [ 1 21]
 [ 2 22]
 [ 3 23]
 [ 4 24]
 [ 5 25]
 [ 6 26]
 [ 7 27]
 [ 8 28]
 [ 9 29]]
>>> t0 = c[:, 0]
>>> t1 = c[:, 1]
>>> t1[t0 > 7] = 0
>>> print c
[[ 0 20]
 [ 1 21]
 [ 2 22]
 [ 3 23]
 [ 4 24]
 [ 5 25]
 [ 6 26]
 [ 7 27]
 [ 8  0]
 [ 9  0]]

Not sure this helps in your real code though.

Alok

-- 
Alok Singhal (as8ca at virginia.edu)          *   *          
Graduate Student, dept. of Astronomy   *           *     *
University of Virginia                                    
http://www.astro.virginia.edu/~as8ca/              *    * 



From perry at stsci.edu  Wed May 26 12:03:06 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Wed May 26 12:03:06 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <20040526171845.GA18671@virginia.edu>
Message-ID: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>

> > try 
> > 
> > x[m1[0][m2]] = array([10,20])
> > 
> > instead. The intent here is to provide x with the net index array
> > by indexing m1 first rather than indexing x first.
> > (note the odd use of m1[0]; this is necessary since where() will
> > return a tuple of index arrays (to allow use in multidimensional 
> > cases as indices, so the m1[0] extracts the array from the tuple;
> > Since m1 is a tuple, indexing it with another index array (well, 
> > tuple containing an index array) doesn't work).
> 
> This works, but for the fact that in my real code I *am* dealing with
> multidimensional arrays.  But this is a nice trick to remember.
> 
> (So, the following "does not work":
> 
>     x = arange(9)
>     x.shape=(3,3)
>     m1 = where(x > 4)
>     m2 = where(x[m1] < 7)
>     x[m1[0][m2]]
> )
 
correct. You'd have to break apart the m1 tuple and
index all the components, e.g.,

m11, m12 = m1
x[m11[m2],m12[m2]] = ...

This gets clumsier with the more dimensions that must
be handled, but you still can do it. It would be most
useful if the indexed array is very large, the number
of items selected is relatively small and one
doesn't want to incur the memory overhead of all the
mask arrays of the admittedly much nicer notational
approach that Francesc illustrated.

Perry
 



From falted at pytables.org  Thu May 27 00:47:04 2004
From: falted at pytables.org (Francesc Alted)
Date: Thu May 27 00:47:04 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>
References: <CEELJPECNGEGKFNDLHFFCEJECDAA.perry@stsci.edu>
Message-ID: <200405270946.06228.falted@pytables.org>

A Dimecres 26 Maig 2004 21:01, Perry Greenfield va escriure:
> correct. You'd have to break apart the m1 tuple and
> index all the components, e.g.,
> 
> m11, m12 = m1
> x[m11[m2],m12[m2]] = ...
> 
> This gets clumsier with the more dimensions that must
> be handled, but you still can do it. It would be most
> useful if the indexed array is very large, the number
> of items selected is relatively small and one
> doesn't want to incur the memory overhead of all the
> mask arrays of the admittedly much nicer notational
> approach that Francesc illustrated.

Well, boolean arrays have the property that they use very little memory
(only 1 byte / element), and normally perform quite well doing indexing.
Some timings:

>>> import timeit
>>> t1 = timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2],m12[m2]]","from numarray import arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t2 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t1.repeat(3,1000)
[3.1320240497589111, 3.1235389709472656, 3.1198310852050781]
>>> t2.repeat(3,1000)
[1.1218469142913818, 1.117638111114502, 1.1156759262084961]

i.e. using boolean arrays for indexing is roughly 3 times faster.

For larger arrays this difference is even more noticeable:

>>> t3 = timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2],m12[m2]]","from numarray import arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t4 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
>>> t3.repeat(3,10)
[3.1818649768829346, 3.20477294921875, 3.190640926361084]
>>> t4.repeat(3,10)
[0.42328095436096191, 0.42140507698059082, 0.41979002952575684]

as you see, now the difference is almost an order of magnitude (!).

So, perhaps assuming the small memory overhead, in most of cases it is
better to use boolean selections. However, it would be nice to know the
ultimate reason of why this happens, because the Perry approach seems
intuitively faster.

-- 
Francesc Alted




From perry at stsci.edu  Thu May 27 10:48:00 2004
From: perry at stsci.edu (Perry Greenfield)
Date: Thu May 27 10:48:00 2004
Subject: [Numpy-discussion] numarray.where confusion
In-Reply-To: <200405270946.06228.falted@pytables.org>
Message-ID: <CEELJPECNGEGKFNDLHFFKEJHCDAA.perry@stsci.edu>

Francesc Alted va escriure:

> A Dimecres 26 Maig 2004 21:01, Perry Greenfield va escriure:
> > correct. You'd have to break apart the m1 tuple and
> > index all the components, e.g.,
> > 
> > m11, m12 = m1
> > x[m11[m2],m12[m2]] = ...
> > 
> > This gets clumsier with the more dimensions that must
> > be handled, but you still can do it. It would be most
> > useful if the indexed array is very large, the number
> > of items selected is relatively small and one
> > doesn't want to incur the memory overhead of all the
> > mask arrays of the admittedly much nicer notational
> > approach that Francesc illustrated.
> 
> Well, boolean arrays have the property that they use very little memory
> (only 1 byte / element), and normally perform quite well doing indexing.
> Some timings:
> 
> >>> import timeit
> >>> t1 = 
> timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2]
> ,m12[m2]]","from numarray import 
> arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t2 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import 
> arange,where;dim=3;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t1.repeat(3,1000)
> [3.1320240497589111, 3.1235389709472656, 3.1198310852050781]
> >>> t2.repeat(3,1000)
> [1.1218469142913818, 1.117638111114502, 1.1156759262084961]
> 
> i.e. using boolean arrays for indexing is roughly 3 times faster.
> 
> For larger arrays this difference is even more noticeable:
> 
> >>> t3 = 
> timeit.Timer("m1=where(x>4);m2=where(x[m1]<7);m11,m12=m1;x[m11[m2]
> ,m12[m2]]","from numarray import 
> arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t4 = timeit.Timer("x[(x>4) & (x<7)]","from numarray import 
> arange,where;dim=1000;x=arange(dim*dim);x.shape=(dim,dim)")
> >>> t3.repeat(3,10)
> [3.1818649768829346, 3.20477294921875, 3.190640926361084]
> >>> t4.repeat(3,10)
> [0.42328095436096191, 0.42140507698059082, 0.41979002952575684]
> 
> as you see, now the difference is almost an order of magnitude (!).
> 
> So, perhaps assuming the small memory overhead, in most of cases it is
> better to use boolean selections. However, it would be nice to know the
> ultimate reason of why this happens, because the Perry approach seems
> intuitively faster.
>
Yes I agree. It was good of you to post these timings. I don't 
think we had actually compared the two approaches though the
results don't surprise me (though I suspect the results may change
if the first mask has a very small percentage of elements; the
large timing test has nearly all elements selected for the first
mask).

Perry



From kck93jom at inland.net  Mon May 31 23:18:01 2004
From: kck93jom at inland.net (Cheryle Rosalyn)
Date: Mon May 31 23:18:01 2004
Subject: [Numpy-discussion] Little Pen-i-sMake U Shy To Screw?? Hehe where
Message-ID: <gfo4pmqf8p4hpcmxxq@creaturewill>

mountain season laughed perfect done justice now 
tie maybe excite my loose experience listen conscience across likely slipped step ache thought 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20040531/4d320c2e/attachment-0001.html>