From gael.varoquaux at normalesup.org  Wed Oct  1 01:26:26 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 1 Oct 2008 07:26:26 +0200
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360809301510r485b8ebfxd25303693bc85643@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<20080930213423.GB22173@phare.normalesup.org>
	<ce557a360809301510r485b8ebfxd25303693bc85643@mail.gmail.com>
Message-ID: <20081001052626.GA22435@phare.normalesup.org>

On Tue, Sep 30, 2008 at 06:10:46PM -0400, Anne Archibald wrote:
> > k=None in the third call to T.query seems redundant. It should be
> > possible do put some logics so that the call is simply

> > distances, indices = T.query(xs, distance_upper_bound=1.0)

> Well, the problem with this is that you often want to provide a
> distance upper bound as well as a number of nearest neighbors.

Absolutely. I just think k should default to None, when
distance_upper_bound is specified. k=None could be interpreted as k=1
when distance_uppper_bound is not specified.

My 2 cents,

Ga?l


From barrywark at gmail.com  Wed Oct  1 02:08:24 2008
From: barrywark at gmail.com (Barry Wark)
Date: Tue, 30 Sep 2008 23:08:24 -0700
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
Message-ID: <cd7634ce0809302308y2ce3f02etbfb9e63d4baae33@mail.gmail.com>

On Mon, Sep 29, 2008 at 8:24 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> Hi,
>
> Once again there has been a thread on the numpy/scipy mailing lists
> requesting (essentially) some form of spatial data structure. Pointers
> have been posted to ANN (sadly LGPLed and in C++) as well as a handful
> of pure-python implementations of kd-trees. I suggest the creation of
> a new submodule of scipy, scipy.spatial, to contain spatial data
> structures and algorithms. Specifically, I propose it contain a
> kd-tree implementation providing nearest-neighbor, approximate
> nearest-neighbor, and all-points-near queries. I have a few other
> suggestions for things it might contain, but kd-trees seem like a good
> first step.
>
> 2008/9/27 Nathan Bell <wnbell at gmail.com>:
>> On Sat, Sep 27, 2008 at 11:18 PM, Anne Archibald
>> <peridot.faceted at gmail.com> wrote:
>>>
>>> I think a kd-tree implementation would be a valuable addition to
>>> scipy, perhaps in a submodule scipy.spatial that might eventually
>>> contain other spatial data structures and algorithms. What do you
>>> think? Should we have one? Should it be based on Sturla Molden's code,
>>> if the license permits? I am willing to contribute one, if not.
>>
>> +1
>
> Judging that your vote and mine are enough in the absence of
> dissenting voices, I have written an implementation based on yours,
> Sturla Molden's, and the algorithms described by the authors of the
> ANN library. Before integrating it into scipy, though, I'd like to
> send it around for comments.
>
> Particular issues:
>
> * It's pure python right now, but with some effort it could be
> partially or completely cythonized. This is probably a good idea in
> the long run. In the meantime I have crudely vectorized it so that
> users can at least avoid looping in their own code.
> * It is able to return the r nearest neighbors, optionally subject to
> a maximum distance limit, as well as approximate nearest neighbors.
> * It is not currently set up to conveniently return all points within
> some fixed distance of the target point, but this could easily be
> added.
> * It returns distances and indices of nearest neighbors in the original array.
> * The testing code is, frankly, a mess. I need to look into using nose
> in a sensible fashion.
> * The license is the scipy license.
>
> I am particularly concerned about providing a convenient return
> format. The natural return from a query is a list of neighbors, since
> it may have variable length (there may not be r elements in the tree,
> or you might have supplied a maximum distance which doesn't contain r
> points). For a single query, it's simple to return a python list
> (should it be sorted? currently it's a heap). But if you want to
> vectorize the process, storing the results in an array becomes
> cumbersome. One option is an object array full of lists; another,
> which, I currently use, is an array with nonexistent points marked
> with an infinite distance and an invalid index. A third would be to
> return masked arrays. How do you recommend handling this variable (or
> potentially-variable) sized output?
>
>> If you're implementing one, I would highly recommend the
>> "left-balanced" kd-tree.
>> http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/2535/pdf/imm2535.pdf
>
> Research suggests that at least in high dimension a more geometric
> balancing criterion can produce vastly better results. I used the
> "sliding midpoint" rule, which doesn't allow such a numerical
> balancing but does ensure that you don't have too many long skinny
> cells (since a sphere tends to cut very many of these).
>
> I've also been thinking about what else would go in scipy.spatial. I
> think it would be valuable to have a reasonably efficient distance
> matrix function (we seem to get that question a lot, and the answer's
> not trivial) as well as a sparse distance matrix function based on the
> kd-trees. The module could potentially also swallow the (currently
> sandboxed?) delaunay code.
>
> Anne

Anne,

Thanks for taking this on. The scikits.ann has licensing issues (as
noted above), so it would be nice to have a clean-room implementation
in scipy. I am happy to port the scikits.ann API to the final API that
you choose, however, if you think that would be helpful.

Cheers,
Barry


From wnbell at gmail.com  Wed Oct  1 02:10:13 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Wed, 1 Oct 2008 02:10:13 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <20081001052626.GA22435@phare.normalesup.org>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<20080930213423.GB22173@phare.normalesup.org>
	<ce557a360809301510r485b8ebfxd25303693bc85643@mail.gmail.com>
	<20081001052626.GA22435@phare.normalesup.org>
Message-ID: <d05265cb0809302310w3161d3j8985e23e08d84d0b@mail.gmail.com>

On Wed, Oct 1, 2008 at 1:26 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
>
> Absolutely. I just think k should default to None, when
> distance_upper_bound is specified. k=None could be interpreted as k=1
> when distance_uppper_bound is not specified.
>

Why not expose the various possibilities through different names?

# nearest k points (possibly fewer)
query_nearest(pt, k=1)

# all points within given distance
query_sphere(pt, distance)

#nearest k points within given distance (possibly fewer)
query(pt, k, distance)

Few people will use the last form, but it's useful nevertheless.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From schut at sarvision.nl  Wed Oct  1 04:47:09 2008
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 01 Oct 2008 10:47:09 +0200
Subject: [Numpy-discussion] xml-rpc with numpy arrays
In-Reply-To: <e7ba66e40809302016q26b4bc63i502ed50e66f11d4@mail.gmail.com>
References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu>	<200809301318.36701.binet@cern.ch>	<e7ba66e40809301524h10b26d41vd93efcf83c4229b8@mail.gmail.com>	<200809301542.44608.binet@cern.ch>	<1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu>
	<e7ba66e40809302016q26b4bc63i502ed50e66f11d4@mail.gmail.com>
Message-ID: <gbvdee$e5u$1@ger.gmane.org>

Lisandro Dalcin wrote:
> 
> I believe xmlrpclib is currently the simpler approach. Some day I'll
> have the time to implement something similar using MPI communication
> with mpi4py. However, I believe it can be done even better: local,
> client-side proxies should automatically provide access to all
> members/methods of remote, server-side instances. The registering step
> needed within xmlrpclib is a bit ugly ;-)
> 
> 
Try pyro: <http://pyro.sourceforge.net/> or rpyc: 
<http://rpyc.wikispaces.com/>, both of which if I recall correctly, do 
implement this.

VS.



From peridot.faceted at gmail.com  Wed Oct  1 05:41:54 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 1 Oct 2008 05:41:54 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <20081001052626.GA22435@phare.normalesup.org>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<20080930213423.GB22173@phare.normalesup.org>
	<ce557a360809301510r485b8ebfxd25303693bc85643@mail.gmail.com>
	<20081001052626.GA22435@phare.normalesup.org>
Message-ID: <ce557a360810010241s1f7de2eeqf515301e48ff897b@mail.gmail.com>

2008/10/1 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Tue, Sep 30, 2008 at 06:10:46PM -0400, Anne Archibald wrote:
>> > k=None in the third call to T.query seems redundant. It should be
>> > possible do put some logics so that the call is simply
>
>> > distances, indices = T.query(xs, distance_upper_bound=1.0)
>
>> Well, the problem with this is that you often want to provide a
>> distance upper bound as well as a number of nearest neighbors.
>
> Absolutely. I just think k should default to None, when
> distance_upper_bound is specified. k=None could be interpreted as k=1
> when distance_uppper_bound is not specified.

That seems very confusing. Better perhaps to have a function
query_all_neighbors, even if it's just a wrapper.

Anne


From bblais at bryant.edu  Wed Oct  1 05:50:47 2008
From: bblais at bryant.edu (Brian Blais)
Date: Wed, 1 Oct 2008 05:50:47 -0400
Subject: [Numpy-discussion] xml-rpc with numpy arrays
In-Reply-To: <e7ba66e40809302016q26b4bc63i502ed50e66f11d4@mail.gmail.com>
References: <98DE1D44-A0B6-4CC6-8FD0-11F187D20862@bryant.edu>
	<200809301318.36701.binet@cern.ch>
	<e7ba66e40809301524h10b26d41vd93efcf83c4229b8@mail.gmail.com>
	<200809301542.44608.binet@cern.ch>
	<1861310C-ED7D-4554-83BF-5751549041E7@bryant.edu>
	<e7ba66e40809302016q26b4bc63i502ed50e66f11d4@mail.gmail.com>
Message-ID: <7793B4DF-E7AA-4D33-8643-A667EC137C1C@bryant.edu>

On Sep 30, 2008, at 23:16 , Lisandro Dalcin wrote:

> On Tue, Sep 30, 2008 at 9:27 PM, Brian Blais <bblais at bryant.edu>
>> thanks for all of the help.  My initial solution is to pickle my  
>> object,
>> with the text-based version of pickle, and send it across rpc.  I  
>> do this
>> because the actual thing I am sending is a dictionary, with lots  
>> of arrays,
>> and other things.  I'll have a look at the format that Robert  
>> sent, because
>> that looks useful for other things I am doing.
>
> Did you try to send binary pickles (protocol=2)? Perhaps it works,
> give a try! Of course, you need the client and server machines having
> the same arch.

I tried that first, and marshal doesn't handle binary streams, so  
that's when I tried the text version.  I'm not sending a lot of info,  
so it's not that big of a deal.  Most of the time is spent on the  
simulation on the other end.


			bb

-- 
Brian Blais
bblais at bryant.edu
http://web.bryant.edu/~bblais



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/a73e7195/attachment.html>

From peridot.faceted at gmail.com  Wed Oct  1 05:57:51 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 1 Oct 2008 05:57:51 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <cd7634ce0809302308y2ce3f02etbfb9e63d4baae33@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<cd7634ce0809302308y2ce3f02etbfb9e63d4baae33@mail.gmail.com>
Message-ID: <ce557a360810010257qc46c4bah77637ff1b9eaceba@mail.gmail.com>

2008/10/1 Barry Wark <barrywark at gmail.com>:

> Thanks for taking this on. The scikits.ann has licensing issues (as
> noted above), so it would be nice to have a clean-room implementation
> in scipy. I am happy to port the scikits.ann API to the final API that
> you choose, however, if you think that would be helpful.

That's a nice idea. I'm not totally sure yet how much it's going to be
possible for different implementations to be plug-in replacements, but
it sure would be nice if users could use ANN transparently.

Anne


From philbinj at gmail.com  Wed Oct  1 08:40:20 2008
From: philbinj at gmail.com (James Philbin)
Date: Wed, 1 Oct 2008 13:40:20 +0100
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
Message-ID: <2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>

> distances, indices = T.query(xs) # single nearest neighbor
I'm not sure if it's implied, but can xs be a NxD matrix here i.e.
query for all N points rather than just one. This will reduce the
python call overhead for large queries.

Also, I have some c++ code for locality sensitive hashing which might be useful?

James


From dmitrey.kroshko at scipy.org  Wed Oct  1 09:04:27 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Wed, 01 Oct 2008 16:04:27 +0300
Subject: [Numpy-discussion] why type(array(1).tolist()) is int?
Message-ID: <48E3755B.2050805@scipy.org>

hi all,
why array(1).tolist() returns 1? I expected to get [1] instead.
D.



From dmitrey.kroshko at scipy.org  Wed Oct  1 09:34:30 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Wed, 01 Oct 2008 16:34:30 +0300
Subject: [Numpy-discussion] why type(array(1).tolist()) is int?
In-Reply-To: <48E3755B.2050805@scipy.org>
References: <48E3755B.2050805@scipy.org>
Message-ID: <48E37C66.3020805@scipy.org>

let me also note that list(array((1))) returns

Traceback (innermost last):
 File "<stdin>", line 1, in <module>
TypeError: iteration over a 0-d array

D.

dmitrey wrote:
> hi all,
> why array(1).tolist() returns 1? I expected to get [1] instead.
> D.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>   



From dmitrey.kroshko at scipy.org  Wed Oct  1 09:38:37 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Wed, 01 Oct 2008 16:38:37 +0300
Subject: [Numpy-discussion] will array(Python set) be ever implemented as
	cast method?
Message-ID: <48E37D5D.2060903@scipy.org>

hi all,
will array(Python set) (and asarray, asfarray etc) ever be implemented 
as cast method?

Now it just puts the set into 1st element:

 >>> asarray(set([11, 12, 13, 14]))
array(set([11, 12, 13, 14]), dtype=object)

 >>> array(set([11, 12, 13, 14]))
array(set([11, 12, 13, 14]), dtype=object)

Currently I use array(list(my_set)) instead.

D.


From aisaac at american.edu  Wed Oct  1 09:46:47 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 01 Oct 2008 09:46:47 -0400
Subject: [Numpy-discussion] why type(array(1).tolist()) is int?
In-Reply-To: <48E3755B.2050805@scipy.org>
References: <48E3755B.2050805@scipy.org>
Message-ID: <48E37F47.2080208@american.edu>

On 10/1/2008 9:04 AM dmitrey apparently wrote:
> why array(1).tolist() returns 1? I expected to get [1] instead.

I guess I would expect it not to work at all.
Given that it does work, this seems the best result.
What list shape matches the shape of a 0-d array?

What is the use case that makes this seem wrong?

Alan Isaac



From oliphant at enthought.com  Wed Oct  1 09:58:32 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 01 Oct 2008 08:58:32 -0500
Subject: [Numpy-discussion] why type(array(1).tolist()) is int?
In-Reply-To: <48E37C66.3020805@scipy.org>
References: <48E3755B.2050805@scipy.org> <48E37C66.3020805@scipy.org>
Message-ID: <48E38208.2090104@enthought.com>

dmitrey wrote:
> let me also note that list(array((1))) returns
>
> Traceback (innermost last):
>  File "<stdin>", line 1, in <module>
> TypeError: iteration over a 0-d array
>
> D.
>   
This is expected.   0-d arrays are currently not iterable.

-Travis



From oliphant at enthought.com  Wed Oct  1 09:59:41 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 01 Oct 2008 08:59:41 -0500
Subject: [Numpy-discussion] will array(Python set) be ever implemented
 as	cast method?
In-Reply-To: <48E37D5D.2060903@scipy.org>
References: <48E37D5D.2060903@scipy.org>
Message-ID: <48E3824D.5020503@enthought.com>

dmitrey wrote:
> hi all,
> will array(Python set) (and asarray, asfarray etc) ever be implemented 
> as cast method?
>   
Use fromiter instead.    We could special case set objects in array(...) 
if that is deemed desirable.  

-Travis




From dmitrey.kroshko at scipy.org  Wed Oct  1 09:57:26 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Wed, 01 Oct 2008 16:57:26 +0300
Subject: [Numpy-discussion] why type(array(1).tolist()) is int?
In-Reply-To: <48E37F47.2080208@american.edu>
References: <48E3755B.2050805@scipy.org> <48E37F47.2080208@american.edu>
Message-ID: <48E381C6.8010800@scipy.org>

Alan G Isaac wrote:
> On 10/1/2008 9:04 AM dmitrey apparently wrote:
>   
>> why array(1).tolist() returns 1? I expected to get [1] instead.
>>     
>
> I guess I would expect it not to work at all.
> Given that it does work, this seems the best result.
> What list shape matches the shape of a 0-d array?
>
> What is the use case that makes this seem wrong?
>   
Because I just expect something.tolist() return *Type* list, not *Type* 
integer.  tolist documentation says "Return the array as a list or 
nested lists" and nothing about possibility to return anything else.

As for my situation I store the list in my data field and then call
for item from prob.my_list: do_something()

D.
> Alan Isaac
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>   



From travis at enthought.com  Wed Oct  1 10:36:54 2008
From: travis at enthought.com (Travis Vaught)
Date: Wed, 1 Oct 2008 09:36:54 -0500
Subject: [Numpy-discussion] Texas Python Regional Unconference Reminders
Message-ID: <0107962E-D762-497B-BCEC-24CBD78B381B@enthought.com>

Greetings,

The Texas Python Regional Unconference is coming up this weekend  
(October 4-5) and I wanted to send out some more details of the  
meeting.  The web page for the meeting is here:

http://www.scipy.org/TXUncon2008

The meeting is _absolutely free_, so please add yourself to the  
Attendees page if you're able to make it.  Also, if you're planning to  
attend, please send me the following information (to travis at enthought.com 
) so I can request wireless access for you during the meeting:

  - Full Name
  - Phone or email
  - Address
  - Affiliation

There are still opportunities to present your pet projects at the  
meeting, so feel free to sign up on the presentation schedule here:

http://www.scipy.org/TXUncon2008Schedule

For those who are in town Friday evening, we're planning to get  
together for a casual dinner in downtown Austin that night.  We'll  
meet at Enthought offices (http://www.enthought.com/contact/map-directions.php 
) and walk to a casual restaurant nearby.  Show up as early as 5:30pm  
and you can hang out and tour the Enthought offices--we'll head out to  
eat at 7:00pm sharp.

Best,

Travis




From lists_ravi at lavabit.com  Wed Oct  1 15:21:29 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 1 Oct 2008 15:21:29 -0400
Subject: [Numpy-discussion] PyArray_Resize reference counting
In-Reply-To: <48E2AE1D.9080802@enthought.com>
References: <200809301548.10782.lists_ravi@lavabit.com>
	<200809301720.53958.lists_ravi@lavabit.com>
	<48E2AE1D.9080802@enthought.com>
Message-ID: <200810011521.29476.lists_ravi@lavabit.com>

On Tuesday 30 September 2008 18:54:21 Travis E. Oliphant wrote:
> I just went to the code and noticed that PyArray_Resize returns None. ?
> So, you certainly don't want to point array to it. ?The array does not
> get any reference count changes.

Thanks for the very clear explanation.

> PyObject *dummy;
> dummy = PyArray_Resize(array, ...)
> if (dummy == NULL) goto fail;
> Py_DECREF(dummy)
>
> is what you need to do.

If dummy is NULL, is the original array guaranteed to be unchanged (strong 
exception safety with rollback semantics)? If not, is the original array 
guaranteed to be at least in an internally consistent state (weak exception 
safety)?

In general, do functions modifying numpy arrays provide at least weak 
exception safety guarantee? Or do they go one step further and provide 
rollback semantics in case of exceptions?

Regards,
Ravi




From dblubaugh at belcan.com  Wed Oct  1 16:04:13 2008
From: dblubaugh at belcan.com (Blubaugh, David A.)
Date: Wed, 1 Oct 2008 16:04:13 -0400
Subject: [Numpy-discussion] f2py IS NOW WORKING
Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F380544D48C@AWMAIL04.belcan.com>

To all,


I have now been able to develop a stable file via f2py!!  However, I had
to execute the following:

1.) First, I had to copy all required library files from my selected
Compaq visual Fortran compiler under python's scripts directory along
with f2py itself.  


2.)  I also had to include a dll from my compiler under python's dll
directory as well.  


I know that the reason as to why I needed to execute these actions, is
that I do not know as to what should be my correct environmental
variables within windows XP running Compaq Visual Fortran 6.6.  


Once again, I would appreciate to know as to what are the correct
environmental variables should I set my windows xp under, given that the
compiler I must utilize is a Compaq Visual Fortran Compiler 6.6??? 


Thanks,

David Blubaugh

This e-mail transmission contains information that is confidential and may be 
privileged. It is intended only for the addressee(s) named above. If you receive 
this e-mail in error, please do not read, copy or disseminate it in any manner. 
If you are not the intended recipient, any disclosure, copying, distribution or 
use of the contents of this information is prohibited. Please reply to the 
message immediately by informing the sender that the message was misdirected. 
After replying, please erase it from your computer system. Your assistance in 
correcting this error is appreciated.



From oc-spam66 at laposte.net  Wed Oct  1 16:46:10 2008
From: oc-spam66 at laposte.net (oc-spam66)
Date: Wed,  1 Oct 2008 22:46:10 +0200 (CEST)
Subject: [Numpy-discussion] Vectorization of the product of
	several	matrices ?
Message-ID: <17478156.887841222893970667.JavaMail.www@wwinf8216>

Hello and thank you for your answer.

> There are at least three methods I can think of, but choosing the best one
> requires more information. How long are the lists? Do the arrays have
> variable dimensions? The simplest and most adaptable method is probably

The lists would be made of 4x4 matrices :
LM = [M_i, i=1..N], M_i 4x4 matrix
LN = [N_i, i=1..N], N_i 4x4 matrix.

N would be 1000 or more (why not 100000... if the computation is not too long) 


> In [3]: P = [m*n for m,n in zip(M,N)]

Thank you for this one. I am curious about other possibilities.

And also : is there a document about how the python interpreter works ? (order of the operations, typical timings, ...)

Regards,

O.C.


 Cr?ez votre adresse ?lectronique prenom.nom at laposte.net 
 1 Go d'espace de stockage, anti-spam et anti-virus int?gr?s.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/a5ddbbb2/attachment.html>

From mailinglist.honeypot at gmail.com  Wed Oct  1 20:56:18 2008
From: mailinglist.honeypot at gmail.com (Steve Lianoglou)
Date: Wed, 1 Oct 2008 20:56:18 -0400
Subject: [Numpy-discussion] Texas Python Regional Unconference Reminders
In-Reply-To: <0107962E-D762-497B-BCEC-24CBD78B381B@enthought.com>
References: <0107962E-D762-497B-BCEC-24CBD78B381B@enthought.com>
Message-ID: <E6A23298-A582-48B0-81DA-002C8866C163@gmail.com>

Hi,

Are there any plans to tape the presentations? Unfortunately some of  
us can't make it down to Texas, but the talks look quite interesting.

Thanks,
-steve

On Oct 1, 2008, at 10:36 AM, Travis Vaught wrote:

> Greetings,
>
> The Texas Python Regional Unconference is coming up this weekend
> (October 4-5) and I wanted to send out some more details of the
> meeting.  The web page for the meeting is here:
>
> http://www.scipy.org/TXUncon2008
>
> The meeting is _absolutely free_, so please add yourself to the
> Attendees page if you're able to make it.  Also, if you're planning to
> attend, please send me the following information (to travis at enthought.com
> ) so I can request wireless access for you during the meeting:
>
>  - Full Name
>  - Phone or email
>  - Address
>  - Affiliation
>
> There are still opportunities to present your pet projects at the
> meeting, so feel free to sign up on the presentation schedule here:
>
> http://www.scipy.org/TXUncon2008Schedule
>
> For those who are in town Friday evening, we're planning to get
> together for a casual dinner in downtown Austin that night.  We'll
> meet at Enthought offices (http://www.enthought.com/contact/map-directions.php
> ) and walk to a casual restaurant nearby.  Show up as early as 5:30pm
> and you can hang out and tour the Enthought offices--we'll head out to
> eat at 7:00pm sharp.
>
> Best,
>
> Travis
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion






From charlesr.harris at gmail.com  Wed Oct  1 21:16:02 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 1 Oct 2008 19:16:02 -0600
Subject: [Numpy-discussion] Vectorization of the product of several
	matrices ?
In-Reply-To: <17478156.887841222893970667.JavaMail.www@wwinf8216>
References: <17478156.887841222893970667.JavaMail.www@wwinf8216>
Message-ID: <e06186140810011816p1ed2a6c5qe86ed65ff81009a6@mail.gmail.com>

On Wed, Oct 1, 2008 at 2:46 PM, oc-spam66 <oc-spam66 at laposte.net> wrote:

> Hello and thank you for your answer.
>
> > There are at least three methods I can think of, but choosing the best
> one
> > requires more information. How long are the lists? Do the arrays have
> > variable dimensions? The simplest and most adaptable method is probably
>
> The lists would be made of 4x4 matrices :
> LM = [M_i, i=1..N], M_i 4x4 matrix
> LN = [N_i, i=1..N], N_i 4x4 matrix.
>
> N would be 1000 or more (why not 100000... if the computation is not too
> long)
>
>
> > In [3]: P = [m*n for m,n in zip(M,N)]
>
> Thank you for this one. I am curious about other possibilities.
>

For stacks of small matrices of same dimensions stored in three dimensional
arrays (not matrices), you can do

In [1]: M = array([eye(4)]*2)

In [2]: N = array([eye(4)]*2)

In [3]: P = (M[...,:,:,newaxis]*N[...,newaxis,::]).sum(axis=-2)

In [4]: P
Out[4]:
array([[[ 1.,  0.,  0.,  0.],
        [ 0.,  1.,  0.,  0.],
        [ 0.,  0.,  1.,  0.],
        [ 0.,  0.,  0.,  1.]],

       [[ 1.,  0.,  0.,  0.],
        [ 0.,  1.,  0.,  0.],
        [ 0.,  0.,  1.,  0.],
        [ 0.,  0.,  0.,  1.]]])

This will use more memory than the first alternative. Whether or not it
might be faster and what the tradeoffs are will depend on the problem.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/123e73d8/attachment.html>

From f.yw at hotmail.com  Wed Oct  1 21:27:27 2008
From: f.yw at hotmail.com (frank wang)
Date: Wed, 1 Oct 2008 19:27:27 -0600
Subject: [Numpy-discussion] Help to process a large data file
In-Reply-To: <ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
References: <gauint$tc2$1@ger.gmane.org>
	<9911419a0809232336n89d6e4djcc59ad71f28732f5@mail.gmail.com>
	<9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com>
	<BLU105-W64E4D0675617417B0A9B989450@phx.gbl>
	<9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com>
	<BLU105-W46A55D24F82955D1366B1F89400@phx.gbl>
	<48E14D27.5040604@enthought.com>
	<BLU105-W57AE036687AE3C43C4D30E89430@phx.gbl>
	<3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com>
	<BLU105-W2339FEAFBA459B6703E4B989430@phx.gbl> 
	<ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
Message-ID: <BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>


Hi, 
 
I have a large data file which contains 2 columns of data. The two columns only have zero and one. Now I want to cound how many one in between if both columns are one. For example, if my data is:
 
1 0
0 0
1 1
0 0     
0 1    x
0 1    x
0 0
0 1    x
1 1
0 0
0 1    x
0 1    x
1 1
 
Then my count will be 3 and 2 (the numbers with x).
 
Are there an efficient way to do this? My data file is pretty big.
 
Thanks
 
Frank
_________________________________________________________________
See how Windows connects the people, information, and fun that are part of your life.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/f0a919dc/attachment.html>

From charlesr.harris at gmail.com  Wed Oct  1 21:42:36 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 1 Oct 2008 19:42:36 -0600
Subject: [Numpy-discussion] Vectorization of the product of several
	matrices ?
In-Reply-To: <e06186140810011816p1ed2a6c5qe86ed65ff81009a6@mail.gmail.com>
References: <17478156.887841222893970667.JavaMail.www@wwinf8216>
	<e06186140810011816p1ed2a6c5qe86ed65ff81009a6@mail.gmail.com>
Message-ID: <e06186140810011842o46f54a5csbae0bf1d2e289eab@mail.gmail.com>

On Wed, Oct 1, 2008 at 7:16 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Wed, Oct 1, 2008 at 2:46 PM, oc-spam66 <oc-spam66 at laposte.net> wrote:
>
>> Hello and thank you for your answer.
>>
>> > There are at least three methods I can think of, but choosing the best
>> one
>> > requires more information. How long are the lists? Do the arrays have
>> > variable dimensions? The simplest and most adaptable method is probably
>>
>> The lists would be made of 4x4 matrices :
>> LM = [M_i, i=1..N], M_i 4x4 matrix
>> LN = [N_i, i=1..N], N_i 4x4 matrix.
>>
>> N would be 1000 or more (why not 100000... if the computation is not too
>> long)
>>
>>
>> > In [3]: P = [m*n for m,n in zip(M,N)]
>>
>> Thank you for this one. I am curious about other possibilities.
>>
>
> For stacks of small matrices of same dimensions stored in three dimensional
> arrays (not matrices), you can do
>
> In [1]: M = array([eye(4)]*2)
>
> In [2]: N = array([eye(4)]*2)
>
> In [3]: P = (M[...,:,:,newaxis]*N[...,newaxis,::]).sum(axis=-2)
>

Should be

In [5]: P = (M[...,:,:,newaxis]*N[...,newaxis,:,:]).sum(axis=-2)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/f642fd55/attachment.html>

From charlesr.harris at gmail.com  Wed Oct  1 22:41:31 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 1 Oct 2008 20:41:31 -0600
Subject: [Numpy-discussion] Portable functions for nans, signbit, etc.
Message-ID: <e06186140810011941s67bbe76bwb1de4e27f8c3427a@mail.gmail.com>

The normal signbit function of gcc without the -std=c99 flag doesn't work
correctly for nans and infs. I found the following code on a boost mailing
list and it might be helpful here for portability.

const boost::uint32_t signbit_mask
     = binary_cast<boost::uint32_t>(1.0f)
     ^ binary_cast<boost::uint32_t>(-1.0f);

inline bool signbit(float x)
{
     return binary_cast<boost::uint32_t>(x) & signbit_mask;
}

inline bool signbit(double x)
{
     return signbit(static_cast<float>(x));
}
inline bool signbit(long double x)
{
     return signbit(static_cast<float>(x));
}

Which is rather clever. I think binary_cast will require some pointer abuse.
There are a bunch of other boost functions here
<http://tinyurl.com/3tvj9u>that might prove useful. This file

 <goog_1222871357985>floating_point_utilities_v3.zip
<http://www.boostpro.com/vault/index.php?action=downloadfile&filename=floating_point_utilities_v3.zip&directory=Math%20-%20Numerics&PHPSESSID=64b789bb8092caa29b11839ec7526611>
portable isnan, fpclassify, signbit etc. + facets for portable handling of
infinity and NaN in text streams

Looks particularly interesting. It's a bit large for the mailing list so I
won't attach it. The Boost license should be compatible with numpy.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/068efd7f/attachment.html>

From johngu at gmail.com  Thu Oct  2 00:15:38 2008
From: johngu at gmail.com (John Gu)
Date: Wed, 1 Oct 2008 23:15:38 -0500
Subject: [Numpy-discussion] complex numpy.ndarray dtypes
Message-ID: <79258dea0810012115o1a77aebbg429d471d42e06d3c@mail.gmail.com>

Hello,

I am using numpy in conjunction with pyTables.  The data that I read in from
pyTables seem to have the following dtype:

p = hdf5.root.myTable.read()

p.__class__
<type 'numpy.ndarray'>

p[0].__class__
<type 'numpy.void'>

p.dtype
dtype([('time', '<f4'), ('obs1', '<f4'), ('obs2', '<f8'), ('obs3', '<f4')])

p.shape
(61230,)

The manner in which I access a particular column is p['time'] or p['obs1'].
I have a couple of questions regarding this data structure: 1) how do I
restructure the array into a 61230 x 4 array that can be indexed using [r,c]
notation?  2) What kind of dtype is pyTables using?  How do I create a
similar array that can be indexed by a named column?  I tried various ways:

a = array([[1,2],[3,4]], dtype=dtype([('obs1','<f4'),('obs2','<f4')]))
---------------------------------------------------------------------------
<type 'exceptions.TypeError'>             Traceback (most recent call last)

p:\AsiaDesk\johngu\projects\deltaForce\<ipython console> in <module>()

<type 'exceptions.TypeError'>: expected a readable buffer object

I did find some documentation about array type descriptors when reading from
files... it seems like these array types are specific to arrays created when
reading from some sort of file / buffer?  Any help is appreciated.  Thanks!

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081001/59a9b7dc/attachment.html>

From charlesr.harris at gmail.com  Thu Oct  2 02:03:43 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 2 Oct 2008 00:03:43 -0600
Subject: [Numpy-discussion] nan, sign, and all that
Message-ID: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>

Hi All,

I've added ufuncs fmin and fmax that behave as follows:

In [3]: a = array([NAN, 0, NAN, 1])

In [4]: b = array([0, NAN, NAN, 0])

In [5]: fmax(a,b)
Out[5]: array([  0.,   0.,  NaN,   1.])

In [6]: fmin(a,b)
Out[6]: array([  0.,   0.,  NaN,   0.])

In [7]: fmax.reduce(a)
Out[7]: 1.0

In [8]: fmin.reduce(a)
Out[8]: 0.0

In [9]: fmax.reduce([NAN,NAN])
Out[9]: nan

In [10]: fmin.reduce([NAN,NAN])
Out[10]: nan

I also made the sign ufunc return the sign of nan. That works, but I'm not
sure it is the way to go because there doesn't seem to be any spec as to
what sign nan takes. The current np.nan on my machine is negative and 0/0,
inf/inf all return negative nan. So it doesn't look like the actual sign of
nan makes any sense. Currently sign(NAN) returns 0, which doesn't look right
either, so I think the thing to do is return nan but this will be a change
in numpy behavior. Any thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081002/e5f55bbf/attachment.html>

From stefan at sun.ac.za  Thu Oct  2 03:37:12 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 2 Oct 2008 09:37:12 +0200
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
Message-ID: <9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>

Hi Charles,

2008/10/2 Charles R Harris <charlesr.harris at gmail.com>:
> In [3]: a = array([NAN, 0, NAN, 1])
> In [4]: b = array([0, NAN, NAN, 0])
>
> In [5]: fmax(a,b)
> Out[5]: array([  0.,   0.,  NaN,   1.])
>
> In [6]: fmin(a,b)
> Out[6]: array([  0.,   0.,  NaN,   0.])

These are great, many thanks!

My only gripe is that they have the same NaN-handling as amin and
friends, which I consider to be broken.  Others also mentioned that
this should be changed, and I think David C wrote a patch for it (but
I am not informed as to the speed implications).

If I had to choose, this would be my preferred output:

In [5]: fmax(a,b)
Out[5]: array([  NaN,   NaN,  NaN,   1.])

Cheers
St?fan


From robert.kern at gmail.com  Thu Oct  2 03:42:58 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 2 Oct 2008 02:42:58 -0500
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
Message-ID: <3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>

On Thu, Oct 2, 2008 at 02:37, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Charles,
>
> 2008/10/2 Charles R Harris <charlesr.harris at gmail.com>:
>> In [3]: a = array([NAN, 0, NAN, 1])
>> In [4]: b = array([0, NAN, NAN, 0])
>>
>> In [5]: fmax(a,b)
>> Out[5]: array([  0.,   0.,  NaN,   1.])
>>
>> In [6]: fmin(a,b)
>> Out[6]: array([  0.,   0.,  NaN,   0.])
>
> These are great, many thanks!
>
> My only gripe is that they have the same NaN-handling as amin and
> friends, which I consider to be broken.

No, these follow well-defined C99 semantics of the fmin() and fmax()
functions in libm. If exactly one of the arguments is a NaN, the
non-NaN argument is returned. This is *not* the current behavior of
amin() et al., which just do naive comparisons.

> Others also mentioned that
> this should be changed, and I think David C wrote a patch for it (but
> I am not informed as to the speed implications).
>
> If I had to choose, this would be my preferred output:
>
> In [5]: fmax(a,b)
> Out[5]: array([  NaN,   NaN,  NaN,   1.])

Chuck proposes letting minimum() and maximum() have that behavior.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From nwagner at iam.uni-stuttgart.de  Thu Oct  2 03:48:24 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 02 Oct 2008 09:48:24 +0200
Subject: [Numpy-discussion] loadtxt
Message-ID: <web-112187411@uni-stuttgart.de>

Hi all,

how can I load ASCII data if the file contains characters 
instead of floats

Traceback (most recent call last):
   File "test_csv.py", line 2, in <module>
     A = loadtxt('ca6_sets.csv',dtype=char ,delimiter=';')
NameError: name 'char' is not defined
  
Nils


From stefan at sun.ac.za  Thu Oct  2 04:10:49 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 2 Oct 2008 10:10:49 +0200
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
	<3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
Message-ID: <9457e7c80810020110i29986675j3d1ccb1f91c6cd98@mail.gmail.com>

2008/10/2 Robert Kern <robert.kern at gmail.com>:
>> My only gripe is that they have the same NaN-handling as amin and
>> friends, which I consider to be broken.
>
> No, these follow well-defined C99 semantics of the fmin() and fmax()
> functions in libm. If exactly one of the arguments is a NaN, the
> non-NaN argument is returned. This is *not* the current behavior of
> amin() et al., which just do naive comparisons.

Let me rephrase: I'm not convinced that these C99 semantics provide an
optimal user experience.  It worries me greatly that NaN's pop up in
operations and then disappear again.  It is entirely possible for a
script to run without failure and spew out garbage without the user
ever knowing.

>> Others also mentioned that
>> this should be changed, and I think David C wrote a patch for it (but
>> I am not informed as to the speed implications).
>>
>> If I had to choose, this would be my preferred output:
>>
>> In [5]: fmax(a,b)
>> Out[5]: array([  NaN,   NaN,  NaN,   1.])
>
> Chuck proposes letting minimum() and maximum() have that behavior.

That would be a good start, which would be complemented by educating
the user via some appropriate mechanism (I still don't know if one
exists; there is no NumPy Paperclip TM that states "You have decided
to commit scientific suicide.  Would you like me to cut your
wrists?").  That's meant only half-tongue-in-cheekedly :)

Thanks for your comments,

Cheers
St?fan


From cournape at gmail.com  Thu Oct  2 04:23:59 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 2 Oct 2008 17:23:59 +0900
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
Message-ID: <5b8d13220810020123h16fb733dp7aea5085402a9344@mail.gmail.com>

On Thu, Oct 2, 2008 at 4:37 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
>
> These are great, many thanks!
>
> My only gripe is that they have the same NaN-handling as amin and
> friends, which I consider to be broken.  Others also mentioned that
> this should be changed, and I think David C wrote a patch for it (but
> I am not informed as to the speed implications).

Hopefully, Chuck and me synchronised a bit on this :) The idea is that
before, I thought that there was a nan ignoring and nan propagating
behavior. Robert later mentioned that fmin/fmax has a third, well
specified behavior in C99. All those three are useful, and as such
have been more or less implemented by Chuck or me.

I think having the new C functions by Chuck makes sense as a new
python API, to follow C99 fmax/fmin. They could be used for the new
max/min, but then, it feels it a bit strange compared to
nanmax/nanmin, so I would prefer having the *current* numpy.max and
numpy.min propagate the NaN, and nanmax/nanmin ignoring the NaN
altogether.

Also note that matlab does not propagate NaN for max/min.

The last question is FPU status flag handling: I thought comparing NaN
directly with < would throw a FPE_INVALID. But this is not the case
(at least on Linux with glibc and Mac OS X). This is confusing because
I thought the whole point of C99 macro isgreater was to not throw
this. This is also how I understand both glibc manual and mac os x man
isgreater. Robert, do you have any insight on this ?

David

From faltet at pytables.org  Thu Oct  2 04:28:23 2008
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 2 Oct 2008 10:28:23 +0200
Subject: [Numpy-discussion] complex numpy.ndarray dtypes
In-Reply-To: <79258dea0810012115o1a77aebbg429d471d42e06d3c@mail.gmail.com>
References: <79258dea0810012115o1a77aebbg429d471d42e06d3c@mail.gmail.com>
Message-ID: <200810021028.24155.faltet@pytables.org>

A Thursday 02 October 2008, John Gu escrigu?:
> Hello,
>
> I am using numpy in conjunction with pyTables.  The data that I read
> in from pyTables seem to have the following dtype:
>
> p = hdf5.root.myTable.read()
>
> p.__class__
> <type 'numpy.ndarray'>
>
> p[0].__class__
> <type 'numpy.void'>
>
> p.dtype
> dtype([('time', '<f4'), ('obs1', '<f4'), ('obs2', '<f8'), ('obs3',
> '<f4')])
>
> p.shape
> (61230,)
>
> The manner in which I access a particular column is p['time'] or
> p['obs1']. I have a couple of questions regarding this data
> structure: 1) how do I restructure the array into a 61230 x 4 array
> that can be indexed using [r,c] notation?

In your example, the table (record array in NumPy jargon) is 
inhomogeneous (all fields are 'f4' except 'obs2' which is 'f8').  In 
that case, you can obtain an homogeneous array by doing something like:

In [44]: a = numpy.array([(1,2),(3,4)], dtype=[('obs1','<f4'),
('obs2','<f8')])

In [45]: b = numpy.array([(val['obs1'], val['obs2']) for val in a], 
dtype='<f4')

In [46]: b
Out[46]:
array([[ 1.,  2.],
       [ 3.,  4.]], dtype=float32)

In case your table would be homegenous, there is a simpler way:

In [41]: a = numpy.array([(1,2),(3,4)], dtype=[('obs1','<f4'),
('obs2','<f4')])

In [42]: d = a.view(('<f4',2))

In [43]: d
Out[43]:
array([[ 1.,  2.],
       [ 3.,  4.]], dtype=float32)

which is faster:

In [68]: timeit d = a.view(('<f4',2))
100000 loops, best of 3: 11.5 ?s per loop

In [69]: timeit b=numpy.array([(val['obs1'], val['obs2']) for val in a], 
dtype='<f4')
10000 loops, best of 3: 39.8 ?s per loop

> 2) What kind of dtype is 
> pyTables using?  How do I create a similar array that can be indexed
> by a named column?  I tried various ways:
>
> a = array([[1,2],[3,4]],
> dtype=dtype([('obs1','<f4'),('obs2','<f4')]))
> ---------------------------------------------------------------------
>------ <type 'exceptions.TypeError'>             Traceback (most
> recent call last)
>
> p:\AsiaDesk\johngu\projects\deltaForce\<ipython console> in
> <module>()
>
> <type 'exceptions.TypeError'>: expected a readable buffer object

Yeah, the error message is too terse in this case.  Record array 
constructor needs to be sure where your records start and end, and this 
is achieved by mapping tuples to records.  So, your example must be 
rewritten as:

In [70]: a = numpy.array([(1,2),(3,4)], dtype=[('obs1','<f4'),
('obs2','<f4')])

In [71]: a
Out[71]:
array([(1.0, 2.0), (3.0, 4.0)],
      dtype=[('obs1', '<f4'), ('obs2', '<f4')])

Have a look at:

http://www.scipy.org/RecordArrays

for more info on record arrays.

> I did find some documentation about array type descriptors when
> reading from files... it seems like these array types are specific to
> arrays created when reading from some sort of file / buffer?  Any
> help is appreciated.  Thanks!

I'm not sure on what you are asking here.  At any rate, it might be 
useful to have a look at complex dtype examples in:

http://www.scipy.org/Numpy_Example_List#head-f9175c69cccd74b9e4ee92e2a060af27c7447b76

Hope that helps,

-- 
Francesc Alted


From faltet at pytables.org  Thu Oct  2 04:31:08 2008
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 2 Oct 2008 10:31:08 +0200
Subject: [Numpy-discussion] loadtxt
In-Reply-To: <web-112187411@uni-stuttgart.de>
References: <web-112187411@uni-stuttgart.de>
Message-ID: <200810021031.08853.faltet@pytables.org>

A Thursday 02 October 2008, Nils Wagner escrigu?:
> Hi all,
>
> how can I load ASCII data if the file contains characters
> instead of floats
>
> Traceback (most recent call last):
>    File "test_csv.py", line 2, in <module>
>      A = loadtxt('ca6_sets.csv',dtype=char ,delimiter=';')
> NameError: name 'char' is not defined

You would need to specify the length of your strings.  Try with 
dtype="SN", where N is the expected length of the strings.

Cheers,

-- 
Francesc Alted


From cournape at gmail.com  Thu Oct  2 04:41:43 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 2 Oct 2008 17:41:43 +0900
Subject: [Numpy-discussion] Portable functions for nans, signbit, etc.
In-Reply-To: <e06186140810011941s67bbe76bwb1de4e27f8c3427a@mail.gmail.com>
References: <e06186140810011941s67bbe76bwb1de4e27f8c3427a@mail.gmail.com>
Message-ID: <5b8d13220810020141w74d207cen9809a0363232346b@mail.gmail.com>

On Thu, Oct 2, 2008 at 11:41 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:

>
> Which is rather clever. I think binary_cast will require some pointer abuse.

Yep (the funny thing is that the bit twiddling will likely end up more
readable than this C++ stuff)

cheers,

David


From stefan at sun.ac.za  Thu Oct  2 05:15:46 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 2 Oct 2008 11:15:46 +0200
Subject: [Numpy-discussion] loadtxt
In-Reply-To: <200810021031.08853.faltet@pytables.org>
References: <web-112187411@uni-stuttgart.de>
	<200810021031.08853.faltet@pytables.org>
Message-ID: <9457e7c80810020215h3aed755doa8a301e6cbb38761@mail.gmail.com>

2008/10/2 Francesc Alted <faltet at pytables.org>:
>> how can I load ASCII data if the file contains characters
>> instead of floats
>
> You would need to specify the length of your strings.  Try with
> dtype="SN", where N is the expected length of the strings.

Other options include:

- using converters to convert the character to a value:

  np.loadtxt('/tmp/bleh.dat', converters={2: lambda x: 0})

- Skipping the specified column:

  np.loadtxt('/tmp/bleh.dat', usecols=(0,1))

Cheers
St?fan


From pete.forman at westerngeco.com  Thu Oct  2 05:50:59 2008
From: pete.forman at westerngeco.com (Pete Forman)
Date: Thu, 02 Oct 2008 10:50:59 +0100
Subject: [Numpy-discussion] nan, sign, and all that
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
	<3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
	<9457e7c80810020110i29986675j3d1ccb1f91c6cd98@mail.gmail.com>
Message-ID: <ljx7pbp8.fsf@wgmail2.gatwick.eur.slb.com>

"St?fan van der Walt" <stefan at sun.ac.za> writes:

 > Let me rephrase: I'm not convinced that these C99 semantics provide
 > an optimal user experience.  It worries me greatly that NaN's pop
 > up in operations and then disappear again.  It is entirely possible
 > for a script to run without failure and spew out garbage without
 > the user ever knowing.

By default NaNs are propagated through operations on them.  At the end
of this discussion we ought to end up with a list of functions such as
fmax, isnan, and copysign that are the exceptions.

I think that it is right to defer to IEEE for their decisions on the
behavior of NaNs, etc.  That is what C and Fortran are doing.  I have
not checked but I would guess that CPUs and FPUs behave that way too.
So it should be easier and faster to follow IEEE.

Note that in the just released Python 2.6 floating point support of
IEEE 754 has been beefed up.
-- 
Pete Forman                -./\.-  Disclaimer: This post is originated
WesternGeco                  -./\.-   by myself and does not represent
pete.forman at westerngeco.com    -./\.-   the opinion of Schlumberger or
http://petef.22web.net           -./\.-   WesternGeco.



From charlesr.harris at gmail.com  Thu Oct  2 09:22:28 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 2 Oct 2008 07:22:28 -0600
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
	<3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
Message-ID: <e06186140810020622u78c165edm96102a19445fd5ee@mail.gmail.com>

On Thu, Oct 2, 2008 at 1:42 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Thu, Oct 2, 2008 at 02:37, St?fan van der Walt <stefan at sun.ac.za>
> wrote:
> > Hi Charles,
> >
> > 2008/10/2 Charles R Harris <charlesr.harris at gmail.com>:
> >> In [3]: a = array([NAN, 0, NAN, 1])
> >> In [4]: b = array([0, NAN, NAN, 0])
> >>
> >> In [5]: fmax(a,b)
> >> Out[5]: array([  0.,   0.,  NaN,   1.])
> >>
> >> In [6]: fmin(a,b)
> >> Out[6]: array([  0.,   0.,  NaN,   0.])
> >
> > These are great, many thanks!
> >
> > My only gripe is that they have the same NaN-handling as amin and
> > friends, which I consider to be broken.
>
> No, these follow well-defined C99 semantics of the fmin() and fmax()
> functions in libm. If exactly one of the arguments is a NaN, the
> non-NaN argument is returned. This is *not* the current behavior of
> amin() et al., which just do naive comparisons.
>
> > Others also mentioned that
> > this should be changed, and I think David C wrote a patch for it (but
> > I am not informed as to the speed implications).
> >
> > If I had to choose, this would be my preferred output:
> >
> > In [5]: fmax(a,b)
> > Out[5]: array([  NaN,   NaN,  NaN,   1.])
>
> Chuck proposes letting minimum() and maximum() have that behavior.
>

Yes. If there is any agreement on this I would like to go ahead and do it.
It does change the current behavior of maximum and minimum.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081002/bb8316ab/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Oct  2 09:11:15 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 02 Oct 2008 22:11:15 +0900
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <e06186140810020622u78c165edm96102a19445fd5ee@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>	<3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
	<e06186140810020622u78c165edm96102a19445fd5ee@mail.gmail.com>
Message-ID: <48E4C873.6080200@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
> Yes. If there is any agreement on this I would like to go ahead and do
> it. It does change the current behavior of maximum and minimum.

If you do it, please do it with as many tests as possible (it should not
be difficult to have a comprehensive test with *all* float data types),
because this is likely to cause problems on some platforms.

thanks,

David


From charlesr.harris at gmail.com  Thu Oct  2 09:31:29 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 2 Oct 2008 07:31:29 -0600
Subject: [Numpy-discussion] Portable functions for nans, signbit, etc.
In-Reply-To: <5b8d13220810020141w74d207cen9809a0363232346b@mail.gmail.com>
References: <e06186140810011941s67bbe76bwb1de4e27f8c3427a@mail.gmail.com>
	<5b8d13220810020141w74d207cen9809a0363232346b@mail.gmail.com>
Message-ID: <e06186140810020631y1926e087p2858d478f23dde8e@mail.gmail.com>

On Thu, Oct 2, 2008 at 2:41 AM, David Cournapeau <cournape at gmail.com> wrote:

> On Thu, Oct 2, 2008 at 11:41 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>
> >
> > Which is rather clever. I think binary_cast will require some pointer
> abuse.
>
> Yep (the funny thing is that the bit twiddling will likely end up more
> readable than this C++ stuff)
>

The zip file has the bit twiddling, which is worth looking at if only for
the note on the PPC extended precision. Motorola seems to  be a problem but
I don't think we support any of the 66xx series.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081002/3c8f6dd4/attachment.html>

From david.huard at gmail.com  Thu Oct  2 09:46:19 2008
From: david.huard at gmail.com (David Huard)
Date: Thu, 2 Oct 2008 09:46:19 -0400
Subject: [Numpy-discussion] Help to process a large data file
In-Reply-To: <BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>
References: <gauint$tc2$1@ger.gmane.org>
	<BLU105-W64E4D0675617417B0A9B989450@phx.gbl>
	<9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com>
	<BLU105-W46A55D24F82955D1366B1F89400@phx.gbl>
	<48E14D27.5040604@enthought.com>
	<BLU105-W57AE036687AE3C43C4D30E89430@phx.gbl>
	<3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com>
	<BLU105-W2339FEAFBA459B6703E4B989430@phx.gbl>
	<ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
	<BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>
Message-ID: <91cf711d0810020646h125c7589v21ae89fb93ab9eed@mail.gmail.com>

Frank,

How about that:

x = np.loadtxt('file')

z = x.sum(1)   # Reduce data to an array of 0,1,2

rz = z[z>0]   # Remove all 0s since you don't want to count those.

loc = np.where(rz==2)[0]  # The location of the (1,1)s

count = np.diff(loc) - 1  # The spacing between those (1,1)s, ie, the number
of elements that have one 1.


HTH,

David


On Wed, Oct 1, 2008 at 9:27 PM, frank wang <f.yw at hotmail.com> wrote:

>  Hi,
>
> I have a large data file which contains 2 columns of data. The two columns
> only have zero and one. Now I want to cound how many one in between if both
> columns are one. For example, if my data is:
>
> 1 0
> 0 0
> 1 1
> 0 0
> 0 1    x
> 0 1    x
> 0 0
> 0 1    x
> 1 1
> 0 0
> 0 1    x
> 0 1    x
> 1 1
>
> Then my count will be 3 and 2 (the numbers with x).
>
> Are there an efficient way to do this? My data file is pretty big.
>
> Thanks
>
> Frank
>
> ------------------------------
> See how Windows connects the people, information, and fun that are part of
> your life. See Now<http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081002/901852c7/attachment.html>

From orionbelt2 at gmail.com  Thu Oct  2 11:43:37 2008
From: orionbelt2 at gmail.com (orionbelt2 at gmail.com)
Date: Thu, 2 Oct 2008 17:43:37 +0200
Subject: [Numpy-discussion] Help to process a large data file
In-Reply-To: <BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>
References: <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com>
	<BLU105-W64E4D0675617417B0A9B989450@phx.gbl>
	<9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com>
	<BLU105-W46A55D24F82955D1366B1F89400@phx.gbl>
	<48E14D27.5040604@enthought.com>
	<BLU105-W57AE036687AE3C43C4D30E89430@phx.gbl>
	<3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com>
	<BLU105-W2339FEAFBA459B6703E4B989430@phx.gbl>
	<ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
	<BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>
Message-ID: <20081002154337.GA20908@ulb.ac.be>

Frank,

I would imagine that you cannot get a much better performance in python 
than this, which avoids string conversions:

c = []
count = 0
for line in open('foo'):
    if line == '1 1\n':
        c.append(count)
        count = 0
    else:
        if '1' in line: count += 1

One could do some numpy trick like:

a = np.loadtxt('foo',dtype=int)
a = np.sum(a,axis=1)    # Add the two columns horizontally
b = np.where(a==2)[0]   # Find with sum == 2 (1 + 1)
count = []
for i,j in zip(b[:-1],b[1:]):
    count.append( a[i+1:j].sum() )  # Calculate number of lines with 1

but on my machine the numpy version takes about 20 sec for a 'foo' file 
of 2,500,000 lines versus 1.2 sec for the pure python version...

As a side note, if i replace "line == '1 1\n'" with "line.startswith('1 
1')", the pure python version goes up to 1.8 sec... Isn't this a bit 
weird, i'd think startswith() should be faster...

Chris

On Wed, Oct 01, 2008 at 07:27:27PM -0600, frank wang wrote:

>    Hi,
> 
>    I have a large data file which contains 2 columns of data. The two 
>    columns only have zero and one. Now I want to cound how many one in 
>    between if both columns are one. For example, if my data is:
> 
>    1 0
>    0 0
>    1 1
>    0 0
>    0 1    x
>    0 1    x
>    0 0
>    0 1    x
>    1 1
>    0 0
>    0 1    x
>    0 1    x
>    1 1
> 
>    Then my count will be 3 and 2 (the numbers with x).
> 
>    Are there an efficient way to do this? My data file is pretty big.
> 
>    Thanks
> 
>    Frank


From f.yw at hotmail.com  Thu Oct  2 15:20:09 2008
From: f.yw at hotmail.com (frank wang)
Date: Thu, 2 Oct 2008 13:20:09 -0600
Subject: [Numpy-discussion] Help to process a large data file
In-Reply-To: <20081002154337.GA20908@ulb.ac.be>
References: <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com>
	<BLU105-W64E4D0675617417B0A9B989450@phx.gbl>
	<9457e7c80809250051y755e232bl8ca79847db186b2@mail.gmail.com>
	<BLU105-W46A55D24F82955D1366B1F89400@phx.gbl>
	<48E14D27.5040604@enthought.com>
	<BLU105-W57AE036687AE3C43C4D30E89430@phx.gbl>
	<3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com>
	<BLU105-W2339FEAFBA459B6703E4B989430@phx.gbl>
	<ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
	<BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl> 
	<20081002154337.GA20908@ulb.ac.be>
Message-ID: <BLU105-W392535B3193305C229930E893D0@phx.gbl>


Thans David and Chris for providing the nice solution.
 
Both method works gread. I could not tell the speed difference between the two solutions. My data size is 1048577 lines.
 
I did not try the second solution from Chris since it is too slow as Chris stated.
 
Frank
 
> Date: Thu, 2 Oct 2008 17:43:37 +0200> From: orionbelt2 at gmail.com> To: numpy-discussion at scipy.org> CC: orionbelt2 at gmail.com> Subject: Re: [Numpy-discussion] Help to process a large data file> > Frank,> > I would imagine that you cannot get a much better performance in python > than this, which avoids string conversions:> > c = []> count = 0> for line in open('foo'):> if line == '1 1\n':> c.append(count)> count = 0> else:> if '1' in line: count += 1> > One could do some numpy trick like:> > a = np.loadtxt('foo',dtype=int)> a = np.sum(a,axis=1) # Add the two columns horizontally> b = np.where(a==2)[0] # Find with sum == 2 (1 + 1)> count = []> for i,j in zip(b[:-1],b[1:]):> count.append( a[i+1:j].sum() ) # Calculate number of lines with 1> > but on my machine the numpy version takes about 20 sec for a 'foo' file > of 2,500,000 lines versus 1.2 sec for the pure python version...> > As a side note, if i replace "line == '1 1\n'" with "line.startswith('1 > 1')", the pure python version goes up to 1.8 sec... Isn't this a bit > weird, i'd think startswith() should be faster...> > Chris> > On Wed, Oct 01, 2008 at 07:27:27PM -0600, frank wang wrote:> > > Hi,> > > > I have a large data file which contains 2 columns of data. The two > > columns only have zero and one. Now I want to cound how many one in > > between if both columns are one. For example, if my data is:> > > > 1 0> > 0 0> > 1 1> > 0 0> > 0 1 x> > 0 1 x> > 0 0> > 0 1 x> > 1 1> > 0 0> > 0 1 x> > 0 1 x> > 1 1> > > > Then my count will be 3 and 2 (the numbers with x).> > > > Are there an efficient way to do this? My data file is pretty big.> > > > Thanks> > > > Frank> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
See how Windows connects the people, information, and fun that are part of your life.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081002/18e945d8/attachment.html>

From travis at enthought.com  Thu Oct  2 15:27:12 2008
From: travis at enthought.com (Travis Vaught)
Date: Thu, 2 Oct 2008 14:27:12 -0500
Subject: [Numpy-discussion] Texas Python Regional Unconference Reminders
In-Reply-To: <E6A23298-A582-48B0-81DA-002C8866C163@gmail.com>
References: <0107962E-D762-497B-BCEC-24CBD78B381B@enthought.com>
	<E6A23298-A582-48B0-81DA-002C8866C163@gmail.com>
Message-ID: <97CF029B-DE73-47A7-8320-044F6AA6E082@enthought.com>

Hey Steve,

I'll bring my camera and try to recruit a volunteer.  No guarantees,  
but we should at least be able to record things (any volunteers to  
transcode a pile of scipy videos? ;-) ).

Best,

Travis


On Oct 1, 2008, at 7:56 PM, Steve Lianoglou wrote:

> Hi,
>
> Are there any plans to tape the presentations? Unfortunately some of
> us can't make it down to Texas, but the talks look quite interesting.
>
> Thanks,
> -steve
>
> On Oct 1, 2008, at 10:36 AM, Travis Vaught wrote:
>
>> Greetings,
>>
>> The Texas Python Regional Unconference is coming up this weekend
>> (October 4-5) and I wanted to send out some more details of the
>> meeting.  The web page for the meeting is here:
>>
>> http://www.scipy.org/TXUncon2008
>>
>> The meeting is _absolutely free_, so please add yourself to the
>> Attendees page if you're able to make it.  Also, if you're planning  
>> to
>> attend, please send me the following information (to travis at enthought.com
>> ) so I can request wireless access for you during the meeting:
>>
>> - Full Name
>> - Phone or email
>> - Address
>> - Affiliation
>>
>> There are still opportunities to present your pet projects at the
>> meeting, so feel free to sign up on the presentation schedule here:
>>
>> http://www.scipy.org/TXUncon2008Schedule
>>
>> For those who are in town Friday evening, we're planning to get
>> together for a casual dinner in downtown Austin that night.  We'll
>> meet at Enthought offices (http://www.enthought.com/contact/map-directions.php
>> ) and walk to a casual restaurant nearby.  Show up as early as 5:30pm
>> and you can hang out and tour the Enthought offices--we'll head out  
>> to
>> eat at 7:00pm sharp.
>>
>> Best,
>>
>> Travis
>>
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



From robert.kern at gmail.com  Thu Oct  2 15:40:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 2 Oct 2008 14:40:32 -0500
Subject: [Numpy-discussion] nan, sign, and all that
In-Reply-To: <e06186140810020622u78c165edm96102a19445fd5ee@mail.gmail.com>
References: <e06186140810012303r3806ae82k7963fb00d85375a1@mail.gmail.com>
	<9457e7c80810020037s13020d14p8bc6ff67726af845@mail.gmail.com>
	<3d375d730810020042i5a20da38sc3f65bef813fefcf@mail.gmail.com>
	<e06186140810020622u78c165edm96102a19445fd5ee@mail.gmail.com>
Message-ID: <3d375d730810021240n111a1014sc62f883d48ec8e41@mail.gmail.com>

On Thu, Oct 2, 2008 at 08:22, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Thu, Oct 2, 2008 at 1:42 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Thu, Oct 2, 2008 at 02:37, St?fan van der Walt <stefan at sun.ac.za>
>> wrote:
>> > Hi Charles,
>> >
>> > 2008/10/2 Charles R Harris <charlesr.harris at gmail.com>:
>> >> In [3]: a = array([NAN, 0, NAN, 1])
>> >> In [4]: b = array([0, NAN, NAN, 0])
>> >>
>> >> In [5]: fmax(a,b)
>> >> Out[5]: array([  0.,   0.,  NaN,   1.])
>> >>
>> >> In [6]: fmin(a,b)
>> >> Out[6]: array([  0.,   0.,  NaN,   0.])
>> >
>> > These are great, many thanks!
>> >
>> > My only gripe is that they have the same NaN-handling as amin and
>> > friends, which I consider to be broken.
>>
>> No, these follow well-defined C99 semantics of the fmin() and fmax()
>> functions in libm. If exactly one of the arguments is a NaN, the
>> non-NaN argument is returned. This is *not* the current behavior of
>> amin() et al., which just do naive comparisons.
>>
>> > Others also mentioned that
>> > this should be changed, and I think David C wrote a patch for it (but
>> > I am not informed as to the speed implications).
>> >
>> > If I had to choose, this would be my preferred output:
>> >
>> > In [5]: fmax(a,b)
>> > Out[5]: array([  NaN,   NaN,  NaN,   1.])
>>
>> Chuck proposes letting minimum() and maximum() have that behavior.
>
> Yes. If there is any agreement on this I would like to go ahead and do it.
> It does change the current behavior of maximum and minimum.

I think the position we've held is that in the presence of NaNs, the
behavior of these functions have been left unspecified, so I think it
is okay to change them.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From bolme1234 at comcast.net  Thu Oct  2 15:40:07 2008
From: bolme1234 at comcast.net (David Bolme)
Date: Thu, 2 Oct 2008 13:40:07 -0600
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
Message-ID: <5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>

I also like the idea of a scipy.spatial library.  For the research I  
do in machine learning and computer vision we are often interested in  
specifying different distance measures.  It would be nice to have a  
way to specify the distance measure.  I would like to see a standard  
set included: City Block, Euclidean, Correlation, etc as well as a  
capability for a user defined distance or similarity function.




From matthieu.brucher at gmail.com  Thu Oct  2 16:01:51 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 2 Oct 2008 22:01:51 +0200
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
Message-ID: <e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>

2008/10/2 David Bolme <bolme1234 at comcast.net>:
> I also like the idea of a scipy.spatial library.  For the research I
> do in machine learning and computer vision we are often interested in
> specifying different distance measures.  It would be nice to have a
> way to specify the distance measure.  I would like to see a standard
> set included: City Block, Euclidean, Correlation, etc as well as a
> capability for a user defined distance or similarity function.

You mean similarity or dissimilarity ? Distance is a dissimilarity but
correlation is a similarity measure.

Matthieu
-- 
French PhD student
Information System Engineer
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From dblubaugh at belcan.com  Thu Oct  2 16:20:38 2008
From: dblubaugh at belcan.com (Blubaugh, David A.)
Date: Thu, 2 Oct 2008 16:20:38 -0400
Subject: [Numpy-discussion] f2py IS NOW WORKING
In-Reply-To: <27CC3060AF71DA40A5DC85F7D5B70F380521CBBE@AWMAIL04.belcan.com>
References: <27CC3060AF71DA40A5DC85F7D5B70F380521CBBE@AWMAIL04.belcan.com>
Message-ID: <27CC3060AF71DA40A5DC85F7D5B70F38054B6973@AWMAIL04.belcan.com>

To all,


I have now been able to develop a stable file via f2py!!  However, I had
to execute the following:

1.) First, I had to copy all required library files from my selected
Compaq visual Fortran compiler under python's scripts directory along
with f2py itself.  


2.)  I also had to include a dll from my compiler under python's dll
directory as well.  


I know that the reason as to why I needed to execute these actions, is
that I do not know as to what should be my correct environmental
variables within windows XP running Compaq Visual Fortran 6.6.  


Once again, I would appreciate to know as to what are the correct
environmental variables should I set my windows xp under, given that the
compiler I must utilize is a Compaq Visual Fortran Compiler 6.6??? 


Thanks,

David Blubaugh

This e-mail transmission contains information that is confidential and may be 
privileged. It is intended only for the addressee(s) named above. If you receive 
this e-mail in error, please do not read, copy or disseminate it in any manner. 
If you are not the intended recipient, any disclosure, copying, distribution or 
use of the contents of this information is prohibited. Please reply to the 
message immediately by informing the sender that the message was misdirected. 
After replying, please erase it from your computer system. Your assistance in 
correcting this error is appreciated.



From Chris.Barker at noaa.gov  Thu Oct  2 17:45:20 2008
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu, 02 Oct 2008 17:45:20 -0400
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
Message-ID: <48E540F0.2080102@noaa.gov>

Jarrod Millman wrote:
> The 1.2.0rc2 is now available:
> http://svn.scipy.org/svn/numpy/tags/1.2.0rc2

what's the status of this?

> Here are the Window's binaries:
> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.5.exe

this appears to be a dead link.

thanks,
-Chris



From robert.kern at gmail.com  Thu Oct  2 17:52:26 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 2 Oct 2008 16:52:26 -0500
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <48E540F0.2080102@noaa.gov>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
	<48E540F0.2080102@noaa.gov>
Message-ID: <3d375d730810021452o18efc1e2l9636ee35f14c11dc@mail.gmail.com>

On Thu, Oct 2, 2008 at 16:45, Chris Barker <Chris.Barker at noaa.gov> wrote:
> Jarrod Millman wrote:
>> The 1.2.0rc2 is now available:
>> http://svn.scipy.org/svn/numpy/tags/1.2.0rc2
>
> what's the status of this?

Superceded by the 1.2.0 release. See the thread "ANN: NumPy 1.2.0".

>> Here are the Window's binaries:
>> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0rc2-win32-superpack-python2.5.exe
>
> this appears to be a dead link.

Superceded by http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0-win32-superpack-python2.5.exe

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bolme1234 at comcast.net  Thu Oct  2 17:17:13 2008
From: bolme1234 at comcast.net (David Bolme)
Date: Thu, 2 Oct 2008 15:17:13 -0600
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
Message-ID: <CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>

It may be useful to have an interface that handles both cases:  
similarity and dissimilarity.  Often I have seen "Nearest Neighbor"  
algorithms that look for maximum similarity instead of minimum  
distance.  In my field (biometrics) we often deal with very  
specialized distance or similarity measures.  I would like to see  
support for user defined distance and similarity functions.  It should  
be easy to implement by passing a function object to the KNN class.  I  
am not sure if kd-trees or other fast algorithms are compatible with  
similarities or non-euclidian norms, however I would be willing to  
implement an exhaustive search KNN that would support user defined  
functions.

On Oct 2, 2008, at 2:01 PM, Matthieu Brucher wrote:

> 2008/10/2 David Bolme <bolme1234 at comcast.net>:
>> I also like the idea of a scipy.spatial library.  For the research I
>> do in machine learning and computer vision we are often interested in
>> specifying different distance measures.  It would be nice to have a
>> way to specify the distance measure.  I would like to see a standard
>> set included: City Block, Euclidean, Correlation, etc as well as a
>> capability for a user defined distance or similarity function.
>
> You mean similarity or dissimilarity ? Distance is a dissimilarity but
> correlation is a similarity measure.
>
> Matthieu
> -- 
> French PhD student
> Information System Engineer
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From Chris.Barker at noaa.gov  Thu Oct  2 19:29:35 2008
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu, 02 Oct 2008 19:29:35 -0400
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <3d375d730810021452o18efc1e2l9636ee35f14c11dc@mail.gmail.com>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
	<48E540F0.2080102@noaa.gov>
	<3d375d730810021452o18efc1e2l9636ee35f14c11dc@mail.gmail.com>
Message-ID: <48E5595F.6000304@noaa.gov>

Robert Kern wrote:
> Superceded by the 1.2.0 release. See the thread "ANN: NumPy 1.2.0".

I thought I'd seen that, but when I went to:

http://www.scipy.org/Download

And I still got 1.1

> Superceded by http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.2.0-win32-superpack-python2.5.exe

thanks,

-Chris





From peridot.faceted at gmail.com  Thu Oct  2 21:57:08 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 2 Oct 2008 21:57:08 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
Message-ID: <ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>

2008/10/2 David Bolme <bolme1234 at comcast.net>:
> It may be useful to have an interface that handles both cases:
> similarity and dissimilarity.  Often I have seen "Nearest Neighbor"
> algorithms that look for maximum similarity instead of minimum
> distance.  In my field (biometrics) we often deal with very
> specialized distance or similarity measures.  I would like to see
> support for user defined distance and similarity functions.  It should
> be easy to implement by passing a function object to the KNN class.  I
> am not sure if kd-trees or other fast algorithms are compatible with
> similarities or non-euclidian norms, however I would be willing to
> implement an exhaustive search KNN that would support user defined
> functions.

kd-trees can only work for distance measures which have certain
special properties (in particular, you have to be able to bound them
based on coordinate differences). This is just fine for all the
Minkowski p-norms (so in particular, Euclidean distance,
maximum-coordinate-difference, and Manhattan distance) and in fact the
current implementation already supports all of these. I don't think
that correlation can be made into such a distance measure - the
neighborhoods are the wrong shape. In fact the basic space is
projective n-1 space rather than affine n-space, so I think you're
going to need some very different algorithm. If you make a metric
space out of it - define d(A,B) to be the angle between A and B - then
cover trees can serve as a spatial data structure for nearest-neighbor
search. Cover trees may be worth implementing, as they're a very
generic data structure, suitable for (among other things)
low-dimensional data in high-dimensional spaces.

Anne


From josef.pktd at gmail.com  Thu Oct  2 22:14:33 2008
From: josef.pktd at gmail.com (joep)
Date: Thu, 2 Oct 2008 19:14:33 -0700 (PDT)
Subject: [Numpy-discussion] numpy.random.hypergeometric - strange results
In-Reply-To: <e6224d81-e04f-4e98-8ca1-b91fbe1e4aba@d77g2000hsb.googlegroups.com>
References: <e6224d81-e04f-4e98-8ca1-b91fbe1e4aba@d77g2000hsb.googlegroups.com>
Message-ID: <49eebbd7-3bed-4ad9-a95f-b7fd53c75d1d@k30g2000hse.googlegroups.com>

see http://scipy.org/scipy/numpy/ticket/921

I think I found the error

http://scipy.org/scipy/numpy/browser/trunk/numpy/random/mtrand/distributions.c

{{{
805 	    /* this is a correction to HRUA* by Ivan Frohne in rv.py */
806 	    if (good > bad) Z = m - Z;
}}}

Quickly looking at the referenced program,
downloaded from: http://pal.ece.iisc.ernet.in/~dhani/frohne/rv.py

Notation: alpha = bad, beta = good:

{{{
               if alpha > beta:           # Error in HRUA*, this is
correct.
                  z = m - z
}}}

As you can see, if my interpretation is correct, then line 806 should
have good and bad reversed, i.e.

{{{
806 	    if (bad > good) Z = m - Z;
}}}

Can you verify this? I never tried to build numpy from source.

Josef

On Sep 25, 4:18?pm, joep <josef.p... at gmail.com> wrote:
> In my fuzz testing of scipy stats, I get sometimes a test failure. I
> think there is something
> wrong with numpy.random.hypergeometric for some cases:
>
> Josef
>
> >>> import numpy.random as mtrand
> >>> mtrand.hypergeometric(3,17,12,size=10) ? # there are only 3 good balls in urn
>
> array([16, 17, 16, 16, 15, 16, 17, 16, 17, 16])>>> mtrand.hypergeometric(17,3,12,size=10) ? #negative result
>
> array([-3, -4, -3, -4, -3, -3, -4, -4, -5, -4])
>
> >>> mtrand.hypergeometric(4,3,12,size=10)
> >>> np.version.version
>
> '1.2.0rc2'
>
> I did not find any clear pattern when trying out different parameter
> values:
>
> >>> mtrand.hypergeometric(10,10,12,size=10)
>
> array([5, 6, 4, 4, 8, 5, 4, 6, 7, 4])>>> mtrand.hypergeometric(10,10,20,size=10)
>
> array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10])>>> mtrand.hypergeometric(10,10,19,size=10)
>
> array([10, ?9, ?9, ?9, ?9, ?9, 10, ?9, ?9, ?9])>>> mtrand.hypergeometric(10,10,5,size=10)
>
> array([3, 5, 2, 2, 1, 2, 2, 4, 3, 1])>>> mtrand.hypergeometric(10,2,5,size=10)
>
> array([4, 5, 4, 5, 5, 5, 4, 3, 4, 4])>>> mtrand.hypergeometric(2,10,5,size=10)
>
> array([0, 2, 1, 0, 2, 2, 1, 1, 1, 1])
>
> >>> mtrand.hypergeometric(17,3,12,size=10)
>
> array([-5, -3, -4, -4, -4, -3, -4, -4, -3, -3])>>> mtrand.hypergeometric(3,17,12,size=10)
>
> array([15, 16, 17, 16, 15, 16, 15, 15, 17, 17])>>> mtrand.hypergeometric(18,3,12,size=10)
>
> array([-5, -6, -6, -4, -4, -4, -5, -3, -5, -5])
>
> >>> mtrand.hypergeometric(18,3,5,size=10)
>
> array([4, 5, 5, 5, 5, 5, 4, 5, 4, 3])>>> mtrand.hypergeometric(18,3,19,size=10)
>
> array([1, 1, 2, 1, 1, 1, 1, 3, 1, 1])
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


From millman at berkeley.edu  Fri Oct  3 01:04:45 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 2 Oct 2008 22:04:45 -0700
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <48E5595F.6000304@noaa.gov>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
	<48E540F0.2080102@noaa.gov>
	<3d375d730810021452o18efc1e2l9636ee35f14c11dc@mail.gmail.com>
	<48E5595F.6000304@noaa.gov>
Message-ID: <c7009a550810022204m3887cfd1xb525a358f322e41c@mail.gmail.com>

On Thu, Oct 2, 2008 at 4:29 PM, Chris Barker <Chris.Barker at noaa.gov> wrote:
> Robert Kern wrote:
>> Superceded by the 1.2.0 release. See the thread "ANN: NumPy 1.2.0".
>
> I thought I'd seen that, but when I went to:
>
> http://www.scipy.org/Download
>
> And I still got 1.1

I updated the page to point to the sourceforge page.  Thanks for catching that.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From josef.pktd at gmail.com  Fri Oct  3 01:17:28 2008
From: josef.pktd at gmail.com (joep)
Date: Thu, 2 Oct 2008 22:17:28 -0700 (PDT)
Subject: [Numpy-discussion] numpy.random.logseries - incorrect
	convergence for k=1, k=2
In-Reply-To: <c614f335-dffe-477e-9243-303ca2c3e541@e53g2000hsa.googlegroups.com>
References: <c614f335-dffe-477e-9243-303ca2c3e541@e53g2000hsa.googlegroups.com>
Message-ID: <95144c91-3f49-48e8-898e-d3cfa225a108@p25g2000hsf.googlegroups.com>

Filed as http://scipy.org/scipy/numpy/ticket/923

and I think i finally tracked down the source of the incorrect random
numbers, a reversed inequality in
http://scipy.org/scipy/numpy/browser/trunk/numpy/random/mtrand/distributions.c
line 871, see my last comment to the trac ticket.

Josef


On Sep 27, 2:12?pm, joep <josef.p... at gmail.com> wrote:
> random numbers generated by numpy.random.logseries do not converge to
> theoretical distribution:
>
> for probability paramater pr = 0.8, the random number generator
> converges to a
> frequency for k=1 at 39.8 %, while the theoretical probability mass is
> 49.71
> k=2 is oversampled, other k's look ok
>
> check frequency of k=1 and k=2 at N = ?1000000
> 0.398406 0.296465
> pmf at k = 1 and k=2 with formula
> [ 0.4971 ?0.1988]
>
> for probability paramater pr = 0.3, the results are not as bad, but
> still off:
> frequency for k=1 at 82.6 %, while the theoretical probability mass is
> 84.11
>
> check frequency of k=1 and k=2 at N = ?1000000
> 0.826006 0.141244
> pmf at k = 1 and k=2 with formula
> [ 0.8411 ?0.1262]
>
> below is a quick script for checking this
>
> Josef
>
> {{{
> import numpy as np
> from scipy import stats
>
> pr = 0.8
> np.set_printoptions(precision=2, suppress=True)
>
> # calculation for N=1million takes some time
> for N in [1000, 10000, 10000, 1000000]:
> ? ? rvsn=np.random.logseries(pr,size=N)
> ? ? fr=stats.itemfreq(rvsn)
> ? ? pmfs=stats.logser.pmf(fr[:,0],pr)*100
> ? ? print 'log series sample frequency and pmf (in %) with N = ', N
> ? ? print np.column_stack((fr[:,0],fr[:,1]*100.0/N,pmfs))
>
> np.set_printoptions(precision=4, suppress=True)
>
> print 'check frequency of k=1 and k=2 at N = ', N
> print np.sum(rvsn==1)/float(N),
> print np.sum(rvsn==2)/float(N)
>
> k = np.array([1,2])
> print 'pmf at k = 1 and k=2 with formula'
> print -pr**k * 1.0 / k / np.log(1-pr)}}}
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


From faltet at pytables.org  Fri Oct  3 03:35:41 2008
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 3 Oct 2008 09:35:41 +0200
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <c7009a550810022204m3887cfd1xb525a358f322e41c@mail.gmail.com>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
	<48E5595F.6000304@noaa.gov>
	<c7009a550810022204m3887cfd1xb525a358f322e41c@mail.gmail.com>
Message-ID: <200810030935.41460.faltet@pytables.org>

A Friday 03 October 2008, Jarrod Millman escrigu?:
> On Thu, Oct 2, 2008 at 4:29 PM, Chris Barker <Chris.Barker at noaa.gov> 
wrote:
> > Robert Kern wrote:
> >> Superceded by the 1.2.0 release. See the thread "ANN: NumPy
> >> 1.2.0".
> >
> > I thought I'd seen that, but when I went to:
> >
> > http://www.scipy.org/Download
> >
> > And I still got 1.1
>
> I updated the page to point to the sourceforge page.  Thanks for
> catching that.

It would be nice if you can update the PYPI package index too.  Perhaps 
having a list of places on where to announce NumPy on every release 
would be handy.

Thanks!

-- 
Francesc Alted


From michel.dupront at hotmail.fr  Fri Oct  3 06:13:26 2008
From: michel.dupront at hotmail.fr (Michel Dupront)
Date: Fri, 3 Oct 2008 12:13:26 +0200
Subject: [Numpy-discussion] =?iso-8859-1?q?Probl=E8me_pour_construire_les_?=
	=?iso-8859-1?q?tests_Numpy-Swig?=
Message-ID: <BAY144-W140E2ADD289B62965C09F0F43C0@phx.gbl>



Bonjour,

Je viens d'installer Numpy. Je suis int?ress? par l'utilisation de swig.
Lorsque je tente de construire les tests j'ai cette erreur:

"""
swig -c++ -python Array.i
:9: Error: Macro '%typecheck' expects 1 argument
:36: Error: Macro '%typecheck' expects 1 argument
:64: Error: Macro '%typecheck' expects 1 argument
:92: Error: Macro '%typecheck' expects 1 argument
:119: Error: Macro '%typecheck' expects 1 argument
:148: Error: Macro '%typecheck' expects 1 argument
:177: Error: Macro '%typecheck' expects 1 argument
:206: Error: Macro '%typecheck' expects 1 argument
:235: Error: Macro '%typecheck' expects 1 argument
.......
"""

Il semble que ca provienne des directives %numpy_typemaps a la fin du fichier numpy.i:

"""
/* Concrete instances of the %numpy_typemaps() macro: Each invocation
 * below applies all of the typemaps above to the specified data type.
 */
%numpy_typemaps(signed char       , NPY_BYTE     , int)
%numpy_typemaps(unsigned char     , NPY_UBYTE    , int)*/
%numpy_typemaps(short             , NPY_SHORT    , int)
/*%numpy_typemaps(unsigned short    , NPY_USHORT   , int)
%numpy_typemaps(int               , NPY_INT      , int)
%numpy_typemaps(unsigned int      , NPY_UINT     , int)
%numpy_typemaps(long              , NPY_LONG     , int)
%numpy_typemaps(unsigned long     , NPY_ULONG    , int)
%numpy_typemaps(long long         , NPY_LONGLONG , int)
%numpy_typemaps(unsigned long long, NPY_ULONGLONG, int)
%numpy_typemaps(float             , NPY_FLOAT    , int)
%numpy_typemaps(double            , NPY_DOUBLE   , int)
"""

Est ce que quelqu'un a rencontr? ce probl?me ?
Merci de bien vouloir m'aider.
Amicalement
Michel
_________________________________________________________________
Email envoy? avec Windows Live Hotmail. Dites adieux aux spam et virus, passez ? Hotmail?! C'est gratuit !
http://www.windowslive.fr/hotmail/default.asp

From michel.dupront at hotmail.fr  Fri Oct  3 06:21:20 2008
From: michel.dupront at hotmail.fr (Michel Dupront)
Date: Fri, 3 Oct 2008 12:21:20 +0200
Subject: [Numpy-discussion] =?iso-8859-1?q?_RE=3A__Probl=E8me_pour_constru?=
 =?iso-8859-1?q?ire_les_tests_Numpy-Swig?=
In-Reply-To: <BAY144-W140E2ADD289B62965C09F0F43C0@phx.gbl>
References: <BAY144-W140E2ADD289B62965C09F0F43C0@phx.gbl>
Message-ID: <BAY144-W30CF4E4025AB83023AF217F43C0@phx.gbl>



Oh sorry I wrote my first email in french 

Hello,
I just installed Numpy. I am interested in using Swig. 
When I try to build the tests I get the following error message:

"""
swig -c++ -python Array.i
:9: Error: Macro '%typecheck' expects 1 argument
:36: Error: Macro '%typecheck' expects 1 argument
:64: Error: Macro '%typecheck' expects 1 argument
:92: Error: Macro '%typecheck' expects 1 argument
:119: Error: Macro '%typecheck' expects 1 argument
:148: Error: Macro '%typecheck' expects 1 argument
:177: Error: Macro '%typecheck' expects 1 argument
:206: Error: Macro '%typecheck' expects 1 argument
:235: Error: Macro '%typecheck' expects 1 argument
.......
"""

It seems that the directive %numpy_typemaps is responsible for this error:
"""
/* Concrete instances of the %numpy_typemaps() macro: Each invocation
 * below applies all of the typemaps above to the specified data type.
 */
%numpy_typemaps(signed char       , NPY_BYTE     , int)
%numpy_typemaps(unsigned char     , NPY_UBYTE    , int)*/
%numpy_typemaps(short             , NPY_SHORT    , int)
/*%numpy_typemaps(unsigned short    , NPY_USHORT   , int)
%numpy_typemaps(int               , NPY_INT      , int)
%numpy_typemaps(unsigned int      , NPY_UINT     , int)
%numpy_typemaps(long              , NPY_LONG     , int)
%numpy_typemaps(unsigned long     , NPY_ULONG    , int)
%numpy_typemaps(long long         , NPY_LONGLONG , int)
%numpy_typemaps(unsigned long long, NPY_ULONGLONG, int)
%numpy_typemaps(float             , NPY_FLOAT    , int)
%numpy_typemaps(double            , NPY_DOUBLE   , int)
"""

Somebody already faced this problem ?
Thank you very much for any help.
Friendly,
Michel


----------------------------------------
> From: michel.dupront at hotmail.fr
> To: numpy-discussion at scipy.org
> Date: Fri, 3 Oct 2008 12:13:26 +0200
> Subject: [Numpy-discussion] Probl?me pour construire les tests Numpy-Swig
> 
> 
> 
> Bonjour,
> 
> Je viens d'installer Numpy. Je suis int?ress? par l'utilisation de swig.
> Lorsque je tente de construire les tests j'ai cette erreur:
> 
> """
> swig -c++ -python Array.i
> :9: Error: Macro '%typecheck' expects 1 argument
> :36: Error: Macro '%typecheck' expects 1 argument
> :64: Error: Macro '%typecheck' expects 1 argument
> :92: Error: Macro '%typecheck' expects 1 argument
> :119: Error: Macro '%typecheck' expects 1 argument
> :148: Error: Macro '%typecheck' expects 1 argument
> :177: Error: Macro '%typecheck' expects 1 argument
> :206: Error: Macro '%typecheck' expects 1 argument
> :235: Error: Macro '%typecheck' expects 1 argument
> .......
> """
> 
> Il semble que ca provienne des directives %numpy_typemaps a la fin du fichier numpy.i:
> 
> """
> /* Concrete instances of the %numpy_typemaps() macro: Each invocation
>  * below applies all of the typemaps above to the specified data type.
>  */
> %numpy_typemaps(signed char       , NPY_BYTE     , int)
> %numpy_typemaps(unsigned char     , NPY_UBYTE    , int)*/
> %numpy_typemaps(short             , NPY_SHORT    , int)
> /*%numpy_typemaps(unsigned short    , NPY_USHORT   , int)
> %numpy_typemaps(int               , NPY_INT      , int)
> %numpy_typemaps(unsigned int      , NPY_UINT     , int)
> %numpy_typemaps(long              , NPY_LONG     , int)
> %numpy_typemaps(unsigned long     , NPY_ULONG    , int)
> %numpy_typemaps(long long         , NPY_LONGLONG , int)
> %numpy_typemaps(unsigned long long, NPY_ULONGLONG, int)
> %numpy_typemaps(float             , NPY_FLOAT    , int)
> %numpy_typemaps(double            , NPY_DOUBLE   , int)
> """
> 
> Est ce que quelqu'un a rencontr? ce probl?me ?
> Merci de bien vouloir m'aider.
> Amicalement
> Michel
> _________________________________________________________________
> Email envoy? avec Windows Live Hotmail. Dites adieux aux spam et virus, passez ? Hotmail ! C'est gratuit !
> http://www.windowslive.fr/hotmail/default.asp
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_________________________________________________________________
T?l?phonez gratuitement ? tous vos proches avec Windows Live Messenger? !? T?l?chargez-le maintenant !
http://www.windowslive.fr/messenger/1.asp

From bertle at smoerz.org  Fri Oct  3 09:14:02 2008
From: bertle at smoerz.org (Roman Bertle)
Date: Fri, 3 Oct 2008 15:14:02 +0200
Subject: [Numpy-discussion] choose() broadcasting, and Trac
In-Reply-To: <20081003111152.GA7228@smoerz.org>
References: <20081003111152.GA7228@smoerz.org>
Message-ID: <20081003131402.GA7652@smoerz.org>

Hello,

I have found something I call a bug in the numpy choose() method and
wanted to report it in trac.
http://scipy.org/BugReport states that "SciPy and NumPy Developer Pages
use the same login/password". However, I (username "smoerz") can log in
with my Scipy account at the Scipy Developer Page
(http://projects.scipy.org/scipy/scipy/), but not at the Numpy Developer
Page (http://projects.scipy.org/scipy/numpy/).

Whatever, porting some code from numarray to numpy, I found a regression
in the broadcasting of choose():

import numarray, numpy

numarray.choose([[0,0,1], [0,0,1]], ([2,2,2], [3,3,3]))
array([[2, 2, 3],
       [2, 2, 3]])

numarray.choose([0,0,1], ([[2,2,2],[2,2,2]], [[3,3,3],[3,3,3]]))
array([[2, 2, 3],
       [2, 2, 3]])

numarray.choose([0,0,1], ([2,2,2], [[3,3,3],[3,3,3]]))
array([[2, 2, 3],
       [2, 2, 3]])

Of these 3 cases, only the first one works for numpy, for the other ones
I get:

/usr/lib/python2.5/site-packages/numpy/core/fromnumeric.pyc in choose(a, choices, out, mode)
    167         choose = a.choose
    168     except AttributeError:
--> 169         return _wrapit(a, 'choose', choices, out=out, mode=mode)
    170     return choose(choices, out=out, mode=mode)
    171 

/usr/lib/python2.5/site-packages/numpy/core/fromnumeric.pyc in _wrapit(obj, method, *args, **kwds)
     35     except AttributeError:
     36         wrap = None
---> 37     result = getattr(asarray(obj),method)(*args, **kwds)
     38     if wrap:
     39         if not isinstance(result, mu.ndarray):

ValueError: too many dimensions

I consider this as a bad regression from numarray to numpy, because the
failing broadcast examples seem to be more important than the working
one.

Best Regards, Roman


From millman at berkeley.edu  Fri Oct  3 09:38:02 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 3 Oct 2008 06:38:02 -0700
Subject: [Numpy-discussion] 1.2.0rc2 tagged! --PLEASE TEST--
In-Reply-To: <200810030935.41460.faltet@pytables.org>
References: <c7009a550809131600k13daab97sac305cefabe0458@mail.gmail.com>
	<48E5595F.6000304@noaa.gov>
	<c7009a550810022204m3887cfd1xb525a358f322e41c@mail.gmail.com>
	<200810030935.41460.faltet@pytables.org>
Message-ID: <c7009a550810030638s2b2dd2e8sa13e80c4eb5c176f@mail.gmail.com>

On Fri, Oct 3, 2008 at 12:35 AM, Francesc Alted <faltet at pytables.org> wrote:
> It would be nice if you can update the PYPI package index too.  Perhaps
> having a list of places on where to announce NumPy on every release
> would be handy.

Done.  Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david.huard at gmail.com  Fri Oct  3 09:48:52 2008
From: david.huard at gmail.com (David Huard)
Date: Fri, 3 Oct 2008 09:48:52 -0400
Subject: [Numpy-discussion] Help to process a large data file
In-Reply-To: <BLU105-W392535B3193305C229930E893D0@phx.gbl>
References: <9457e7c80809240810l6bf656aey70434d893200aa98@mail.gmail.com>
	<BLU105-W46A55D24F82955D1366B1F89400@phx.gbl>
	<48E14D27.5040604@enthought.com>
	<BLU105-W57AE036687AE3C43C4D30E89430@phx.gbl>
	<3d375d730809291822x2c4fb34dq19aab6e9466e2d64@mail.gmail.com>
	<BLU105-W2339FEAFBA459B6703E4B989430@phx.gbl>
	<ce557a360809292225x49d6e715pd74c4f0d9e529cbd@mail.gmail.com>
	<BLU105-W344F4A183DEF23AABDC0BA893D0@phx.gbl>
	<20081002154337.GA20908@ulb.ac.be>
	<BLU105-W392535B3193305C229930E893D0@phx.gbl>
Message-ID: <91cf711d0810030648i6199e512t9bd4e806f3720cd4@mail.gmail.com>

Frank, On Thu, Oct 2, 2008 at 3:20 PM, frank wang <f.yw at hotmail.com> wrote:

>
> Thans David and Chris for providing the nice solution.
>
>

Glad it helped.


> Both method works gread. I could not tell the speed difference between the
> two solutions. My data size is 1048577 lines.
>

I'd be curious to know what happens for larger files (~ 10 M lines). I'd
guess Chris solution would be the fastest since it works incrementally and
does not load the entire data in memory.  If you ever try, I'll be
interested to know how it turns out.

David


> I did not try the second solution from Chris since it is too slow as Chris
> stated.
>
> Frank
>
>
> > Date: Thu, 2 Oct 2008 17:43:37 +0200
> > From: orionbelt2 at gmail.com
> > To: numpy-discussion at scipy.org
> > CC: orionbelt2 at gmail.com
> > Subject: Re: [Numpy-discussion] Help to process a large data file
>
> >
> > Frank,
> >
> > I would imagine that you cannot get a much better performance in python
> > than this, which avoids string conversions:
> >
> > c = []
> > count = 0
> > for line in open('foo'):
> > if line == '1 1\n':
> > c.append(count)
> > count = 0
> > else:
> > if '1' in line: count += 1
> >
> > One could do some numpy trick like:
> >
> > a = np.loadtxt('foo',dtype=int)
> > a = np.sum(a,axis=1) # Add the two columns horizontally
> > b = np.where(a==2)[0] # Find with sum == 2 (1 + 1)
> > count = []
> > for i,j in zip(b[:-1],b[1:]):
> > count.append( a[i+1:j].sum() ) # Calculate number of lines with 1
> >
> > but on my machine the numpy version takes about 20 sec for a 'foo' file
> > of 2,500,000 lines versus 1.2 sec for the pure python version...
> >
> > As a side note, if i replace "line == '1 1\n'" with "line.startswith('1
> > 1')", the pure python version goes up to 1.8 sec... Isn't this a bit
> > weird, i'd think startswith() should be faster...
> >
> > Chris
> >
> > On Wed, Oct 01, 2008 at 07:27:27PM -0600, frank wang wrote:
> >
> > > Hi,
> > >
> > > I have a large data file which contains 2 columns of data. The two
> > > columns only have zero and one. Now I want to cound how many one in
> > > between if both columns are one. For example, if my data is:
> > >
> > > 1 0
> > > 0 0
> > > 1 1
> > > 0 0
> > > 0 1 x
> > > 0 1 x
> > > 0 0
> > > 0 1 x
> > > 1 1
> > > 0 0
> > > 0 1 x
> > > 0 1 x
> > > 1 1
> > >
> > > Then my count will be 3 and 2 (the numbers with x).
> > >
> > > Are there an efficient way to do this? My data file is pretty big.
> > >
> > > Thanks
> > >
> > > Frank
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ------------------------------
> See how Windows connects the people, information, and fun that are part of
> your life. See Now<http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081003/68f49a69/attachment.html>

From oliphant at enthought.com  Fri Oct  3 10:52:13 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 03 Oct 2008 09:52:13 -0500
Subject: [Numpy-discussion] choose() broadcasting, and Trac
In-Reply-To: <20081003131402.GA7652@smoerz.org>
References: <20081003111152.GA7228@smoerz.org>
	<20081003131402.GA7652@smoerz.org>
Message-ID: <48E6319D.7050702@enthought.com>

Roman Bertle wrote:
> Hello,
>
> I have found something I call a bug in the numpy choose() method and
> wanted to report it in trac.
>   
Thanks for your report.  I'm not sure why you are having trouble with 
Trac, but I've created a ticket for this problem.

-Travis



From michel.dupront at hotmail.fr  Fri Oct  3 11:21:41 2008
From: michel.dupront at hotmail.fr (Michel Dupront)
Date: Fri, 3 Oct 2008 17:21:41 +0200
Subject: [Numpy-discussion] =?iso-8859-1?q?_RE=3A___RE=3A__Probl=E8me_pour?=
 =?iso-8859-1?q?_construire_les_tests_Numpy-Swig?=
In-Reply-To: <BAY144-W30CF4E4025AB83023AF217F43C0@phx.gbl>
References: <BAY144-W140E2ADD289B62965C09F0F43C0@phx.gbl>
	<BAY144-W30CF4E4025AB83023AF217F43C0@phx.gbl>
Message-ID: <BAY144-W41026DEFC26C0BDEB97378F43C0@phx.gbl>




I was using swig 1.3.24.
I installed the last swig version 1.3.36 and now it is working fine !
and it makes me very very happy !!!


----------------------------------------
> From: michel.dupront at hotmail.fr
> To: numpy-discussion at scipy.org
> Date: Fri, 3 Oct 2008 12:21:20 +0200
> Subject: [Numpy-discussion]  RE:  Probl?me pour construire les tests Numpy-Swig
> 
> 
> 
> Oh sorry I wrote my first email in french 
> 
> Hello,
> I just installed Numpy. I am interested in using Swig. 
> When I try to build the tests I get the following error message:
> 
> """
> swig -c++ -python Array.i
> :9: Error: Macro '%typecheck' expects 1 argument
> :36: Error: Macro '%typecheck' expects 1 argument
> :64: Error: Macro '%typecheck' expects 1 argument
> :92: Error: Macro '%typecheck' expects 1 argument
> :119: Error: Macro '%typecheck' expects 1 argument
> :148: Error: Macro '%typecheck' expects 1 argument
> :177: Error: Macro '%typecheck' expects 1 argument
> :206: Error: Macro '%typecheck' expects 1 argument
> :235: Error: Macro '%typecheck' expects 1 argument
> .......
> """
> 
> It seems that the directive %numpy_typemaps is responsible for this error:
> """
> /* Concrete instances of the %numpy_typemaps() macro: Each invocation
>  * below applies all of the typemaps above to the specified data type.
>  */
> %numpy_typemaps(signed char       , NPY_BYTE     , int)
> %numpy_typemaps(unsigned char     , NPY_UBYTE    , int)*/
> %numpy_typemaps(short             , NPY_SHORT    , int)
> /*%numpy_typemaps(unsigned short    , NPY_USHORT   , int)
> %numpy_typemaps(int               , NPY_INT      , int)
> %numpy_typemaps(unsigned int      , NPY_UINT     , int)
> %numpy_typemaps(long              , NPY_LONG     , int)
> %numpy_typemaps(unsigned long     , NPY_ULONG    , int)
> %numpy_typemaps(long long         , NPY_LONGLONG , int)
> %numpy_typemaps(unsigned long long, NPY_ULONGLONG, int)
> %numpy_typemaps(float             , NPY_FLOAT    , int)
> %numpy_typemaps(double            , NPY_DOUBLE   , int)
> """
> 
> Somebody already faced this problem ?
> Thank you very much for any help.
> Friendly,
> Michel
> 
> 
> ----------------------------------------
> > From: michel.dupront at hotmail.fr
> > To: numpy-discussion at scipy.org
> > Date: Fri, 3 Oct 2008 12:13:26 +0200
> > Subject: [Numpy-discussion] Probl?me pour construire les tests Numpy-Swig
> > 
> > 
> > 
> > Bonjour,
> > 
> > Je viens d'installer Numpy. Je suis int?ress? par l'utilisation de swig.
> > Lorsque je tente de construire les tests j'ai cette erreur:
> > 
> > """
> > swig -c++ -python Array.i
> > :9: Error: Macro '%typecheck' expects 1 argument
> > :36: Error: Macro '%typecheck' expects 1 argument
> > :64: Error: Macro '%typecheck' expects 1 argument
> > :92: Error: Macro '%typecheck' expects 1 argument
> > :119: Error: Macro '%typecheck' expects 1 argument
> > :148: Error: Macro '%typecheck' expects 1 argument
> > :177: Error: Macro '%typecheck' expects 1 argument
> > :206: Error: Macro '%typecheck' expects 1 argument
> > :235: Error: Macro '%typecheck' expects 1 argument
> > .......
> > """
> > 
> > Il semble que ca provienne des directives %numpy_typemaps a la fin du fichier numpy.i:
> > 
> > """
> > /* Concrete instances of the %numpy_typemaps() macro: Each invocation
> >  * below applies all of the typemaps above to the specified data type.
> >  */
> > %numpy_typemaps(signed char       , NPY_BYTE     , int)
> > %numpy_typemaps(unsigned char     , NPY_UBYTE    , int)*/
> > %numpy_typemaps(short             , NPY_SHORT    , int)
> > /*%numpy_typemaps(unsigned short    , NPY_USHORT   , int)
> > %numpy_typemaps(int               , NPY_INT      , int)
> > %numpy_typemaps(unsigned int      , NPY_UINT     , int)
> > %numpy_typemaps(long              , NPY_LONG     , int)
> > %numpy_typemaps(unsigned long     , NPY_ULONG    , int)
> > %numpy_typemaps(long long         , NPY_LONGLONG , int)
> > %numpy_typemaps(unsigned long long, NPY_ULONGLONG, int)
> > %numpy_typemaps(float             , NPY_FLOAT    , int)
> > %numpy_typemaps(double            , NPY_DOUBLE   , int)
> > """
> > 
> > Est ce que quelqu'un a rencontr? ce probl?me ?
> > Merci de bien vouloir m'aider.
> > Amicalement
> > Michel
> > _________________________________________________________________
> > Email envoy? avec Windows Live Hotmail. Dites adieux aux spam et virus, passez ? Hotmail ! C'est gratuit !
> > http://www.windowslive.fr/hotmail/default.asp
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> _________________________________________________________________
> T?l?phonez gratuitement ? tous vos proches avec Windows Live Messenger  !  T?l?chargez-le maintenant !
> http://www.windowslive.fr/messenger/1.asp
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

_________________________________________________________________
Installez gratuitement les 20 ?m?ticones Windows Live Messenger les plus fous ! Cliquez ici !
http://www.emoticones-messenger.fr/

From kpvincent at hotmail.com  Fri Oct  3 12:39:39 2008
From: kpvincent at hotmail.com (Kelly Vincent)
Date: Fri, 3 Oct 2008 12:39:39 -0400
Subject: [Numpy-discussion] Array shape
Message-ID: <BLU145-W3373867BAED06D90FBC0BB93C0@phx.gbl>


I'm using Numpy to do some basic array manipulation, and I'm getting some unexpected behavior from shape. Specifically, I have some 3x3 and 2x2 matrices, and shape gives me (5, 3) and (3, 2) for their respective sizes. I was expecting (3, 3) and (2, 2), for number of rows, number of columns. I'm assuming I must either be misunderstanding what shape gives you or doing something wrong. Can anybody give me any advice? I'm using Python 2.5 and Numpy 1.1.0.

Thanks,
Kelly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081003/67b9bfd1/attachment.html>

From rmay31 at gmail.com  Fri Oct  3 12:44:23 2008
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 03 Oct 2008 11:44:23 -0500
Subject: [Numpy-discussion] Array shape
In-Reply-To: <BLU145-W3373867BAED06D90FBC0BB93C0@phx.gbl>
References: <BLU145-W3373867BAED06D90FBC0BB93C0@phx.gbl>
Message-ID: <48E64BE7.1010800@gmail.com>

Kelly Vincent wrote:
> I'm using Numpy to do some basic array manipulation, and I'm getting
> some unexpected behavior from shape. Specifically, I have some 3x3 and
> 2x2 matrices, and shape gives me (5, 3) and (3, 2) for their respective
> sizes. I was expecting (3, 3) and (2, 2), for number of rows, number of
> columns. I'm assuming I must either be misunderstanding what shape gives
> you or doing something wrong. Can anybody give me any advice? I'm using
> Python 2.5 and Numpy 1.1.0.

Can you post a complete, minimal example that shows the problem you
have?  For an array object A, A.shape should give the shape you're
expecting.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From wnbell at gmail.com  Fri Oct  3 12:56:19 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Fri, 3 Oct 2008 12:56:19 -0400
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Probl=E8me_pour_construire_les_?=
	=?iso-8859-1?q?tests_Numpy-Swig?=
In-Reply-To: <BAY144-W41026DEFC26C0BDEB97378F43C0@phx.gbl>
References: <BAY144-W140E2ADD289B62965C09F0F43C0@phx.gbl>
	<BAY144-W30CF4E4025AB83023AF217F43C0@phx.gbl>
	<BAY144-W41026DEFC26C0BDEB97378F43C0@phx.gbl>
Message-ID: <d05265cb0810030956p62975e8frc790e98524e66c6c@mail.gmail.com>

On Fri, Oct 3, 2008 at 11:21 AM, Michel Dupront
<michel.dupront at hotmail.fr> wrote:
>
> I was using swig 1.3.24.
> I installed the last swig version 1.3.36 and now it is working fine !
> and it makes me very very happy !!!
>

SWIG often has that effect on people :)

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/
Satisfied SWIG customer


From gael.varoquaux at normalesup.org  Fri Oct  3 12:59:02 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 3 Oct 2008 18:59:02 +0200
Subject: [Numpy-discussion] making numpy.dot faster
Message-ID: <20081003165902.GB28423@phare.normalesup.org>

I am doing a calculation where one call numpy.dot ends up taking 90% of
the time (the array is huge: (61373, 500) ).

Any chance I can make this faster? I would believe BLAS/ATLAS would be
behind this, but from my quick analysis (ldd on numpy/core/multiarray.so)
it doesn't seem so. Have I done something stupid when building numpy
(disclaimer: I am on a system I don't know well --Mandriva--, so I could
very well have done something stupid).

Cheers,

Ga?l


From charlesr.harris at gmail.com  Fri Oct  3 13:14:08 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 3 Oct 2008 11:14:08 -0600
Subject: [Numpy-discussion] making numpy.dot faster
In-Reply-To: <20081003165902.GB28423@phare.normalesup.org>
References: <20081003165902.GB28423@phare.normalesup.org>
Message-ID: <e06186140810031014w68bac296vb61c60fe39583d92@mail.gmail.com>

On Fri, Oct 3, 2008 at 10:59 AM, Gael Varoquaux <
gael.varoquaux at normalesup.org> wrote:

> I am doing a calculation where one call numpy.dot ends up taking 90% of
> the time (the array is huge: (61373, 500) ).
>
> Any chance I can make this faster? I would believe BLAS/ATLAS would be
> behind this, but from my quick analysis (ldd on numpy/core/multiarray.so)
> it doesn't seem so. Have I done something stupid when building numpy
> (disclaimer: I am on a system I don't know well --Mandriva--, so I could
> very well have done something stupid).
>

What does  np.__config__.show() show?  What exactly are you multiplying?
What is the original problem?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081003/fa4e6f77/attachment.html>

From rwgk at yahoo.com  Fri Oct  3 14:49:21 2008
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Fri, 3 Oct 2008 11:49:21 -0700 (PDT)
Subject: [Numpy-discussion] Co-existing Numeric and numpy?
Message-ID: <979257.99821.qm@web31403.mail.mud.yahoo.com>

Hi,

We have some older 3rd party packages that require Numeric (24.2), but like to also use newer
3rd party packages that require a recent numpy. Can Numeric and numpy co-exist in the same
process? -- I'm mainly worried about clashes at the C API level.

Thanks!

Ralf


From Chris.Barker at noaa.gov  Fri Oct  3 15:12:48 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 03 Oct 2008 12:12:48 -0700
Subject: [Numpy-discussion] Co-existing Numeric and numpy?
In-Reply-To: <979257.99821.qm@web31403.mail.mud.yahoo.com>
References: <979257.99821.qm@web31403.mail.mud.yahoo.com>
Message-ID: <48E66EB0.6080301@noaa.gov>

Ralf W. Grosse-Kunstleve wrote:
> We have some older 3rd party packages that require Numeric (24.2), but like to also use newer
> 3rd party packages that require a recent numpy. Can Numeric and numpy co-exist in the same
> process? -- I'm mainly worried about clashes at the C API level.

nope, there are no problems. In fact, converting between the two arrays 
types with asarray() works very efficiently.

I sure wish everyone would make the transition, though.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From rwgk at yahoo.com  Fri Oct  3 15:36:30 2008
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Fri, 3 Oct 2008 12:36:30 -0700 (PDT)
Subject: [Numpy-discussion] Co-existing Numeric and numpy?
Message-ID: <519676.31370.qm@web31406.mail.mud.yahoo.com>

Thank you very much for the information!
There are weird but important reasons why we are stuck with Numeric 24.2 for the
foreseeable future. It is extremely valuable that you provide a smooth upgrade
path.
Ralf



----- Original Message ----
From: Christopher Barker <Chris.Barker at noaa.gov>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Sent: Friday, October 3, 2008 12:12:48 PM
Subject: Re: [Numpy-discussion] Co-existing Numeric and numpy?

Ralf W. Grosse-Kunstleve wrote:
> We have some older 3rd party packages that require Numeric (24.2), but like to also use newer
> 3rd party packages that require a recent numpy. Can Numeric and numpy co-exist in the same
> process? -- I'm mainly worried about clashes at the C API level.

nope, there are no problems. In fact, converting between the two arrays 
types with asarray() works very efficiently.

I sure wish everyone would make the transition, though.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From pav at iki.fi  Fri Oct  3 17:11:58 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 3 Oct 2008 21:11:58 +0000 (UTC)
Subject: [Numpy-discussion] making numpy.dot faster
References: <20081003165902.GB28423@phare.normalesup.org>
Message-ID: <gc61qu$cjt$1@ger.gmane.org>

Fri, 03 Oct 2008 18:59:02 +0200, Gael Varoquaux wrote:

> I am doing a calculation where one call numpy.dot ends up taking 90% of
> the time (the array is huge: (61373, 500) ).
> 
> Any chance I can make this faster? I would believe BLAS/ATLAS would be
> behind this, but from my quick analysis (ldd on
> numpy/core/multiarray.so) it doesn't seem so. Have I done something
> stupid when building numpy (disclaimer: I am on a system I don't know
> well --Mandriva--, so I could very well have done something stupid).

AFAIK, multiarray.so is never linked against ATLAS. The accelerated dot 
implementation is in _dotblas.so, and can be toggled with alterdot/
restoredot (but the ATLAS one should be active by default).

>>> numpy.dot.__module__
'numpy.core._dotblas'

Are your arrays appropriately contiguous? Numpy needs to copy the data if 
they are not; though I'm not sure if this could account for what you see.

-- 
Pauli Virtanen



From bolme1234 at comcast.net  Fri Oct  3 20:25:25 2008
From: bolme1234 at comcast.net (David Bolme)
Date: Fri, 3 Oct 2008 18:25:25 -0600
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
Message-ID: <26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>

I remember reading a paper or book that stated that for data that has  
been normalized correlation and Euclidean are equivalent and will  
produce the same knn results.  To this end I spent a couple hours this  
afternoon doing the math.  This document is the result.

http://www.cs.colostate.edu/~bolme/bolme08euclidean.pdf

I believe that with mean subtracted and unit length vectors, a  
Euclidean knn algorithm will produces the same result as if the  
vectors were compared using correlation.  I am not sure if kd-trees  
will perform well on the normalized vectors as they have a very  
specific geometry.  If my math checks out it may be worth adding  
Pearson's correlation as a default option or as a separate class.

I have also spent a little time looking at kd-trees and the kdtree  
code.  It looks good.  As I understand it kd-trees only work well when  
the number of datapoints (N) is larger than 2^D, where D is the  
dimensionality of those points.  This will not work well for many of  
my computer vision problems because often D is large.  As Anne  
suggested I will probably look at cover trees because often times the  
data are "low-dimensional data in high-dimensional spaces".  I have  
been told though that with a large D there is know known fast  
algorithm for knn.

Another problem is that the distances and similarity measures used in  
biometrics and computer vision are often very specialized and may or  
may not conform to the underlying assumptions of fast algorithms.  I  
think for this reason I will need an exhaustive search algorithm.  I  
will code it up modeled after Anne's interface and hopefully it will  
make it into the spatial module.

I think that kd-trees and the spatial module are a good contribution  
to scipy.  I have also enjoyed learning more about norms, correlation,  
and fast knn algorithms.

Thanks,
Dave


From peridot.faceted at gmail.com  Fri Oct  3 23:03:08 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 3 Oct 2008 23:03:08 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
	<26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
Message-ID: <ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>

2008/10/3 David Bolme <bolme1234 at comcast.net>:
> I remember reading a paper or book that stated that for data that has
> been normalized correlation and Euclidean are equivalent and will
> produce the same knn results.  To this end I spent a couple hours this
> afternoon doing the math.  This document is the result.
>
> http://www.cs.colostate.edu/~bolme/bolme08euclidean.pdf

Yes, you're right, even without the mean subtraction they all lie on a
hypersphere in Euclidean space. It's a little more awkward if you want
to identify x and -x.

> I believe that with mean subtracted and unit length vectors, a
> Euclidean knn algorithm will produces the same result as if the
> vectors were compared using correlation.  I am not sure if kd-trees
> will perform well on the normalized vectors as they have a very
> specific geometry.  If my math checks out it may be worth adding
> Pearson's correlation as a default option or as a separate class.

Actually it's probably easier if the user just does the prenormalization.

> I have also spent a little time looking at kd-trees and the kdtree
> code.  It looks good.  As I understand it kd-trees only work well when
> the number of datapoints (N) is larger than 2^D, where D is the
> dimensionality of those points.  This will not work well for many of
> my computer vision problems because often D is large.  As Anne
> suggested I will probably look at cover trees because often times the
> data are "low-dimensional data in high-dimensional spaces".  I have
> been told though that with a large D there is know known fast
> algorithm for knn.

Pretty much true. Though if the intrinsic dimensionality is low, cover
trees should be all right.

> Another problem is that the distances and similarity measures used in
> biometrics and computer vision are often very specialized and may or
> may not conform to the underlying assumptions of fast algorithms.  I
> think for this reason I will need an exhaustive search algorithm.  I
> will code it up modeled after Anne's interface and hopefully it will
> make it into the spatial module.

Metric spaces are quite general - edit distance for strings, for
example, is an adequate distance measure. But brute-force is
definitely worth having too.

If I get the test suite cleaned up, it should be possible to just drop
an arbitrary k-nearest-neighbors class into it and get a reasonably
thorough collection of unit tests.

Anne


From gael.varoquaux at normalesup.org  Sat Oct  4 07:00:18 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 4 Oct 2008 13:00:18 +0200
Subject: [Numpy-discussion] making numpy.dot faster
In-Reply-To: <e06186140810031014w68bac296vb61c60fe39583d92@mail.gmail.com>
References: <20081003165902.GB28423@phare.normalesup.org>
	<e06186140810031014w68bac296vb61c60fe39583d92@mail.gmail.com>
Message-ID: <20081004110018.GD834@phare.normalesup.org>

On Fri, Oct 03, 2008 at 09:11:58PM +0000, Pauli Virtanen wrote:
> Fri, 03 Oct 2008 18:59:02 +0200, Gael Varoquaux wrote:

> > I am doing a calculation where one call numpy.dot ends up taking 90%
> > of
> > the time (the array is huge: (61373, 500) ).

> > Any chance I can make this faster? I would believe BLAS/ATLAS would
> > be
> > behind this, but from my quick analysis (ldd on
> > numpy/core/multiarray.so) it doesn't seem so. Have I done something
> > stupid when building numpy (disclaimer: I am on a system I don't know
> > well --Mandriva--, so I could very well have done something stupid).

> AFAIK, multiarray.so is never linked against ATLAS. The accelerated dot 
> implementation is in _dotblas.so, and can be toggled with alterdot/
> restoredot (but the ATLAS one should be active by default).

> >>> numpy.dot.__module__
> 'numpy.core._dotblas'

OK, thanks, that's useful info.

I am not at work right now, and I can't log in the boxes at work, but I
am pretty sure that 'numpy.dot.__module__' returned
'numpy.core.mutliarray', that's why I tried an ldd on it. On an Ubuntu
box at home, using the off-the-shelf numpy package I have the same
results, althought the numpy package has an optional dependency on atlas.
I have 1.0.3 on the Ubuntu box and something more recent on the work box
-- not sure what, but I seem to remember I grabbed trunk. Playing with
restoredot/alterdot didn't change anything.

> Are your arrays appropriately contiguous? Numpy needs to copy the data
> if they are not; though I'm not sure if this could account for what you
> see.


On Fri, Oct 03, 2008 at 11:14:08AM -0600, Charles R Harris wrote:
>    What does  np.__config__.show() show? 

That's on my home box (where things and also quite slow), but nothing
good:

In [4]: numpy.__config__.show()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77

lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77

atlas_threads_info:
  NOT AVAILABLE

blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]

atlas_blas_threads_info:
  NOT AVAILABLE

lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]

atlas_info:
  NOT AVAILABLE

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

atlas_blas_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

This seems to tell that numpy has been build without altas. Hum, maybe we
need to work with the Debian guys to make sure that numpy is available
with altas.

My home box is a 4-years old AMD64 (single core) and it is slightly
quicker than the bran new 8-cores, super-cool box we have at the lab
(intel CPUs). I am quite puzzled. This is not the first time I see this.
could it be because I am running a 64 bit distro? I haven't checked if
PAE is enabled on the box at the lab, but I definitely know it is not
64 bit.

> What exactly are you multiplying?

Right now numy.random.random to do my tests, but my original problem
(very noisy matrices coming from neuroimaging data) showed the same
behavior.

>    What is the original problem?

I am building a correlation matrix as a first step for a PCA. I have a
matrix M made of m=61373 rows giving time-series of length n=500. I am
calculating 

X = np.dot(M.T, M)

and doing an SVD on X. This is not code I have writen, and I am slowly
warming up to a new field, so I might be missing some obvious points, or
talking nonsense.

Cheers,

Ga?l


From josef.pktd at gmail.com  Sat Oct  4 10:31:21 2008
From: josef.pktd at gmail.com (joep)
Date: Sat, 4 Oct 2008 07:31:21 -0700 (PDT)
Subject: [Numpy-discussion] converting nan to an integer
Message-ID: <05249a34-b378-4bf8-b703-af50ad9c5e85@h60g2000hsg.googlegroups.com>

Why does converting nan to an integer not throw an exception (as with
inf), instead numpy silently replaces nan by zero?


>>> inti = np.array([0,1,2])
>>> inti.dtype
dtype('int32')
>>> inti[1] = np.inf
Traceback (most recent call last):
  File "<pyshell#34>", line 1, in ?
    inti[1] = np.inf
OverflowError: cannot convert float infinity to long
>>> inti[1] = np.nan
>>> inti
array([0, 0, 2])

This is based on an example in numpy.stats.distribution.py

Josef


From lists_ravi at lavabit.com  Sat Oct  4 10:56:04 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Sat, 4 Oct 2008 10:56:04 -0400
Subject: [Numpy-discussion] PyArray_New bug?
Message-ID: <200810041056.06065.lists_ravi@lavabit.com>

Hi,
  PyArray_New seems to return a fortran contiguous array regardless of the 
requested array type. I am using numpy version 1.1.1.

PyObject *obj = PyArray_New( &PyArray_Type, 2, dims, /*whatever*/, NULL,
                                 NULL, 0, NPY_CARRAY, NULL );
PyObject *obj = PyArray_New( &PyArray_Type, 2, dims, /*whatever*/, NULL,
                                 NULL, 0, NPY_FARRAY, NULL );

Both the above return a array who PyArray_ISFORTRAN( obj ) succeeds. I can 
verify this by checking bits 0 and 1 (LSB is bit 0) of PyArray_FLAGS.

Regards,
Ravi




From simpson at math.toronto.edu  Sat Oct  4 11:23:09 2008
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Sat, 4 Oct 2008 11:23:09 -0400
Subject: [Numpy-discussion] fast matrix vector operations
Message-ID: <0D67E423-F549-4DA5-882C-4D6AF10E0FDD@math.toronto.edu>

Suppose I have a toeplitz matrix, A.  There is a well known algorithm  
for computing the matrix vector product Ax, in NlogN operations.  An  
exact reference escapes me, but it may be in Golub & van Loan's book.

My question is, how could I best take advantage of this algorithm  
within numpy/scipy?

I could code it python.  However, since python is a high level  
language, it's not clear to me that I'd see an execution time benefit  
over numpy.dot(A,x).  Alternatively, I could write it in a compiled  
language and build python bindings to it.

Thoughts?

-gideon



From opossumnano at gmail.com  Sat Oct  4 11:59:40 2008
From: opossumnano at gmail.com (Tiziano Zito)
Date: Sat, 4 Oct 2008 17:59:40 +0200
Subject: [Numpy-discussion] making numpy.dot faster
In-Reply-To: <20081004110018.GD834@phare.normalesup.org>
References: <20081003165902.GB28423@phare.normalesup.org>
	<e06186140810031014w68bac296vb61c60fe39583d92@mail.gmail.com>
	<20081004110018.GD834@phare.normalesup.org>
Message-ID: <e195e9cb0810040859s532b4cb4n8efab49bdbb2a294@mail.gmail.com>

Hi,

> This seems to tell that numpy has been build without altas. Hum, maybe we
> need to work with the Debian guys to make sure that numpy is available
> with altas.
>

we had recently a discussion regarding this issue on this mailinglist, see:
http://groups.google.com/group/Numpy-discussion/browse_thread/thread/507e7722f99406fa/
and on the debian bug tracker:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=489253

in debian testing and unstable numpy is now built with a patch that
enables atlas support. I don't know the status of the ubuntu package,
but I presume they may apply a similar patch until the numpy building
system's checks are relaxed ina way that no patch is needed anymore.

cheers,
tiziano


From xavier.gnata at gmail.com  Sat Oct  4 12:13:54 2008
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Sat, 04 Oct 2008 18:13:54 +0200
Subject: [Numpy-discussion] fast matrix vector operations
In-Reply-To: <0D67E423-F549-4DA5-882C-4D6AF10E0FDD@math.toronto.edu>
References: <0D67E423-F549-4DA5-882C-4D6AF10E0FDD@math.toronto.edu>
Message-ID: <48E79642.5040307@gmail.com>


> Suppose I have a toeplitz matrix, A.  There is a well known algorithm  
> for computing the matrix vector product Ax, in NlogN operations.  An  
> exact reference escapes me, but it may be in Golub & van Loan's book.
>
> My question is, how could I best take advantage of this algorithm  
> within numpy/scipy?
>
> I could code it python.  However, since python is a high level  
> language, it's not clear to me that I'd see an execution time benefit  
> over numpy.dot(A,x).  Alternatively, I could write it in a compiled  
> language and build python bindings to it.
>
> Thoughts?
>
> -gideon
>
>   

http://www.scipy.org/PerformancePython
weave.inline or pyrex.

BTW, is there a good introduction to weave.inline ?
I had an hard time to learn/guess it.

Xavier



From charlesr.harris at gmail.com  Sat Oct  4 12:16:37 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 4 Oct 2008 10:16:37 -0600
Subject: [Numpy-discussion] Warnings for Travis
Message-ID: <e06186140810040916q515c2cc7n176155c4dbc2410d@mail.gmail.com>

Hi Travis,

The following turned up on the 64-bit windows buildbot. I don't know what
triggers them as they don't normally show, but they look legit.

numpy\core\src\ufuncobject.c(1700) : warning C4244: '=' : conversion
from 'npy_intp' to 'int', possible loss of data
numpy\core\src\ufuncobject.c(1701) : warning C4244: '=' : conversion
from 'npy_intp' to 'int', possible loss of data
numpy\core\src\ufuncobject.c(2422) : warning C4244: '=' : conversion
from 'npy_intp' to 'int', possible loss of data

They involve some of the parameters in the loop object. If this object is
not exposed I would suggest making them npy_intp. That said, I don't think
casting to int is dangerous in this case.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081004/c99f7725/attachment.html>

From oliphant at enthought.com  Sat Oct  4 12:24:48 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sat, 04 Oct 2008 11:24:48 -0500
Subject: [Numpy-discussion] PyArray_New bug?
In-Reply-To: <200810041056.06065.lists_ravi@lavabit.com>
References: <200810041056.06065.lists_ravi@lavabit.com>
Message-ID: <48E798D0.2000900@enthought.com>

Ravi wrote:
> Hi,
>   PyArray_New seems to return a fortran contiguous array regardless of the 
> requested array type. I am using numpy version 1.1.1.
>
> PyObject *obj = PyArray_New( &PyArray_Type, 2, dims, /*whatever*/, NULL,
>                                  NULL, 0, NPY_CARRAY, NULL );
> PyObject *obj = PyArray_New( &PyArray_Type, 2, dims, /*whatever*/, NULL,
>                                  NULL, 0, NPY_FARRAY, NULL );
>
> Both the above return a array who PyArray_ISFORTRAN( obj ) succeeds. I can 
> verify this by checking bits 0 and 1 (LSB is bit 0) of PyArray_FLAGS.
>
>   
The C-API for PyArray_New is not what you were expecting.    When data 
is NULL so that NumPy creates the array, then the flags argument is a 
toggle switch (i.e. any non-zero value for the flags argument means 
construct a Fortran array).

Sorry for the confusion.

-Travis



From gael.varoquaux at normalesup.org  Sat Oct  4 14:08:53 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 4 Oct 2008 20:08:53 +0200
Subject: [Numpy-discussion] making numpy.dot faster
In-Reply-To: <e195e9cb0810040859s532b4cb4n8efab49bdbb2a294@mail.gmail.com>
References: <20081003165902.GB28423@phare.normalesup.org>
	<e06186140810031014w68bac296vb61c60fe39583d92@mail.gmail.com>
	<20081004110018.GD834@phare.normalesup.org>
	<e195e9cb0810040859s532b4cb4n8efab49bdbb2a294@mail.gmail.com>
Message-ID: <20081004180853.GA23581@phare.normalesup.org>

On Sat, Oct 04, 2008 at 05:59:40PM +0200, Tiziano Zito wrote:
> Hi,

> > This seems to tell that numpy has been build without altas. Hum, maybe we
> > need to work with the Debian guys to make sure that numpy is available
> > with altas.


> we had recently a discussion regarding this issue on this mailinglist, see:
> http://groups.google.com/group/Numpy-discussion/browse_thread/thread/507e7722f99406fa/
> and on the debian bug tracker:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=489253

OK, thanks. I thought I had seen something like this go by, but I wasn't
sure.

> in debian testing and unstable numpy is now built with a patch that
> enables atlas support. I don't know the status of the ubuntu package,
> but I presume they may apply a similar patch until the numpy building
> system's checks are relaxed ina way that no patch is needed anymore.

I'll can make sure that this patch makes its way into next ubuntu.

Thanks a lot for recalling this discussion, and forgive me for not paying
enough attention. My attention span is limited because I am doing too
many things.

Ga?l


From daniel.wolff at gmail.com  Sat Oct  4 14:14:06 2008
From: daniel.wolff at gmail.com (Daniel Wolff)
Date: Sun, 5 Oct 2008 02:14:06 +0800
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on window
Message-ID: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>

Hello, I recently upgraded to python 2.6. I cannot seem to find the
appropriate numpy install file. I tried to make use of the .exe built for
python 2.5 but the installer hung. Any suggestions? Regards, Daniel 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081005/01e0d477/attachment.html>

From robert.kern at gmail.com  Sat Oct  4 19:28:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 4 Oct 2008 18:28:01 -0500
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on
	window
In-Reply-To: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
Message-ID: <3d375d730810041628q8b7a68eg25cccf2cab2ef9ee@mail.gmail.com>

On Sat, Oct 4, 2008 at 13:14, Daniel Wolff <daniel.wolff at gmail.com> wrote:
> Hello, I recently upgraded to python 2.6. I cannot seem to find the
> appropriate numpy install file. I tried to make use of the .exe built for
> python 2.5 but the installer hung. Any suggestions? Regards, Daniel

Right, you can't use a binary installer meant for Python 2.5 with
Python 2.6. The binaries aren't compatible. If you do not want to
build from source, you will have to wait for the volunteer
binary-builders to get around to making a Python 2.6 installer.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From alan.mcintyre at gmail.com  Sat Oct  4 19:28:13 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sat, 4 Oct 2008 16:28:13 -0700
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on
	window
In-Reply-To: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
Message-ID: <1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>

On Sat, Oct 4, 2008 at 11:14 AM, Daniel Wolff <daniel.wolff at gmail.com> wrote:
> Hello, I recently upgraded to python 2.6. I cannot seem to find the
> appropriate numpy install file. I tried to make use of the .exe built for
> python 2.5 but the installer hung. Any suggestions? Regards, Daniel

Hi Daniel,

Last I heard, there are still some things that need to be ironed out
for NumPy on Python 2.6, so you will probably have to stick with 2.5
for a while.  (Somebody please correct me if I'm wrong and there's an
imminent release for 2.6 coming up ;)

Cheers,
Alan


From david at ar.media.kyoto-u.ac.jp  Sun Oct  5 04:25:39 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 05 Oct 2008 17:25:39 +0900
Subject: [Numpy-discussion] Merged clean_math_config branch
Message-ID: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>

Hi there,

    Just to mention that I merged back my changes from the
clean_math_config branch into trunk. The main point of the branch is to
clean our math configuration. If this causes problems, please report it.
I  built and tested on mac os x, linux 32 bits and windows (both mingw32
and VS 2003). It breaks windows 64 bits ATM, but this will be fixed
soon. The numscons built is broken as well, but the missing features are
already backported from numpy.distutils to numscons; a new working
version of numscons is about to be released.

Some details for the record:
    - more code is platform independent, with platform specifics math
functions at one location
    - instead of platform-specific heuristics, every function is tested
at the configuration stage. Some distutils.command.config
functionalities were improved to make this process as fast as before for
relatively standard-compliant platforms (mac os X and linux are as fast
to build as before).
    - the distutils.command.config.check_func can now detect MS
intrinsics (functions which are not visible without optimization flags).
    - should also be more robust for non standard platforms (MS
compilers, mostly).

In the process, a few hundred lines of C code have been removed.
cheers,

David


From dmitrey.kroshko at scipy.org  Sun Oct  5 05:31:51 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Sun, 05 Oct 2008 12:31:51 +0300
Subject: [Numpy-discussion] asscalar(number) - why yields error,
	why can't return the number?!
Message-ID: <48E88987.8010905@scipy.org>

hi all,
I wonder why numpy.asscalar(1.5) yields error, why it can't just return 
1.5? Is it intended to be ever changed?

 >>> numpy.__version__
'1.3.0.dev5864'

D.


From hanni.ali at gmail.com  Sun Oct  5 05:38:19 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Sun, 5 Oct 2008 10:38:19 +0100
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on
	window
In-Reply-To: <1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
	<1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>
Message-ID: <789d27b10810050238h21776111l910838cfb2fa0657@mail.gmail.com>

Hi Alan,

I've been using numpy with 2.6 on amd64 for about 3 months no problem and
far more stable on 64-bit than the same combination with 2.5.

Daniel, just download the source and compile it yourself. You need to use VS
9.0 for best compatibility with Python and I haven't yet sorted out fast
blas I've just been using the internal numpy one.

Hanni



2008/10/5 Alan McIntyre <alan.mcintyre at gmail.com>

> On Sat, Oct 4, 2008 at 11:14 AM, Daniel Wolff <daniel.wolff at gmail.com>
> wrote:
> > Hello, I recently upgraded to python 2.6. I cannot seem to find the
> > appropriate numpy install file. I tried to make use of the .exe built for
> > python 2.5 but the installer hung. Any suggestions? Regards, Daniel
>
> Hi Daniel,
>
> Last I heard, there are still some things that need to be ironed out
> for NumPy on Python 2.6, so you will probably have to stick with 2.5
> for a while.  (Somebody please correct me if I'm wrong and there's an
> imminent release for 2.6 coming up ;)
>
> Cheers,
> Alan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081005/160e704c/attachment.html>

From cournape at gmail.com  Sun Oct  5 05:42:09 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 5 Oct 2008 18:42:09 +0900
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on
	window
In-Reply-To: <1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
	<1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>
Message-ID: <5b8d13220810050242u7a221a58uf087c18150db6bc6@mail.gmail.com>

On Sun, Oct 5, 2008 at 8:28 AM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Sat, Oct 4, 2008 at 11:14 AM, Daniel Wolff <daniel.wolff at gmail.com> wrote:
>> Hello, I recently upgraded to python 2.6. I cannot seem to find the
>> appropriate numpy install file. I tried to make use of the .exe built for
>> python 2.5 but the installer hung. Any suggestions? Regards, Daniel
>
> Hi Daniel,
>
> Last I heard, there are still some things that need to be ironed out
> for NumPy on Python 2.6, so you will probably have to stick with 2.5
> for a while.  (Somebody please correct me if I'm wrong and there's an
> imminent release for 2.6 coming up ;)

One thing which was a relatively major stopper was nose, our test
infrastructure, was not 2.6 compatible, but that has just changed. I
will look on windows to see if 1.2 can be built with 2008 express,

cheers,

David


From hanni.ali at gmail.com  Sun Oct  5 08:08:48 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Sun, 5 Oct 2008 13:08:48 +0100
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6 on
	window
In-Reply-To: <5b8d13220810050242u7a221a58uf087c18150db6bc6@mail.gmail.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>
	<1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>
	<5b8d13220810050242u7a221a58uf087c18150db6bc6@mail.gmail.com>
Message-ID: <789d27b10810050508s72c04c5fwac308005bccca45e@mail.gmail.com>

Hi David,

Yeah nose was an issue, thanks for letting me know about nose compatibility
being sorted that's good news as I use it a bit with my testing.

I last built 1.1.1 with express for 32bit it went fine (although I think you
do have to comment out , I was also able to compile with 2008 full for
64-bit platform, my next goal is to see if i can sort out blas etc. external
libs on 64bit.

>From the "Installation info" thread around the 30th May I had reported the
one oddity I  had encountered with the VS compiler

> In order to get numpy to compile I
> have commented out a small part which was causing compilation to fail:
>
> numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
> numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
>
> This relates to this section of code:
>
> #ifndef HAVE_FREXPF
> static float frexpf(float x, int * i)
> {
>     return (float)frexp((double)(x), i);
> }
> #endif
> #ifndef HAVE_LDEXPF
> static float ldexpf(float x, int i)
> {
>     return (float)ldexp((double)(x), i);
> }
> #endif

At the time I had tried to send further output following a checkout, but
couldn't get it to post to the list, I think the message was too big or
something. I will probably be having a go with 1.2.0, when I get some time.
I'll let you know how it goes.

Cheers,

Hanni


2008/10/5 David Cournapeau <cournape at gmail.com>

> On Sun, Oct 5, 2008 at 8:28 AM, Alan McIntyre <alan.mcintyre at gmail.com>
> wrote:
> > On Sat, Oct 4, 2008 at 11:14 AM, Daniel Wolff <daniel.wolff at gmail.com>
> wrote:
> >> Hello, I recently upgraded to python 2.6. I cannot seem to find the
> >> appropriate numpy install file. I tried to make use of the .exe built
> for
> >> python 2.5 but the installer hung. Any suggestions? Regards, Daniel
> >
> > Hi Daniel,
> >
> > Last I heard, there are still some things that need to be ironed out
> > for NumPy on Python 2.6, so you will probably have to stick with 2.5
> > for a while.  (Somebody please correct me if I'm wrong and there's an
> > imminent release for 2.6 coming up ;)
>
> One thing which was a relatively major stopper was nose, our test
> infrastructure, was not 2.6 compatible, but that has just changed. I
> will look on windows to see if 1.2 can be built with 2008 express,
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081005/90ca2170/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Sun Oct  5 09:07:10 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 05 Oct 2008 22:07:10 +0900
Subject: [Numpy-discussion] cannot find numpy 1.2 for python 2.6
	on	window
In-Reply-To: <789d27b10810050508s72c04c5fwac308005bccca45e@mail.gmail.com>
References: <48e7b281.044e6e0a.2ae2.2e7c@mx.google.com>	<1d36917a0810041628v50464128wfe49ff8d7d915f8b@mail.gmail.com>	<5b8d13220810050242u7a221a58uf087c18150db6bc6@mail.gmail.com>
	<789d27b10810050508s72c04c5fwac308005bccca45e@mail.gmail.com>
Message-ID: <48E8BBFE.1030808@ar.media.kyoto-u.ac.jp>

Hanni Ali wrote:
> Hi David,
>
> Yeah nose was an issue, thanks for letting me know about nose
> compatibility being sorted that's good news as I use it a bit with my
> testing.
>
> I last built 1.1.1 with express for 32bit it went fine (although I
> think you do have to comment out , I was also able to compile with
> 2008 full for 64-bit platform, my next goal is to see if i can sort
> out blas etc. external libs on 64bit.

That's the major difficulty: no open source blas/lapack (that I am aware
of) are buildable with MS compilers, and there is no official open
source compiler (mingw) on 64 bits. Also, cygwin itself is not available
in 64 bits (only available on WOW, the windows subsystem to run 32 bits
on 64 bits OS).

Atlas, in particular, is not buildable for 64 bits windows AFAIK.
>
>
> >
> > #ifndef HAVE_FREXPF
> > static float frexpf(float x, int * i)
> > {
> >     return (float)frexp((double)(x), i);
> > }
> > #endif
> > #ifndef HAVE_LDEXPF
> > static float ldexpf(float x, int i)
> > {
> >     return (float)ldexp((double)(x), i);
> > }
> > #endif
>
> At the time I had tried to send further output following a checkout,
> but couldn't get it to post to the list, I think the message was too
> big or something. I will probably be having a go with 1.2.0, when I
> get some time. I'll let you know how it goes.

I did some heavy refactoring for the above problems, and it should be
now easier to handle (in the trunk). I could build 1.2.0 with VS 2008
express on 32 bits (wo blas/lapack), and there are some test errors -
albeit relatively minor at first sight. I have not tried on 64 bits.

cheers,

David


From tjhnson at gmail.com  Sun Oct  5 16:53:22 2008
From: tjhnson at gmail.com (T J)
Date: Sun, 5 Oct 2008 13:53:22 -0700
Subject: [Numpy-discussion] Test failures on 2.6
Message-ID: <b02c60870810051353w602daa99nc1dac1b5f3e1eab9@mail.gmail.com>

Hi,

I'm getting a couple of test failures with Python 2.6, Numpy 1.2.0, Nose 0.10.4:



nose version 0.10.4
..........................................................................................................................................................................................................................................................................................................................................................................................................................................F................K........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:68:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(c.readlines(),
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./share/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1315:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(store._mask, True)
/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1322:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(store._mask, True)
/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py:1989:
SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(test.mask, [0,1,0,0,0,0,0,0,0,0])
...............................................E................................................................................................................................................................................
======================================================================
ERROR: Tests the min/max functions with explicit outputs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/tests/test_core.py",
line 653, in test_minmax_funcs_with_output
    result = npfunc(xm,axis=0,out=nout)
  File "/home/me/usr/lib/python2.6/site-packages/numpy/core/fromnumeric.py",
line 1525, in amin
    return amin(axis, out)
  File "/home/me/usr/lib/python2.6/site-packages/numpy/ma/core.py",
line 2978, in min
    np.putmask(out, newmask, np.nan)
ValueError: cannot convert float NaN to integer

======================================================================
FAIL: test_umath.TestComplexFunctions.test_against_cmath
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/me/usr/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/nose/case.py",
line 182, in runTest
    self.test(*self.arg)
  File "/home/me/usr/lib/python2.6/site-packages/numpy/core/tests/test_umath.py",
line 268, in test_against_cmath
    assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b)
AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath:
(1.57079632679+1.31695789692j)

----------------------------------------------------------------------
Ran 1726 tests in 8.856s

FAILED (KNOWNFAIL=1, errors=1, failures=1)
<nose.result.TextTestResult run=1726 errors=1 failures=1>


From millman at berkeley.edu  Sun Oct  5 17:57:30 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 5 Oct 2008 14:57:30 -0700
Subject: [Numpy-discussion] Merged clean_math_config branch
In-Reply-To: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>
References: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550810051457m7532fae1m8502e8484ec70801@mail.gmail.com>

On Sun, Oct 5, 2008 at 1:25 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
>    Just to mention that I merged back my changes from the
> clean_math_config branch into trunk. The main point of the branch is to
> clean our math configuration. If this causes problems, please report it.
> I  built and tested on mac os x, linux 32 bits and windows (both mingw32
> and VS 2003). It breaks windows 64 bits ATM, but this will be fixed
> soon. The numscons built is broken as well, but the missing features are
> already backported from numpy.distutils to numscons; a new working
> version of numscons is about to be released.

Excellent.  Thanks for working on this.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From oliphant at enthought.com  Sun Oct  5 18:17:59 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sun, 05 Oct 2008 17:17:59 -0500
Subject: [Numpy-discussion] Merged clean_math_config branch
In-Reply-To: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>
References: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>
Message-ID: <48E93D17.2000207@enthought.com>

David Cournapeau wrote:
> Hi there,
>
>     Just to mention that I merged back my changes from the
> clean_math_config branch into trunk. The main point of the branch is to
> clean our math configuration. If this causes problems, please report it.
> I  built and tested on mac os x, linux 32 bits and windows (both mingw32
> and VS 2003). It breaks windows 64 bits ATM, but this will be fixed
> soon. The numscons built is broken as well, but the missing features are
> already backported from numpy.distutils to numscons; a new working
> version of numscons is about to be released.
>   
This is a really good thing and a lot of work.  Thank you, David for 
doing all the heavy lifting.

-Travis



From millman at berkeley.edu  Sun Oct  5 22:59:28 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 5 Oct 2008 19:59:28 -0700
Subject: [Numpy-discussion] Please backport fixes to the 1.2.x branch
Message-ID: <c7009a550810051959n23d45f55ja06416cb1ae15b62@mail.gmail.com>

Hello,

I would like to get a 1.2.1 release out ASAP.  There are several
bug-fixes on the trunk that need to be backported.  If you have made a
bug-fix to the trunk that you have been waiting to backport to the
1.2.x branch, please do so now:
http://svn.scipy.org/svn/numpy/branches/1.2.x

Ideally, I would like to freeze the branch for the 1.2.1 release in
about 1 week.  Please let me know if you need more time or if there is
something in particular that you would like to see backported.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From roeland at huysmus.be  Mon Oct  6 04:00:25 2008
From: roeland at huysmus.be (Roeland Huys)
Date: Mon, 6 Oct 2008 10:00:25 +0200 (CEST)
Subject: [Numpy-discussion] install without admin rights
Message-ID: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>


Hi,

I am very interested in my work to switch from Matlab to Numpy.  However,
I want to test some algorithms first.
I tried to install NumPy on one of our corporate servers.  However, I do
not have root access, so i did following:

tar -xvzf numpy-1.2.0.tar.gz   [OK]
cd numpy-1.2.0
python setup.py build   [OK]
set PYTHONPATH=$HOME/pythonlib
mv numpy ~/pythonlib
python
import numpy   [ FAILS!]

Could you tell me how I can install NumPy on my local account, (without
admin rights) ?

Thanks,

Roeland.






From matthieu.brucher at gmail.com  Mon Oct  6 04:10:27 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 6 Oct 2008 10:10:27 +0200
Subject: [Numpy-discussion] install without admin rights
In-Reply-To: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
References: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
Message-ID: <e76aa17f0810060110l49c26ebdt537444249298e1bc@mail.gmail.com>

python setup.py install --prefix=$HOME/local/lib
set PYTHONPATH=$HOME/local/lib/python2.5/site-packages

and you're good to go. This is how I install everything and it's been
working for several years now ;)

Matthieu

2008/10/6 Roeland Huys <roeland at huysmus.be>:
>
> Hi,
>
> I am very interested in my work to switch from Matlab to Numpy.  However,
> I want to test some algorithms first.
> I tried to install NumPy on one of our corporate servers.  However, I do
> not have root access, so i did following:
>
> tar -xvzf numpy-1.2.0.tar.gz   [OK]
> cd numpy-1.2.0
> python setup.py build   [OK]
> set PYTHONPATH=$HOME/pythonlib
> mv numpy ~/pythonlib
> python
> import numpy   [ FAILS!]
>
> Could you tell me how I can install NumPy on my local account, (without
> admin rights) ?
>
> Thanks,
>
> Roeland.
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, PhD
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From stefan at sun.ac.za  Mon Oct  6 04:27:46 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 6 Oct 2008 10:27:46 +0200
Subject: [Numpy-discussion] install without admin rights
In-Reply-To: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
References: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
Message-ID: <9457e7c80810060127o5246d16bub23b4822729a64ad@mail.gmail.com>

Hi Roeland

2008/10/6 Roeland Huys <roeland at huysmus.be>:
> tar -xvzf numpy-1.2.0.tar.gz   [OK]
> cd numpy-1.2.0
> python setup.py build   [OK]

python setup.py install --prefix=${HOME}

> set PYTHONPATH=$HOME/pythonlib

export PYTHONPATH=${HOME}/lib/python2.5/site-packages

Cheers
St?fan


From roeland at huysmus.be  Mon Oct  6 05:23:08 2008
From: roeland at huysmus.be (Roeland Huys)
Date: Mon, 6 Oct 2008 11:23:08 +0200 (CEST)
Subject: [Numpy-discussion] install without admin rights
In-Reply-To: <9457e7c80810060127o5246d16bub23b4822729a64ad@mail.gmail.com>
References: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
	<9457e7c80810060127o5246d16bub23b4822729a64ad@mail.gmail.com>
Message-ID: <a94e4935beaef3a625ce47a6f0e8d6e7.squirrel@webmail01.one.com>

Thanks!

Unfortunately, I got following error:

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File
"/imec/other/neuray2/lib64/python2.3/site-packages/numpy/__init__.py",
line 125, in ?
    import add_newdocs
  File
"/imec/other/neuray2/lib64/python2.3/site-packages/numpy/add_newdocs.py",
line 9, in ?
    from lib import add_newdoc
  File
"/imec/other/neuray2/lib64/python2.3/site-packages/numpy/lib/__init__.py",
line 22, in ?
    from arrayterator import *
  File "numpy/lib/arrayterator.py", line 89
    slice_ = tuple(slice(*t) for t in zip(
                               ^
SyntaxError: invalid syntax


I have python 2.3 installed on the server.

Thanks,

Roeland






On Mon, October 6, 2008 10:27 am, St?fan van der Walt wrote:
> Hi Roeland
>
> 2008/10/6 Roeland Huys <roeland at huysmus.be>:
>> tar -xvzf numpy-1.2.0.tar.gz   [OK]
>> cd numpy-1.2.0
>> python setup.py build   [OK]
>
> python setup.py install --prefix=${HOME}
>
>> set PYTHONPATH=$HOME/pythonlib
>
> export PYTHONPATH=${HOME}/lib/python2.5/site-packages
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


-- 
------------------------------------
Roeland Huys
http://www.huysmus.be

http://en.nothingisreal.com/wiki/Please_don't_send_me_Microsoft_Word_documents
http://badvista.fsf.org/



From matthieu.brucher at gmail.com  Mon Oct  6 05:26:27 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 6 Oct 2008 11:26:27 +0200
Subject: [Numpy-discussion] install without admin rights
In-Reply-To: <a94e4935beaef3a625ce47a6f0e8d6e7.squirrel@webmail01.one.com>
References: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>
	<9457e7c80810060127o5246d16bub23b4822729a64ad@mail.gmail.com>
	<a94e4935beaef3a625ce47a6f0e8d6e7.squirrel@webmail01.one.com>
Message-ID: <e76aa17f0810060226y58ecc544l9d5b15ea66124c23@mail.gmail.com>

Hi again,

That's because Python 2.3 is not supported by numpy 1.2 anymore (see
the release notes) :|
Try installing a Python 2.5 locally, as with numpy.

Matthieu

2008/10/6 Roeland Huys <roeland at huysmus.be>:
> Thanks!
>
> Unfortunately, I got following error:
>
>>>> import numpy
> Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File
> "/imec/other/neuray2/lib64/python2.3/site-packages/numpy/__init__.py",
> line 125, in ?
>    import add_newdocs
>  File
> "/imec/other/neuray2/lib64/python2.3/site-packages/numpy/add_newdocs.py",
> line 9, in ?
>    from lib import add_newdoc
>  File
> "/imec/other/neuray2/lib64/python2.3/site-packages/numpy/lib/__init__.py",
> line 22, in ?
>    from arrayterator import *
>  File "numpy/lib/arrayterator.py", line 89
>    slice_ = tuple(slice(*t) for t in zip(
>                               ^
> SyntaxError: invalid syntax
>
>
> I have python 2.3 installed on the server.
>
> Thanks,
>
> Roeland
>
>
>
>
>
>
> On Mon, October 6, 2008 10:27 am, St?fan van der Walt wrote:
>> Hi Roeland
>>
>> 2008/10/6 Roeland Huys <roeland at huysmus.be>:
>>> tar -xvzf numpy-1.2.0.tar.gz   [OK]
>>> cd numpy-1.2.0
>>> python setup.py build   [OK]
>>
>> python setup.py install --prefix=${HOME}
>>
>>> set PYTHONPATH=$HOME/pythonlib
>>
>> export PYTHONPATH=${HOME}/lib/python2.5/site-packages
>>
>> Cheers
>> St?fan
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> --
> ------------------------------------
> Roeland Huys
> http://www.huysmus.be
>
> http://en.nothingisreal.com/wiki/Please_don't_send_me_Microsoft_Word_documents
> http://badvista.fsf.org/
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, PhD
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From david at ar.media.kyoto-u.ac.jp  Mon Oct  6 08:14:40 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 06 Oct 2008 21:14:40 +0900
Subject: [Numpy-discussion] install without admin rights
In-Reply-To: <e76aa17f0810060226y58ecc544l9d5b15ea66124c23@mail.gmail.com>
References: <4502aff9450d71c9649d171c6e7e9c91.squirrel@webmail01.one.com>	<9457e7c80810060127o5246d16bub23b4822729a64ad@mail.gmail.com>	<a94e4935beaef3a625ce47a6f0e8d6e7.squirrel@webmail01.one.com>
	<e76aa17f0810060226y58ecc544l9d5b15ea66124c23@mail.gmail.com>
Message-ID: <48EA0130.6090808@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> Hi again,
>
> That's because Python 2.3 is not supported by numpy 1.2 anymore (see
> the release notes) :|
> Try installing a Python 2.5 locally, as with numpy.
>   

Alternatively, you can install numpy 1.1.x (numpy 1.1* still support
python 2.3).

cheers,

David


From cournape at gmail.com  Mon Oct  6 12:02:36 2008
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 7 Oct 2008 01:02:36 +0900
Subject: [Numpy-discussion] Merged clean_math_config branch
In-Reply-To: <48E93D17.2000207@enthought.com>
References: <48E87A03.1020603@ar.media.kyoto-u.ac.jp>
	<48E93D17.2000207@enthought.com>
Message-ID: <5b8d13220810060902w739a3bdet1f24e0cad47921ce@mail.gmail.com>

On Mon, Oct 6, 2008 at 7:17 AM, Travis E. Oliphant
<oliphant at enthought.com> wrote:
>>
> This is a really good thing and a lot of work.  Thank you, David for
> doing all the heavy lifting.

Thanks. That's really a first step, though, as the long-term goal is
to start support of dynamically loaded optimzed SIMD (SSE and co)
code.

cheers,

David


From john at saponara.net  Mon Oct  6 15:32:55 2008
From: john at saponara.net (John)
Date: Mon, 06 Oct 2008 15:32:55 -0400
Subject: [Numpy-discussion] multi-comparison expressions
Message-ID: <48EA67E7.1090900@saponara.net>

hi,

why does the ValueError appear below, and how can i make that 2<a<5 
expression work when a is an array?

thanks.

 >>> from numpy import reshape,arange
 >>> a=reshape(arange(9),(3,3))
 >>> a
array([[0, 1, 2],
      [3, 4, 5],
      [6, 7, 8]])
 >>> 2<a
array([[False, False, False],
      [ True,  True,  True],
      [ True,  True,  True]], dtype=bool)
 >>> a<5
array([[ True,  True,  True],
      [ True,  True, False],
      [False, False, False]], dtype=bool)
 >>> 2<a<5
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()

nor does it work with constant arrays:

 >>> from numpy import zeros
 >>> twos=zeros(a.shape)+2
 >>> fives=zeros(a.shape)+5
 >>> twos<a
array([[False, False, False],
      [ True,  True,  True],
      [ True,  True,  True]], dtype=bool)
 >>> a<fives
array([[ True,  True,  True],
      [ True,  True, False],
      [False, False, False]], dtype=bool)
 >>> twos<a<fives
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()

but it works with builtin numbers:

 >>> b=3
 >>> 2<b<5
True




From nadavh at visionsense.com  Mon Oct  6 15:49:20 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Mon, 6 Oct 2008 21:49:20 +0200
Subject: [Numpy-discussion] multi-comparison expressions
References: <48EA67E7.1090900@saponara.net>
Message-ID: <710F2847B0018641891D9A216027636029C2C2@ex3.envision.co.il>

a < b < c 

is equivalent to:

(a < b) and (b < c)

If you wand an element-wise operation you have to use the & operator, so the expression is:

(a > 2) & (a < 5)

   Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? John
????: ? 06-???????-08 21:32
??: numpy-discussion at scipy.org
????: [Numpy-discussion] multi-comparison expressions
 
hi,

why does the ValueError appear below, and how can i make that 2<a<5 
expression work when a is an array?

thanks.

 >>> from numpy import reshape,arange
 >>> a=reshape(arange(9),(3,3))
 >>> a
array([[0, 1, 2],
      [3, 4, 5],
      [6, 7, 8]])
 >>> 2<a
array([[False, False, False],
      [ True,  True,  True],
      [ True,  True,  True]], dtype=bool)
 >>> a<5
array([[ True,  True,  True],
      [ True,  True, False],
      [False, False, False]], dtype=bool)
 >>> 2<a<5
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()

nor does it work with constant arrays:

 >>> from numpy import zeros
 >>> twos=zeros(a.shape)+2
 >>> fives=zeros(a.shape)+5
 >>> twos<a
array([[False, False, False],
      [ True,  True,  True],
      [ True,  True,  True]], dtype=bool)
 >>> a<fives
array([[ True,  True,  True],
      [ True,  True, False],
      [False, False, False]], dtype=bool)
 >>> twos<a<fives
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()

but it works with builtin numbers:

 >>> b=3
 >>> 2<b<5
True


_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3376 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081006/9533aac9/attachment.bin>

From robert.kern at gmail.com  Mon Oct  6 15:59:34 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 6 Oct 2008 14:59:34 -0500
Subject: [Numpy-discussion] multi-comparison expressions
In-Reply-To: <48EA67E7.1090900@saponara.net>
References: <48EA67E7.1090900@saponara.net>
Message-ID: <3d375d730810061259r311ccea4s258ed8962f9889b3@mail.gmail.com>

On Mon, Oct 6, 2008 at 14:32, John <john at saponara.net> wrote:
> hi,
>
> why does the ValueError appear below, and how can i make that 2<a<5
> expression work when a is an array?

(2<a<5) evaluates to ((2<a) and (a<5)). The "and" keyword coerces its
arguments to actual boolean True or False objects, so this is actually
(bool(2<a) and bool(a<5)). Arrays implement rich comparisons, so if
'a' is an array, then (2<a) is an array of bools. bool(some_array) is
ambiguous; sometimes you want it to be True if any of the elements are
True and sometimes you only want it to be True if all of the elements
are True. Consequently, numpy refuses to guess what you want and makes
you specify with the .any() and .all() methods. Unfortunately, this
means that (2<a<5) will never work for arrays.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Mon Oct  6 16:02:48 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 06 Oct 2008 13:02:48 -0700
Subject: [Numpy-discussion] multi-comparison expressions
In-Reply-To: <48EA67E7.1090900@saponara.net>
References: <48EA67E7.1090900@saponara.net>
Message-ID: <48EA6EE8.9000606@noaa.gov>

John wrote:
> hi,
> 
> why does the ValueError appear below,

because python short circuits this expression, and numpy can't overwrite 
that -- the same reason a and b doesn't work for a and b arrays.

 > and how can i make that 2<a<5
> expression work when a is an array?

 >>> (a > 2) & (a < 5)

array([[False, False, False],
        [ True,  True, False],
        [False, False, False]], dtype=bool)

& is the bitwise-and, which is overridden by numpy, and works like and 
for boolean data.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Mon Oct  6 16:58:12 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 6 Oct 2008 14:58:12 -0600
Subject: [Numpy-discussion] multi-comparison expressions
In-Reply-To: <3d375d730810061259r311ccea4s258ed8962f9889b3@mail.gmail.com>
References: <48EA67E7.1090900@saponara.net>
	<3d375d730810061259r311ccea4s258ed8962f9889b3@mail.gmail.com>
Message-ID: <e06186140810061358y1ddcf558h7c3cb519fbbc59aa@mail.gmail.com>

On Mon, Oct 6, 2008 at 1:59 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Oct 6, 2008 at 14:32, John <john at saponara.net> wrote:
> > hi,
> >
> > why does the ValueError appear below, and how can i make that 2<a<5
> > expression work when a is an array?
>
> (2<a<5) evaluates to ((2<a) and (a<5)). The "and" keyword coerces its
> arguments to actual boolean True or False objects, so this is actually
> (bool(2<a) and bool(a<5)). Arrays implement rich comparisons, so if
> 'a' is an array, then (2<a) is an array of bools. bool(some_array) is
> ambiguous; sometimes you want it to be True if any of the elements are
> True and sometimes you only want it to be True if all of the elements
> are True. Consequently, numpy refuses to guess what you want and makes
> you specify with the .any() and .all() methods. Unfortunately, this
> means that (2<a<5) will never work for arrays.
>

This reminds me that if we add matrix operators to python it would also be
nice to have some for the logical operators.  Having && and || would be
handy.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081006/46264fb0/attachment.html>

From nadavh at visionsense.com  Mon Oct  6 23:45:44 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Tue, 7 Oct 2008 05:45:44 +0200
Subject: [Numpy-discussion] Scipy in Python 2.6
References: <df39e47d0810061545w4992ce53l6df5b179ace0350f@mail.gmail.com>
Message-ID: <710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>


Scipy from svn compiles with python2.6 with no problem. Problems that came later:
1. Could not install packages of scikits: I got a strange error about unknown name: log
2. When trying to run from "idle" warnings become errors. I think that is a bug of idle.

  Nadav.

-----????? ??????-----
???: jah [mailto:jah.mailinglist at gmail.com]
????: ? 07-???????-08 00:45
??: Nadav Horesh
????: Scipy in Python 2.6
 
Hi Nadav,

I saw:

http://projects.scipy.org/pipermail/numpy-discussion/2008-July/035968.html

where you reported some compilation problems with scipy in Python 2.6.  Did
you resolve this issue?  Do you have any knowledge if the current svn still
has this problem?

Thanks



From robert.kern at gmail.com  Mon Oct  6 23:52:28 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 6 Oct 2008 22:52:28 -0500
Subject: [Numpy-discussion] Scipy in Python 2.6
In-Reply-To: <710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>
References: <df39e47d0810061545w4992ce53l6df5b179ace0350f@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>
Message-ID: <3d375d730810062052o4ff9de49q183de24d27b4561a@mail.gmail.com>

2008/10/6 Nadav Horesh <nadavh at visionsense.com>:
>
> Scipy from svn compiles with python2.6 with no problem. Problems that came later:
> 1. Could not install packages of scikits: I got a strange error about unknown name: log

It's a bug in the version of setuptools that you have. I believe it
has been fixed in the latest release. But please, when you report an
error message, copy-and-paste the traceback rather than paraphrasing
it.

> 2. When trying to run from "idle" warnings become errors. I think that is a bug of idle.

Probably.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Tue Oct  7 01:08:24 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 7 Oct 2008 07:08:24 +0200
Subject: [Numpy-discussion] Scipy in Python 2.6
In-Reply-To: <710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>
References: <df39e47d0810061545w4992ce53l6df5b179ace0350f@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>
Message-ID: <20081007050824.GA8091@phare.normalesup.org>

On Tue, Oct 07, 2008 at 05:45:44AM +0200, Nadav Horesh wrote:

> Scipy from svn compiles with python2.6 with no problem. Problems that came later:
> 1. Could not install packages of scikits: I got a strange error about unknown name: log

Changes are this is a setuptools problem. Could you post the traceback?
In which case upgrade to latest release: 0.6c9, should fix the problem.

Ga?l


From nadavh at visionsense.com  Tue Oct  7 03:15:12 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Tue, 07 Oct 2008 09:15:12 +0200
Subject: [Numpy-discussion] Scipy in Python 2.6
In-Reply-To: <3d375d730810062052o4ff9de49q183de24d27b4561a@mail.gmail.com>
References: <df39e47d0810061545w4992ce53l6df5b179ace0350f@mail.gmail.com>
	<710F2847B0018641891D9A216027636029C2C3@ex3.envision.co.il>
	<3d375d730810062052o4ff9de49q183de24d27b4561a@mail.gmail.com>
Message-ID: <1223363712.18980.3.camel@nadav.envision.co.il>

1. The latest release of setuptools solved the problem of scikits
2. Idle error log example:

>>> from pylab import *

Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    from pylab import *
  File "/usr/local/lib/python2.6/site-packages/pylab.py", line 1, in
<module>
    from matplotlib.pylab import *
  File "/usr/local/lib/python2.6/site-packages/matplotlib/__init__.py",
line 96, in <module>
    import md5, os, re, shutil, sys, warnings
  File "/usr/local/lib/python2.6/md5.py", line 8, in <module>
    DeprecationWarning, 2)
  File "/usr/local/lib/python2.6/warnings.py", line 29, in _show_warning
    file.write(formatwarning(message, category, filename, lineno, line))
TypeError: idle_formatwarning_subproc() takes exactly 4 arguments (5
given)

3. I have the feeling that python2.6 goes in the way of python 2.2 that
had to pass two bug-fix revisions to be really stable.

  Nadav.


On Mon, 2008-10-06 at 22:52 -0500, Robert Kern wrote:

> 2008/10/6 Nadav Horesh <nadavh at visionsense.com>:
> >
> > Scipy from svn compiles with python2.6 with no problem. Problems that came later:
> > 1. Could not install packages of scikits: I got a strange error about unknown name: log
> 
> It's a bug in the version of setuptools that you have. I believe it
> has been fixed in the latest release. But please, when you report an
> error message, copy-and-paste the traceback rather than paraphrasing
> it.
> 
> > 2. When trying to run from "idle" warnings become errors. I think that is a bug of idle.
> 
> Probably.
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081007/e652e18f/attachment.html>

From plucek at ssaris.com  Tue Oct  7 13:42:30 2008
From: plucek at ssaris.com (Paul Lucek)
Date: Tue, 7 Oct 2008 13:42:30 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
Message-ID: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>

I get the following errors in umathmodule.c.src:

 

D:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe
/c /nolog

o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
-Inumpy\cor

e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
-Inumpy\core\src -I

numpy\core\include -Id:\python26\include -Id:\python26\PC
/Tcbuild\src.win-amd64

-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Release\build\src.

win-amd64-2.6\numpy\core\src\umathmodule.obj

umathmodule.c

numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error :
'type'

numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error :
'type'

numpy\core\src\ufuncobject.c(1701) : warning C4244: '=' : conversion
from 'npy_i

ntp' to 'int', possible loss of data

numpy\core\src\ufuncobject.c(2422) : warning C4244: '=' : conversion
from 'npy_i

ntp' to 'int', possible loss of data

error: Command "D:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\amd64\

cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
-Ibuild\src.win-amd64-2.6\numpy\core

\src -Inumpy\core\include
-Ibuild\src.win-amd64-2.6\numpy\core\include/numpy -In

umpy\core\src -Inumpy\core\include -Id:\python26\include
-Id:\python26\PC /Tcbui

ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
/Fobuild\temp.win-amd64-2.6\Re

lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed
with exit s

tatus 2

 

 

 

 

 

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081007/ff9fbf0f/attachment.html>

From charlesr.harris at gmail.com  Tue Oct  7 14:19:21 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 7 Oct 2008 12:19:21 -0600
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
In-Reply-To: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
Message-ID: <e06186140810071119x7f8d0656x70794cbee328e256@mail.gmail.com>

On Tue, Oct 7, 2008 at 11:42 AM, Paul Lucek <plucek at ssaris.com> wrote:

>  I get the following errors in umathmodule.c.src:
>
>
>
> D:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
> /nolog
>
> o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
> -Inumpy\cor
>
> e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
> -Inumpy\core\src -I
>
> numpy\core\include -Id:\python26\include -Id:\python26\PC
> /Tcbuild\src.win-amd64
>
> -2.6\numpy\core\src\umathmodule.c
> /Fobuild\temp.win-amd64-2.6\Release\build\src.
>
> win-amd64-2.6\numpy\core\src\umathmodule.obj
>
> Could you compress win-amd64-2.6\numpy\core\src\umathmodule.c -- it should
be in the build directory -- and attach it if possible, or at least that
part that seems to be a problem? Mind that the list has a rather small size
limit.

> umathmodule.c
>
> numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
>
> numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
>
> numpy\core\src\ufuncobject.c(1701) : warning C4244: '=' : conversion from
> 'npy_i
>
> ntp' to 'int', possible loss of data
>
> numpy\core\src\ufuncobject.c(2422) : warning C4244: '=' : conversion from
> 'npy_i
>
> ntp' to 'int', possible loss of data
>
> The warnings are known.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081007/c37d4c2e/attachment.html>

From paultaney at yahoo.com  Tue Oct  7 14:28:43 2008
From: paultaney at yahoo.com (paul taney)
Date: Tue, 7 Oct 2008 11:28:43 -0700 (PDT)
Subject: [Numpy-discussion] collecting the bluest pixels
Message-ID: <813570.19734.qm@web53905.mail.re2.yahoo.com>

Hi,

I have this silly color filter that Stefan gave me:


def vanderwalt(image, f):
    """colorfilter, thanks to Stefan van der Walt"""
    RED, GRN, BLU = 0, 1, 2
    bluemask = (image[...,BLU] > f*image[...,GRN]) & \
               (image[...,BLU] > f*image[...,RED])

    return bluemask


To collect the right number of the bluest pixels I am calling it from this arduous successive approximation routine.  It occured to me that someone on this list knows how to do this in a couple of lines...


def successive_approximation(image, density, width, height, bpp):
    """keep calling vanderwalt till line length is within 10% of density target"""
    count = 0
    failsafe = 20  # max iterations
    gimp.progress_init("this is the fun part...")
    init_high = 1.400001
    init_low = 1.399991
    high_guesses = [(init_high, width*height*bpp)]  # we collect recent stats in this list
    low_guesses  = [(init_low, 0)]
    crossed_high = crossed_low = False
    factor = 1.4
    delta_high = .18   # adjust the factor by this much when it is too high
    delta_low = .14    # adjust the factor by this much when it is too low 
                       # if they were the same amount then they"d cancel
    xlength = 0
    while not ((xlength < density*6.3) and (xlength > density*5.7)):
        bluemask = vanderwalt(image, factor)
        line = np.array(bluemask.nonzero()).swapaxes(0,1).tolist()

        xlength = len(line) 
        # spread the search outward from initial condition (1.4) 
        # until we cross target. we collect 6 times the desired 
        # line length because we will take an average y per x
        # (unless they are too spread out and then we skip this column (despeckling))
        # My dataset averages 6 blue pixels per column, so...
        if xlength > density*6:                                               
            """too many points so raise the factor"""  
            print "LOOP1.%i: factor too low =%.5f, target=%i, line length = %i" % (count, factor, density*6, xlength)
            crossed_low = True
            low_guesses.append((factor, xlength))      
            factor += delta_low
            if crossed_high:   
                crossed_high = False
                delta_high *= .38  # cut the stepsizes
                delta_low *= .38                      
                print "crossed target.  factor now=%.5f" % (factor)
        else:
            """not enuf points so lower the factor"""
            print "LOOP1.%i: factor too high=%.5f, target=%i, line length = %i" % (count, factor, density*6, xlength)
            crossed_high = True
            high_guesses.append((factor, xlength))
            factor -= delta_high
            if factor < 0:
                lineno = getframeinfo(currentframe())[1]  # Beaz p 157
                print "algorithm failed with factor=%.5f (line %i)" % (factor, lineno)
                break
            if crossed_low:   
                crossed_low = False
                delta_high *= .38  # cut the stepsizes 
                delta_low *= .38                       
                print "crossed target.  factor now=%.5f" % (factor)
        count += 1
        gimp.progress_update(float(count) / failsafe)
        #print "count=%i, factor=%f\nhigh_guesses=%r\nlow_guesses=%r" % (count, factor, high_guesses, low_guesses)
        if count == failsafe:
             print "failsafe: level 1 failed to converge."
             break
    
    if (xlength < density*6.3) and (xlength > density*5.7):
        print "success at density*%1.3f" % (xlength/float(density))
    else:
        pdb.gimp_message("maybe not enuf blue pixels for this algorithm.")

    return bluemask
# end successive_approximation




From john.m.harrold at gmail.com  Tue Oct  7 15:23:58 2008
From: john.m.harrold at gmail.com (John Harrold)
Date: Tue, 7 Oct 2008 15:23:58 -0400
Subject: [Numpy-discussion] ImportError: No module named ma
Message-ID: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>

Howdy,

I'm trying to run some scripts associated with a paper I was reading. They use
the matplotlib to generate figures which in turn calles numpy. I'm new to
python, but I'm very familiar with perl. I'm getting the following error, and
I'm not quite sure what is causing it. Can anyone here offer any suggestions?

jmh at dhcp068-156:$ python2.5  Bb-plot.py 
Traceback (most recent call last):
  File "Bb-plot.py", line 7, in <module>
    from pylab import *
  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/pylab.py", line 1, in <module>
    from matplotlib.pylab import *
  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/__init__.py", line 128, in <module>
    from rcsetup import defaultParams, validate_backend, validate_toolbar
  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/rcsetup.py", line 19, in <module>
    from matplotlib.colors import is_color_like
  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/colors.py", line 39, in <module>
    import matplotlib.cbook as cbook
  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/cbook.py", line 9, in <module>
    import numpy.ma as ma
ImportError: No module named ma


-- 
 John M Harrold              


From stefan at sun.ac.za  Tue Oct  7 15:28:46 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 7 Oct 2008 21:28:46 +0200
Subject: [Numpy-discussion] collecting the bluest pixels
In-Reply-To: <813570.19734.qm@web53905.mail.re2.yahoo.com>
References: <813570.19734.qm@web53905.mail.re2.yahoo.com>
Message-ID: <9457e7c80810071228r5daa71c3yd6e1b43edc855882@mail.gmail.com>

Hi Paul

2008/10/7 paul taney <paultaney at yahoo.com>:
> I have this silly color filter that Stefan gave me:
>
> def vanderwalt(image, f):
>    """colorfilter, thanks to Stefan van der Walt"""
>    RED, GRN, BLU = 0, 1, 2
>    bluemask = (image[...,BLU] > f*image[...,GRN]) & \
>               (image[...,BLU] > f*image[...,RED])
>
>    return bluemask

Here's a heuristic method that may work.  The above statement is equivalent to

image[...,BLU] > f*max(image[...,GRN], image[...,RED])

So I would construct a new array with the maximum of the red and green
values.  Then, divide that new image by the blue image -- this gives
you all the different "f" values above.  Calculate the histogram of
the "f"s, and pick an f with the right percentile.

Regards
St?fan


From peridot.faceted at gmail.com  Tue Oct  7 15:30:44 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 7 Oct 2008 15:30:44 -0400
Subject: [Numpy-discussion] collecting the bluest pixels
In-Reply-To: <813570.19734.qm@web53905.mail.re2.yahoo.com>
References: <813570.19734.qm@web53905.mail.re2.yahoo.com>
Message-ID: <ce557a360810071230u6f8a54bau912f40673e9b7520@mail.gmail.com>

2008/10/7 paul taney <paultaney at yahoo.com>:
> Hi,
>
> I have this silly color filter that Stefan gave me:
>
>
> def vanderwalt(image, f):
>    """colorfilter, thanks to Stefan van der Walt"""
>    RED, GRN, BLU = 0, 1, 2
>    bluemask = (image[...,BLU] > f*image[...,GRN]) & \
>               (image[...,BLU] > f*image[...,RED])
>
>    return bluemask
>
>
> To collect the right number of the bluest pixels I am calling it from this arduous successive approximation routine.  It occured to me that someone on this list knows how to do this in a couple of lines...

Well, I can see several approaches. The most direct way to do what
you're asking is to use scipy.optimize.bisect to implement the
successive approximations. That'll be almost as slow as your current
approach, though. Instead, I'd first write a function that measures
the "blueness" of each pixel:

def blueness(image):
      A = np.empty(image.shape[:-1],dtype=np.float32)
      np.divide(image[...,BLU],image[...,RED],A) # use three-argument
divide to reduce the number of float temporaries
      B = np.empty(image.shape[:-1],dtype=np.float32)
      np.divide(image[...,BLU],image[...,GRN],B)
      return np.minimum(A,B)

Now, once you have the bluenesses, you can sort them and pull out the
blueness that gives you the percent you want:

bluenesses = np.sort(blueness(image),axis=None) # axis=None flattens the array
factor = bluenesses[int((1-wanted_fraction)*len(bluenesses))]

If you want a smarter blueness filter, you could look into using HSV:
you could then specify a distance in hue and a distance in saturation,
based on how relatively important you think they are.

Anne


From pgmdevlist at gmail.com  Tue Oct  7 15:23:41 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 7 Oct 2008 15:23:41 -0400
Subject: [Numpy-discussion] ImportError: No module named ma
In-Reply-To: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
References: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
Message-ID: <200810071523.41387.pgmdevlist@gmail.com>

On Tuesday 07 October 2008 15:23:58 John Harrold wrote:

> "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/c
>book.py", line 9, in <module> import numpy.ma as ma
> ImportError: No module named ma

John,
Are you sure you have numpy installed ? What version do you have ?



From stefan at sun.ac.za  Tue Oct  7 15:32:44 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 7 Oct 2008 21:32:44 +0200
Subject: [Numpy-discussion] ImportError: No module named ma
In-Reply-To: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
References: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
Message-ID: <9457e7c80810071232l70d5db5g5bdac5f649cd13f5@mail.gmail.com>

Hi John

2008/10/7 John Harrold <john.m.harrold at gmail.com>:
> I'm trying to run some scripts associated with a paper I was reading. They use
> the matplotlib to generate figures which in turn calles numpy. I'm new to
> python, but I'm very familiar with perl. I'm getting the following error, and
> I'm not quite sure what is causing it. Can anyone here offer any suggestions?
>
> jmh at dhcp068-156:$ python2.5  Bb-plot.py
> Traceback (most recent call last):
>  File "Bb-plot.py", line 7, in <module>
>    from pylab import *
>  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/pylab.py", line 1, in <module>
>    from matplotlib.pylab import *
>  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/__init__.py", line 128, in <module>
>    from rcsetup import defaultParams, validate_backend, validate_toolbar
>  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/rcsetup.py", line 19, in <module>
>    from matplotlib.colors import is_color_like
>  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/colors.py", line 39, in <module>
>    import matplotlib.cbook as cbook
>  File "/Library/Python/2.5/site-packages/matplotlib-0.98.3-py2.5.egg/matplotlib/cbook.py", line 9, in <module>
>    import numpy.ma as ma
> ImportError: No module named ma

Could you give us the version of NumPy that you are using?

>>> import numpy
>>> print numpy.__version__

As far as I recall, the masked array module was always available as
numpy.ma, but maybe I'm wrong.  Could you try the following:

>>> import numpy
>>> print numpy.core.ma

Thanks!
St?fan


From john.m.harrold at gmail.com  Tue Oct  7 15:40:25 2008
From: john.m.harrold at gmail.com (John Harrold)
Date: Tue, 7 Oct 2008 15:40:25 -0400
Subject: [Numpy-discussion] ImportError: No module named ma
In-Reply-To: <9457e7c80810071232l70d5db5g5bdac5f649cd13f5@mail.gmail.com>
References: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
	<9457e7c80810071232l70d5db5g5bdac5f649cd13f5@mail.gmail.com>
Message-ID: <20081007194025.GB42793@dhcp057-187.openport.buffalo.edu>

Sometime in October, St?fan van der Walt proposed the following: 

| Could you give us the version of NumPy that you are using?
| 
| >>> import numpy
| >>> print numpy.__version__
| 
| As far as I recall, the masked array module was always available as
| numpy.ma, but maybe I'm wrong.  Could you try the following:
| 
| >>> import numpy
| >>> print numpy.core.ma
| 

Howdy Stefan and Pierre

This should answer both of your questions:

>>> import numpy
>>> print numpy.__version__
1.0.1
>>> print numpy.core.ma
<module 'numpy.core.ma' from
'/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy/core/ma.pyc'>
>>> 

I actually installed 1.2.0 using the installation package for OS X.
However, I have no idea where it was installed to and how to instruct
python to use the new one over the old.

-- 
 John M Harrold              


From stefan at sun.ac.za  Tue Oct  7 15:42:38 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 7 Oct 2008 21:42:38 +0200
Subject: [Numpy-discussion] Merge of generalised ufuncs branch
Message-ID: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>

Hi all,

The generalised ufuncs branch was made available before SciPy'08.  We
solicited comments on its implementation and structuring, but received
very little feedback.  Unless there are any further comments from the
community, I propose that we merge it.

It is unfortunate that we have so many patches waiting for review
(SciPy suffers worst, I'm afraid); clearly there are too few hours in
a day.  Nothing discourages contributions as much as a stale project,
and I hope we can avoid that.

Regards,
St?fan


From millman at berkeley.edu  Tue Oct  7 15:44:14 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 7 Oct 2008 12:44:14 -0700
Subject: [Numpy-discussion] Merge of generalised ufuncs branch
In-Reply-To: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
References: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
Message-ID: <c7009a550810071244p2fe8df8bnd2845533c16e1c4a@mail.gmail.com>

On Tue, Oct 7, 2008 at 12:42 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> The generalised ufuncs branch was made available before SciPy'08.  We
> solicited comments on its implementation and structuring, but received
> very little feedback.  Unless there are any further comments from the
> community, I propose that we merge it.

+1

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From Chris.Barker at noaa.gov  Tue Oct  7 15:47:46 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 07 Oct 2008 12:47:46 -0700
Subject: [Numpy-discussion] ImportError: No module named ma
In-Reply-To: <20081007194025.GB42793@dhcp057-187.openport.buffalo.edu>
References: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
	<9457e7c80810071232l70d5db5g5bdac5f649cd13f5@mail.gmail.com>
	<20081007194025.GB42793@dhcp057-187.openport.buffalo.edu>
Message-ID: <48EBBCE2.1090305@noaa.gov>

John Harrold wrote:
> This should answer both of your questions:
> 
>>>> import numpy
>>>> print numpy.__version__
> 1.0.1
>>>> print numpy.core.ma
> <module 'numpy.core.ma' from
> '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/numpy/core/ma.pyc'>
> 
> I actually installed 1.2.0 using the installation package for OS X.
> However, I have no idea where it was installed to and how to instruct
> python to use the new one over the old.

the installation package for OS-X is for the python.org build -- from 
the above, I can see you are running Apple's python, which came with a 
version of numpy.

I don't remember how you installed matplotlib, but in general, most 
extrenal packages are built for the python or python, so what I wold do is:

download and install python 2.5.2 from python.org (2.6 is brand new, and 
not yet well supported by external packages)

re-install numpy 1.2.0 (may not be necessary, but it won't hurt)

re-install MPL.

you can tell what python you are running by typing "python" n the 
command line -- it should be 2.5.2 if it's the python.org one. If that's 
not what you get, you may need to edit your .bash_profile file to add it 
to your PATH.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From peridot.faceted at gmail.com  Tue Oct  7 16:10:25 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 7 Oct 2008 16:10:25 -0400
Subject: [Numpy-discussion] Merge of generalised ufuncs branch
In-Reply-To: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
References: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
Message-ID: <ce557a360810071310j5536255eoc14bb2e7cfce4853@mail.gmail.com>

2008/10/7 St?fan van der Walt <stefan at sun.ac.za>:

> The generalised ufuncs branch was made available before SciPy'08.  We
> solicited comments on its implementation and structuring, but received
> very little feedback.  Unless there are any further comments from the
> community, I propose that we merge it.

Sounds good to me - I've counted at least three or four threads on the
mailing lists wishing for the ufuncized linear algebra this would
allow since it was put forward. (Of course, these won't appear until
someone implements them - perhaps a start would be for someone to
write a tutorial on using the new generalized ufunc code...)

> It is unfortunate that we have so many patches waiting for review
> (SciPy suffers worst, I'm afraid); clearly there are too few hours in
> a day.  Nothing discourages contributions as much as a stale project,
> and I hope we can avoid that.

The problem may perhaps be that not many people feel they are in a
position to actually do the reviews, so that everyone is waiting on an
imagined few "real" developers to place the Official Stamp of
Approval. Perhaps an informal rule of thumb for acceptance of patches?
How about: posted to list, reviewed by someone not the author who's
had at least one patch accepted before, and no objections on the list?
Anything that receives objections can fall through to the usual
decision by discussion on the mailing list, this is just intended for
those patches that everyone just kind of shrugs and says "looks all
right to me".

Anne

From paultaney at yahoo.com  Tue Oct  7 16:27:55 2008
From: paultaney at yahoo.com (paul taney)
Date: Tue, 7 Oct 2008 13:27:55 -0700 (PDT)
Subject: [Numpy-discussion] collecting the bluest pixels
In-Reply-To: <ce557a360810071230u6f8a54bau912f40673e9b7520@mail.gmail.com>
Message-ID: <260495.50905.qm@web53912.mail.re2.yahoo.com>

Thank you Stefan and Anne for such quick replies.

I am writing a gimp plugin, and if anybody is interested in how do that -- there are only about 10-20 examples that I"ve found -- this plugin is attempting to do raster-to-vector conversion on the bluest pixels.  It outputs SVG and a python tuple.  Current version is at 

    http://pastebin.com/m2bd8113f

I dunno np.histogram yet, so rather than look that up I tried Anne"s approach.

I get a divide by zero error on 

    np.divide(image[...,BLU], image[...,GRN], B)

...and I dont understand this well enough to diagnose.  Any ideas?


Thanks in advance,
paul


----


Anne1 = """Well, I can see several approaches. The most direct way to do what you're asking is to use scipy.optimize.bisect to implement the successive approximations. That'll be almost as slow as your current approach, though. Instead, I'd first write a function that measures the "blueness" of each pixel:"""

def blueness(image, linedensity):
    A = np.empty(image.shape[:-1], dtype=np.float32)
    np.divide(image[...,BLU], image[...,RED], A)  # use three-argument
    # divide to reduce the number of float temporaries
    B = np.empty(image.shape[:-1], dtype=np.float32)
    np.divide(image[...,BLU], image[...,GRN], B)
    return np.minimum(A, B)

    Anne2 = """
Now, once you have the bluenesses, you can sort them and pull out the
blueness that gives you the percent you want:
"""
    bluenesses = np.sort(blueness(image), axis=None)  # axis=None flattens the array
    #factor = bluenesses[int((1-wanted_fraction)*len(bluenesses))]
    w, h, bpp = image.shape
    factor = bluenesses[int((1-(linedensity*6/float(w*h))) * len(bluenesses))] 

    Anne3 = """
If you want a smarter blueness filter, you could look into using HSV:
you could then specify a distance in hue and a distance in saturation,
based on how relatively important you think they are.
Anne
"""
    print "blueness returning factor = %r" % (factor)
    return factor




From charlesr.harris at gmail.com  Tue Oct  7 16:52:21 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 7 Oct 2008 14:52:21 -0600
Subject: [Numpy-discussion] Merge of generalised ufuncs branch
In-Reply-To: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
References: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
Message-ID: <e06186140810071352i5f3de24fq6502c5e107b14756@mail.gmail.com>

On Tue, Oct 7, 2008 at 1:42 PM, St?fan van der Walt <stefan at sun.ac.za>wrote:

> Hi all,
>
> The generalised ufuncs branch was made available before SciPy'08.  We
> solicited comments on its implementation and structuring, but received
> very little feedback.  Unless there are any further comments from the
> community, I propose that we merge it.
>
> It is unfortunate that we have so many patches waiting for review
> (SciPy suffers worst, I'm afraid); clearly there are too few hours in
> a day.  Nothing discourages contributions as much as a stale project,
> and I hope we can avoid that.
>

I've been thinking it's about time to get to this. Can you start by merging
as much of the current trunk as you can so we can concentrate on the
differences?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081007/c1a04d1e/attachment.html>

From Chris.Barker at noaa.gov  Tue Oct  7 16:53:44 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 07 Oct 2008 13:53:44 -0700
Subject: [Numpy-discussion] collecting the bluest pixels
In-Reply-To: <260495.50905.qm@web53912.mail.re2.yahoo.com>
References: <260495.50905.qm@web53912.mail.re2.yahoo.com>
Message-ID: <48EBCC58.2040402@noaa.gov>

paul taney wrote:
> Thank you Stefan and Anne for such quick replies.
> 
> I am writing a gimp plugin

cool!

> I get a divide by zero error on 
> 
>     np.divide(image[...,BLU], image[...,GRN], B)
> 
> ...and I dont understand this well enough to diagnose.  Any ideas?

you're dividing the value of Blue by the value of green, so if there is 
zero green, yes, you will get a divide by zero error.

I wonder if the euclidian norm would make sense for this application:

HowFarFromBlue = np.sqrt((255-image[...,BLU])**2 +
                          image[...,GRN]**2 +
                          image[...,RED]**2)

smaller numbers would be bluest -- pure blue would be 0, pure red 360, 
etc...

One thing I like about this is that your "blue" may not exactly be an 
RBG blue -- so you could see how "far" a given pixel was from any given 
color -- in your case, whatever your blue is. Then it would be something 
like:

r, g, b = ref_color

HowFarFromRefColor = np.sqrt((r - image[...,RED])**2 +
                              (g - image[...,GRN])**2 +
                              (b - image[...,BLU])**2
                              )


NOTE: I know nothing of image prcessing -- I"ll be there is are some 
standard ways to determine how "close" two colors are.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Tue Oct  7 16:53:37 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 7 Oct 2008 14:53:37 -0600
Subject: [Numpy-discussion] Merge of generalised ufuncs branch
In-Reply-To: <e06186140810071352i5f3de24fq6502c5e107b14756@mail.gmail.com>
References: <9457e7c80810071242u3df734f4n94c935cc9affe771@mail.gmail.com>
	<e06186140810071352i5f3de24fq6502c5e107b14756@mail.gmail.com>
Message-ID: <e06186140810071353t64a5e119p1c8e64d6eb17e7f7@mail.gmail.com>

On Tue, Oct 7, 2008 at 2:52 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Tue, Oct 7, 2008 at 1:42 PM, St?fan van der Walt <stefan at sun.ac.za>wrote:
>
>> Hi all,
>>
>> The generalised ufuncs branch was made available before SciPy'08.  We
>> solicited comments on its implementation and structuring, but received
>> very little feedback.  Unless there are any further comments from the
>> community, I propose that we merge it.
>>
>> It is unfortunate that we have so many patches waiting for review
>> (SciPy suffers worst, I'm afraid); clearly there are too few hours in
>> a day.  Nothing discourages contributions as much as a stale project,
>> and I hope we can avoid that.
>>
>
> I've been thinking it's about time to get to this. Can you start by merging
> as much of the current trunk as you can so we can concentrate on the
> differences?
>

Or maybe just start a new branch and see if the patch still applies.

Chuck

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081007/3b6feb42/attachment.html>

From john.m.harrold at gmail.com  Tue Oct  7 18:18:48 2008
From: john.m.harrold at gmail.com (John Harrold)
Date: Tue, 7 Oct 2008 18:18:48 -0400
Subject: [Numpy-discussion] ImportError: No module named ma
In-Reply-To: <48EBBCE2.1090305@noaa.gov>
References: <20081007192358.GA42793@dhcp057-187.openport.buffalo.edu>
	<9457e7c80810071232l70d5db5g5bdac5f649cd13f5@mail.gmail.com>
	<20081007194025.GB42793@dhcp057-187.openport.buffalo.edu>
	<48EBBCE2.1090305@noaa.gov>
Message-ID: <20081007221848.GC41169@dhcp057-187.openport.buffalo.edu>

Sometime in October, Christopher Barker proposed the following: 

| the installation package for OS-X is for the python.org build -- from 
| the above, I can see you are running Apple's python, which came with a 
| version of numpy.
| 
| I don't remember how you installed matplotlib, but in general, most 
| extrenal packages are built for the python or python, so what I wold do is:
| 
| download and install python 2.5.2 from python.org (2.6 is brand new, and 
| not yet well supported by external packages)
| 
| re-install numpy 1.2.0 (may not be necessary, but it won't hurt)
| 
| re-install MPL.
| 
| you can tell what python you are running by typing "python" n the 
| command line -- it should be 2.5.2 if it's the python.org one. If that's 
| not what you get, you may need to edit your .bash_profile file to add it 
| to your PATH.
| 

Thanks Chris,

This worked great. Along with the above mentioned stuff, I also  had a bit
of trouble installing the MPL egg which seems to be mac specific.  This was
also pretty useful in fixing that:

http://www.nabble.com/OS-X-(Intel-Mac)-installation-woes-td19011120.html

The important part being: 
  That said, the build on the matplotlib website is self-sufficient save  
  for numpy, use the -N flag to easy_install (and don't forget to run it  
  with sudo). This should take care of things. 

-- 
 John M Harrold              


From peridot.faceted at gmail.com  Tue Oct  7 20:10:07 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 7 Oct 2008 20:10:07 -0400
Subject: [Numpy-discussion] collecting the bluest pixels
In-Reply-To: <48EBCC58.2040402@noaa.gov>
References: <260495.50905.qm@web53912.mail.re2.yahoo.com>
	<48EBCC58.2040402@noaa.gov>
Message-ID: <ce557a360810071710o18a79509wa320be4e493a00e8@mail.gmail.com>

2008/10/7 Christopher Barker <Chris.Barker at noaa.gov>:

> I wonder if the euclidian norm would make sense for this application:
>
> HowFarFromBlue = np.sqrt((255-image[...,BLU])**2 +
>                          image[...,GRN]**2 +
>                          image[...,RED]**2)
>
> smaller numbers would be bluest -- pure blue would be 0, pure red 360,
> etc...
>
> One thing I like about this is that your "blue" may not exactly be an
> RBG blue -- so you could see how "far" a given pixel was from any given
> color -- in your case, whatever your blue is. Then it would be something
> like:
>
> r, g, b = ref_color
>
> HowFarFromRefColor = np.sqrt((r - image[...,RED])**2 +
>                              (g - image[...,GRN])**2 +
>                              (b - image[...,BLU])**2
>                              )
>
>
> NOTE: I know nothing of image prcessing -- I"ll be there is are some
> standard ways to determine how "close" two colors are.

It's a tricky problem, but if you're serious about it you can use
Euclidean distance in the CIELUV colour space, or if you're really
serious Euclidean distance in CIELAB. Both probably overkill for this
project.

Anne


From david at ar.media.kyoto-u.ac.jp  Wed Oct  8 03:39:32 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 08 Oct 2008 16:39:32 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
Message-ID: <48EC63B4.5030307@ar.media.kyoto-u.ac.jp>

Paul Lucek wrote:
>
> I get the following errors in umathmodule.c.src:
>
>  
>
> D:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe
> /c /nolog
>
> o /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64-2.6\numpy\core\src
> -Inumpy\cor
>
> e\include -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy
> -Inumpy\core\src -I
>
> numpy\core\include -Id:\python26\include -Id:\python26\PC
> /Tcbuild\src.win-amd64
>
> -2.6\numpy\core\src\umathmodule.c
> /Fobuild\temp.win-amd64-2.6\Release\build\src.
>
> win-amd64-2.6\numpy\core\src\umathmodule.obj
>
> umathmodule.c
>
> numpy\core\src\umathmodule.c.src(64) : error C2059: syntax error : 'type'
>
> numpy\core\src\umathmodule.c.src(70) : error C2059: syntax error : 'type'
>
> numpy\core\src\ufuncobject.c(1701) : warning C4244: '=' : conversion
> from 'npy_i
>
> ntp' to 'int', possible loss of data
>
> numpy\core\src\ufuncobject.c(2422) : warning C4244: '=' : conversion
> from 'npy_i
>
> ntp' to 'int', possible loss of data
>
> error: Command "D:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\amd64\
>
> cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG
> -Ibuild\src.win-amd64-2.6\numpy\core
>
> \src -Inumpy\core\include
> -Ibuild\src.win-amd64-2.6\numpy\core\include/numpy -In
>
> umpy\core\src -Inumpy\core\include -Id:\python26\include
> -Id:\python26\PC /Tcbui
>
> ld\src.win-amd64-2.6\numpy\core\src\umathmodule.c
> /Fobuild\temp.win-amd64-2.6\Re
>
> lease\build\src.win-amd64-2.6\numpy\core\src\umathmodule.obj" failed
> with exit s
>
> tatus 2
>
>  
>

Note that 1.2.0 does not officially support python 2.6, specially on
windows. Because python 2.6 uses a new toolchain (VS 2008 by default
instead of 2003), it requires some non trivial changes. I hope that all
those changes will be in place for numpy 1.3,

cheers,

David


From y.copin at ipnl.in2p3.fr  Wed Oct  8 08:21:04 2008
From: y.copin at ipnl.in2p3.fr (Yannick Copin)
Date: Wed, 08 Oct 2008 14:21:04 +0200
Subject: [Numpy-discussion] Wrong behavior of atleast_3d?
Message-ID: <48ECA5B0.2060709@ipnl.in2p3.fr>

Hi,

I wonder if there's a "behavior bug" in atleast_3d, with respect to 
atleast_2d. In atleast_2d, the results on a 1D-array or on a single-element 
list of 1D-arrays are the same:

In [22]: atleast_2d(arange(3)).shape
Out[22]: (1, 3)

In [23]: atleast_2d([arange(3),]).shape
Out[23]: (1, 3)

On the contrary, with atleast_3d, the results differ between a 2D-array and a 
single-element list of 2D-arrays:

In [24]: atleast_3d(randn(3,3)).shape
Out[24]: (3, 3, 1)

In [25]: atleast_3d([randn(3,3),]).shape
Out[25]: (1, 3, 3)

Wouldn't it be more logical wrt atleast_2d that atleast_3d(randn(3,3)) return 
an array of shape (1,3,3)? (even though the docstring makes it clear that it 
*will* return a (3,3,1)-array).

I mostly use atleast_nd to be able to iterate over input arrays without having 
to know if it's a single array, or a list of arrays. For that, at_least2d has 
the correct behavior, but not atleast_3d.

Cheers.
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucleaire de Lyon (IN2P3 - France)
   // \\  Tel: (33/0) 472 431 968     AIM: YnCopin ICQ: 236931013
  /(   )\ http://snovae.in2p3.fr/ycopin/
   ^`~'^


From lists_ravi at lavabit.com  Wed Oct  8 09:41:23 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 8 Oct 2008 09:41:23 -0400
Subject: [Numpy-discussion]
 =?iso-8859-1?q?can=27t_build_numpy_1=2E2=2E0_u?=
 =?iso-8859-1?q?nder_python_2=2E6_=28windows-amd64=29_using_VS9?=
In-Reply-To: <48EC63B4.5030307@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<48EC63B4.5030307@ar.media.kyoto-u.ac.jp>
Message-ID: <200810080941.23489.lists_ravi@lavabit.com>

On Wednesday 08 October 2008 03:39:32 David Cournapeau wrote:
> Note that 1.2.0 does not officially support python 2.6, specially on
> windows. Because python 2.6 uses a new toolchain (VS 2008 by default
> instead of 2003), it requires some non trivial changes.

How extensive are these non-trivial changes? I could help test any changes. My 
problem is one that should be familiar to most people working in a large 
organization. I am trying to sneak in python+scipy as a replacement for 
Matlab, mostly by ensuring that python is used as the embedded scripting 
language for a signal processing simulation platform. (The ability to embed 
python, especially with seamless interoperability between numpy arrays and 
boost ublas vectors/matrices, is something Matlab cannot match.) 

The majority of my colleagues work on Windows and are very resistant to 
toolset changes. Hence, from my perspective, whenever a new project starts, it 
is very important to start with the latest version of any software packages 
used. Usually, over the typical 3-4 year lifetime of a project, the tools are 
never updated unless there is an absolutely critical bug fix. We are still on 
python 2.2 for a couple of currently active projects (neither of which uses 
numpy). For the next project, I was hoping to use Python 2.6 + numpy 1.2 as 
the base versions, but that seems unworkable now.

> I hope that all those changes will be in place for numpy 1.3,

Is there an idea of the timeframe for numpy 1.3?

Regards,
Ravi





From numpy-discussion at maubp.freeserve.co.uk  Wed Oct  8 09:52:50 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Wed, 8 Oct 2008 14:52:50 +0100
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
In-Reply-To: <200810080941.23489.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<48EC63B4.5030307@ar.media.kyoto-u.ac.jp>
	<200810080941.23489.lists_ravi@lavabit.com>
Message-ID: <320fb6e00810080652r3cef3e6p35d75c2211a9a763@mail.gmail.com>

On Wed, Oct 8, 2008 at 2:41 PM, Ravi <lists_ravi at lavabit.com> wrote:
> The majority of my colleagues work on Windows and are very resistant to
> toolset changes. Hence, from my perspective, whenever a new project starts, it
> is very important to start with the latest version of any software packages
> used. Usually, over the typical 3-4 year lifetime of a project, the tools are
> never updated unless there is an absolutely critical bug fix. We are still on
> python 2.2 for a couple of currently active projects (neither of which uses
> numpy). For the next project, I was hoping to use Python 2.6 + numpy 1.2 as
> the base versions, but that seems unworkable now.

How about python 2.5 and numpy 1.2 instead?  Python 2.6 makes some
important changes to python 2.5 (in preparation for Python 3.0), so
you may find several other packages will take a couple of months to
work 100% with python 2.6 - so check this out if you do plan to use
more than just numpy.  There are sometimes drawbacks to using brand
new releases ;)

Peter


From david at ar.media.kyoto-u.ac.jp  Wed Oct  8 09:40:58 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 08 Oct 2008 22:40:58 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <320fb6e00810080652r3cef3e6p35d75c2211a9a763@mail.gmail.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<48EC63B4.5030307@ar.media.kyoto-u.ac.jp>	<200810080941.23489.lists_ravi@lavabit.com>
	<320fb6e00810080652r3cef3e6p35d75c2211a9a763@mail.gmail.com>
Message-ID: <48ECB86A.1080006@ar.media.kyoto-u.ac.jp>

Peter wrote:
>
> How about python 2.5 and numpy 1.2 instead?  Python 2.6 makes some
> important changes to python 2.5 (in preparation for Python 3.0), so
> you may find several other packages will take a couple of months to
> work 100% with python 2.6 - so check this out if you do plan to use
> more than just numpy.  There are sometimes drawbacks to using brand
> new releases ;)

If I understand Ravi right, one problem with 2.5 is that it relies on an
old toolset (VS 2003, not available anymore). OTOH, 2.6 depends on the
most recent toolset, which is also available for free in a limited
version (VS 2008 express). If you think about supporting things for a
couple of years, relying on VS 2008 sounds safer than 2003.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Wed Oct  8 09:44:25 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 08 Oct 2008 22:44:25 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <200810080941.23489.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<48EC63B4.5030307@ar.media.kyoto-u.ac.jp>
	<200810080941.23489.lists_ravi@lavabit.com>
Message-ID: <48ECB939.8010406@ar.media.kyoto-u.ac.jp>

Ravi wrote:
> On Wednesday 08 October 2008 03:39:32 David Cournapeau wrote:
>   
>> Note that 1.2.0 does not officially support python 2.6, specially on
>> windows. Because python 2.6 uses a new toolchain (VS 2008 by default
>> instead of 2003), it requires some non trivial changes.
>>     
>
> How extensive are these non-trivial changes? I could help test any changes. 

For one, the manifest madness introduced with VS 2005 may be painful to
handle, since it severly lacks any serious documentation. I am still
unsure whether we will need to care at all, though.

Another problem is that python headers are not "clean", they pollute the
namespace quite heavily; a new version of python means new names added:
those should be trivial, though.

Other things are the usual: brokenness of MS tools with respect to
standards (basic C99 functions not available, etc...). Every version of
MS tool is broken, but in a different way.

cheers,

David


From plucek at ssaris.com  Wed Oct  8 10:13:50 2008
From: plucek at ssaris.com (Paul Lucek)
Date: Wed, 8 Oct 2008 10:13:50 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
Message-ID: <3633FE192D10364D889D84EB77185F3E73AE97@rxrex1.ssaris.com>

>Could you compress win-amd64-2.6\numpy\core\src\umathmodule.c -- it 

>should be in the build directory -- and attach it if possible, or at 

>least that part that seems to be a problem? Mind that the list has a
rather small size limit.

 

umathmodule.zip attached


NOTICE- This communication (including any attachments) contains confidential and/or privileged information and is intended only for the use of the individual(s) to whom it is addressed for a specific purpose and is protected by law. Any review, use, distribution, disclosure, alteration, copying, transmittal or re-transmittal by persons who are not intended recipients of this communication may be a violation of law and is strictly prohibited. If you are not the intended recipient, please permanently delete all copies of this communication and any attachments from your computer system, destroy any hard copies, and immediately notify the sender or SSARIS Advisors, LLC at compliance at ssaris.com or (203) 328-7200. No waiver of confidentiality or privilege is made by mistransmission.

Any views expressed in this communication are those of the individual sender. This communication and any attachments hereto are for informational purposes only and should not be construed as an offer to sell interests or shares in any investment vehicle managed by SSARIS Advisors, LLC or its affiliates. Any information regarding trading performance must be considered in conjunction with the appropriate disclosure documents. Past performance is not necessarily indicative of future results.

SSARIS Advisors, LLC reserves the right to monitor all communications through its networks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081008/c866ff81/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: umathmodule.zip
Type: application/x-zip-compressed
Size: 18431 bytes
Desc: umathmodule.zip
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081008/c866ff81/attachment.bin>

From plucek at ssaris.com  Wed Oct  8 10:14:46 2008
From: plucek at ssaris.com (Paul Lucek)
Date: Wed, 8 Oct 2008 10:14:46 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
Message-ID: <3633FE192D10364D889D84EB77185F3E73AE99@rxrex1.ssaris.com>

>Could you compress win-amd64-2.6\numpy\core\src\umathmodule.c -- it 

>should be in the build directory -- and attach it if possible, or at 

>least that part that seems to be a problem? Mind that the list has a
rather small size limit.

 

I think this is the pertinent section:

 

#ifndef HAVE_FREXPF

static float frexpf(float x, int * i)

{

    return (float)frexp((double)(x), i);

}

#endif

#ifndef HAVE_LDEXPF

static float ldexpf(float x, int i)

{

    return (float)ldexp((double)(x), i);

}

#endif

#define tanhf nc_tanhf

#endif


NOTICE- This communication (including any attachments) contains confidential and/or privileged information and is intended only for the use of the individual(s) to whom it is addressed for a specific purpose and is protected by law. Any review, use, distribution, disclosure, alteration, copying, transmittal or re-transmittal by persons who are not intended recipients of this communication may be a violation of law and is strictly prohibited. If you are not the intended recipient, please permanently delete all copies of this communication and any attachments from your computer system, destroy any hard copies, and immediately notify the sender or SSARIS Advisors, LLC at compliance at ssaris.com or (203) 328-7200. No waiver of confidentiality or privilege is made by mistransmission.

Any views expressed in this communication are those of the individual sender. This communication and any attachments hereto are for informational purposes only and should not be construed as an offer to sell interests or shares in any investment vehicle managed by SSARIS Advisors, LLC or its affiliates. Any information regarding trading performance must be considered in conjunction with the appropriate disclosure documents. Past performance is not necessarily indicative of future results.

SSARIS Advisors, LLC reserves the right to monitor all communications through its networks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081008/d3635aa5/attachment.html>

From lists_ravi at lavabit.com  Wed Oct  8 10:17:42 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 8 Oct 2008 10:17:42 -0400
Subject: [Numpy-discussion]
 =?iso-8859-1?q?can=27t_build_numpy_1=2E2=2E0_u?=
 =?iso-8859-1?q?nder_python_2=2E6_=28windows-amd64=29_using_VS9?=
In-Reply-To: <48ECB86A.1080006@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<320fb6e00810080652r3cef3e6p35d75c2211a9a763@mail.gmail.com>
	<48ECB86A.1080006@ar.media.kyoto-u.ac.jp>
Message-ID: <200810081017.43024.lists_ravi@lavabit.com>

On Wednesday 08 October 2008 09:40:58 David Cournapeau wrote:
> > How about python 2.5 and numpy 1.2 instead? ?Python 2.6 makes some
> > important changes to python 2.5 (in preparation for Python 3.0), so
> > you may find several other packages will take a couple of months to
> > work 100% with python 2.6 - so check this out if you do plan to use
> > more than just numpy. ?There are sometimes drawbacks to using brand
> > new releases ;)
>
> If I understand Ravi right, one problem with 2.5 is that it relies on an
> old toolset (VS 2003, not available anymore). OTOH, 2.6 depends on the
> most recent toolset, which is also available for free in a limited
> version (VS 2008 express). If you think about supporting things for a
> couple of years, relying on VS 2008 sounds safer than 2003.

Absolutely correct. VS 2003 is one of the biggest problems. I know C++ is not 
a favored language in these parts, but a lot of our code relies on Boost, for 
platform neutrality (like filesystem, threading, stdint) and for many other 
utilities (like ublas, wave, tuples, bind). VS 2003 requires too many 
workarounds in my code and support for it may be dropped in Boost some time 
soon: VS 7.0 is already scheduled for to be dropped and VS 7.1 (a.k.a. VS 
2003) is not likely to be supported for much longer. In fact python 2.5.x's 
requirement of VS 2003 is probably the main reason that this compiler is still 
used in a lot of places.

The only python packages that I use are numpy/scipy, matplotlib, pyhdf5, and 
an interprocess communication link with VisIt. To the best of my knowledge, 
all of these work today (in some fashion) with python 2.6 on Linux. Therefore, 
any changes required to make them build on Windows are likely to be compiler-
specific rather than to fix inherent problems related to python 2.6.

Regards,
Ravi




From lists_ravi at lavabit.com  Wed Oct  8 10:29:43 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 8 Oct 2008 10:29:43 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
In-Reply-To: <48ECB939.8010406@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<200810080941.23489.lists_ravi@lavabit.com>
	<48ECB939.8010406@ar.media.kyoto-u.ac.jp>
Message-ID: <200810081029.43569.lists_ravi@lavabit.com>

On Wednesday 08 October 2008 09:44:25 David Cournapeau wrote:
> For one, the manifest madness introduced with VS 2005 may be painful to
> handle, since it severly lacks any serious documentation. I am still
> unsure whether we will need to care at all, though.
>
> Another problem is that python headers are not "clean", they pollute the
> namespace quite heavily; a new version of python means new names added:
> those should be trivial, though.
>
> Other things are the usual: brokenness of MS tools with respect to
> standards (basic C99 functions not available, etc...). Every version of
> MS tool is broken, but in a different way.

The reasons above are why I don't try to do anything on Windows unless there 
is support from some external source, e.g., CMake taking care of build issues. 
The reasons above are also why I admire your heroic efforts at making Windows 
binaries available. But, then, I sometimes wonder about the motivation for an 
unpaid volunteer to take on an utterly thankless job in which help is never 
forthcoming from users; for any of my code out in the wild, the only help, bug 
fixes and improvements I have ever received have been from Linux and Solris 
users.

Thank for taking on this arduous task.

Regards,
Ravi




From david at ar.media.kyoto-u.ac.jp  Wed Oct  8 10:25:14 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 08 Oct 2008 23:25:14 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <200810081029.43569.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810080941.23489.lists_ravi@lavabit.com>	<48ECB939.8010406@ar.media.kyoto-u.ac.jp>
	<200810081029.43569.lists_ravi@lavabit.com>
Message-ID: <48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>

Ravi wrote:
>
> The reasons above are why I don't try to do anything on Windows unless there 
> is support from some external source, e.g., CMake taking care of build issues. 
> The reasons above are also why I admire your heroic efforts at making Windows 
> binaries available. But, then, I sometimes wonder about the motivation for an 
> unpaid volunteer to take on an utterly thankless job in which help is never 
> forthcoming from users;

I think numpy and scipy have a wonderful potential, and that currently,
installation is the biggest hurdle. I can show some awesome things in
numpy/scipy and co that people used to matlab would only dream of. But
if it takes more than 2 minutes and a few clicks to install, it is of no
use. I have some people who ask me how to install numpy/scipy, and I
have no simple answer: I think this is by far the biggest barrier of
entry for numpy and scipy ATM. That's why I am interested in making
numpy and scipy installation easy.

>
> Thank for taking on this arduous task.

Just want to mention I am certainly not the only one involved here for
windows binaries. This is really a collective work,

cheers,

David


From aisaac at american.edu  Wed Oct  8 10:45:44 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 08 Oct 2008 10:45:44 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <200810081029.43569.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810080941.23489.lists_ravi@lavabit.com>	<48ECB939.8010406@ar.media.kyoto-u.ac.jp>
	<200810081029.43569.lists_ravi@lavabit.com>
Message-ID: <48ECC798.9030603@american.edu>

On 10/8/2008 10:29 AM Ravi apparently wrote:
> I sometimes wonder about the motivation for an 
> unpaid volunteer to take on an utterly thankless job in which help is never 
> forthcoming from users  ...
> Thank for taking on this arduous task.

See, it is not entirely thankless.  ;-)
I would like to add my thanks as well,
and those of my students who are mostly
Windows users who rely on these binaries.
Alan Isaac



From hanni.ali at gmail.com  Wed Oct  8 10:56:02 2008
From: hanni.ali at gmail.com (Hanni Ali)
Date: Wed, 8 Oct 2008 15:56:02 +0100
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
In-Reply-To: <48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<200810080941.23489.lists_ravi@lavabit.com>
	<48ECB939.8010406@ar.media.kyoto-u.ac.jp>
	<200810081029.43569.lists_ravi@lavabit.com>
	<48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>
Message-ID: <789d27b10810080756r293ee6b4r5b7705df24aa1ccc@mail.gmail.com>

Just to note, on the compilation issue, I encountered this a while ago with
numpy 1.1.1 and I think Python 2.6b2, again because we wanted to skip Python
2.5 in my organization, largely because it was an issue to get working on
64-bit. I couldn't find anywhere 7.1 was available.

We discussed errors you are encountering a few months ago, they are related
to the compiler directives.

> #ifndef HAVE_FREXPF
> static float frexpf(float x, int * i)
> {
>     return (float)frexp((double)(x), i);
> }
> #endif
> #ifndef HAVE_LDEXPF
> static float ldexpf(float x, int i)
> {
>     return (float)ldexp((double)(x), i);
> }
> #endif

Commenting out this section at line 64 allow compilation and has no ill
effects.

Hanni



2008/10/8 David Cournapeau <david at ar.media.kyoto-u.ac.jp>

> Ravi wrote:
> >
> > The reasons above are why I don't try to do anything on Windows unless
> there
> > is support from some external source, e.g., CMake taking care of build
> issues.
> > The reasons above are also why I admire your heroic efforts at making
> Windows
> > binaries available. But, then, I sometimes wonder about the motivation
> for an
> > unpaid volunteer to take on an utterly thankless job in which help is
> never
> > forthcoming from users;
>
> I think numpy and scipy have a wonderful potential, and that currently,
> installation is the biggest hurdle. I can show some awesome things in
> numpy/scipy and co that people used to matlab would only dream of. But
> if it takes more than 2 minutes and a few clicks to install, it is of no
> use. I have some people who ask me how to install numpy/scipy, and I
> have no simple answer: I think this is by far the biggest barrier of
> entry for numpy and scipy ATM. That's why I am interested in making
> numpy and scipy installation easy.
>
> >
> > Thank for taking on this arduous task.
>
> Just want to mention I am certainly not the only one involved here for
> windows binaries. This is really a collective work,
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081008/d3ccb647/attachment.html>

From hetland at tamu.edu  Wed Oct  8 13:25:32 2008
From: hetland at tamu.edu (Rob Hetland)
Date: Wed, 8 Oct 2008 12:25:32 -0500
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
	<26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
	<ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
Message-ID: <7F4C2565-D598-4733-9392-64EBD054D5FA@tamu.edu>


My version of a wrapper of ANN is attached.  I wrote it when I had  
some issues installing the scikits.ann package.  It uses ctypes, and  
might be useful in deciding on an API.

Please feel free to take what you like,

-Rob


-------------- next part --------------
A non-text attachment was scrubbed...
Name: ann.tar.gz
Type: application/x-gzip
Size: 6640 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081008/662f1539/attachment.bin>
-------------- next part --------------

----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331




From hetland at tamu.edu  Wed Oct  8 15:11:14 2008
From: hetland at tamu.edu (Rob Hetland)
Date: Wed, 8 Oct 2008 14:11:14 -0500
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810081136j4cc7d3d6w42a45a89097cb2dd@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
	<26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
	<ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
	<7F4C2565-D598-4733-9392-64EBD054D5FA@tamu.edu>
	<ce557a360810081136j4cc7d3d6w42a45a89097cb2dd@mail.gmail.com>
Message-ID: <A803DE7C-4ADF-4955-BFF7-33E237C42A2C@tamu.edu>


On Oct 8, 2008, at 1:36 PM, Anne Archibald wrote:

> How important did you find the ability to select priority versus
> non-priority search?
>
> How important did you find the ability to select other splitting  
> rules?


In both cases, I included those things just because they were options  
in ANN, rather than any real need on my part.

The things that I found important (and lacking in some of the other  
many kd_tree implementations) were the ability to select an arbitrary  
number of nearest neighbors returned and specifying a max search radius.

I can see that having a choice of splitting rules may be handy for  
unusual point distributions, but I have always used this on fairly  
regularly spaced data.

-Rob

----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331





From zachary.pincus at yale.edu  Wed Oct  8 21:00:16 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 8 Oct 2008 21:00:16 -0400
Subject: [Numpy-discussion] 2D (or n-d) fancy indexing?
Message-ID: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>

Hello all,

I'm doing something silly with images and am unable to figure out the  
right way to express this with "fancy indexing" -- or anything other  
than a brute for-loop for that matter.

The basic gist is that I have an array representing n images, of shape  
(n, x, y). I also have a "map" of shape (x, y), which contains indices  
in the range [0, n-1]. I then want to construct the "composite" image,  
of shape (x, y), with pixels selected from the n source images as per  
the indices in the map, i.e.:

composite[x, y] = images[map[x, y], x, y]
for all (x, y).

Now, I can't figure out if there's an easy way to express this in  
numpy. For that matter, I can't even figure out a simple way to do the  
1D version of the same:

composite[i] = images[map[i], i]
where composite and map have shape (m,), and images has shape (n, m).

Can anyone assist? Surely there's something simple that I'm just not  
seeing.

Thanks,

Zach


From robert.kern at gmail.com  Wed Oct  8 21:25:55 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 8 Oct 2008 20:25:55 -0500
Subject: [Numpy-discussion] 2D (or n-d) fancy indexing?
In-Reply-To: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
Message-ID: <3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>

On Wed, Oct 8, 2008 at 20:00, Zachary Pincus <zachary.pincus at yale.edu> wrote:
> Hello all,
>
> I'm doing something silly with images and am unable to figure out the
> right way to express this with "fancy indexing" -- or anything other
> than a brute for-loop for that matter.
>
> The basic gist is that I have an array representing n images, of shape
> (n, x, y). I also have a "map" of shape (x, y), which contains indices
> in the range [0, n-1]. I then want to construct the "composite" image,
> of shape (x, y), with pixels selected from the n source images as per
> the indices in the map, i.e.:
>
> composite[x, y] = images[map[x, y], x, y]
> for all (x, y).
>
> Now, I can't figure out if there's an easy way to express this in
> numpy. For that matter, I can't even figure out a simple way to do the
> 1D version of the same:
>
> composite[i] = images[map[i], i]
> where composite and map have shape (m,), and images has shape (n, m).
>
> Can anyone assist? Surely there's something simple that I'm just not
> seeing.

You need to give an array for each axis. Each of these arrays will be
broadcast against each other to form three arrays of the desired shape
of composite. This is discussed in the manual here:

  http://mentat.za.net/numpy/refguide/indexing.xhtml#indexing-multi-dimensional-arrays

Conceptually, you need arrays A, B, and C such that

  composite[x,y] == images[A[x,y], B[x,y], C[x,y]]
  for all x,y

You already have A, you just need to construct B and C correctly.

Here is an example:

In [26]: from numpy import *

In [27]: Nx = 480

In [28]: Ny = 640

In [29]: N = 100

In [30]: images = random.randint(0, 1000, size=(N, Nx, Ny))

In [31]: map = random.randint(0, N, size=(Nx, Ny))

In [32]: composite = images[map, arange(Nx)[:,newaxis], arange(Ny)]

In [33]: for x in range(Nx):
   ....:     for y in range(Ny):
   ....:         assert composite[x,y] == images[map[x,y],x,y]
   ....:
   ....:

In [34]:


When arange(Nx)[:,newaxis] and arange(Ny) get broadcasted with map,
you get (480,640) arrays like you would get with mgrid[0:Nx,0:Ny].

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From zachary.pincus at yale.edu  Wed Oct  8 21:44:15 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 8 Oct 2008 21:44:15 -0400
Subject: [Numpy-discussion] 2D (or n-d) fancy indexing?
In-Reply-To: <3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>
Message-ID: <19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>

> You need to give an array for each axis. Each of these arrays will be
> broadcast against each other to form three arrays of the desired shape
> of composite. This is discussed in the manual here:
>
>  http://mentat.za.net/numpy/refguide/indexing.xhtml#indexing-multi-dimensional-arrays
>
> Conceptually, you need arrays A, B, and C such that
>
>  composite[x,y] == images[A[x,y], B[x,y], C[x,y]]
>  for all x,y

Aha -- thanks especially for the clear illustration of what B and C  
need to be. That really helps.

> In [32]: composite = images[map, arange(Nx)[:,newaxis], arange(Ny)]
>
> When arange(Nx)[:,newaxis] and arange(Ny) get broadcasted with map,
> you get (480,640) arrays like you would get with mgrid[0:Nx,0:Ny].

That's very handy indeed. Thanks for your help!

Zach


From stefan at sun.ac.za  Thu Oct  9 03:53:52 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 9 Oct 2008 09:53:52 +0200
Subject: [Numpy-discussion] 2D (or n-d) fancy indexing?
In-Reply-To: <19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>
	<19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>
Message-ID: <9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>

Hi Zach

2008/10/9 Zachary Pincus <zachary.pincus at yale.edu>:
>> Conceptually, you need arrays A, B, and C such that
>>
>>  composite[x,y] == images[A[x,y], B[x,y], C[x,y]]
>>  for all x,y
>
> Aha -- thanks especially for the clear illustration of what B and C
> need to be. That really helps.

I also summarised some of the posts on this topic between Jack Cooke
and Robert Kern in my SciPy'08 slides:

http://mentat.za.net/numpy/numpy_advanced_slides/

I don't know if they're much use without the dialogue.  Or maybe
they're better :)

Cheers
St?fan


From ndbecker2 at gmail.com  Thu Oct  9 08:04:52 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 09 Oct 2008 08:04:52 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a matrix
Message-ID: <gcks15$7bt$1@ger.gmane.org>

Suppose I have a function (I wrote in c++) that accepts a numpy 1-d vector.  What is the recommended way to apply it to each row of a matrix, returning a new matrix result?  (Assume the function has signature newvec = f (oldvec))



From david.huard at gmail.com  Thu Oct  9 09:24:45 2008
From: david.huard at gmail.com (David Huard)
Date: Thu, 9 Oct 2008 09:24:45 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
In-Reply-To: <gcks15$7bt$1@ger.gmane.org>
References: <gcks15$7bt$1@ger.gmane.org>
Message-ID: <91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>

Neal,

Look at: apply_along_axis


David

On Thu, Oct 9, 2008 at 8:04 AM, Neal Becker <ndbecker2 at gmail.com> wrote:

> Suppose I have a function (I wrote in c++) that accepts a numpy 1-d vector.
>  What is the recommended way to apply it to each row of a matrix, returning
> a new matrix result?  (Assume the function has signature newvec = f
> (oldvec))
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081009/db91dc90/attachment.html>

From ndbecker2 at gmail.com  Thu Oct  9 09:40:06 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 09 Oct 2008 09:40:06 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
References: <gcks15$7bt$1@ger.gmane.org>
	<91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>
Message-ID: <gcl1jn$sgr$1@ger.gmane.org>

David Huard wrote:

> Neal,
> 
> Look at: apply_along_axis
> 
> 
I guess it'd be:

b = empty_like(a)
for row in a.shape[0]:
  b[row,:] = apply_along_axis (func, row, a)

I don't suppose there is a way to do this without explicitly writing a loop.



From zachary.pincus at yale.edu  Thu Oct  9 10:50:29 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Thu, 9 Oct 2008 10:50:29 -0400
Subject: [Numpy-discussion] 2D (or n-d) fancy indexing?
In-Reply-To: <9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>
	<19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>
	<9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>
Message-ID: <B058DC22-0507-40C0-9A8C-5B0C3ABF433A@yale.edu>

> http://mentat.za.net/numpy/numpy_advanced_slides/

Those slides are really useful! Thanks a ton.


From lists.20.chth at xoxy.net  Thu Oct  9 11:45:32 2008
From: lists.20.chth at xoxy.net (ctw)
Date: Thu, 9 Oct 2008 11:45:32 -0400
Subject: [Numpy-discussion] dtype behavior
Message-ID: <c78dc03d0810090845n334773b1w6114b16299c25962@mail.gmail.com>

Hi -- Can somebody here explain the following behavior:

In [1]: tst = np.array([5.])

In [2]: tst
Out[2]: array([ 5.])

In [3]: tst.shape
Out[3]: (1,)

In [4]: tst.dtype
Out[4]: dtype('float64')

In [5]: tst.dtype = np.int

In [6]: tst
Out[6]: array([         0, 1075052544])

In [7]: tst.dtype
Out[7]: dtype('int32')

In [8]: tst.shape
Out[8]: (2,)


Is this a bug? I'm running numpy version 1.1.1 and was trying to
convert the float array([5.]) to an int array([5]).


From oliphant at enthought.com  Thu Oct  9 11:53:27 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 09 Oct 2008 10:53:27 -0500
Subject: [Numpy-discussion] dtype behavior
In-Reply-To: <c78dc03d0810090845n334773b1w6114b16299c25962@mail.gmail.com>
References: <c78dc03d0810090845n334773b1w6114b16299c25962@mail.gmail.com>
Message-ID: <48EE28F7.10203@enthought.com>

ctw wrote:
> Hi -- Can somebody here explain the following behavior:
>
> In [1]: tst = np.array([5.])
>
> In [2]: tst
> Out[2]: array([ 5.])
>
> In [3]: tst.shape
> Out[3]: (1,)
>
> In [4]: tst.dtype
> Out[4]: dtype('float64')
>
> In [5]: tst.dtype = np.int
>
> In [6]: tst
> Out[6]: array([         0, 1075052544])
>
> In [7]: tst.dtype
> Out[7]: dtype('int32')
>
> In [8]: tst.shape
> Out[8]: (2,)
>
>
>   
Setting attributes of the array always just change the information about 
the array, they never change the memory the array points to.

In this case you've taken the bits that represent float64 and 
re-interpreted them as int32 (that's why you know have a length 2 array).

So, you are exploring the floating-point bit-pattern on your computer.

If you want to "cast" to another data-type, then you need to use the 
astype method:

tst = tst.astype(np.int)

-Travis



From plucek at ssaris.com  Thu Oct  9 12:17:52 2008
From: plucek at ssaris.com (Paul Lucek)
Date: Thu, 9 Oct 2008 12:17:52 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
Message-ID: <3633FE192D10364D889D84EB77185F3E73AF27@rxrex1.ssaris.com>

Thanks Hanni!  That did it.  Numpy builds and installs by commenting
out:

 

#ifndef HAVE_FREXPF

static float frexpf(float x, int * i)

{

    return (float)frexp((double)(x), i);

}

#endif

#ifndef HAVE_LDEXPF

static float ldexpf(float x, int i)

{

    return (float)ldexp((double)(x), i);

}

#endif

 

in numpy-1.2.0\numpy\core\src\umathmodule.c.src


NOTICE- This communication (including any attachments) contains confidential and/or privileged information and is intended only for the use of the individual(s) to whom it is addressed for a specific purpose and is protected by law. Any review, use, distribution, disclosure, alteration, copying, transmittal or re-transmittal by persons who are not intended recipients of this communication may be a violation of law and is strictly prohibited. If you are not the intended recipient, please permanently delete all copies of this communication and any attachments from your computer system, destroy any hard copies, and immediately notify the sender or SSARIS Advisors, LLC at compliance at ssaris.com or (203) 328-7200. No waiver of confidentiality or privilege is made by mistransmission.

Any views expressed in this communication are those of the individual sender. This communication and any attachments hereto are for informational purposes only and should not be construed as an offer to sell interests or shares in any investment vehicle managed by SSARIS Advisors, LLC or its affiliates. Any information regarding trading performance must be considered in conjunction with the appropriate disclosure documents. Past performance is not necessarily indicative of future results.

SSARIS Advisors, LLC reserves the right to monitor all communications through its networks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081009/0f4a6747/attachment.html>

From david.huard at gmail.com  Thu Oct  9 13:01:33 2008
From: david.huard at gmail.com (David Huard)
Date: Thu, 9 Oct 2008 13:01:33 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
In-Reply-To: <gcl1jn$sgr$1@ger.gmane.org>
References: <gcks15$7bt$1@ger.gmane.org>
	<91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>
	<gcl1jn$sgr$1@ger.gmane.org>
Message-ID: <91cf711d0810091001n6abf6fe8ie2cb145d8406e137@mail.gmail.com>

On Thu, Oct 9, 2008 at 9:40 AM, Neal Becker <ndbecker2 at gmail.com> wrote:

> David Huard wrote:
>
> > Neal,
> >
> > Look at: apply_along_axis
> >
> >
> I guess it'd be:
>
> b = empty_like(a)
> for row in a.shape[0]:
>  b[row,:] = apply_along_axis (func, row, a)
>

> I don't suppose there is a way to do this without explicitly writing a
> loop.


Have you tried

b = apply_along_axis(func, 1, a)

It should work.


>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081009/6b8038c3/attachment.html>

From aisaac at american.edu  Thu Oct  9 13:39:19 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 09 Oct 2008 13:39:19 -0400
Subject: [Numpy-discussion] OT (NumPy slides)
In-Reply-To: <B058DC22-0507-40C0-9A8C-5B0C3ABF433A@yale.edu>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>	<19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>	<9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>
	<B058DC22-0507-40C0-9A8C-5B0C3ABF433A@yale.edu>
Message-ID: <48EE41C7.5000103@american.edu>

 >> http://mentat.za.net/numpy/numpy_advanced_slides/

Zachary Pincus wrote:
> Those slides are really useful! Thanks a ton.

Nice content!
And I have to add,
S5 produces a beautiful show.

Alan Isaac

PS What did you use to produce the 3d figures?

PPS Do you know why the display get muddled if
you switch to full screen on FireFox?  It seems
to be a problem with the fixed-width font display?
This is the first time I've ever seen something look
better on IE than FireFox.  Is it just the display
I am using at work?


From ndbecker2 at gmail.com  Thu Oct  9 14:48:57 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 09 Oct 2008 14:48:57 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
References: <gcks15$7bt$1@ger.gmane.org>
	<91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>
	<gcl1jn$sgr$1@ger.gmane.org>
	<91cf711d0810091001n6abf6fe8ie2cb145d8406e137@mail.gmail.com>
Message-ID: <gcljn7$52a$1@ger.gmane.org>

David Huard wrote:

> On Thu, Oct 9, 2008 at 9:40 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> 
>> David Huard wrote:
>>
>> > Neal,
>> >
>> > Look at: apply_along_axis
>> >
>> >
>> I guess it'd be:
>>
>> b = empty_like(a)
>> for row in a.shape[0]:
>>  b[row,:] = apply_along_axis (func, row, a)
>>
> 
>> I don't suppose there is a way to do this without explicitly writing a
>> loop.
> 
> 
> Have you tried
> 
> b = apply_along_axis(func, 1, a)
> 
> It should work.
> 
Yes, thanks.

The doc for apply_along_axis is not clear.

For one thing, it says:
The output array. The shape of outarr depends on the return value of func1d. If it returns arrays with the same shape as the input arrays it receives, outarr has the same shape as arr.

What happens if the 'if' clause is not true?




From aisaac at american.edu  Thu Oct  9 14:54:32 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 09 Oct 2008 14:54:32 -0400
Subject: [Numpy-discussion] OT (NumPy slides)
In-Reply-To: <48EE41C7.5000103@american.edu>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>	<19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>	<9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>	<B058DC22-0507-40C0-9A8C-5B0C3ABF433A@yale.edu>
	<48EE41C7.5000103@american.edu>
Message-ID: <48EE5368.8060106@american.edu>

>>> http://mentat.za.net/numpy/numpy_advanced_slides/ 

Alan G Isaac wrote:
> Do you know why the display get muddled if 
> you switch to full screen on FireFox? 

I received this reply:

         Whenever you resize an S5 display (switch to
         fullscreen or just resize the window), you have to
         reload the page (Ctrl-R or Cmd-R). S5 sizes text
         proportionally to the display size once when loaded.

And it works.

Cheers,
Alan Isaac



From bolme1234 at comcast.net  Thu Oct  9 16:28:34 2008
From: bolme1234 at comcast.net (David Bolme)
Date: Thu, 9 Oct 2008 14:28:34 -0600
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
	<26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
	<ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
Message-ID: <F28BD98D-E284-44DD-91BF-5C288F714456@comcast.net>

I have written up basic nearest neighbor algorithm.  It does a brute  
force search so it will be slower than kdtrees as the number of points  
gets large.  It should however work well for high dimensional data.  I  
have also added the option for user defined distance measures.  The  
user can set a default "p".  "p" has the same functionality if it is a  
float.  "p" can also be a function that computes a distance matrix or  
the measure can be selected using the strings: "Manhattan",  
"Euclidean", or "Correlation".

https://pyvision.svn.sourceforge.net/svnroot/pyvision/trunk/src/pyvision/vector/knn.py

The interface is similar to Anne's code and in many cases can be used  
as a drop in replacement.  I have posted the code to my own project  
because I have a short term need and I do not have access to the scipy  
repository.  Feel free to include the code with scipy under the scipy  
license.

I did find a typo your documentation.
typo "trie -> tree" - ... kd-tree is a binary trie, each of whose ...

Also I found the use of k in the documentation some what confusing as  
it is the dimensionality of the data points in the kd-tree and the  
number of neighbors for k-nearest neighbors.

>> I believe that with mean subtracted and unit length vectors, a
>> Euclidean knn algorithm will produces the same result as if the
>> vectors were compared using correlation.  I am not sure if kd-trees
>> will perform well on the normalized vectors as they have a very
>> specific geometry.  If my math checks out it may be worth adding
>> Pearson's correlation as a default option or as a separate class.
>
> Actually it's probably easier if the user just does the  
> prenormalization.

I agree.  I think we should keep your class as-is and maybe create a  
class that wraps the kdtree and handles the normalization for  
correlation.  I would also like to look at cover trees, however that  
will have to wait a couple months until I have more time.

Dave
---
http://www.cs.colostate.edu/~bolme


From lists_ravi at lavabit.com  Thu Oct  9 17:07:37 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Thu, 9 Oct 2008 17:07:37 -0400
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
	(windows-amd64) using VS9
In-Reply-To: <789d27b10810080756r293ee6b4r5b7705df24aa1ccc@mail.gmail.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>
	<789d27b10810080756r293ee6b4r5b7705df24aa1ccc@mail.gmail.com>
Message-ID: <200810091707.38689.lists_ravi@lavabit.com>

On Wednesday 08 October 2008 10:56:02 Hanni Ali wrote:
> We discussed errors you are encountering a few months ago, they are related
> to the compiler directives.
>
> > #ifndef HAVE_FREXPF
> > static float frexpf(float x, int * i)
> > {
> > ? ? return (float)frexp((double)(x), i);
> > }
> > #endif
> > #ifndef HAVE_LDEXPF
> > static float ldexpf(float x, int i)
> > {
> > ? ? return (float)ldexp((double)(x), i);
> > }
> > #endif
>
> Commenting out this section at line 64 allow compilation and has no ill
> effects.

Given that commenting out the section above allows numpy to compile without 
any apparent side effects, is there any chance we could get "experimental" 
binaries of numpy 1.2.0 for python 2.6? I do understand that a negative answer 
is very likely and the reasons therefor.

Regards,
Ravi




From stefan at sun.ac.za  Thu Oct  9 17:14:09 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 9 Oct 2008 23:14:09 +0200
Subject: [Numpy-discussion] OT (NumPy slides)
In-Reply-To: <48EE41C7.5000103@american.edu>
References: <E486C8AF-CFE0-4E1D-8F17-FE5739631A51@yale.edu>
	<3d375d730810081825x75ca88c1n4efe0c1fe9a73027@mail.gmail.com>
	<19CF3A93-8B73-4E92-97B8-7C76A1596C8E@yale.edu>
	<9457e7c80810090053u61b4886cu2e1342b5f576fe61@mail.gmail.com>
	<B058DC22-0507-40C0-9A8C-5B0C3ABF433A@yale.edu>
	<48EE41C7.5000103@american.edu>
Message-ID: <9457e7c80810091414r4ffb1c23hf2631cbc7d3d6de1@mail.gmail.com>

Hi Alan

2008/10/9 Alan G Isaac <aisaac at american.edu>:
>  >> http://mentat.za.net/numpy/numpy_advanced_slides/
> Nice content!

Thanks!  As you can see, I enjoyed myself at SciPy'08 :)

> And I have to add,
> S5 produces a beautiful show.

This slide show incorporates the changes from S5 Reloaded:

http://www.netzgesta.de/S5/references.php

I put the sources for generating the slides on

http://mentat.za.net/numpy/numpy_advanced_slides_sources.tar.bz2

This includes the custom rst2s5 script that does Python highlighting
with Pyglet, and which executes all code snippets as tests.

> PS What did you use to produce the 3d figures?

I'm not sure I want to mention it out loud, but... Google Sketchup,
scripted using Ruby.

I hope someone finds it useful.

Regards
St?fan


From millman at berkeley.edu  Thu Oct  9 18:01:37 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 9 Oct 2008 15:01:37 -0700
Subject: [Numpy-discussion] Please backport fixes to the 1.2.x branch
In-Reply-To: <c7009a550810051959n23d45f55ja06416cb1ae15b62@mail.gmail.com>
References: <c7009a550810051959n23d45f55ja06416cb1ae15b62@mail.gmail.com>
Message-ID: <c7009a550810091501y56f46985h2a3f6c459d51d6eb@mail.gmail.com>

On Sun, Oct 5, 2008 at 7:59 PM, Jarrod Millman <millman at berkeley.edu> wrote:
> I would like to get a 1.2.1 release out ASAP.  There are several
> bug-fixes on the trunk that need to be backported.  If you have made a
> bug-fix to the trunk that you have been waiting to backport to the
> 1.2.x branch, please do so now:
> http://svn.scipy.org/svn/numpy/branches/1.2.x
>
> Ideally, I would like to freeze the branch for the 1.2.1 release in
> about 1 week.  Please let me know if you need more time or if there is
> something in particular that you would like to see backported.

Hey,

Is anyone planning to back port anymore fixes to the 1.2.x branch?

So this is all that has been back ported:

bug fix for subclassing object arrays:
http://projects.scipy.org/scipy/numpy/changeset/5891

MaskedArray fixes:
http://projects.scipy.org/scipy/numpy/changeset/5936

Python 2.4 compatible lookfor:
http://projects.scipy.org/scipy/numpy/changeset/5945

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Thu Oct  9 18:14:09 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 9 Oct 2008 15:14:09 -0700
Subject: [Numpy-discussion] Please backport fixes to the 1.2.x branch
In-Reply-To: <c7009a550810091501y56f46985h2a3f6c459d51d6eb@mail.gmail.com>
References: <c7009a550810051959n23d45f55ja06416cb1ae15b62@mail.gmail.com>
	<c7009a550810091501y56f46985h2a3f6c459d51d6eb@mail.gmail.com>
Message-ID: <c7009a550810091514ifb5ba7fpc5599d50cc873760@mail.gmail.com>

I would also like to back port revision 5833:
http://projects.scipy.org/scipy/numpy/changeset/5833

Are there any other fixes that should be back ported?

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david at ar.media.kyoto-u.ac.jp  Fri Oct 10 03:06:26 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 10 Oct 2008 16:06:26 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <200810091707.38689.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>	<789d27b10810080756r293ee6b4r5b7705df24aa1ccc@mail.gmail.com>
	<200810091707.38689.lists_ravi@lavabit.com>
Message-ID: <48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>

Ravi wrote:
>
>
> Given that commenting out the section above allows numpy to compile without 
> any apparent side effects, is there any chance we could get "experimental" 
> binaries of numpy 1.2.0 for python 2.6? I do understand that a negative answer 
> is very likely and the reasons therefor.
>   

I started a wiki page on the issues related to windows, 64 bits and
python 2.6 (those issues are somewhat related at some level):

http://scipy.org/scipy/numpy/wiki/MicrosoftToolchainSupport

If you want to help, you can try solving one problem. In particular, if
you know how to build ATLAS with Visual studio (for 64 bits support), it
would be really helpful,

cheers,

David


From scotta_2002 at yahoo.com  Fri Oct 10 07:02:21 2008
From: scotta_2002 at yahoo.com (Scott Askey)
Date: Fri, 10 Oct 2008 04:02:21 -0700 (PDT)
Subject: [Numpy-discussion] matlab ss2tf in python/symbols in linear
	algebra/transfer functions
Message-ID: <893083.2737.qm@web36501.mail.mud.yahoo.com>




> Subject: matlab ss2tf in python/symbol in linear algebra/transfer functions
> To: numpy-discussion at scipy.org
> Date: Thursday, October 9, 2008, 5:36 PM
> Is there a clever way to do symbolic linear algebra with
> python? What is the prober tool for doing linear algebra
> with matrices containing symbols? 
> 
> 
> V/R
> 
> Scott


      


From niki.spahiev at gmail.com  Fri Oct 10 07:48:46 2008
From: niki.spahiev at gmail.com (Niki Spahiev)
Date: Fri, 10 Oct 2008 14:48:46 +0300
Subject: [Numpy-discussion] NURBS to arc
Message-ID: <gcnfeq$g4n$1@ger.gmane.org>

Hello,

can i use numpy for handling NURBS splines? Especially i like to make 
arc approximation of NURBS.

thanks,

Niki



From gael.varoquaux at normalesup.org  Fri Oct 10 07:56:47 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 10 Oct 2008 13:56:47 +0200
Subject: [Numpy-discussion] matlab ss2tf in python/symbols in
	linear	algebra/transfer functions
In-Reply-To: <893083.2737.qm@web36501.mail.mud.yahoo.com>
References: <893083.2737.qm@web36501.mail.mud.yahoo.com>
Message-ID: <20081010115647.GA26097@phare.normalesup.org>

On Fri, Oct 10, 2008 at 04:02:21AM -0700, Scott Askey wrote:
> > Subject: matlab ss2tf in python/symbol in linear algebra/transfer functions
> > To: numpy-discussion at scipy.org
> > Date: Thursday, October 9, 2008, 5:36 PM
> > Is there a clever way to do symbolic linear algebra with
> > python? What is the prober tool for doing linear algebra
> > with matrices containing symbols? 

Have a look at sympy: http://code.google.com/p/sympy/

Ga?l


From gael.varoquaux at normalesup.org  Fri Oct 10 08:02:35 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 10 Oct 2008 14:02:35 +0200
Subject: [Numpy-discussion] Vectorizing "dot" on the two last axis
Message-ID: <20081010120235.GB26097@phare.normalesup.org>

Hi list,

I have been unable to vectorize the following operation::

    window_size = 10
    nb_windows = 819
    nb_clusters = 501
    restricted_series = np.random.random(size=(window_size, nb_clusters, 
						    nb_windows))
    this_cov = np.zeros((nb_windows, nb_clusters, nb_clusters))
    print >>sys.stderr, "alive"
    for index, window in enumerate(restricted_series):
        this_cov[index, ...] = np.dot(window, window.T)


The last for loop is the one I am unhappy with.

Note that it is fairly easy to get huge arrays when trying the vectorize
this through np.dot or np.tensordot.

I am unhappy with myself: I feel that it should be possible to vectorize
this operation, but I cannot figure it out.

Cheers,

Ga?l


From dagss at student.matnat.uio.no  Fri Oct 10 08:03:42 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 10 Oct 2008 14:03:42 +0200
Subject: [Numpy-discussion] matlab ss2tf in python/symbols
 in	linear	algebra/transfer functions
In-Reply-To: <20081010115647.GA26097@phare.normalesup.org>
References: <893083.2737.qm@web36501.mail.mud.yahoo.com>
	<20081010115647.GA26097@phare.normalesup.org>
Message-ID: <48EF449E.2070806@student.matnat.uio.no>

Gael Varoquaux wrote:
> On Fri, Oct 10, 2008 at 04:02:21AM -0700, Scott Askey wrote:
>   
>>> Subject: matlab ss2tf in python/symbol in linear algebra/transfer functions
>>> To: numpy-discussion at scipy.org
>>> Date: Thursday, October 9, 2008, 5:36 PM
>>> Is there a clever way to do symbolic linear algebra with
>>> python? What is the prober tool for doing linear algebra
>>> with matrices containing symbols? 
>>>       
>
> Have a look at sympy: http://code.google.com/p/sympy/
>   
Also, depending on usecase, perhaps Sage: www.sagemath.org

If the symbols happen to be restricted to polynomials over the same 
variables, for instance, then I think Sage likely has highly optimized 
algorithms for dealing with those (though I didn't try it).

Dag Sverre


From michael.abshoff at googlemail.com  Fri Oct 10 08:12:37 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 10 Oct 2008 05:12:37 -0700
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python
 2.6	(windows-amd64) using VS9
In-Reply-To: <48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<48ECC2CA.5010002@ar.media.kyoto-u.ac.jp>	<789d27b10810080756r293ee6b4r5b7705df24aa1ccc@mail.gmail.com>	<200810091707.38689.lists_ravi@lavabit.com>
	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>
Message-ID: <48EF46B5.6040801@gmail.com>

David Cournapeau wrote:

Hi David,

> 
> I started a wiki page on the issues related to windows, 64 bits and
> python 2.6 (those issues are somewhat related at some level):
> 
> http://scipy.org/scipy/numpy/wiki/MicrosoftToolchainSupport

Cool.

> If you want to help, you can try solving one problem. In particular, if
> you know how to build ATLAS with Visual studio (for 64 bits support), it
> would be really helpful,

The problem with 64 bit ATLAS support for MSVC is that as is that ATLAS 
uses (AFAIK) assembly code that is not compilable with the MSVC 
toolchain. Since the official MinGW cannot create 64 bit code (there is 
some experimental support I have not tried yet) the only hope at the 
moment (without converting the assembly) to use the Intel toolchain. I 
have not tried that yet.

The current ATLAS code even requires Cygwin, but there was recent 
activity on the ATLAS mailing list to support MinGW only. There are also 
issues with threading support with MinGW to winpthread, so there is some 
work ahead to fully support ATLAS with MSVC.

Clint Whaley is supposed to speak at Sage Days 11 in Austin in about a 
month and I had planned to investigate the possibilities of native 64 
bit ATLAS support for VC9. I had planned to spend some days after SD 11 
at Enthought and work on MSVC build issues as well as 64 bit OSX stuff 
for example, but I need to make plans shortly since I need to buy my 
plane ticket soon

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From peridot.faceted at gmail.com  Fri Oct 10 09:03:14 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 10 Oct 2008 09:03:14 -0400
Subject: [Numpy-discussion] Vectorizing "dot" on the two last axis
In-Reply-To: <20081010120235.GB26097@phare.normalesup.org>
References: <20081010120235.GB26097@phare.normalesup.org>
Message-ID: <ce557a360810100603j1e69c064o7918260abd052a89@mail.gmail.com>

2008/10/10 Gael Varoquaux <gael.varoquaux at normalesup.org>:

> I have been unable to vectorize the following operation::
>
>    window_size = 10
>    nb_windows = 819
>    nb_clusters = 501
>    restricted_series = np.random.random(size=(window_size, nb_clusters,
>                                                    nb_windows))
>    this_cov = np.zeros((nb_windows, nb_clusters, nb_clusters))
>    print >>sys.stderr, "alive"
>    for index, window in enumerate(restricted_series):
>        this_cov[index, ...] = np.dot(window, window.T)
>
>
> The last for loop is the one I am unhappy with.
>
> Note that it is fairly easy to get huge arrays when trying the vectorize
> this through np.dot or np.tensordot.
>
> I am unhappy with myself: I feel that it should be possible to vectorize
> this operation, but I cannot figure it out.

I am pretty sure that, unfortunately, there is no way to vectorize
this without an intermediate array substantially larger than either
the inputs or the outputs. (Add one to the tally of people wishing for
ufunc-like linear algebra.) From a computational point of view, this
isn't particularly a problem: the intermediate array cannot be large
enough to be a problem without the slices along at least one axis
being large enough that for loop overhead is pretty minor. So you can
just loop over this axis. Coding-wise, of course, this is a royal
pain: unless you know ahead of time which axis is the smallest, you
need to code several versions of your routine, and each version must
include a for loop (just what numpy is supposed to help avoid).

So: ufunc-like linear algebra would be great, but in the meantime, I
guess we all learn to live with for loops.

Anne


From stefan at sun.ac.za  Fri Oct 10 09:25:18 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 10 Oct 2008 15:25:18 +0200
Subject: [Numpy-discussion] NURBS to arc
In-Reply-To: <gcnfeq$g4n$1@ger.gmane.org>
References: <gcnfeq$g4n$1@ger.gmane.org>
Message-ID: <9457e7c80810100625s552c6862uf76eab07118a3d6c@mail.gmail.com>

Hi Niki

2008/10/10 Niki Spahiev <niki.spahiev at gmail.com>:
> can i use numpy for handling NURBS splines? Especially i like to make
> arc approximation of NURBS.

I have Runten Tenfjord's NURBs module, modified for NumPy, as part of
my (incomplete) super-resolution library:

http://mentat.za.net/supreme

Regards
St?fan


From lists_ravi at lavabit.com  Fri Oct 10 10:00:48 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Fri, 10 Oct 2008 10:00:48 -0400
Subject: [Numpy-discussion]
 =?iso-8859-1?q?can=27t_build_numpy_1=2E2=2E0_u?=
 =?iso-8859-1?q?nder_python_2=2E6_=28windows-amd64=29_using_VS9?=
In-Reply-To: <48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>
	<200810091707.38689.lists_ravi@lavabit.com>
	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>
Message-ID: <200810101000.48555.lists_ravi@lavabit.com>

On Friday 10 October 2008 03:06:26 David Cournapeau wrote:
> > Given that commenting out the section above allows numpy to compile
> > without any apparent side effects, is there any chance we could get
> > "experimental" binaries of numpy 1.2.0 for python 2.6? I do understand
> > that a negative answer is very likely and the reasons therefor.
> > ?
>
> I started a wiki page on the issues related to windows, 64 bits and
> python 2.6 (those issues are somewhat related at some level):
>
> http://scipy.org/scipy/numpy/wiki/MicrosoftToolchainSupport
>
> If you want to help, you can try solving one problem. In particular, if
> you know how to build ATLAS with Visual studio (for 64 bits support), it
> would be really helpful,

Michael Abshoff already responded to the ATLAS question. I don't have access 
to a 64-bit Windows. Given the volume of legacy 32-bit applications where I 
work, there is no chance of 64-bit Windows access for me for at least 2 years.

VS2008 with 32-bit windows should not have any problems (as you mentioned on 
the Wiki page referenced above). What needs to be done to figure out msvc9 
support on mingw and how can I help? I am a Windows n00b (mostly by choice) 
when it comes to platform-specific issues.

Regards,
Ravi




From david at ar.media.kyoto-u.ac.jp  Fri Oct 10 09:58:49 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 10 Oct 2008 22:58:49 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <200810101000.48555.lists_ravi@lavabit.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>
	<200810101000.48555.lists_ravi@lavabit.com>
Message-ID: <48EF5F99.3010403@ar.media.kyoto-u.ac.jp>

Ravi wrote:
>
> Michael Abshoff already responded to the ATLAS question. I don't have access 
> to a 64-bit Windows. Given the volume of legacy 32-bit applications where I 
> work, there is no chance of 64-bit Windows access for me for at least 2 years.

Windows 64 actually has a very nice feature: WoW (windows on windows).
It can executes any 32 bits software, AFAIK (which does not run in ring
0 of course).

>
> VS2008 with 32-bit windows should not have any problems (as you mentioned on 
> the Wiki page referenced above).

I wish it were true :) I can't build numpy with mingw ATM, because of
bugs in mingw. Things like:

http://bugs.python.org/issue3308

>  What needs to be done to figure out msvc9 
> support on mingw and how can I help? I am a Windows n00b (mostly by choice) 
> when it comes to platform-specific issues.

Then, I am afraid you won't be of much help, ufortunately.

cheers,

David


From michael.abshoff at googlemail.com  Fri Oct 10 10:51:28 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 10 Oct 2008 07:51:28 -0700
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <48EF5F99.3010403@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>	<200810101000.48555.lists_ravi@lavabit.com>
	<48EF5F99.3010403@ar.media.kyoto-u.ac.jp>
Message-ID: <48EF6BF0.9080708@gmail.com>

David Cournapeau wrote:
> Ravi wrote:

Hi,

>> Michael Abshoff already responded to the ATLAS question. I don't have access 
>> to a 64-bit Windows. Given the volume of legacy 32-bit applications where I 
>> work, there is no chance of 64-bit Windows access for me for at least 2 years.
> 
> Windows 64 actually has a very nice feature: WoW (windows on windows).
> It can executes any 32 bits software, AFAIK (which does not run in ring
> 0 of course).

Well, I think that having a 64 bit native build of numpy/scipy using an 
efficient and non-commercial licensed BLAS/Lapack (i.e. not Intel MKL) 
can't be a bad thing :)

>> VS2008 with 32-bit windows should not have any problems (as you mentioned on 
>> the Wiki page referenced above).
> 
> I wish it were true :) I can't build numpy with mingw ATM, because of
> bugs in mingw. Things like:
> 
> http://bugs.python.org/issue3308

It might be possible to build python [2.5|2.6|3.0] with MinGW itself to 
avoid the runtime issue. At least Python 2.4 had problems when building 
it with MinGW and I never investigated if 2.5.x had fixed those issues.

>>  What needs to be done to figure out msvc9 
>> support on mingw and how can I help? I am a Windows n00b (mostly by choice) 
>> when it comes to platform-specific issues.
> 
> Then, I am afraid you won't be of much help, ufortunately.
> 
> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From david at ar.media.kyoto-u.ac.jp  Fri Oct 10 11:05:16 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 11 Oct 2008 00:05:16 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <48EF6BF0.9080708@gmail.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>	<200810101000.48555.lists_ravi@lavabit.com>	<48EF5F99.3010403@ar.media.kyoto-u.ac.jp>
	<48EF6BF0.9080708@gmail.com>
Message-ID: <48EF6F2C.6040509@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
>
> Well, I think that having a 64 bit native build of numpy/scipy using an 
> efficient and non-commercial licensed BLAS/Lapack (i.e. not Intel MKL) 
> can't be a bad thing :)

Yes, of course. But it is useful to be able to use a 32 bits toolchain
to produce 64 bits software.

>
> It might be possible to build python [2.5|2.6|3.0] with MinGW itself to 
> avoid the runtime issue. At least Python 2.4 had problems when building 
> it with MinGW and I never investigated if 2.5.x had fixed those issues.

Not being ABI compatible with Python from python.org is not an option,
and building it with mingw would make it ABI incompatible for sure. I
certainly wished they used an open source compiler to build the official
python binary, but that's not the case.

cheers,

David


From david.huard at gmail.com  Fri Oct 10 11:27:24 2008
From: david.huard at gmail.com (David Huard)
Date: Fri, 10 Oct 2008 11:27:24 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
In-Reply-To: <gcljn7$52a$1@ger.gmane.org>
References: <gcks15$7bt$1@ger.gmane.org>
	<91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>
	<gcl1jn$sgr$1@ger.gmane.org>
	<91cf711d0810091001n6abf6fe8ie2cb145d8406e137@mail.gmail.com>
	<gcljn7$52a$1@ger.gmane.org>
Message-ID: <91cf711d0810100827u735c98d8od32ea85d671680f0@mail.gmail.com>

On Thu, Oct 9, 2008 at 2:48 PM, Neal Becker <ndbecker2 at gmail.com> wrote:

> David Huard wrote:
>
> > On Thu, Oct 9, 2008 at 9:40 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> >
> >> David Huard wrote:
> >>
> >> > Neal,
> >> >
> >> > Look at: apply_along_axis
> >> >
> >> >
> >> I guess it'd be:
> >>
> >> b = empty_like(a)
> >> for row in a.shape[0]:
> >>  b[row,:] = apply_along_axis (func, row, a)
> >>
> >
> >> I don't suppose there is a way to do this without explicitly writing a
> >> loop.
> >
> >
> > Have you tried
> >
> > b = apply_along_axis(func, 1, a)
> >
> > It should work.
> >
> Yes, thanks.
>
> The doc for apply_along_axis is not clear.
>

> For one thing, it says:
> The output array. The shape of outarr depends on the return value of
> func1d. If it returns arrays with the same shape as the input arrays it
> receives, outarr has the same shape as arr.
>
> What happens if the 'if' clause is not true?
>

The shape along the axis is determined by the result's shape of your
function.

def func(x):
   ...:     return x[::2]
   ...:

> [3]: a = random.rand(3,4)

> [4]: a
 <[4]:
array([[ 0.95979758,  0.37350614,  0.77423741,  0.62520089],
       [ 0.69060211,  0.91480227,  0.60105525,  0.20184552],
       [ 0.31540644,  0.19919848,  0.72567385,  0.63987393]])

> [5]: apply_along_axis(func, 1, a)
 <[5]:
array([[ 0.95979758,  0.77423741],
       [ 0.69060211,  0.60105525],
       [ 0.31540644,  0.72567385]])

I've edited the docstring at
http://sd-2116.dedibox.fr/pydocweb/doc/numpy.lib.shape_base.apply_along_axis/

Feel free to improve on it.

David


>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081010/d47dc191/attachment.html>

From michael.abshoff at googlemail.com  Fri Oct 10 13:05:58 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 10 Oct 2008 10:05:58 -0700
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <48EF6F2C.6040509@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>	<200810101000.48555.lists_ravi@lavabit.com>	<48EF5F99.3010403@ar.media.kyoto-u.ac.jp>	<48EF6BF0.9080708@gmail.com>
	<48EF6F2C.6040509@ar.media.kyoto-u.ac.jp>
Message-ID: <48EF8B76.7010200@gmail.com>

David Cournapeau wrote:

Hi David,

> Michael Abshoff wrote:
>> Well, I think that having a 64 bit native build of numpy/scipy using an 
>> efficient and non-commercial licensed BLAS/Lapack (i.e. not Intel MKL) 
>> can't be a bad thing :)
> 
> Yes, of course. But it is useful to be able to use a 32 bits toolchain
> to produce 64 bits software.

Sure, but there isn't even a 32 bit gcc out there that can produce 64 
bit PE binaries (aside from the MinGW fork that AFAIK does not work 
particularly well and allegedly has issues with the cleanliness of some 
of the code which is allegedly the reason that the official MinGW people 
will not touch the code base) . It has been rumored for a while that 
there is a new version of SFU by Microsoft in the works based on gcc 
4.2.x that will be able create 64 bit PE binaries, but I have not 
actually talked to anybody who has access, so it could be just a rumor.

>> It might be possible to build python [2.5|2.6|3.0] with MinGW itself to 
>> avoid the runtime issue. At least Python 2.4 had problems when building 
>> it with MinGW and I never investigated if 2.5.x had fixed those issues.
> 
> Not being ABI compatible with Python from python.org is not an option,
> and building it with mingw would make it ABI incompatible for sure. I
> certainly wished they used an open source compiler to build the official
> python binary, but that's not the case.

Ok, that is a concern I usually do not have since I tend to build my own 
Python :).

I am pretty sure that building Python with MinGW will break ABI 
compatibility with Python 2.6. As has been discussed on this list more 
than once not even Python 2.5 build with MSVC 2003 is really compatible 
with C++ extensions build with MinGW.

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From lseltzer at alumni.caltech.edu  Fri Oct 10 17:26:06 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Fri, 10 Oct 2008 17:26:06 -0400 (EDT)
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
Message-ID: <1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>

Hello Group Members:

Can someone please answer a question about installing Numerical Python and
getting it to work.  I downloaded the files.  They are in
C:\Program Files\Python25

I have the python code and also the new file
numpy-1.2.0-win32-superpack-python2.5

There is also a text file, numpy.wininst, and the last line of this file is:
200 File Copy: C:\Program
Files\Python25\Lib\site-packages\numpy\add_newdocs.pyo

I have these lines in my program:
import Numeric
from Numeric import *

The problem is that when I run my python program in IDLE, I receive the
following error message:

Traceback (most recent call last):
  File "C:\Documents and Settings\music1\My
Documents\Aerostream\aerostream\Reed
Solomon\python-rscode\python-rscode\GF16genroot2a.py", line 18, in
<module>
    import Numeric
ImportError: No module named Numeric
>>>

I would be most appreciative of any clear, detailed instructions on where
the Numeric Python files should be located and how to get this package to
work.

lseltzer at alumni.caltech.edu




From robert.kern at gmail.com  Fri Oct 10 17:32:36 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 10 Oct 2008 16:32:36 -0500
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
Message-ID: <3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>

On Fri, Oct 10, 2008 at 16:26, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
> Hello Group Members:
>
> Can someone please answer a question about installing Numerical Python and
> getting it to work.  I downloaded the files.  They are in
> C:\Program Files\Python25
>
> I have the python code and also the new file
> numpy-1.2.0-win32-superpack-python2.5
>
> There is also a text file, numpy.wininst, and the last line of this file is:
> 200 File Copy: C:\Program
> Files\Python25\Lib\site-packages\numpy\add_newdocs.pyo
>
> I have these lines in my program:
> import Numeric
> from Numeric import *
>
> The problem is that when I run my python program in IDLE, I receive the
> following error message:
>
> Traceback (most recent call last):
>  File "C:\Documents and Settings\music1\My
> Documents\Aerostream\aerostream\Reed
> Solomon\python-rscode\python-rscode\GF16genroot2a.py", line 18, in
> <module>
>    import Numeric
> ImportError: No module named Numeric
>>>>
>
> I would be most appreciative of any clear, detailed instructions on where
> the Numeric Python files should be located and how to get this package to
> work.

Numeric is an old, obsolete package. numpy is the successor. You
installed numpy, not Numeric. Unfortunately, there is no Python 2.5
binary available for Numeric. I am not familiar with the Reed Solomon
coding package that you are using, so I don't know if there is an
updated version that uses numpy instead of Numeric. If it is open
source, and you can show us the source, we can help you convert it to
use numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From numpy-discussion at maubp.freeserve.co.uk  Fri Oct 10 17:36:16 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Fri, 10 Oct 2008 22:36:16 +0100
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
Message-ID: <320fb6e00810101436u74cb9a0oe73c514f8f8cf0d0@mail.gmail.com>

On Fri, Oct 10, 2008 at 10:26 PM, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
>
> Hello Group Members:
>
> Can someone please answer a question about installing Numerical Python and
> getting it to work.  I downloaded the ... new file
> numpy-1.2.0-win32-superpack-python2.5 [.exe I assume?]

You have installed "numpy" which is the successor to "Numeric", see:
http://numpy.scipy.org/#older_array

Confusingly both have been known as "Numerical Python"!

> I have these lines in my program:
> import Numeric
> from Numeric import *

If you are trying to use an old program written to use Numeric, then
you could install Numeric instead - but it is no longer being
maintained.  Or try replacing those import statements with the
backwards compatiblity layer, e.g.

from Numeric import *

becomes:

from numpy.oldnumeric import *

Peter



Peter


From numpy-discussion at maubp.freeserve.co.uk  Fri Oct 10 17:42:59 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Fri, 10 Oct 2008 22:42:59 +0100
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
	<3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>
Message-ID: <320fb6e00810101442s4a35f1fbxfadef1696aad782e@mail.gmail.com>

Rober Kern wrote:
> Numeric is an old, obsolete package. numpy is the successor. You
> installed numpy, not Numeric. Unfortunately, there is no Python 2.5
> binary available for Numeric.

Actually, there is - but not from the numpy/Numeric/SciPy team as they
want to encourage people to move over to numpy ;)

If you don't mind trusting a third party, you can use this Numeric for
python 2.5 setup exe:
http://biopython.org/DIST/Numeric-24.2.win32-py2.5.exe

But just to be clear - Numeric is not being maintained anymore, so you
should try and move over to numpy.

Peter


From kammeyer at enthought.com  Fri Oct 10 19:00:54 2008
From: kammeyer at enthought.com (Dave Kammeyer)
Date: Fri, 10 Oct 2008 18:00:54 -0500
Subject: [Numpy-discussion] Build problem using mingw on Win32
Message-ID: <48EFDEA6.7010801@enthought.com>

Hi All,

Using the current trunk, I am getting the following build error:

creating build\temp.win32-2.5\Release\build
creating build\temp.win32-2.5\Release\build\src.win32-2.5
creating build\temp.win32-2.5\Release\build\src.win32-2.5\numpy
creating build\temp.win32-2.5\Release\build\src.win32-2.5\numpy\core
creating build\temp.win32-2.5\Release\build\src.win32-2.5\numpy\core\src
compile options: '-Ibuild\src.win32-2.5\numpy\core\src 
-Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core\i
nclude/numpy -Inumpy\core\src -Inumpy\core\include -Ic:\python25\include 
-Ic:\python25\PC -c'
gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes 
-Ibuild\src.win32-2.5\numpy\core\src -Inumpy\core\include 
-Ibuild\src.win32-2.5\numpy\core\include/numpy -Inumpy\core\src 
-Inumpy\core\include -Ic:\python25\include -Ic:\python25\PC -c 
build\src.win32-2.5\numpy\core\src\umathmodule.c -o 
build\temp.win32-2.5\Release\build\src.win32-2.5\numpy\core\src\umathmodule.o 
numpy\core\src\umathmodule.c.src:1451: warning: return type defaults to 
`int'
In file included from numpy\core\src\umathmodule.c.src:1510:
build/src.win32-2.5/numpy/core/include/numpy/__umath_generated.c:173: 
warning: initialization from incompatible pointer type
build/src.win32-2.5/numpy/core/include/numpy/__umath_generated.c: In 
function `InitOperators':
build/src.win32-2.5/numpy/core/include/numpy/__umath_generated.c:285: 
error: `atanhf' undeclared (first use in this function)
build/src.win32-2.5/numpy/core/include/numpy/__umath_generated.c:285: 
error: (Each undeclared identifier is reported only once
build/src.win32-2.5/numpy/core/include/numpy/__umath_generated.c:285: 
error: for each function it appears in.)
numpy\core\src\umathmodule.c.src: At top level:
numpy\core\src\umathmodule.c.src:608: warning: 'BOOL_ones_like' defined 
but not used
error: Command "gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes 
-Ibuild\src.win32-2.5\numpy\core\src -Inumpy\cor
e\include -Ibuild\src.win32-2.5\numpy\core\include/numpy 
-Inumpy\core\src -Inumpy\core\include -Ic:\python25\include 
-Ic:\python25\PC -c build\src.win32-2.5\numpy\core\src\umathmodule.c -o 
build\temp.win32-2.5\Release\build\src.win32-2.5\numpy\core\src\umathmodule.o" 
failed with exit status 1
DEBUG: return value for "c:\python25\python.exe setup.py build 
--compiler=mingw32 config --compiler=mingw32" is 1



From lseltzer at alumni.caltech.edu  Fri Oct 10 22:42:37 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Fri, 10 Oct 2008 22:42:37 -0400 (EDT)
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
Message-ID: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>

This worked:
from numpy.oldnumeric import *

Should I use
from numpy import *
(Does the first statement use outdated software?)

I want to use 2-D arrays.  Please advise me on the best way to do this.



From robert.kern at gmail.com  Sat Oct 11 00:02:37 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 10 Oct 2008 23:02:37 -0500
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
Message-ID: <3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>

On Fri, Oct 10, 2008 at 21:42, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
> This worked:
> from numpy.oldnumeric import *

To be clear, you mean that this worked to replace the statement "from
Numeric import *" in the older code that you mentioned in your
previous thread.

> Should I use
> from numpy import *
> (Does the first statement use outdated software?)

The first statement exposes an API which is mostly compatible with the
older Numeric API. The underlying software is up-to-date, but the
interface is not.

> I want to use 2-D arrays.  Please advise me on the best way to do this.

If you are writing new code, you should use "import numpy" (as an
orthogonal note, "from <foo> import *" is discouraged). Only use
numpy.oldnumeric if you need to use old code and do not have the time
or resources to update it to use the new API.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lseltzer at alumni.caltech.edu  Sat Oct 11 01:16:13 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Sat, 11 Oct 2008 01:16:13 -0400 (EDT)
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
Message-ID: <1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>

I would appreciate it if someone could answer my question without
referring to subjects such as APIs and interfaces, since I am only
concerned with a mathematical application at this time.
In most tutorials, array examples are of the form:
a = array([1, 2, 3], [4, 5, 6] )
The problem with this is that I have an array of size 256 x 256, and I
have had applications with 3-d arrays of size 128x128x128.  In video, you
can arrays such as 640x480, etc.  So filling in the value of each element
is not practical.  I wish that tutorials provided real world examples.
I would appreciate it if someone could give me the actual statements
needed to define and initialize a 2-D array of size NxN, where N can be
any large number, where the initial values of the array elements are all
zeros, but will be changed by the program.
In Matlab, this is done as a = zeros(256,256).  If I try this in python,
it won't let the program overwrite the zeros.




From wbaxter at gmail.com  Sat Oct 11 01:26:39 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Sat, 11 Oct 2008 14:26:39 +0900
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
Message-ID: <e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>

On Sat, Oct 11, 2008 at 2:16 PM, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
> I would appreciate it if someone could answer my question without
> referring to subjects such as APIs and interfaces, since I am only
> concerned with a mathematical application at this time.
> In most tutorials, array examples are of the form:
> a = array([1, 2, 3], [4, 5, 6] )
> The problem with this is that I have an array of size 256 x 256, and I
> have had applications with 3-d arrays of size 128x128x128.  In video, you
> can arrays such as 640x480, etc.  So filling in the value of each element
> is not practical.  I wish that tutorials provided real world examples.
> I would appreciate it if someone could give me the actual statements
> needed to define and initialize a 2-D array of size NxN, where N can be
> any large number, where the initial values of the array elements are all
> zeros, but will be changed by the program.
> In Matlab, this is done as a = zeros(256,256).  If I try this in python,
> it won't let the program overwrite the zeros.

In numpy it's
   import numpy as npy
   a = npy.zeros((256,256))

   a[0,0] = 1.0
   a[200,123] = -42.0
   # etc...

I think you were just missing the extra parentheses in the numpy
version of "zeros"

--bb


From lseltzer at alumni.caltech.edu  Sat Oct 11 01:38:54 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Sat, 11 Oct 2008 01:38:54 -0400 (EDT)
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
Message-ID: <1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>

Thank you. It produced this error message:
NameError: global name 'npy' is not defined
> In numpy it's
>    import numpy as npy
>    a = npy.zeros((256,256))
>
>    a[0,0] = 1.0
>    a[200,123] = -42.0
>    # etc...
>
> I think you were just missing the extra parentheses in the numpy
> version of "zeros"
>
> --bb



From robert.kern at gmail.com  Sat Oct 11 01:46:53 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 11 Oct 2008 00:46:53 -0500
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
Message-ID: <3d375d730810102246q64fe43d2ra7420814f993a40b@mail.gmail.com>

On Sat, Oct 11, 2008 at 00:38, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
> Thank you. It produced this error message:
> NameError: global name 'npy' is not defined

Note the line:

  import numpy as npy

>> In numpy it's
>>    import numpy as npy
>>    a = npy.zeros((256,256))
>>
>>    a[0,0] = 1.0
>>    a[200,123] = -42.0
>>    # etc...
>>
>> I think you were just missing the extra parentheses in the numpy
>> version of "zeros"
>>
>> --bb

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From numpy-discussion at maubp.freeserve.co.uk  Sat Oct 11 07:08:23 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Sat, 11 Oct 2008 12:08:23 +0100
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <1315.71.231.103.249.1223686712.squirrel@mail.alumni.caltech.edu>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
	<3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>
	<320fb6e00810101442s4a35f1fbxfadef1696aad782e@mail.gmail.com>
	<1315.71.231.103.249.1223686712.squirrel@mail.alumni.caltech.edu>
Message-ID: <320fb6e00810110408q60292511r661ff30f1872b4c2@mail.gmail.com>

On Sat, Oct 11, 2008 at 1:58 AM, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
>
> I didn't know there was a difference between Numeric and numpy.  Could
> someone please provide clear, detailed instructions for how to get
> *something* (numeric, numpy) to work so that I can get my work (matrices)
> done?

Hi Linda,

You only sent this to me BTW, I have copiedg this reply back to the list.

>From your original email, I have the impression you are using some
existing code that someone else wrote:

Traceback (most recent call last):
 File "C:\Documents and Settings\music1\My
Documents\Aerostream\aerostream\Reed
Solomon\python-rscode\python-rscode\GF16genroot2a.py", line 18, in
<module>
   import Numeric
ImportError: No module named Numeric

I suggest you try one of the following (in order of preferrence):

I would start by emailing the author(s) and asking if they have any
plans to move their code from Numeric to numpy.

Otherwise, if you are reasonably experienced with python, you could
try modifying their code to switch it from Numeric to numpy.

Finally, what is probably the easiest answer in the short term, just
install Numeric for Python 2.5 using the Windows installer I linked
to.

Peter

P.S. You can have both Numeric and numpy installed at the same time -
that won't cause a problem.


From gael.varoquaux at normalesup.org  Sat Oct 11 07:15:59 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 11 Oct 2008 13:15:59 +0200
Subject: [Numpy-discussion] Vectorizing "dot" on the two last axis
In-Reply-To: <ce557a360810100603j1e69c064o7918260abd052a89@mail.gmail.com>
References: <20081010120235.GB26097@phare.normalesup.org>
	<ce557a360810100603j1e69c064o7918260abd052a89@mail.gmail.com>
Message-ID: <20081011111559.GB22415@phare.normalesup.org>

On Fri, Oct 10, 2008 at 09:03:14AM -0400, Anne Archibald wrote:
> I am pretty sure that, unfortunately, there is no way to vectorize
> this without an intermediate array substantially larger than either
> the inputs or the outputs.

OK, this is what I had found.

> From a computational point of view, this isn't particularly a problem:
> the intermediate array cannot be large enough to be a problem without
> the slices along at least one axis being large enough that for loop
> overhead is pretty minor. 

OK, I here what you are saying: the loop is not costing me that much,
because anyhow the operation would be slow, and the overhead is small. I
have just grown to find that vector operations are easier to read than
loops (to a trained eye).

Thank for confirming my findings,

Ga?l


From david at ar.media.kyoto-u.ac.jp  Sat Oct 11 08:10:34 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 11 Oct 2008 21:10:34 +0900
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <48EF8B76.7010200@gmail.com>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>	<200810101000.48555.lists_ravi@lavabit.com>	<48EF5F99.3010403@ar.media.kyoto-u.ac.jp>	<48EF6BF0.9080708@gmail.com>	<48EF6F2C.6040509@ar.media.kyoto-u.ac.jp>
	<48EF8B76.7010200@gmail.com>
Message-ID: <48F097BA.1080100@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
>
> Sure, but there isn't even a 32 bit gcc out there that can produce 64 
> bit PE binaries (aside from the MinGW fork that AFAIK does not work 
> particularly well and allegedly has issues with the cleanliness of some 
> of the code which is allegedly the reason that the official MinGW people 
> will not touch the code base) .

The biggest problem is that officially, there is still no gcc 4 release
for mingw. I saw a gcc 4 section in cygwin, though, so maybe it is about
to be released. There is no support at all for 64 bits PE in the 3 serie.

I think binutils officially support 64 bits PE (I can build a linux
hosted binutils for 64 bits PE with x86_64-pc-mingw32 as a target, and
it seems to work: disassembling and co). gcc 4 can work, too (you can
build a bootstrap C compiler which targets windows 64 bits IICR). The
biggest problem AFAICS is the runtime (mingw64, which is indeed legally
murky).

>
> Ok, that is a concern I usually do not have since I tend to build my own 
> Python :).

I would say that if you can build python by yourself on windows, you can
certainly build numpy by yourself :) It took me quite a time to be able
to build python on windows by myself from scratch.

cheers,

David


From lseltzer at alumni.caltech.edu  Sat Oct 11 10:19:34 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Sat, 11 Oct 2008 10:19:34 -0400 (EDT)
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <320fb6e00810110408q60292511r661ff30f1872b4c2@mail.gmail.com>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
	<3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>
	<320fb6e00810101442s4a35f1fbxfadef1696aad782e@mail.gmail.com>
	<1315.71.231.103.249.1223686712.squirrel@mail.alumni.caltech.edu>
	<320fb6e00810110408q60292511r661ff30f1872b4c2@mail.gmail.com>
Message-ID: <2394.71.231.103.249.1223734774.squirrel@mail.alumni.caltech.edu>

I did not give permission to have the e-mail sent to the list.
Please do not forawrd anyone's e-mail to a list without permission.
Also, kindly do not send an entire list a message falsely stating that
someone may be using code written by someone else.
I have been reading various tutorials on the Internet, and they offer very
incomplete information.  Also, it is necessary to minimize time spent on
installation and system administration and maximize time spent on math.
>
> Hi Linda,
>
> You only sent this to me BTW, I have copiedg this reply back to the list.
>
> From your original email, I have the impression you are using some
> existing code that someone else wrote:
>
> Traceback (most recent call last):
>  File "C:\Documents and Settings\music1\My
> Documents\Aerostream\aerostream\Reed
> Solomon\python-rscode\python-rscode\GF16genroot2a.py", line 18, in
> <module>
>    import Numeric
> ImportError: No module named Numeric
>
> I suggest you try one of the following (in order of preferrence):
>
> I would start by emailing the author(s) and asking if they have any
> plans to move their code from Numeric to numpy.
>
> Otherwise, if you are reasonably experienced with python, you could
> try modifying their code to switch it from Numeric to numpy.
>
> Finally, what is probably the easiest answer in the short term, just
> install Numeric for Python 2.5 using the Windows installer I linked
> to.
>
> Peter
>
> P.S. You can have both Numeric and numpy installed at the same time -
> that won't cause a problem.
>



From aisaac at american.edu  Sat Oct 11 10:38:45 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 11 Oct 2008 10:38:45 -0400
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <2394.71.231.103.249.1223734774.squirrel@mail.alumni.caltech.edu>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>	<3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>	<320fb6e00810101442s4a35f1fbxfadef1696aad782e@mail.gmail.com>	<1315.71.231.103.249.1223686712.squirrel@mail.alumni.caltech.edu>	<320fb6e00810110408q60292511r661ff30f1872b4c2@mail.gmail.com>
	<2394.71.231.103.249.1223734774.squirrel@mail.alumni.caltech.edu>
Message-ID: <48F0BA75.40200@american.edu>

On 10/11/2008 10:19 AM Linda Seltzer apparently wrote:
> Please do not forawrd anyone's e-mail to a list without 
> permission. 


I'm afraid that is a reversal of discussion list practice.
People on this list are not offering to participate in
private conversations.  They are offering to participate in
public conversations.  If you send a response to an
individual without an apology or at least a request for
privacy, we will assume that you meant to send it to the
list.  (It is preferable to assume that than to assume that
you are rudely violating list etiquette.)

Cheers,
Alan Isaac



From numpy-discussion at maubp.freeserve.co.uk  Sat Oct 11 18:23:34 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Sat, 11 Oct 2008 23:23:34 +0100
Subject: [Numpy-discussion] Installation of Numerical Python
In-Reply-To: <2394.71.231.103.249.1223734774.squirrel@mail.alumni.caltech.edu>
References: <mailman.125.1223661054.2877.numpy-discussion@scipy.org>
	<1317.71.231.103.249.1223673966.squirrel@mail.alumni.caltech.edu>
	<3d375d730810101432g650e3211wc50469404d774eea@mail.gmail.com>
	<320fb6e00810101442s4a35f1fbxfadef1696aad782e@mail.gmail.com>
	<1315.71.231.103.249.1223686712.squirrel@mail.alumni.caltech.edu>
	<320fb6e00810110408q60292511r661ff30f1872b4c2@mail.gmail.com>
	<2394.71.231.103.249.1223734774.squirrel@mail.alumni.caltech.edu>
Message-ID: <320fb6e00810111523l7561af7dtac71727a6c8961f4@mail.gmail.com>

On Sat, Oct 11, 2008 at 3:19 PM, Linda Seltzer wrote:
>
> I did not give permission to have the e-mail sent to the list.
> Please do not forawrd anyone's e-mail to a list without permission.

I must apologise for my error - I had honestly assumed your email had
been intended for the whole list.  This was partly because the email
address I use for this mailing list might be confusing
(numpy-discussion at ...), but also your phrasing:

"Could someone please provide clear, detailed instructions for how to ..."

With hindsight this may have been a rhetorical question, but I did
think it was an open question to the whole list.

> Also, kindly do not send an entire list a message falsely stating that
> someone may be using code written by someone else.
> I have been reading various tutorials on the Internet, and they offer very
> incomplete information.  Also, it is necessary to minimize time spent on
> installation and system administration and maximize time spent on math.

I wasn't intending any insult - there is nothing inherently wrong with
using code written by someone else - Numeric, NumPy, SciPy and python
itself are all open source projects.  I merely said "I have the
impression you are using some existing code that someone else wrote",
partly to give context to the advice that followed.

If you are interested in the history of  Numeric and numpy (and the
occasional confusion between the two) see:
http://numpy.scipy.org/#older_array and
http://www.scipy.org/History_of_SciPy

For the actual numpy documentation, see the links on
http://www.scipy.org/Documentation

I hope any frustration getting numpy installed and working is short
lived, and wish you good luck with your work.

Peter


From michael.abshoff at googlemail.com  Sat Oct 11 19:22:14 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 11 Oct 2008 16:22:14 -0700
Subject: [Numpy-discussion] can't build numpy 1.2.0 under python 2.6
 (windows-amd64) using VS9
In-Reply-To: <48F097BA.1080100@ar.media.kyoto-u.ac.jp>
References: <3633FE192D10364D889D84EB77185F3E73AE4D@rxrex1.ssaris.com>	<200810091707.38689.lists_ravi@lavabit.com>	<48EEFEF2.9070208@ar.media.kyoto-u.ac.jp>	<200810101000.48555.lists_ravi@lavabit.com>	<48EF5F99.3010403@ar.media.kyoto-u.ac.jp>	<48EF6BF0.9080708@gmail.com>	<48EF6F2C.6040509@ar.media.kyoto-u.ac.jp>	<48EF8B76.7010200@gmail.com>
	<48F097BA.1080100@ar.media.kyoto-u.ac.jp>
Message-ID: <48F13526.9040300@gmail.com>

David Cournapeau wrote:

> Michael Abshoff wrote:

Hi David,

>> Sure, but there isn't even a 32 bit gcc out there that can produce 64 
>> bit PE binaries (aside from the MinGW fork that AFAIK does not work 
>> particularly well and allegedly has issues with the cleanliness of some 
>> of the code which is allegedly the reason that the official MinGW people 
>> will not touch the code base) .
> 
> The biggest problem is that officially, there is still no gcc 4 release
> for mingw. I saw a gcc 4 section in cygwin, though, so maybe it is about
> to be released. There is no support at all for 64 bits PE in the 3 serie.

Yes, you are correct and I was wrong. I just checked out the mingw-64 
project and there has been a lot of activity the last couple month, 
including a patch to build pthread-win32 in 64 bit mode.

> I think binutils officially support 64 bits PE (I can build a linux
> hosted binutils for 64 bits PE with x86_64-pc-mingw32 as a target, and
> it seems to work: disassembling and co). gcc 4 can work, too (you can
> build a bootstrap C compiler which targets windows 64 bits IICR). The
> biggest problem AFAICS is the runtime (mingw64, which is indeed legally
> murky).

I would really like to find the actual reason *why* the legal status of 
the 64 bit MinGW port is murky (To my knowledge it has to do with taking 
code from the MS Platform toolkit - but that is conjecture), so I guess 
I will do the obvious thing and ask on the MinGW list :)

>> Ok, that is a concern I usually do not have since I tend to build my own 
>> Python :).
> 
> I would say that if you can build python by yourself on windows, you can
> certainly build numpy by yourself :) It took me quite a time to be able
> to build python on windows by myself from scratch.

Sure, I do see your point.

Accidentally someone posted about

    http://debian-interix.net/

on the sage-windows list today. It offers a gcc 4.2 toolchain and AFAIK 
there is at least a patch set for ATLAS to make it work on Interix.

> cheers,
> 
> David

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From lseltzer at alumni.caltech.edu  Sun Oct 12 02:39:03 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Sun, 12 Oct 2008 02:39:03 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
Message-ID: <1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>

I received several pieces of advice concerning my previous question on the
use of 2-D arrays.  However, none of these pieces of advice resulted in
code that works.

The latest suggestion
>> In numpy it's
>>    import numpy as npy
>>    a = npy.zeros((256,256))
produced this error message:
> NameError: global name 'npy' is not defined
These are the import statements I used:
import numpy as npy
from numpy.oldnumeric import *

I would appreciate it if someone could post a **working** example that
creates a 2-D array of a large size, such as 256x256, with the initial
elements all zero.  This file shouold include the import statements and
the lines of code to create the array.
Please, no demeaning statements like "you forgot a parenthesis" or "you
were using someone else's code" - just the lines of code for a file that
actually *works.*



From cournape at gmail.com  Sun Oct 12 04:11:04 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 12 Oct 2008 17:11:04 +0900
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
Message-ID: <5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>

On Sun, Oct 12, 2008 at 3:39 PM, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:

> These are the import statements I used:
> import numpy as npy
> from numpy.oldnumeric import *

Here is an example that works for any working numpy installation:

import numpy as npy
npy.zeros((256, 256)).

If those are the first two statements at python prompt, and it does
not work, your numpy installation is broken. In that case, which
platform and how did you install numpy would be useful information to
help you better.

> Please, no demeaning statements like "you forgot a parenthesis" or "you
> were using someone else's code" - just the lines of code for a file that
> actually *works.*

Those were not demeaning statements, and the line that works was shown
to you. I strongly suspect that either you did not give use enough
information, or that your numpy installation is broken.

cheers,

David


From robert.kern at gmail.com  Sun Oct 12 04:23:21 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 12 Oct 2008 03:23:21 -0500
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
Message-ID: <3d375d730810120123n5a986522i275c732e17b072f8@mail.gmail.com>

On Sun, Oct 12, 2008 at 03:11, David Cournapeau <cournape at gmail.com> wrote:
> On Sun, Oct 12, 2008 at 3:39 PM, Linda Seltzer
> <lseltzer at alumni.caltech.edu> wrote:
>
>> These are the import statements I used:
>> import numpy as npy
>> from numpy.oldnumeric import *
>
> Here is an example that works for any working numpy installation:
>
> import numpy as npy
> npy.zeros((256, 256)).

Well, except for that last period.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Sun Oct 12 10:17:39 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 12 Oct 2008 10:17:39 -0400
Subject: [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <F28BD98D-E284-44DD-91BF-5C288F714456@comcast.net>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<2b1c8c4f0810010540t7167281ev4d5e952093811250@mail.gmail.com>
	<5472729D-2D1B-491C-8DFA-C578AF61A940@comcast.net>
	<e76aa17f0810021301sf4092ectb2a181974e7d629a@mail.gmail.com>
	<CD643BF2-E4D1-40DB-BCCC-DFB768E9A6AA@comcast.net>
	<ce557a360810021857w4bb5f884of42902dd6ad1b4d8@mail.gmail.com>
	<26EB3B10-1BC7-438A-8C1A-6C41A227D6F6@comcast.net>
	<ce557a360810032003i6563cd49n9ffbd3eeb6375058@mail.gmail.com>
	<F28BD98D-E284-44DD-91BF-5C288F714456@comcast.net>
Message-ID: <ce557a360810120717w42f26f46n2d544d1e4f841da2@mail.gmail.com>

2008/10/9 David Bolme <bolme1234 at comcast.net>:
> I have written up basic nearest neighbor algorithm.  It does a brute
> force search so it will be slower than kdtrees as the number of points
> gets large.  It should however work well for high dimensional data.  I
> have also added the option for user defined distance measures.  The
> user can set a default "p".  "p" has the same functionality if it is a
> float.  "p" can also be a function that computes a distance matrix or
> the measure can be selected using the strings: "Manhattan",
> "Euclidean", or "Correlation".
>
> https://pyvision.svn.sourceforge.net/svnroot/pyvision/trunk/src/pyvision/vector/knn.py

This is interesting. I would point out, though, that if you want a
Minkowski norm, it may be more efficient (that is, faster) to use the
new compiled kd-tree implementation with leafsize set to the size of
your data. This is written in compiled code and uses short-circuit
distance evaluation, and may be much faster for high-dimensional
problems.

Given that, this should perhaps go in with other generic metric space
code. I have a functional implementation of ball trees (though I don't
know how efficient they are), and am looking into implementing cover
trees.

> The interface is similar to Anne's code and in many cases can be used
> as a drop in replacement.  I have posted the code to my own project
> because I have a short term need and I do not have access to the scipy
> repository.  Feel free to include the code with scipy under the scipy
> license.
>
> I did find a typo your documentation.
> typo "trie -> tree" - ... kd-tree is a binary trie, each of whose ...

That's not actually a typo: a trie is a tree in which all the data is
stored at leaf nodes. Basic kd-trees use the nodes themselves to
define splitting planes; you can actually construct one with no extra
storage at all, just by choosing an appropriate order for your array
of points. This implementation chooses splitting planes that may not
pass through any point, so all the points get stored in leaves.

> Also I found the use of k in the documentation some what confusing as
> it is the dimensionality of the data points in the kd-tree and the
> number of neighbors for k-nearest neighbors.

That's a good point. I changed the dimension of the kd-tree to m throughout.

Anne


From lseltzer at alumni.caltech.edu  Sun Oct 12 11:11:51 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Sun, 12 Oct 2008 11:11:51 -0400 (EDT)
Subject: [Numpy-discussion] Data types in Numerical Python
In-Reply-To: <5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
Message-ID: <1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>

> Here is an example that works for any working numpy installation:
>
> import numpy as npy
> npy.zeros((256, 256))
This suggestion from David did work so far, and removing the other import
line enabled the program to run.
However, the data types the program used as defaults for variables has
changed, and now I am getting error messages about data types.  It seems
that some variables are getting a default designation as floats.  Before I
installed numpy and needed 2-D arrays, the program was working with the
default types, and I did not have to specify types.
Is there a clear tutorial that describes a means to assign data types for
each variable as in C, so that I don't obtain error messages about data
types?
Because I am simulating code for a DSP processor, the data types I need
are unsigned bytes, unsigned 32-bit ints, and signed 32-bit ints.  In some
cases I can use unsigned and signed 16-bit ints.
Also, what data types are valid for use with local operations such as
exclusive or?


From peridot.faceted at gmail.com  Sun Oct 12 11:26:05 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 12 Oct 2008 11:26:05 -0400
Subject: [Numpy-discussion] Data types in Numerical Python
In-Reply-To: <1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
	<1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
Message-ID: <ce557a360810120826i3a498c4dte7eaac77c5a15fe@mail.gmail.com>

2008/10/12 Linda Seltzer <lseltzer at alumni.caltech.edu>:
>> Here is an example that works for any working numpy installation:
>>
>> import numpy as npy
>> npy.zeros((256, 256))
> This suggestion from David did work so far, and removing the other import
> line enabled the program to run.
> However, the data types the program used as defaults for variables has
> changed, and now I am getting error messages about data types.  It seems
> that some variables are getting a default designation as floats.  Before I
> installed numpy and needed 2-D arrays, the program was working with the
> default types, and I did not have to specify types.
> Is there a clear tutorial that describes a means to assign data types for
> each variable as in C, so that I don't obtain error messages about data
> types?

Python is a dynamically-typed language (unlike C), so variables do not
have type. That is, a variable can refer to an object of any type; if
you need to know what type of object a variable currently refers to
you must inspect the object. You may want to go through one of the
brief introduction-to-python tutorials that are on the python website
just to get comfortable with the language. (For example, understanding
the meaning and syntax of import statements.)

When you create a numpy array, you can specify its type. You can also
explicitly or implicitly convert the types of numpy arrays. I
recommend you select a data type, let's say np.uint32, and make sure
various arrays are created containing that type:

np.zeros((m,n),dtype=np.uint32)
np.arange(10,dtype=np.uint32)
x.astype(np.uint32)
np.array([1,2,3,4.5], dtype=np.uint32)

et cetera. Most operations (addition, multiplication, maximum) will
preserve the data type of arrays they are given (but if you supply two
different data types numpy will attempt to choose the "larger").

> Because I am simulating code for a DSP processor, the data types I need
> are unsigned bytes, unsigned 32-bit ints, and signed 32-bit ints.  In some
> cases I can use unsigned and signed 16-bit ints.
> Also, what data types are valid for use with local operations such as
> exclusive or?

The numpy data types you want are described by "dtype" objects. These
can in principle become quite complicated, but the ones you need are
given names already:

np.uint8
np.uint32
np.int32
np.uint16
np.int16

You can specify any of these as "dtype=" arguments to the various
numpy functions. If you need really rigid typing, python may not be
the ideal language for you.

Good luck,
Anne


From lbrooks at MIT.EDU  Sun Oct 12 11:23:34 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sun, 12 Oct 2008 09:23:34 -0600
Subject: [Numpy-discussion] Data types in Numerical Python
In-Reply-To: <1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
	<1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
Message-ID: <48F21676.3050806@mit.edu>

Linda Seltzer wrote:
>> Here is an example that works for any working numpy installation:
>>
>> import numpy as npy
>> npy.zeros((256, 256))
>>     
> This suggestion from David did work so far, and removing the other import
> line enabled the program to run.
> However, the data types the program used as defaults for variables has
> changed, and now I am getting error messages about data types.  It seems
> that some variables are getting a default designation as floats.  Before I
> installed numpy and needed 2-D arrays, the program was working with the
> default types, and I did not have to specify types.
> Is there a clear tutorial that describes a means to assign data types for
> each variable as in C, so that I don't obtain error messages about data
> types?
> Because I am simulating code for a DSP processor, the data types I need
> are unsigned bytes, unsigned 32-bit ints, and signed 32-bit ints.  In some
> cases I can use unsigned and signed 16-bit ints.
> Also, what data types are valid for use with local operations such as
> exclusive or?
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
You can specify the type in the zeros command

import numpy as npy
npy.zeros((256, 256), npy.uint32)

or you can convert an array between types at any point using the 
.astype(npy.uint16) notation like this

npy.zeros((256,256)).astype(npy.uint16)

I am not sure if there are any tutorials on this, but here are the types 
you are interested in:
npy.uint32
npy.uint16
npy.int32
npy.int16


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081012/9916a541/attachment.html>

From dalke at dalkescientific.com  Sun Oct 12 13:15:34 2008
From: dalke at dalkescientific.com (Andrew Dalke)
Date: Sun, 12 Oct 2008 19:15:34 +0200
Subject: [Numpy-discussion] Data types in Numerical Python
In-Reply-To: <ce557a360810120826i3a498c4dte7eaac77c5a15fe@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
	<1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
	<ce557a360810120826i3a498c4dte7eaac77c5a15fe@mail.gmail.com>
Message-ID: <6FF89053-ADA0-43B8-9C5F-0F69B1C3A87B@dalkescientific.com>

On Oct 12, 2008, at 5:26 PM, Anne Archibald wrote:
> Python is a dynamically-typed language (unlike C), so variables do not
> have type.

Another way to think of it for C people is that all variables
have the same type, which is "reference to Python object."
It's the objects which are typed, and not the variable.

				Andrew
				dalke at dalkescientific.com




From charlesr.harris at gmail.com  Sun Oct 12 13:48:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 12 Oct 2008 11:48:51 -0600
Subject: [Numpy-discussion] Data types in Numerical Python
In-Reply-To: <1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<5b8d13220810120111j32442e04qaa14b060beb8bd31@mail.gmail.com>
	<1455.71.231.103.249.1223824311.squirrel@mail.alumni.caltech.edu>
Message-ID: <e06186140810121048r2ba2390fn352fcc6a4dc99347@mail.gmail.com>

On Sun, Oct 12, 2008 at 9:11 AM, Linda Seltzer
<lseltzer at alumni.caltech.edu>wrote:

> > Here is an example that works for any working numpy installation:
> >
> > import numpy as npy
> > npy.zeros((256, 256))
> This suggestion from David did work so far, and removing the other import
> line enabled the program to run.
> However, the data types the program used as defaults for variables has
> changed, and now I am getting error messages about data types.  It seems
> that some variables are getting a default designation as floats.  Before I
> installed numpy and needed 2-D arrays, the program was working with the
> default types, and I did not have to specify types.


Yes, the default type of the functions zeros and ones have changed from
integer to float. If your program is short you could send it as an
attachment so we could look at it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081012/7ad1e273/attachment.html>

From aisaac at american.edu  Sun Oct 12 17:29:48 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 12 Oct 2008 17:29:48 -0400
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
Message-ID: <48F26C4C.3030504@american.edu>

On 10/12/2008 2:39 AM Linda Seltzer apparently wrote:
> Please, no demeaning statements like "you forgot 
> a parenthesis" or "you were using someone else's code" 
> - just the lines of code for a file that actually *works.* 


Those statements are not demeaning; lighten up.
And the answer was correct.

Start up an interpreter prompt.
Type these in.  What happens?

     >>> import numpy as np
     >>> a = np.zeros((256,256))

Cheers,
Alan Isaac




From lseltzer at alumni.caltech.edu  Mon Oct 13 01:21:52 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 01:21:52 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <48F26C4C.3030504@american.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
Message-ID: <1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>

>
> Those statements are not demeaning; lighten up.
STOP IT.  JUST STOP IT.  STOP IT RIGHT NOW.
Is there a moderator on the list to put a stop to these kinds of statements?
I deserve to be treated with respect.
I deserve to have my questions treated with respect.
I deserve to receive technical information without personal attacks.


From matthew.brett at gmail.com  Mon Oct 13 01:37:02 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 12 Oct 2008 22:37:02 -0700
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
Message-ID: <1e2af89e0810122237mf1efc9apaec1c3f338c414e9@mail.gmail.com>

Friends,

>> Those statements are not demeaning; lighten up.
> STOP IT.  JUST STOP IT.  STOP IT RIGHT NOW.

Let us not go to this place, honestly, there is no need.  Let's go
back to the technical problem again.

Linda, did you have time to try Alan's example?

Best,

Matthew


From lseltzer at alumni.caltech.edu  Mon Oct 13 01:47:19 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 01:47:19 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1e2af89e0810122237mf1efc9apaec1c3f338c414e9@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<1e2af89e0810122237mf1efc9apaec1c3f338c414e9@mail.gmail.com>
Message-ID: <1369.71.231.103.249.1223876839.squirrel@mail.alumni.caltech.edu>

L. Brooks of M.I.T. sent a professional e-mail with a code fragment that
has worked.
> Friends,
>
>>> Those statements are not demeaning; lighten up.
>> STOP IT.  JUST STOP IT.  STOP IT RIGHT NOW.
>
> Let us not go to this place, honestly, there is no need.  Let's go
> back to the technical problem again.
>
> Linda, did you have time to try Alan's example?
>
> Best,
>
> Matthew
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



From pgmdevlist at gmail.com  Mon Oct 13 01:48:38 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 13 Oct 2008 01:48:38 -0400
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
Message-ID: <200810130148.38543.pgmdevlist@gmail.com>

Linda,
If you're familiar with Matlab syntax, you may find this link interesting:
http://www.scipy.org/NumPy_for_Matlab_Users

Here another couple of useful links
http://www.scipy.org/Tentative_NumPy_Tutorial
http://www.scipy.org/Numpy_Functions_by_Category

For your specific example, if you want to create a (256,128) array of unsigned 
integers:

import numpy as np
a = np.zeros((256,128), dtype=np.uint32)

Note that if you intend to fill the array afterwards with other values, it 
might be more efficient to create an 'empty' array instead of an array full 
of zeros:

b=np.empty((256,128), dtype=uint32)



From stefan at sun.ac.za  Mon Oct 13 05:50:30 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 13 Oct 2008 11:50:30 +0200
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
Message-ID: <9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>

Linda,

2008/10/13 Linda Seltzer <lseltzer at alumni.caltech.edu>:
>> Those statements are not demeaning; lighten up.
> STOP IT.  JUST STOP IT.  STOP IT RIGHT NOW.
> Is there a moderator on the list to put a stop to these kinds of statements?
> I deserve to be treated with respect.
> I deserve to have my questions treated with respect.
> I deserve to receive technical information without personal attacks.

I think you'll be hard pressed to find a more friendly, open and
relaxed mailing list than this one.  We're like having pi?a coladas
while we type.  That said, keep in mind that you are asking
professionals to donate *their* valuable time to solve *your* problem.
 They gladly do so, but at the same time they try to be efficient; so
if you sometimes receive a curt answer, it certainly wasn't meant to
be rude.  Many of us also sprinkle our responses with a liberal dose
of Tongue In Cheek :)

It looks like you received some good answers to your question, but let
us know if your problems persist and we'll help you sort it out.

Regards
St?fan


From rmay31 at gmail.com  Mon Oct 13 09:31:32 2008
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 13 Oct 2008 08:31:32 -0500
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>	<48F26C4C.3030504@american.edu>	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
Message-ID: <48F34DB4.8080603@gmail.com>

St?fan van der Walt wrote:
> Linda,
> 
> 2008/10/13 Linda Seltzer <lseltzer at alumni.caltech.edu>:
>>> Those statements are not demeaning; lighten up.
>> STOP IT.  JUST STOP IT.  STOP IT RIGHT NOW.
>> Is there a moderator on the list to put a stop to these kinds of statements?
>> I deserve to be treated with respect.
>> I deserve to have my questions treated with respect.
>> I deserve to receive technical information without personal attacks.
> 
> I think you'll be hard pressed to find a more friendly, open and
> relaxed mailing list than this one.  We're like having pi?a coladas
> while we type.  That said, keep in mind that you are asking
> professionals to donate *their* valuable time to solve *your* problem.
>  They gladly do so, but at the same time they try to be efficient; so
> if you sometimes receive a curt answer, it certainly wasn't meant to
> be rude.  Many of us also sprinkle our responses with a liberal dose
> of Tongue In Cheek :)
> 
> It looks like you received some good answers to your question, but let
> us know if your problems persist and we'll help you sort it out.

Well said.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From Chris.Barker at noaa.gov  Mon Oct 13 12:21:50 2008
From: Chris.Barker at noaa.gov (Chris.Barker)
Date: Mon, 13 Oct 2008 09:21:50 -0700
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
Message-ID: <48F3759E.6060804@noaa.gov>

Bill Baxter wrote:
>    import numpy as npy

Bill,

for what it's worth, I *think* this group has reached a consensus to use:

import numpy as np

We all have different tastes for how they might want to spell it, but 
the more consistent we are, the easier it will be for newbies.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Mon Oct 13 12:35:49 2008
From: Chris.Barker at noaa.gov (Chris.Barker)
Date: Mon, 13 Oct 2008 09:35:49 -0700
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
Message-ID: <48F378E5.5070307@noaa.gov>

Linda Seltzer wrote:
> I would appreciate it if someone could answer my question without
> referring to subjects such as APIs and interfaces, since I am only
> concerned with a mathematical application at this time.

caution: this is a bit rude -- that was an excellent and informative 
answer to your not-very-clear question.

No matter how you slice it, you're going to need to learn a bit about 
computer programming in general, and python in particular, in order to 
be productive with numpy.

> I wish that tutorials provided real world examples.

They certainly do -- just not the one you happen to be looking for. It's 
not good form to criticize folks work that they have generously donated.

> I would appreciate it if someone could give me the actual statements
> needed to define and initialize a 2-D array of size NxN, where N can be
> any large number,

There are a few ways to initialize numpy arrays:

import numpy as np

a = np.zeros((M,N))
a = np.ones ((M,N))
a = np.empty((M,N), dtype=np.float)

> In Matlab, this is done as a = zeros(256,256).

If you are familiar with Matlab, you'll want to take a look at the Wiki 
pages that describe the similarities and differences between numpy and 
Matlab:

http://www.scipy.org/NumPy_for_Matlab_Users

numpy is more complex, but also more powerful than Matlab -- it will 
take a bit of learning, but it's worth it.

Also, read some of the intros to python itself -- you'll need to 
understand importing and name spaces.

a couple quick examples:

1) numpy has many different data types. In the examples above, you will 
get double precision floats by default (like Matlab), but you can also 
get other data types (with your image examples, you'll want that). For 
example, one way to store an RBG image:

a = np.zeros((w,h,3), dtype=np.uint8)

that is, a width x height x 3 array of 8bit unsigned integers.

2) arrays and matrices are different.

3) numpy provides n-d matrices, not just 2-d

4) importing and name spaces.

>  If I try this in python,
> it won't let the program overwrite the zeros.

if something doesn't work as expected, always post your code, exactly as 
you tested it, so we can tell you what's wrong.

Also, post specific questions -- you first question was something like 
"can I work with arrays", which is quite different than this one: "how 
do I create an array of nXn size full of zeros?"

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From pgmdevlist at gmail.com  Mon Oct 13 13:35:23 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 13 Oct 2008 13:35:23 -0400
Subject: [Numpy-discussion] Priority rules between numpy scalars and 0d
	arrays
Message-ID: <200810131335.24577.pgmdevlist@gmail.com>

All,
Sorry to bring back this subject, but I still haven't got any proper answers:

* What are the priority rules between numpy scalars and 0d arrays ?

When multiplying a numpy scalar by a 0d array, shouldn't the __mul__ or 
__rmul__ methods of the array be called ?
Should the result be a numpy scalar, or a 0d array (possibly recasted to the 
higher dtype) ?

The problem occurs with numpy.ma.masked, defined as a 0d, np.float64 
MaskedArray, which has the __mul__ and __rmul__ of a MaskedArray.

np.float(1)*ma.masked gives ma.masked, as it should
np.float(64)* ma.masked gives 0, when ma.masked should have been obtained:
that leads me to think that ma.masked.__rmul__ isn't called. Why ? Are 0d 
arrays that special beasts ?


From oliphant at enthought.com  Mon Oct 13 14:04:33 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 13 Oct 2008 13:04:33 -0500
Subject: [Numpy-discussion] Priority rules between numpy scalars and 0d
 arrays
In-Reply-To: <200810131335.24577.pgmdevlist@gmail.com>
References: <200810131335.24577.pgmdevlist@gmail.com>
Message-ID: <48F38DB1.40508@enthought.com>

Pierre GM wrote:
> All,
> Sorry to bring back this subject, but I still haven't got any proper answers:
>
> * What are the priority rules between numpy scalars and 0d arrays ?
>   
There aren't really any specified.  However, there is behavior that 
emerges from what is specified.

The problem is that there has never been a formal "resolution" (that I 
recall) of when should something be returned as a 0-d array and when it 
should be returned as a scalar.   There is rather an informal 
implementation of what actually happens. 

Their are some "rules of thumb" that have emerged (like array-operations 
--- e.g. reshaping --- should return 0-d arrays and not scalars).

The other issue is that there is the rule that when scalars and arrays 
mix, the "data-type" of the array determines the result, but there 
aren't fixed rules about what the "sub-type" should be.
> The problem occurs with numpy.ma.masked, defined as a 0d, np.float64 
> MaskedArray, which has the __mul__ and __rmul__ of a MaskedArray.
>
> np.float(1)*ma.masked gives ma.masked, as it should
> np.float(64)* ma.masked gives 0, when ma.masked should have been obtained:
> that leads me to think that ma.masked.__rmul__ isn't called. Why ? Are 0d 
> arrays that special beasts ?
>   
Could you post code to describe what you mean?

np.float(64) should be the same type as np.float(1) so I don't get what 
you are saying exactly.

I think the issue is that numpy scalars are currently wrapped into 0-d 
arrays for all math and so the 'priority' issue might really an issue 
between numpy arrays and masked arrays.

-Travis



From lseltzer at alumni.caltech.edu  Mon Oct 13 14:24:29 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 14:24:29 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <48F34DB4.8080603@gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
	<48F34DB4.8080603@gmail.com>
Message-ID: <1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>

Where is the moderator? Please get these condescending, demeaning personal
comments off of this list.  I asked technical question.  Now please send
technical information only.
> St?fan van der Walt wrote:

>> I think you'll be hard pressed to find a more friendly, open and
>> relaxed mailing list than this one.  We're like having pi?a coladas
>> while we type.  That said, keep in mind that you are asking
>> professionals to donate *their* valuable time to solve *your* problem.
>>  They gladly do so, but at the same time they try to be efficient; so
>> if you sometimes receive a curt answer, it certainly wasn't meant to
>> be rude.  Many of us also sprinkle our responses with a liberal dose
>> of Tongue In Cheek :)
>>
>> It looks like you received some good answers to your question, but let
>> us know if your problems persist and we'll help you sort it out.
>
> Well said.
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



From lseltzer at alumni.caltech.edu  Mon Oct 13 14:36:47 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 14:36:47 -0400 (EDT)
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <48F378E5.5070307@noaa.gov>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<48F378E5.5070307@noaa.gov>
Message-ID: <1288.71.231.103.249.1223923007.squirrel@mail.alumni.caltech.edu>

Christopher Barker wrote:
> No matter how you slice it, you're going to need to learn a bit about
> computer programming in general, and python in particular, in order to
> be productive with numpy.
WHERE IS THE MODERATOR?
I deserve not to be insulted in front of the professional community with
personal slurs such as this.
My computer programming background includes thorough training in C at Bell
Labs in the early days, and experience developing more than 65,000 lines
of code.  My software in C was used in the AT&T telephone network, the
Sprint network operations system, and the Silicon Graphics workstation. 
My assembly language DSP software is running in the Coast Guard
communications system that coordinated the ship locations during Katrina.
I got a large memory physical chemistry simulation to run at Princeton
University, which enabled a chemistry lab to obtain results a year ahead
of other universities. I have a long record of doing accurate, high
quality work on algorithms and software and I would appreciate it if
persons on this list would provide technical information only and stop
making snide, superior personal comments and slours about a person's
knowledge, background or ability.  Also, I do not appreciate personal
comments alleging that I am being overly sensitive.  Where is the
moderator during all of this?
I asked professional, technical questions and I expect professional,
technical answers.



From pgmdevlist at gmail.com  Mon Oct 13 14:30:29 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 13 Oct 2008 14:30:29 -0400
Subject: [Numpy-discussion] Priority rules between numpy scalars and 0d
	arrays
In-Reply-To: <48F38DB1.40508@enthought.com>
References: <200810131335.24577.pgmdevlist@gmail.com>
	<48F38DB1.40508@enthought.com>
Message-ID: <200810131430.30836.pgmdevlist@gmail.com>

Travis, 

> The problem is that there has never been a formal "resolution" (that I
> recall) of when should something be returned as a 0-d array and when it
> should be returned as a scalar.   There is rather an informal
> implementation of what actually happens.

Ah. It might be worth putting the current informal rules or rule-of-thumb 
black-on-white (or green-on-black) somewhere

> The other issue is that there is the rule that when scalars and arrays
> mix, the "data-type" of the array determines the result, but there
> aren't fixed rules about what the "sub-type" should be.

I would expect something like "return a scalar unless specified otherwise by a 
subclass".

> Could you post code to describe what you mean?

In MaskedArray, we check whether the output of an operation is 0d: if it is 
and that the result is masked, then ma.masked is output. if the result is 0d 
without a mask, a numpy scalar is output.

ma.masked is defined as MaskedArray(0,mask=True,dtype=np.float)

First, let's check the left multiplication

>>> ma.masked * 1
masked_array(data = --,
      mask = True,
      fill_value=1e+20)
>>> ma.masked * np.float64(1)
masked_array(data = --,
      mask = True,
      fill_value=1e+20)
>>> ma.masked * np.float128(1)
masked_array(data = --,
      mask = True,
      fill_value=1e+20)

Everythng works as planned.

Now, for the right multiplication:
>>> 1.*ma.masked
masked_array(data = --,
      mask = True,
      fill_value=1e+20)
>>> np.float64(1)*ma.masked
0.0
>>> np.float128(1)*ma.masked
0.0

And that's where the problem is. It looks like ma.masked.__rmul__ or 
ma.masked.__mul__ are *NOT* called in the last two cases, when I expected it 
would.

But if we have a 1d array:
>>> np.array(1., dtype=np.float128)*ma.masked
masked_array(data = --,
      mask = True,
      fill_value=1e+20)

> I think the issue is that numpy scalars are currently wrapped into 0-d
> arrays for all math and so the 'priority' issue might really an issue
> between numpy arrays and masked arrays.

I don't think it's a problem of __array__priority__. MaskedArrays have 
currently a __array_priority__ of 15, switching to 1e99 or even np.inf 
doesn't change anything. It looks like the dtype is checked first, which 
dictates which method is being called (ndarray.__mul__ instead of 
MaskedArray.__rmul__).
What surprises me also is that numpy scalar are supposed to have a very low 
priority (negative).

In short, Travis, could you explain me what's happening ?




From dalke at dalkescientific.com  Mon Oct 13 15:11:09 2008
From: dalke at dalkescientific.com (Andrew Dalke)
Date: Mon, 13 Oct 2008 21:11:09 +0200
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
Message-ID: <D18079EF-5FC2-496C-9DD0-09340AA26628@dalkescientific.com>

On Oct 13, 2008, at 7:21 AM, Linda Seltzer wrote:
> Is there a moderator on the list to put a stop to these kinds of  
> statements?

No.


				Andrew
				dalke at dalkescientific.com




From aisaac at american.edu  Mon Oct 13 15:29:00 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 13 Oct 2008 15:29:00 -0400
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>	<48F26C4C.3030504@american.edu>	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>	<48F34DB4.8080603@gmail.com>
	<1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
Message-ID: <48F3A17C.3060602@american.edu>

Linda Seltzer wrote:
> Where is the moderator? Please get these condescending, demeaning personal
> comments off of this list.  I asked technical question.  Now please send
> technical information only.

The problem is, you did not just ask
for technical information.  You also
accused people of being condescending
and demeaning.  But nobody was
condescending or demeaning.  As several
people **politely** explained to you,
you are wrong about that.

If you stop making such accusations,
you will stop receiving such corrections.
There is no reason that list members
should allow your accusations to go
unchallenged.

Stick to technical inquiries *only*, and you
will get responses more to your taste.

Cheers,
Alan Isaac


From robert.kern at gmail.com  Mon Oct 13 15:48:55 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 13 Oct 2008 14:48:55 -0500
Subject: [Numpy-discussion] Using 2-D arrays in Numeric Python (numpy)
In-Reply-To: <1288.71.231.103.249.1223923007.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<48F378E5.5070307@noaa.gov>
	<1288.71.231.103.249.1223923007.squirrel@mail.alumni.caltech.edu>
Message-ID: <3d375d730810131248p273d2071qa99035cdfdc813@mail.gmail.com>

On Mon, Oct 13, 2008 at 13:36, Linda Seltzer
<lseltzer at alumni.caltech.edu> wrote:
> Christopher Barker wrote:
>> No matter how you slice it, you're going to need to learn a bit about
>> computer programming in general, and python in particular, in order to
>> be productive with numpy.
> WHERE IS THE MODERATOR?

There is no moderator. I may be the closest to such a thing, but I
have no intention of silencing anyone at this point in time.

> I deserve not to be insulted in front of the professional community with
> personal slurs such as this.

These are not such. They may be blunt or assume too much, but the
proper response is a plain correction, not a call to a moderator to
silence someone. You have been initially treated with respect, and we
tried to answer your technical questions the best we could, but our
patience is rapidly thinning. You took purely technical responses as
personal insults where nothing of the kind was intended. And now we
are lost in the non-technical weeds of accusation and
counter-accusation.

Whether you believe it or not, this mailing list is about as good as
it gets when it comes to lists for open source software (and you have
received a far gentler treatment given your behavior than you would
have elsewhere), but the medium of email has its limitations. It is
not the subtlest form of communication. Messages may seem blunter than
you might think is appropriate. That's just the way email is, so you
have to give the other person the benefit of the doubt and assume that
they did not intend to offend you. By participating in this list, we
have all implicitly agreed to this rule. If you can not or will not do
the same, then it seems clear to me that further participation on the
list on your part will not serve your interests, much less ours.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jdh2358 at gmail.com  Mon Oct 13 16:04:06 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Mon, 13 Oct 2008 15:04:06 -0500
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <48F3A17C.3060602@american.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
	<48F34DB4.8080603@gmail.com>
	<1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
	<48F3A17C.3060602@american.edu>
Message-ID: <88e473830810131304p1172e997k1c2d3899c2653ee2@mail.gmail.com>

On Mon, Oct 13, 2008 at 2:29 PM, Alan G Isaac <aisaac at american.edu> wrote:

> The problem is, you did not just ask
> for technical information.  You also
> accused people of being condescending
> and demeaning.  But nobody was
> condescending or demeaning.  As several
> people **politely** explained to you,
> you are wrong about that.

Here is a simple example of loading some 2D data into an array and
manipulating the contents

import numpy as np
# load a 2D array of integers
X = np.loadtxt('somefile.txt').astype(int)
print X.shape  # X is a 2D array

# display the contents of X as a string
print '\n'.join([''.join([chr(c) for c in row]) for row in X])

The input file "somefile.txt" is attached

JDH
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: somefile.txt
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081013/9ea9e741/attachment.txt>

From lseltzer at alumni.caltech.edu  Mon Oct 13 17:15:10 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 17:15:10 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <48F3A17C.3060602@american.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<3d375d730810102102w41cd843eua16dbe0c54bf0a14@mail.gmail.com>
	<1653.71.231.103.249.1223702173.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
	<48F34DB4.8080603@gmail.com>
	<1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
	<48F3A17C.3060602@american.edu>
Message-ID: <1091.71.231.103.249.1223932510.squirrel@mail.alumni.caltech.edu>

Your reply is inappropriate.  it is not a "correction."  A request was
made to stop posting mail that did not concern math and you have continued
with your put downs.  Stop it. Just stop it.  Stop it right now.
> Linda Seltzer wrote:
>> Where is the moderator? Please get these condescending, demeaning
>> personal
>> comments off of this list.  I asked technical question.  Now please send
>> technical information only.
>
> The problem is, you did not just ask
> for technical information.  You also
> accused people of being condescending
> and demeaning.  But nobody was
> condescending or demeaning.  As several
> people **politely** explained to you,
> you are wrong about that.
>
> If you stop making such accusations,
> you will stop receiving such corrections.
> There is no reason that list members
> should allow your accusations to go
> unchallenged.
>
> Stick to technical inquiries *only*, and you
> will get responses more to your taste.
>
> Cheers,
> Alan Isaac
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



From lseltzer at alumni.caltech.edu  Mon Oct 13 17:16:47 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 17:16:47 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <88e473830810131304p1172e997k1c2d3899c2653ee2@mail.gmail.com>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
	<48F34DB4.8080603@gmail.com>
	<1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
	<48F3A17C.3060602@american.edu>
	<88e473830810131304p1172e997k1c2d3899c2653ee2@mail.gmail.com>
Message-ID: <1095.71.231.103.249.1223932607.squirrel@mail.alumni.caltech.edu>

Alan, Stop tuyrning this around.  Stop referring to my request as an
accusation and stop referring to your put-downs as a "correction."
> On Mon, Oct 13, 2008 at 2:29 PM, Alan G Isaac <aisaac at american.edu> wrote:
>
>> The problem is, you did not just ask
>> for technical information.  You also
>> accused people of being condescending
>> and demeaning.  But nobody was
>> condescending or demeaning.  As several
>> people **politely** explained to you,
>> you are wrong about that.
>



From daranrife at gmail.com  Mon Oct 13 17:24:51 2008
From: daranrife at gmail.com (Daran Rife)
Date: Mon, 13 Oct 2008 15:24:51 -0600
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
Message-ID: <c2c0fea90810131424w6c70a6c6s134de6a818ab0ab4@mail.gmail.com>

Ordinarily I avoid becoming involved in such acrimony, but I take this
single
opportunity to state clearly that I find Linda Seltzer's behavior utterly
rude
and childish.

Having been a "member" of this mailing list for over 6 years, I take
exception
to the pointless ranting and vitriolic comments of people, like Linda, who
are
using NumPy/SciPy, and demand immediate attention to their special problem,
while hurling false accusations, and creating hard feelings among the comm-
unity. NumPy and SciPy are software packages of the highest caliber that
have
been  produced, supported, and nurtured by a community of -volunteers- at no
cost to the users.

In siutations like this, it may be helpful to review proper etiquette and
the fun-
damental purpose for mailing/support lists of open source tools. Two docum-
ents that help reaffirm what this list is all about can be found below.

Linda, I highly recommend you read the sections titled "Dealing with
rudeness"
and "On not acting like a loser" of the following document:

http://www.catb.org/~esr/faqs/smart-questions.html<http://www.catb.org/%7Eesr/faqs/smart-questions.html>

http://www.catb.org/~esr/faqs/smart-questions.html#keepcool<http://www.catb.org/%7Eesr/faqs/smart-questions.html#keepcool>
http://www.catb.org/~esr/faqs/smart-questions.html#not_losing<http://www.catb.org/%7Eesr/faqs/smart-questions.html#not_losing>

You might also benefit from seeing things from the perspective of
developers,
who again, volunteer their precious time to developing and supporting open
source
software, so that you can do your work more efficiently and smartly. Give
this a
read over a cup of tea:

http://math-atlas.sourceforge.net/faq.html#utone

Perhaps when you calm down, and allow yourself to reflect upon this
experience,
you will realize how badly you've behaved to a group of professional,
friendly
people whose only desire is to support and nuture a great tool.

I will say nothing more about this topic, and under no circumstances will I
reply to
future messages from Linda Seltzer.


Daran
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081013/90cc2ffe/attachment.html>

From lseltzer at alumni.caltech.edu  Mon Oct 13 17:27:47 2008
From: lseltzer at alumni.caltech.edu (Linda Seltzer)
Date: Mon, 13 Oct 2008 17:27:47 -0400 (EDT)
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <c2c0fea90810131424w6c70a6c6s134de6a818ab0ab4@mail.gmail.com>
References: <c2c0fea90810131424w6c70a6c6s134de6a818ab0ab4@mail.gmail.com>
Message-ID: <1110.71.231.103.249.1223933267.squirrel@mail.alumni.caltech.edu>

This ia another example of non-scientific attacking that does not belong
on the list.  As I mentioned earlier: Please keep all non-mathematical or
non-computer science remarks off the list.
> Ordinarily I avoid becoming involved in such acrimony, but I take this
> single
> opportunity to state clearly that I find Linda Seltzer's behavior utterly
> rude
> and childish.
>
> Having been a "member" of this mailing list for over 6 years, I take
> exception
> to the pointless ranting and vitriolic comments of people, like Linda, who
> are
> using NumPy/SciPy, and demand immediate attention to their special
> problem,
> while hurling false accusations, and creating hard feelings among the
> comm-
> unity. NumPy and SciPy are software packages of the highest caliber that
> have
> been  produced, supported, and nurtured by a community of -volunteers- at
> no
> cost to the users.
>
> In siutations like this, it may be helpful to review proper etiquette and
> the fun-
> damental purpose for mailing/support lists of open source tools. Two
> docum-
> ents that help reaffirm what this list is all about can be found below.
>
> Linda, I highly recommend you read the sections titled "Dealing with
> rudeness"
> and "On not acting like a loser" of the following document:
>
> http://www.catb.org/~esr/faqs/smart-questions.html<http://www.catb.org/%7Eesr/faqs/smart-questions.html>
>
> http://www.catb.org/~esr/faqs/smart-questions.html#keepcool<http://www.catb.org/%7Eesr/faqs/smart-questions.html#keepcool>
> http://www.catb.org/~esr/faqs/smart-questions.html#not_losing<http://www.catb.org/%7Eesr/faqs/smart-questions.html#not_losing>
>
> You might also benefit from seeing things from the perspective of
> developers,
> who again, volunteer their precious time to developing and supporting open
> source
> software, so that you can do your work more efficiently and smartly. Give
> this a
> read over a cup of tea:
>
> http://math-atlas.sourceforge.net/faq.html#utone
>
> Perhaps when you calm down, and allow yourself to reflect upon this
> experience,
> you will realize how badly you've behaved to a group of professional,
> friendly
> people whose only desire is to support and nuture a great tool.
>
> I will say nothing more about this topic, and under no circumstances will
> I
> reply to
> future messages from Linda Seltzer.
>
>
> Daran
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



From pwang at enthought.com  Mon Oct 13 17:30:16 2008
From: pwang at enthought.com (Peter Wang)
Date: Mon, 13 Oct 2008 16:30:16 -0500
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <1095.71.231.103.249.1223932607.squirrel@mail.alumni.caltech.edu>
References: <1398.71.231.103.249.1223692957.squirrel@mail.alumni.caltech.edu>
	<e86a5fd00810102226t5259d3acu9680836b8a61c1a1@mail.gmail.com>
	<1717.71.231.103.249.1223703534.squirrel@mail.alumni.caltech.edu>
	<1405.71.231.103.249.1223793543.squirrel@mail.alumni.caltech.edu>
	<48F26C4C.3030504@american.edu>
	<1253.71.231.103.249.1223875312.squirrel@mail.alumni.caltech.edu>
	<9457e7c80810130250o40eb3eaai59054ed5750a89b0@mail.gmail.com>
	<48F34DB4.8080603@gmail.com>
	<1268.71.231.103.249.1223922269.squirrel@mail.alumni.caltech.edu>
	<48F3A17C.3060602@american.edu>
	<88e473830810131304p1172e997k1c2d3899c2653ee2@mail.gmail.com>
	<1095.71.231.103.249.1223932607.squirrel@mail.alumni.caltech.edu>
Message-ID: <49790D98-C518-4EFE-8E0A-9F03518C831C@enthought.com>

On Oct 13, 2008, at 4:16 PM, Linda Seltzer wrote:

> Alan, Stop tuyrning this around.  Stop referring to my request as an
> accusation and stop referring to your put-downs as a "correction."

Linda, from what I can tell, the tone in this discussion thread  
changed from the "professional", "technical" mode with this statement:

> Please, no demeaning statements like "you forgot a parenthesis" or  
> "you
> were using someone else's code" - just the lines of code for a file  
> that
> actually *works.*

The folks on this list are a very friendly and helpful bunch, and I  
think some of them did not take well to your implying that they were  
demeaning to you.  You might not have meant it to be accusatory, but  
it certainly seems to have been interpreted that way.

Further appeals to a non-existent "moderator" and typing in ALL CAPS  
only fanned the flames, and so I think perhaps it would be good if  
folks just stepped back from the keyboard a bit and took some deep  
breaths.

Also, Linda, I would like to stress that although this mailing list is  
an open, unmoderated forum for technical discussion, even the veterans  
don't go about making proclamations about what people should and  
should not post... I think if you stick to asking technical questions,  
people will respond in kind.


-Peter



From robert.kern at gmail.com  Mon Oct 13 17:43:43 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 13 Oct 2008 16:43:43 -0500
Subject: [Numpy-discussion] End of Discussion (was Re: Need **working** code
	example of 2-D arrays)
Message-ID: <3d375d730810131443u5cb4c2cdvc1926d086a6aee25@mail.gmail.com>

Linda has informed me that she has left the mailing list. Please
consider this and all related threads closed.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From matthew.brett at gmail.com  Mon Oct 13 18:28:22 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 13 Oct 2008 15:28:22 -0700
Subject: [Numpy-discussion] Need **working** code example of 2-D arrays
In-Reply-To: <c2c0fea90810131424w6c70a6c6s134de6a818ab0ab4@mail.gmail.com>
References: <c2c0fea90810131424w6c70a6c6s134de6a818ab0ab4@mail.gmail.com>
Message-ID: <1e2af89e0810131528n323f9feaw6a561a473b5ff407@mail.gmail.com>

I know, I know, last one...

> http://www.catb.org/~esr/faqs/smart-questions.html

I had forgotten this wise quote from the smart questions FAQ:

"Be gentle. Problem-related stress can make people seem rude or stupid
even when they're not."

Best,

Matthew


From myeates at jpl.nasa.gov  Mon Oct 13 19:28:28 2008
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Mon, 13 Oct 2008 16:28:28 -0700
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
Message-ID: <48F3D99C.1050301@jpl.nasa.gov>

Is there a routine in scipy for telling whether  a point is inside a 
convex 4 sided polygon?

Mathew



From amcmorl at gmail.com  Mon Oct 13 19:46:05 2008
From: amcmorl at gmail.com (Angus McMorland)
Date: Mon, 13 Oct 2008 19:46:05 -0400
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <48F3D99C.1050301@jpl.nasa.gov>
References: <48F3D99C.1050301@jpl.nasa.gov>
Message-ID: <e85f13c40810131646nd282feeleb9d9f3903096551@mail.gmail.com>

2008/10/13 Mathew Yeates <myeates at jpl.nasa.gov>

> Is there a routine in scipy for telling whether  a point is inside a
> convex 4 sided polygon?


Not specifically in scipy, as far as I know, but there are several
supplementary packages that provide this functionality, including
matplotlib:

http://projects.scipy.org/pipermail/scipy-user/2008-February/015418.html

http://groups.google.com/group/Numpy-discussion/browse_thread/thread/2fca22bd29546ff2

Angus.
-- 
AJC McMorland
Post-doctoral research fellow
Neurobiology, University of Pittsburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081013/38cbee8a/attachment.html>

From pgmdevlist at gmail.com  Mon Oct 13 19:46:01 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 13 Oct 2008 19:46:01 -0400
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <e85f13c40810131646nd282feeleb9d9f3903096551@mail.gmail.com>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<e85f13c40810131646nd282feeleb9d9f3903096551@mail.gmail.com>
Message-ID: <200810131946.02141.pgmdevlist@gmail.com>

> 2008/10/13 Mathew Yeates <myeates at jpl.nasa.gov>
>
> > Is there a routine in scipy for telling whether  a point is inside a
> > convex 4 sided polygon?

Mathew,
You could use OGR (www.gdal.org)

Example
-------------
import osgeo.ogr as ogr

vert = [(0,0),(0,1),(1,1),(1,0)]
listvert = [" %s %s" % (x,y) for (x,y) in vert]
listvert.append(listvert[0])
geo = ogr.CreateGeometryFromWkt("POLYGON ((%s))" % ','.join(listvert))

querypoint = (0.5, 0.5)
qpt = ogr.CreateGeometryFromWkt("POINT(%s %s)" % querypoint)

assert geo.Contains(qpt)

querypoint = (0.5, 1.5)
qpt = ogr.CreateGeometryFromWkt("POINT(%s %s)" % querypoint)

assert not geo.Contains(qpt)



From wbaxter at gmail.com  Mon Oct 13 21:56:48 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 14 Oct 2008 10:56:48 +0900
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <200810131946.02141.pgmdevlist@gmail.com>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<e85f13c40810131646nd282feeleb9d9f3903096551@mail.gmail.com>
	<200810131946.02141.pgmdevlist@gmail.com>
Message-ID: <e86a5fd00810131856v26c0d252v1940612a107df19@mail.gmail.com>

On Tue, Oct 14, 2008 at 8:46 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>> 2008/10/13 Mathew Yeates <myeates at jpl.nasa.gov>
>>
>> > Is there a routine in scipy for telling whether  a point is inside a
>> > convex 4 sided polygon?
>
> Mathew,
> You could use OGR (www.gdal.org)
>
> Example
> -------------
> import osgeo.ogr as ogr
>
> vert = [(0,0),(0,1),(1,1),(1,0)]
> listvert = [" %s %s" % (x,y) for (x,y) in vert]
> listvert.append(listvert[0])
> geo = ogr.CreateGeometryFromWkt("POLYGON ((%s))" % ','.join(listvert))
>
> querypoint = (0.5, 0.5)
> qpt = ogr.CreateGeometryFromWkt("POINT(%s %s)" % querypoint)
>
> assert geo.Contains(qpt)
>
> querypoint = (0.5, 1.5)
> qpt = ogr.CreateGeometryFromWkt("POINT(%s %s)" % querypoint)
>
> assert not geo.Contains(qpt)


If all you really need is a point in convex polygon test, it's
probably a little too trivial to be worth dragging in a dependency on
anything.  But if you may find yourself needing more geometric tests
then it might be a good idea to get a good geom lib now.

As for this test, all you need to do is check that the point is to the
left of each of the edges, taken counter-clockwise.

Here's some code I wrote a while back that does a more general
even-odd test, but should work for your case too:

def inside_shape(p, verts, edges=None):
    """Test whether the point p is inside the specified shape.
    The shape is specified by 'verts' and 'edges'
    Arguments:
    p - the 2d point
    verts - (N,2) array of points
    edges - (N,2) array of vert indices indicating edges
            If edges is None then assumed to be in order. I.e.
              [[0,1], [1,2], [2,3] ... [N-1,0]]

    Returns:
    - True/False based on result of in/out test.

    Uses the 'ray to infinity' even-odd test.
    Let the ray be the horizontal ray starting at p and going to +inf in x.
    """
    verts = npy.asarray(verts)
    if edges is None:
        N = verts.shape[0]
        edges = npy.column_stack([npy.c_[0:N],npy.c_[1:N,0]])

    inside = False
    x,y=p[0],p[1]
    for e in edges:
        v0,v1 = verts[e[0]],verts[e[1]]
        # Check if both verts to the left of ray
        if v0[0]<x and v1[0]<x:
            continue
        # check if both on the same side of ray
        if (v0[1]<y and v1[1]<y) or (v0[1]>y and v1[1]>y):
            continue
        #check for horizontal line - another horz line can't intersect it
        if (v0[1]==v1[1]):
            continue
        # compute x intersection value
        xisect = v0[0] + (v1[0]-v0[0])*((y-v0[1])/(v1[1]-v0[1]))
        if xisect >= x:
            inside = not inside
    return inside

License: public domain.

--bb


From tjhnson at gmail.com  Tue Oct 14 00:40:15 2008
From: tjhnson at gmail.com (T J)
Date: Mon, 13 Oct 2008 21:40:15 -0700
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
Message-ID: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>

Hi,

I'm new to using SWIG and my reading of numpy_swig.pdf tells me that
the following typemap does not exist:

  (int* ARGOUT_ARRAY2, int DIM1, int DIM2)

What is the recommended way to output a 2D array?  It seems like I should use:

  (int* ARGOUT_ARRAY1, int DIM1)

and then provide a python function which reshapes the 1D array?  Is it
correct that there will be insignificant performance disadvantages to
this?  Also, is there any way to do this in an automated fashion?  My
first thought is that I'd need to create this function outside of the
python module that SWIG creates.

Thanks!


From stefan at sun.ac.za  Tue Oct 14 01:56:20 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 14 Oct 2008 07:56:20 +0200
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <48F3D99C.1050301@jpl.nasa.gov>
References: <48F3D99C.1050301@jpl.nasa.gov>
Message-ID: <9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>

Hi Matthew

Here is an implementation in Python, ctypes and in weave:

http://mentat.za.net/source/pnpoly.tar.bz2

Regards
St?fan


From wbaxter at gmail.com  Tue Oct 14 02:10:50 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 14 Oct 2008 15:10:50 +0900
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
Message-ID: <e86a5fd00810132310u1e55ce9fpef235798229ddfe0@mail.gmail.com>

On Tue, Oct 14, 2008 at 2:56 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Matthew
>
> Here is an implementation in Python, ctypes and in weave:
>
> http://mentat.za.net/source/pnpoly.tar.bz2
>

<snarf>
Thanks!  Looks better than what I wrote.

--bb


From haase at msg.ucsf.edu  Tue Oct 14 04:02:36 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 14 Oct 2008 10:02:36 +0200
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
In-Reply-To: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
References: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
Message-ID: <bc657ead0810140102t73c62f29nde72c2a3ed63ee5@mail.gmail.com>

On Tue, Oct 14, 2008 at 6:40 AM, T J <tjhnson at gmail.com> wrote:
> Hi,
>
> I'm new to using SWIG and my reading of numpy_swig.pdf tells me that
> the following typemap does not exist:
>
>  (int* ARGOUT_ARRAY2, int DIM1, int DIM2)
>
> What is the recommended way to output a 2D array?  It seems like I should use:
>
>  (int* ARGOUT_ARRAY1, int DIM1)
>
> and then provide a python function which reshapes the 1D array?  Is it
> correct that there will be insignificant performance disadvantages to
> this?  Also, is there any way to do this in an automated fashion?  My
> first thought is that I'd need to create this function outside of the
> python module that SWIG creates.
>
> Thanks!

Hi tjhnson,

I have been using SWIG for many years with numpy and before with  numarray;
but I still feel somewhat "insecure"...

anyhow:
a) There should be no performance difference between 1d and 2d -- its
sets just a different shape. (right?)
b) I don't want to use Python / numpy API code in the C functions I'm
wrapping - so I limit myself to "input" arrays! Since array memory
does not distinguish between input or output (assuming there is no
copying needed because of alignment or contiguity issues) the only
implication of this is that I have to allocate the array outside the C
functions.  I don't expect any noticable performance hit from this;
only caveat is that I have to know the "output" array SIZE before I go
into the C function.
The advantage of this is that I don't have to worry about using the
correct malloc (or rather equivalent numpy API) function and/or
garbage collection (object destructor)
(( there was a post by Travis Oliphant (Sep.10:: NumPy arrays that use
memory allocated from other libraries or tools) were he mentioned a
blog http://blog.enthought.com/?p=62   where is further discussed ))
An extra 2-line Python wrapper-function handles this
input<->conversion transparently for my --- do you need to handle
non-contiguous arrays ?

HTH, I'm eager to learn here ;-)

- Sebastian Haase


From scotta_2002 at yahoo.com  Tue Oct 14 09:12:59 2008
From: scotta_2002 at yahoo.com (Scott Askey)
Date: Tue, 14 Oct 2008 06:12:59 -0700 (PDT)
Subject: [Numpy-discussion] matlab ss2tf in python/symbols in linear
In-Reply-To: <mailman.748.1223648012.2878.numpy-discussion@scipy.org>
Message-ID: <777975.58824.qm@web36507.mail.mud.yahoo.com>


The numpy poly function work to solve most of my problem.  As soon as I realized the poly in numpy base and matlab perform the same funtion I was ble to efficintly transform my  state space model (A B C D ) into polynomial transfer funtions.  

H=num/den  num=(poly(A-dot(B,C))-poly(A))   den = poly(A)

Symbolically I was able to solve 
from sympy import *
Matrix(s*I -A +dot(B,C)).berkowitz()  
however it was orders of magnitude slower.

V/R

Scott

--- On Fri, 10/10/08, numpy-discussion-request at scipy.org <numpy-discussion-request at scipy.org> wrote:

> From: numpy-discussion-request at scipy.org <numpy-discussion-request at scipy.org>
> Subject: Numpy-discussion Digest, Vol 25, Issue 26
> To: numpy-discussion at scipy.org
> Date: Friday, October 10, 2008, 10:13 AM
> Send Numpy-discussion mailing list submissions to
> 	numpy-discussion at scipy.org
> 
> 
> Message: 2
> Date: Fri, 10 Oct 2008 04:02:21 -0700 (PDT)
> From: Scott Askey <scotta_2002 at yahoo.com>
> Subject: [Numpy-discussion] matlab ss2tf in python/symbols
> in linear
> 	algebra/transfer functions
> To: numpy-discussion at scipy.org
> Message-ID:
> <893083.2737.qm at web36501.mail.mud.yahoo.com>
> Content-Type: text/plain; charset=us-ascii
> 
> 
> 
> 
> > Subject: matlab ss2tf in python/symbol in linear
> algebra/transfer functions
> > To: numpy-discussion at scipy.org
> > Date: Thursday, October 9, 2008, 5:36 PM
> > Is there a clever way to do symbolic linear algebra
> with
> > python? What is the prober tool for doing linear
> algebra
> > with matrices containing symbols? 
> > 
> > 
> > V/R
> > 
> > Scott
> 
> 
>       
> 
> 
> ------------------------------
> 
> Message: 4
> Date: Fri, 10 Oct 2008 13:56:47 +0200
> From: Gael Varoquaux <gael.varoquaux at normalesup.org>
> Subject: Re: [Numpy-discussion] matlab ss2tf in
> python/symbols in
> 	linear	algebra/transfer functions
> To: Scott Askey <scotta_2002 at yahoo.com>
> Cc: numpy-discussion at scipy.org
> Message-ID:
> <20081010115647.GA26097 at phare.normalesup.org>
> Content-Type: text/plain; charset=iso-8859-1
> 
> On Fri, Oct 10, 2008 at 04:02:21AM -0700, Scott Askey
> wrote:
> > > Subject: matlab ss2tf in python/symbol in linear
> algebra/transfer functions
> > > To: numpy-discussion at scipy.org
> > > Date: Thursday, October 9, 2008, 5:36 PM
> > > Is there a clever way to do symbolic linear
> algebra with
> > > python? What is the prober tool for doing linear
> algebra
> > > with matrices containing symbols? 
> 
> Have a look at sympy: http://code.google.com/p/sympy/
> 
> Ga?l
> 
> 


      


From scotta_2002 at yahoo.com  Tue Oct 14 09:22:10 2008
From: scotta_2002 at yahoo.com (Scott Askey)
Date: Tue, 14 Oct 2008 06:22:10 -0700 (PDT)
Subject: [Numpy-discussion] Python equivalent to Matlab control systems/
	LTI/state space model manipulation tools
Message-ID: <240084.15289.qm@web36504.mail.mud.yahoo.com>

Where is a good place to look for python funtions similar to Matlab's
ss , tf, ss2tf as use for transforming a linear time invariant(LTI) model into a state space model. 

V/R

Scott

http://www.mathworks.com/access/helpdesk/help/toolbox/control/


      


From strawman at astraw.com  Tue Oct 14 09:40:38 2008
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 14 Oct 2008 06:40:38 -0700
Subject: [Numpy-discussion] Python equivalent to Matlab control systems/
 LTI/state space model manipulation tools
In-Reply-To: <240084.15289.qm@web36504.mail.mud.yahoo.com>
References: <240084.15289.qm@web36504.mail.mud.yahoo.com>
Message-ID: <48F4A156.30105@astraw.com>

Ryan Krauss has been working on something, although I have not had a
chance to try it.

http://www.siue.edu/~rkrauss/python_intro.html

Scott Askey wrote:
> Where is a good place to look for python funtions similar to Matlab's
> ss , tf, ss2tf as use for transforming a linear time invariant(LTI) model into a state space model. 
> 
> V/R
> 
> Scott
> 
> http://www.mathworks.com/access/helpdesk/help/toolbox/control/
> 
> 
>       
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From stefan at sun.ac.za  Tue Oct 14 09:45:37 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 14 Oct 2008 15:45:37 +0200
Subject: [Numpy-discussion] Python equivalent to Matlab control systems/
	LTI/state space model manipulation tools
In-Reply-To: <240084.15289.qm@web36504.mail.mud.yahoo.com>
References: <240084.15289.qm@web36504.mail.mud.yahoo.com>
Message-ID: <9457e7c80810140645x604cb778p98e4fcf93b4bd8ea@mail.gmail.com>

Hi Scott

2008/10/14 Scott Askey <scotta_2002 at yahoo.com>:
> Where is a good place to look for python funtions similar to Matlab's
> ss , tf, ss2tf as use for transforming a linear time invariant(LTI) model into a state space model.

Look under scipy.signal.  These utilities need a lot of work, so if
you can contribute anything back that would be greatly appreciated.

Regards
St?fan


From ndbecker2 at gmail.com  Tue Oct 14 10:44:12 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 14 Oct 2008 10:44:12 -0400
Subject: [Numpy-discussion] Apply a vector function to each row of a
	matrix
References: <gcks15$7bt$1@ger.gmane.org>
	<91cf711d0810090624s1c465992l6dd922cfaa7c67c7@mail.gmail.com>
	<gcl1jn$sgr$1@ger.gmane.org>
	<91cf711d0810091001n6abf6fe8ie2cb145d8406e137@mail.gmail.com>
	<gcljn7$52a$1@ger.gmane.org>
	<91cf711d0810100827u735c98d8od32ea85d671680f0@mail.gmail.com>
Message-ID: <gd2b7s$bmc$1@ger.gmane.org>

David Huard wrote:
...

> I've edited the docstring at
> http://sd-2116.dedibox.fr/pydocweb/doc/numpy.lib.shape_base.apply_along_axis/
> 
> Feel free to improve on it.

Thanks!  That is better.




From peter_williams at agilent.com  Tue Oct 14 12:12:30 2008
From: peter_williams at agilent.com (peter_williams at agilent.com)
Date: Tue, 14 Oct 2008 10:12:30 -0600
Subject: [Numpy-discussion] basic install question
Message-ID: <4E858FCB361FFB48B8E9816986850ACB0275E916@cos-us-mb03.cos.agilent.com>

Do I need ATLAS to install NumPy?

 

Apologies if this is in the archives somewhere, but I can't find it, and I can't figure it out from http://www.scipy.org/Installing_SciPy/Linux. Clearly you need some kind of BLAS/ LAPACK, but it's not clear if ATLAS required, or is just one option of many.

 

Thanks,

Peter 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081014/83cb804b/attachment.html>

From robert.kern at gmail.com  Tue Oct 14 12:32:33 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 14 Oct 2008 11:32:33 -0500
Subject: [Numpy-discussion] basic install question
In-Reply-To: <4E858FCB361FFB48B8E9816986850ACB0275E916@cos-us-mb03.cos.agilent.com>
References: <4E858FCB361FFB48B8E9816986850ACB0275E916@cos-us-mb03.cos.agilent.com>
Message-ID: <3d375d730810140932o578abcfdu2d57ebeac994fdf0@mail.gmail.com>

On Tue, Oct 14, 2008 at 11:12,  <peter_williams at agilent.com> wrote:
> Do I need ATLAS to install NumPy?

No.

> Apologies if this is in the archives somewhere, but I can't find it, and I
> can't figure it out from http://www.scipy.org/Installing_SciPy/Linux.
> Clearly you need some kind of BLAS/ LAPACK, but it's not clear if ATLAS
> required, or is just one option of many.

If you do not have ATLAS or a similarly accelerated BLAS/LAPACK, numpy
will just use its own implementations; i.e. you do not need an
external library at all unless if you want to use one which is
particularly fast.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From tjhnson at gmail.com  Tue Oct 14 13:51:43 2008
From: tjhnson at gmail.com (T J)
Date: Tue, 14 Oct 2008 10:51:43 -0700
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
In-Reply-To: <bc657ead0810140102t73c62f29nde72c2a3ed63ee5@mail.gmail.com>
References: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
	<bc657ead0810140102t73c62f29nde72c2a3ed63ee5@mail.gmail.com>
Message-ID: <b02c60870810141051l7c55c287vcc096cbee5fbadd@mail.gmail.com>

On Tue, Oct 14, 2008 at 1:02 AM, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> b) I don't want to use Python / numpy API code in the C functions I'm
> wrapping - so I limit myself to "input" arrays! Since array memory
> does not distinguish between input or output (assuming there is no
> copying needed because of alignment or contiguity issues) the only
> implication of this is that I have to allocate the array outside the C
> functions.

To clarify, are you saying that you don't make any use of numpy.i?
The functions I'm writing are not making any explicit use of the
Python/numpy API, but I'm sure it shows up in the *_wrap.c file that
SWIG creates.

> input<->conversion transparently for my --- do you need to handle
> non-contiguous arrays ?

Probably not.

>
> HTH, I'm eager to learn here ;-)
>

Thanks


From Chris.Barker at noaa.gov  Tue Oct 14 14:11:29 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 14 Oct 2008 11:11:29 -0700
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
In-Reply-To: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
References: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
Message-ID: <48F4E0D1.6030709@noaa.gov>

T J wrote:
> I'm new to using SWIG and my reading of numpy_swig.pdf tells me that
> the following typemap does not exist:
> 
>   (int* ARGOUT_ARRAY2, int DIM1, int DIM2)

are you referring to this statement in the docs?

"""
Note that we support DATA_TYPE* argout typemaps in 1D, but not 2D or 3D. 
This is because of a quirk with the SWIG typemap syntax and cannot be 
avoided.
"""
> What is the recommended way to output a 2D array?

I wonder if SWIG has been improved in that regard? I might be worth a 
post to the SWIG list. Otherwise, Bill Spotz is the guy to ask -- if he 
doesn't reply to the soon, he may not be monitoring this list 00 you 
might try contacting him directly.

>  It seems like I should use:
> 
>   (int* ARGOUT_ARRAY1, int DIM1)
> 
> and then provide a python function which reshapes the 1D array?

That probably would be the easiest way to do it.

>  Is it
> correct that there will be insignificant performance disadvantages to
> this?

miniscule performance hit -- re-shaping if you aren't changing the data 
is cheap.

>  Also, is there any way to do this in an automated fashion?

yes -- my SWIG-foo is very rusty, but you can add python code in your 
*.i file, so you could write the file that does that conversion, put it 
there, and it would get added in by SWIG.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Tue Oct 14 14:14:44 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 14 Oct 2008 11:14:44 -0700
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
In-Reply-To: <48F4E0D1.6030709@noaa.gov>
References: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
	<48F4E0D1.6030709@noaa.gov>
Message-ID: <48F4E194.2050907@noaa.gov>

>>  It seems like I should use:
>>
>>   (int* ARGOUT_ARRAY1, int DIM1)
>>
>> and then provide a python function which reshapes the 1D array?

Oh, another option is to use:

( DATA_TYPE* INPLACE_FARRAY2, int DIM1, int DIM2 )

and create the array in python first. This could be done with a little 
wrapper, if you want.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From haase at msg.ucsf.edu  Tue Oct 14 15:58:49 2008
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 14 Oct 2008 21:58:49 +0200
Subject: [Numpy-discussion] SWIG, typemaps, 2D argout arrays
In-Reply-To: <b02c60870810141051l7c55c287vcc096cbee5fbadd@mail.gmail.com>
References: <b02c60870810132140m2d81bcfcod4ef4e00d158ef42@mail.gmail.com>
	<bc657ead0810140102t73c62f29nde72c2a3ed63ee5@mail.gmail.com>
	<b02c60870810141051l7c55c287vcc096cbee5fbadd@mail.gmail.com>
Message-ID: <bc657ead0810141258k615685f4u32df256325528b06@mail.gmail.com>

On Tue, Oct 14, 2008 at 7:51 PM, T J <tjhnson at gmail.com> wrote:
> On Tue, Oct 14, 2008 at 1:02 AM, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>> b) I don't want to use Python / numpy API code in the C functions I'm
>> wrapping - so I limit myself to "input" arrays! Since array memory
>> does not distinguish between input or output (assuming there is no
>> copying needed because of alignment or contiguity issues) the only
>> implication of this is that I have to allocate the array outside the C
>> functions.
>
> To clarify, are you saying that you don't make any use of numpy.i?

I have my own (older) version -- I think Bill Spotz has improved
numpy.i by now to do what mine does (supposedly using cleaner code).

> The functions I'm writing are not making any explicit use of the
> Python/numpy API, but I'm sure it shows up in the *_wrap.c file that
> SWIG creates.

If you are using INPLACE arrays (this is I think the word numpy.i
uses) than there is no additional mem allocation. This is/was the most
important feature of all this numpy/SWIG idea.

>
>> input<->conversion transparently for my --- do you need to handle
>> non-contiguous arrays ?
>
> Probably not.
>
So you are fine;
do you know the output array shape beforehand ?

- Sebastian


From sidky at uchicago.edu  Tue Oct 14 18:02:56 2008
From: sidky at uchicago.edu (emil)
Date: Tue, 14 Oct 2008 17:02:56 -0500
Subject: [Numpy-discussion] memory usage
Message-ID: <48F51710.60103@uchicago.edu>

Hi,
   I'm having a problem with my python code, using numpy, chewing up too
much memory.
In the following, I boiled down my program to the simplest example that
has the problem:

from numpy import *
for i in range(1000):
   a = random.randn(512**2)
   b = a.argsort(kind='quick')

This loop takes a couple of minutes to run on my machine.
While running 'top' concurrently, I see that the memory usage is
increasing as the loop progresses. By the time the loop is finished
the python process is taking over 30% of the memory, and I have 8 GB RAM.

Is there some way to prevent this from happening?
It's fine if the alternative slows the code down a bit.
Im'm using python2.4 and numpy 1.0.1

Thanks in advance,
Emil


From robert.kern at gmail.com  Tue Oct 14 18:11:01 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 14 Oct 2008 17:11:01 -0500
Subject: [Numpy-discussion] memory usage
In-Reply-To: <48F51710.60103@uchicago.edu>
References: <48F51710.60103@uchicago.edu>
Message-ID: <3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>

On Tue, Oct 14, 2008 at 17:02, emil <sidky at uchicago.edu> wrote:
> Hi,
>   I'm having a problem with my python code, using numpy, chewing up too
> much memory.
> In the following, I boiled down my program to the simplest example that
> has the problem:
>
> from numpy import *
> for i in range(1000):
>   a = random.randn(512**2)
>   b = a.argsort(kind='quick')
>
> This loop takes a couple of minutes to run on my machine.
> While running 'top' concurrently, I see that the memory usage is
> increasing as the loop progresses. By the time the loop is finished
> the python process is taking over 30% of the memory, and I have 8 GB RAM.
>
> Is there some way to prevent this from happening?
> It's fine if the alternative slows the code down a bit.
> Im'm using python2.4 and numpy 1.0.1

Can you try upgrading to numpy 1.2.0? On my machine with numpy 1.2.0
on OS X, the memory usage is stable.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From f.yw at hotmail.com  Tue Oct 14 21:23:24 2008
From: f.yw at hotmail.com (frank wang)
Date: Tue, 14 Oct 2008 19:23:24 -0600
Subject: [Numpy-discussion] how to save a large array into a file quickly
In-Reply-To: <3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
References: <48F51710.60103@uchicago.edu>
	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
Message-ID: <BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>


Hi,
 
I have a large ndarray that I want to dump to a file. I know that I can use a for loop to write one data at a time. Since Python is a very powerfully language, I want to find a way that will dump the data fast and clean. The data can be in floating point or integer.
 
Thanks
 
Frank
_________________________________________________________________
Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie.
http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081014/217a6d3f/attachment.html>

From hwchen.mailman at gmail.com  Tue Oct 14 21:24:34 2008
From: hwchen.mailman at gmail.com (Huang-Wen Chen)
Date: Tue, 14 Oct 2008 21:24:34 -0400
Subject: [Numpy-discussion] memory usage
In-Reply-To: <3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
References: <48F51710.60103@uchicago.edu>
	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
Message-ID: <48F54652.4070608@gmail.com>

Robert Kern wrote:
>> from numpy import *
>> for i in range(1000):
>>   a = random.randn(512**2)
>>   b = a.argsort(kind='quick')
>
> Can you try upgrading to numpy 1.2.0? On my machine with numpy 1.2.0
> on OS X, the memory usage is stable.
>   
I tried the code fragment on two platforms and the memory usage is also 
normal.

1. numpy 1.1.1, python 2.5.1 on Vista 32bit
2. numpy 1.2.0, python 2.6 on RedHat 64bit


From efiring at hawaii.edu  Tue Oct 14 21:29:32 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Tue, 14 Oct 2008 15:29:32 -1000
Subject: [Numpy-discussion] how to save a large array into a file quickly
In-Reply-To: <BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>
References: <48F51710.60103@uchicago.edu>
	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
	<BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>
Message-ID: <48F5477C.1020308@hawaii.edu>

frank wang wrote:
> Hi,
>  
> I have a large ndarray that I want to dump to a file. I know that I can 
> use a for loop to write one data at a time. Since Python is a very 
> powerfully language, I want to find a way that will dump the data fast 
> and clean. The data can be in floating point or integer.

Use numpy.save for a single array, or numpy.savez for multiple ndarrays, 
assuming you will want to read them back with numpy.  If you want to 
dump to a text file, use numpy.savetxt.  If you want to dump to a binary 
file to be read by another program, you might want to use the tofile 
method of the ndarray.

Eric


From charlesr.harris at gmail.com  Wed Oct 15 01:23:56 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 14 Oct 2008 23:23:56 -0600
Subject: [Numpy-discussion] LU factorization?
Message-ID: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>

Hi All,

numpy.linalg has qr and cholesky factorizations, but LU factorization is
only available in scipy. That doesn't seem quite right. I think is would
make sense to include the LU factorization in numpy among the basic linalg
operations, and probably LU_solve also. Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081014/6c6b924d/attachment.html>

From stefan at sun.ac.za  Wed Oct 15 02:51:52 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 15 Oct 2008 08:51:52 +0200
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
Message-ID: <9457e7c80810142351qa245328x4f60d21a50e910d0@mail.gmail.com>

2008/10/15 Charles R Harris <charlesr.harris at gmail.com>:
> numpy.linalg has qr and cholesky factorizations, but LU factorization is
> only available in scipy. That doesn't seem quite right. I think is would
> make sense to include the LU factorization in numpy among the basic linalg
> operations, and probably LU_solve also. Thoughts?

I've needed it a lot in the past, and it is a perfect fit for
numpy.linalg.  It also paves the way to a reduced-row-echelon routine
in the Matrix class.  It seems technically feasable, so I am in
favour.

Regards
St?fan


From irving at naml.us  Wed Oct 15 03:20:50 2008
From: irving at naml.us (Geoffrey Irving)
Date: Wed, 15 Oct 2008 00:20:50 -0700
Subject: [Numpy-discussion] dtype comparison and hashing
Message-ID: <7f9d599f0810150020m489b54day4c909f6cbc8c1203@mail.gmail.com>

Hello,

Currently in numpy comparing dtypes for equality with == does an
internal PyArray_EquivTypes check, which means that the dtypes NPY_INT
and NPY_LONG compare as equal in python.  However, the hash function
for dtypes reduces id(), which is therefore inconsistent with ==.
Unfortunately I can't produce a python snippet showing this since I
don't know how to create a NPY_INT dtype in pure python.

Based on the source it looks like hash should raise a type error,
since tp_hash is null but tp_richcompare is not.  Does the following
snippet through an exception for others?

>>> import numpy
>>> hash(numpy.dtype('int'))
5708736

This might be the problem:

/* Macro to get the tp_richcompare field of a type if defined */
#define RICHCOMPARE(t) (PyType_HasFeature((t), Py_TPFLAGS_HAVE_RICHCOMPARE) \
                         ? (t)->tp_richcompare : NULL)

I'm using the default Mac OS X 10.5 installation of python 2.5 and
numpy, so maybe those weren't compiled correctly.  Has anyone else
seen this issue?

Thanks,
Geoffrey


From schut at sarvision.nl  Wed Oct 15 03:24:02 2008
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 15 Oct 2008 09:24:02 +0200
Subject: [Numpy-discussion] memory usage
In-Reply-To: <48F54652.4070608@gmail.com>
References: <48F51710.60103@uchicago.edu>	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
	<48F54652.4070608@gmail.com>
Message-ID: <gd45qj$d50$1@ger.gmane.org>

Huang-Wen Chen wrote:
> Robert Kern wrote:
>>> from numpy import *
>>> for i in range(1000):
>>>   a = random.randn(512**2)
>>>   b = a.argsort(kind='quick')
>> Can you try upgrading to numpy 1.2.0? On my machine with numpy 1.2.0
>> on OS X, the memory usage is stable.
>>   
> I tried the code fragment on two platforms and the memory usage is also 
> normal.
> 
> 1. numpy 1.1.1, python 2.5.1 on Vista 32bit
> 2. numpy 1.2.0, python 2.6 on RedHat 64bit

If I recall correctly, there were some major improvements in python's 
memory management/garbage collection from version 2.4 to 2.5. If you 
could try to upgrade your python to 2.5 (and possibly also your numpy to 
1.2.0), you'd probably see some better behaviour.

Regards,
Vincent.



From rocksportrocker at googlemail.com  Wed Oct 15 06:47:53 2008
From: rocksportrocker at googlemail.com (Uwe Schmitt)
Date: Wed, 15 Oct 2008 03:47:53 -0700 (PDT)
Subject: [Numpy-discussion] Any numpy trick for my problem ?
Message-ID: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>

Hi,

I got a matrix of 2100 lines, and I want to calculate  blockwise mean
vectors.
Each block consists of 10 consecutive rows.

My code looks like this:

   rv = []
   for i in range(0, 2100, 10):
       rv.append( mean(matrix[i:i+10], axis=0))

   return array(rv)

Is there a more elegant and may be faster method to perform this
calculation ?

Greetings, Uwe


From charlesr.harris at gmail.com  Wed Oct 15 06:56:52 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 04:56:52 -0600
Subject: [Numpy-discussion] Any numpy trick for my problem ?
In-Reply-To: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>
References: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>
Message-ID: <e06186140810150356q76a06841oe3d276f463a29484@mail.gmail.com>

On Wed, Oct 15, 2008 at 4:47 AM, Uwe Schmitt <rocksportrocker at googlemail.com
> wrote:

> Hi,
>
> I got a matrix of 2100 lines, and I want to calculate  blockwise mean
> vectors.
> Each block consists of 10 consecutive rows.
>
> My code looks like this:
>
>   rv = []
>   for i in range(0, 2100, 10):
>       rv.append( mean(matrix[i:i+10], axis=0))
>
>   return array(rv)
>
> Is there a more elegant and may be faster method to perform this
> calculation ?
>

Something like

In [1]: M = np.random.ranf((40,5))

In [2]: M.reshape(4,10,5).mean(axis=1)
Out[2]:
array([[ 0.57979278,  0.50013352,  0.66783389,  0.4009187 ,  0.36379445],
       [ 0.46938844,  0.34449102,  0.56419189,  0.49134703,  0.61380198],
       [ 0.5644788 ,  0.61734034,  0.3656104 ,  0.63147275,  0.46319345],
       [ 0.56556899,  0.59012606,  0.39691084,  0.26566127,  0.57107896]])

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/8abcbe94/attachment.html>

From rocksportrocker at googlemail.com  Wed Oct 15 07:00:59 2008
From: rocksportrocker at googlemail.com (Uwe Schmitt)
Date: Wed, 15 Oct 2008 04:00:59 -0700 (PDT)
Subject: [Numpy-discussion] Any numpy trick for my problem ?
In-Reply-To: <e06186140810150356q76a06841oe3d276f463a29484@mail.gmail.com>
References: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>
	<e06186140810150356q76a06841oe3d276f463a29484@mail.gmail.com>
Message-ID: <1df32ff9-4720-4249-9380-1d84f4073940@u27g2000pro.googlegroups.com>

That's cool. Thanks for your fast answer.

Greetings, Uwe

On 15 Okt., 12:56, "Charles R Harris" <charlesr.har... at gmail.com>
wrote:
> On Wed, Oct 15, 2008 at 4:47 AM, Uwe Schmitt <rocksportroc... at googlemail.com
>
>
>
> > wrote:
> > Hi,
>
> > I got a matrix of 2100 lines, and I want to calculate ?blockwise mean
> > vectors.
> > Each block consists of 10 consecutive rows.
>
> > My code looks like this:
>
> > ? rv = []
> > ? for i in range(0, 2100, 10):
> > ? ? ? rv.append( mean(matrix[i:i+10], axis=0))
>
> > ? return array(rv)
>
> > Is there a more elegant and may be faster method to perform this
> > calculation ?
>
> Something like
>
> In [1]: M = np.random.ranf((40,5))
>
> In [2]: M.reshape(4,10,5).mean(axis=1)
> Out[2]:
> array([[ 0.57979278, ?0.50013352, ?0.66783389, ?0.4009187 , ?0.36379445],
> ? ? ? ?[ 0.46938844, ?0.34449102, ?0.56419189, ?0.49134703, ?0.61380198],
> ? ? ? ?[ 0.5644788 , ?0.61734034, ?0.3656104 , ?0.63147275, ?0.46319345],
> ? ? ? ?[ 0.56556899, ?0.59012606, ?0.39691084, ?0.26566127, ?0.57107896]])
>
> Chuck
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


From elcorto at gmx.net  Wed Oct 15 07:01:21 2008
From: elcorto at gmx.net (Steve Schmerler)
Date: Wed, 15 Oct 2008 13:01:21 +0200
Subject: [Numpy-discussion] how to save a large array into a file	quickly
In-Reply-To: <48F5477C.1020308@hawaii.edu>
References: <48F51710.60103@uchicago.edu>
	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
	<BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>
	<48F5477C.1020308@hawaii.edu>
Message-ID: <20081015110121.GA5652@ramrod.starsheriffs.de>

On Oct 14 15:29 -1000, Eric Firing  wrote:
> frank wang wrote:
> > Hi,
> >  
> > I have a large ndarray that I want to dump to a file. I know that I can 
> > use a for loop to write one data at a time. Since Python is a very 
> > powerfully language, I want to find a way that will dump the data fast 
> > and clean. The data can be in floating point or integer.
> 
> Use numpy.save for a single array, or numpy.savez for multiple ndarrays, 
> assuming you will want to read them back with numpy.  If you want to 
> dump to a text file, use numpy.savetxt.  If you want to dump to a binary 
> file to be read by another program, you might want to use the tofile 
> method of the ndarray.
> 

I've just updated [1] to mention scipy.io.npfile as well as numpy.save &
friends.  Now, I hope that all common ways to read/write arrays are
present in one place.

[1] http://scipy.org/Cookbook/InputOutput

best,
steve


From stefan at sun.ac.za  Wed Oct 15 07:05:33 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 15 Oct 2008 13:05:33 +0200
Subject: [Numpy-discussion] Any numpy trick for my problem ?
In-Reply-To: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>
References: <eef669dd-fcce-4518-ac4a-9f595ac5607d@t39g2000prh.googlegroups.com>
Message-ID: <9457e7c80810150405j16b8ebebl59e8df0c36c0ad0c@mail.gmail.com>

Hi Uwe

2008/10/15 Uwe Schmitt <rocksportrocker at googlemail.com>:
> I got a matrix of 2100 lines, and I want to calculate  blockwise mean
> vectors.
> Each block consists of 10 consecutive rows.
>
> My code looks like this:
>
>   rv = []
>   for i in range(0, 2100, 10):
>       rv.append( mean(matrix[i:i+10], axis=0))
>
>   return array(rv)
>
> Is there a more elegant and may be faster method to perform this
> calculation ?

You can use array striding.  See

http://mentat.za.net/numpy/numpy_advanced_slides/

from slide 29 onwards.

Regards
St?fan


From aisaac at american.edu  Wed Oct 15 08:40:20 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 15 Oct 2008 08:40:20 -0400
Subject: [Numpy-discussion] how to save a large array into a file quickly
In-Reply-To: <BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>
References: <48F51710.60103@uchicago.edu>	<3d375d730810141511m515b14fave482bd8e14e9a15@mail.gmail.com>
	<BLU105-W31743B48E543A7BFFC8AD589300@phx.gbl>
Message-ID: <48F5E4B4.2010701@american.edu>

On 10/14/2008 9:23 PM frank wang apparently wrote:
> I have a large ndarray that I want to dump to a file. I know that I can 
> use a for loop to write one data at a time. Since Python is a very 
> powerfully language, I want to find a way that will dump the data fast 
> and clean. The data can be in floating point or integer.

Use the ``tofile()`` method:
http://www.scipy.org/Numpy_Example_List#head-2acd2a84907edbd410bf426847403ce8ea151814

hth,
Alan Isaac





From ggellner at uoguelph.ca  Wed Oct 15 09:56:26 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Wed, 15 Oct 2008 09:56:26 -0400
Subject: [Numpy-discussion] var bias reason?
Message-ID: <20081015135626.GA12837@encolpuis>

Some colleagues noticed that var uses biased formula's by default in numpy,
searching for the reason only brought up:

http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias

which I totally agree with, but there was no response? Any reason for this? Is
there any way I can hide the var/std methods of numpy arrays? I have my own
library which does things the R way, and in my lab letting people discover
this 'bug' using name completion in ipython is a nightmare.

Any help?

thanks, Gabriel


From oliphant at enthought.com  Wed Oct 15 10:45:39 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 15 Oct 2008 09:45:39 -0500
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <20081015135626.GA12837@encolpuis>
References: <20081015135626.GA12837@encolpuis>
Message-ID: <48F60213.2050204@enthought.com>

Gabriel Gellner wrote:
> Some colleagues noticed that var uses biased formula's by default in numpy,
> searching for the reason only brought up:
>
> http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
>
> which I totally agree with, but there was no response? Any reason for this?
I will try to respond to this as it was me who made the change.  I think 
there have been responses, but I think I've preferred to stay quiet 
rather than feed a flame war.   Ultimately, it is a matter of preference 
and I don't think there would be equal weights given to all the 
arguments surrounding the decision by everybody.

I will attempt to articulate my reasons:  dividing by n is the maximum 
likelihood estimator of variance and I prefer that justification more 
than the "un-biased" justification for a default (especially given that 
bias is just one part of the "error" in an estimator).    Having every 
package that computes the mean return the "un-biased" estimate gives it 
more cultural weight than than the concept deserves, I think.  Any 
surprise that is created by the different default should be mitigated by 
the fact that it's an opportunity to learn something about what you are 
doing.    Here is a paper I wrote on the subject that you might find 
useful:

https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
(Hopefully, they will resolve a link problem at the above site soon, but 
you can read the abstract).

I'm not trying to persuade anybody with this email (although if you can 
download the paper at the above link, then I am trying to persuade with 
that).  In this email I'm just trying to give context to the poster as I 
think the question is legitimate.

With that said, there is the ddof parameter so that you can change what 
the divisor is.  I think that is a useful compromise.

I'm unhappy with the internal inconsistency of cov, as I think it was an 
oversight. I'd be happy to see cov changed as well to use the ddof 
argument instead of the bias keyword, but that is an API change and 
requires some transition discussion and work.

The only other argument I've heard against the current situation is 
"unit testing" with MATLAB or R code.   Just use ddof=1 when comparing 
against MATLAB and R code is my suggestion.

Best regards,

-Travis



From cournape at gmail.com  Wed Oct 15 11:19:54 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 16 Oct 2008 00:19:54 +0900
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <48F60213.2050204@enthought.com>
References: <20081015135626.GA12837@encolpuis> <48F60213.2050204@enthought.com>
Message-ID: <5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>

On Wed, Oct 15, 2008 at 11:45 PM, Travis E. Oliphant
<oliphant at enthought.com> wrote:
> Gabriel Gellner wrote:
>> Some colleagues noticed that var uses biased formula's by default in numpy,
>> searching for the reason only brought up:
>>
>> http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
>>
>> which I totally agree with, but there was no response? Any reason for this?
> I will try to respond to this as it was me who made the change.  I think
> there have been responses, but I think I've preferred to stay quiet
> rather than feed a flame war.   Ultimately, it is a matter of preference
> and I don't think there would be equal weights given to all the
> arguments surrounding the decision by everybody.
>
> I will attempt to articulate my reasons:  dividing by n is the maximum
> likelihood estimator of variance and I prefer that justification more
> than the "un-biased" justification for a default (especially given that
> bias is just one part of the "error" in an estimator).    Having every
> package that computes the mean return the "un-biased" estimate gives it
> more cultural weight than than the concept deserves, I think.  Any
> surprise that is created by the different default should be mitigated by
> the fact that it's an opportunity to learn something about what you are
> doing.    Here is a paper I wrote on the subject that you might find
> useful:
>
> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
> (Hopefully, they will resolve a link problem at the above site soon, but
> you can read the abstract).

Yes, I hope too, I would be happy to read the article.

On the limit of unbiasdness, the following document mentions an
example (in a different context than variance estimation):

http://www.stat.columbia.edu/~gelman/research/published/badbayesresponsemain.pdf

AFAIK, even statisticians who consider themselves as "mostly
frequentist" (if that makes any sense) do not advocate unbiasdness as
such an important concept anymore (Larry Wasserman mentions it in his
"all of statistics").

cheers,

David


From pebarrett at gmail.com  Wed Oct 15 11:31:44 2008
From: pebarrett at gmail.com (Paul Barrett)
Date: Wed, 15 Oct 2008 11:31:44 -0400
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>
References: <20081015135626.GA12837@encolpuis> <48F60213.2050204@enthought.com>
	<5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>
Message-ID: <40e64fa20810150831q5852b96co68c5f53741a794a7@mail.gmail.com>

I'm behind Travis on this one.

 -- Paul

On Wed, Oct 15, 2008 at 11:19 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Wed, Oct 15, 2008 at 11:45 PM, Travis E. Oliphant
> <oliphant at enthought.com> wrote:
>> Gabriel Gellner wrote:
>>> Some colleagues noticed that var uses biased formula's by default in numpy,
>>> searching for the reason only brought up:
>>>
>>> http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
>>>
>>> which I totally agree with, but there was no response? Any reason for this?
>> I will try to respond to this as it was me who made the change.  I think
>> there have been responses, but I think I've preferred to stay quiet
>> rather than feed a flame war.   Ultimately, it is a matter of preference
>> and I don't think there would be equal weights given to all the
>> arguments surrounding the decision by everybody.
>>
>> I will attempt to articulate my reasons:  dividing by n is the maximum
>> likelihood estimator of variance and I prefer that justification more
>> than the "un-biased" justification for a default (especially given that
>> bias is just one part of the "error" in an estimator).    Having every
>> package that computes the mean return the "un-biased" estimate gives it
>> more cultural weight than than the concept deserves, I think.  Any
>> surprise that is created by the different default should be mitigated by
>> the fact that it's an opportunity to learn something about what you are
>> doing.    Here is a paper I wrote on the subject that you might find
>> useful:
>>
>> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
>> (Hopefully, they will resolve a link problem at the above site soon, but
>> you can read the abstract).
>
> Yes, I hope too, I would be happy to read the article.
>
> On the limit of unbiasdness, the following document mentions an
> example (in a different context than variance estimation):
>
> http://www.stat.columbia.edu/~gelman/research/published/badbayesresponsemain.pdf
>
> AFAIK, even statisticians who consider themselves as "mostly
> frequentist" (if that makes any sense) do not advocate unbiasdness as
> such an important concept anymore (Larry Wasserman mentions it in his
> "all of statistics").
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From sransom at nrao.edu  Wed Oct 15 11:58:24 2008
From: sransom at nrao.edu (Scott Ransom)
Date: Wed, 15 Oct 2008 11:58:24 -0400
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <40e64fa20810150831q5852b96co68c5f53741a794a7@mail.gmail.com>
References: <20081015135626.GA12837@encolpuis>
	<5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>
	<40e64fa20810150831q5852b96co68c5f53741a794a7@mail.gmail.com>
Message-ID: <200810151158.24268.sransom@nrao.edu>

Me too.

S

On Wednesday 15 October 2008 11:31:44 am Paul Barrett wrote:
> I'm behind Travis on this one.
>
>  -- Paul
>
> On Wed, Oct 15, 2008 at 11:19 AM, David Cournapeau 
<cournape at gmail.com> wrote:
> > On Wed, Oct 15, 2008 at 11:45 PM, Travis E. Oliphant
> >
> > <oliphant at enthought.com> wrote:
> >> Gabriel Gellner wrote:
> >>> Some colleagues noticed that var uses biased formula's by default
> >>> in numpy, searching for the reason only brought up:
> >>>
> >>> http://article.gmane.org/gmane.comp.python.numeric.general/12438/
> >>>match=var+bias
> >>>
> >>> which I totally agree with, but there was no response? Any reason
> >>> for this?
> >>
> >> I will try to respond to this as it was me who made the change.  I
> >> think there have been responses, but I think I've preferred to
> >> stay quiet rather than feed a flame war.   Ultimately, it is a
> >> matter of preference and I don't think there would be equal
> >> weights given to all the arguments surrounding the decision by
> >> everybody.
> >>
> >> I will attempt to articulate my reasons:  dividing by n is the
> >> maximum likelihood estimator of variance and I prefer that
> >> justification more than the "un-biased" justification for a
> >> default (especially given that bias is just one part of the
> >> "error" in an estimator).    Having every package that computes
> >> the mean return the "un-biased" estimate gives it more cultural
> >> weight than than the concept deserves, I think.  Any surprise that
> >> is created by the different default should be mitigated by the
> >> fact that it's an opportunity to learn something about what you
> >> are doing.    Here is a paper I wrote on the subject that you
> >> might find useful:
> >>
> >> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&C
> >>ISOPTR=134&CISOBOX=1&REC=1 (Hopefully, they will resolve a link
> >> problem at the above site soon, but you can read the abstract).
> >
> > Yes, I hope too, I would be happy to read the article.
> >
> > On the limit of unbiasdness, the following document mentions an
> > example (in a different context than variance estimation):
> >
> > http://www.stat.columbia.edu/~gelman/research/published/badbayesres
> >ponsemain.pdf
> >
> > AFAIK, even statisticians who consider themselves as "mostly
> > frequentist" (if that makes any sense) do not advocate unbiasdness
> > as such an important concept anymore (Larry Wasserman mentions it
> > in his "all of statistics").
> >
> > cheers,
> >
> > David
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989


From ggellner at uoguelph.ca  Wed Oct 15 12:09:03 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Wed, 15 Oct 2008 12:09:03 -0400
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <48F60213.2050204@enthought.com>
References: <20081015135626.GA12837@encolpuis> <48F60213.2050204@enthought.com>
Message-ID: <20081015160903.GA6032@encolpuis>

On Wed, Oct 15, 2008 at 09:45:39AM -0500, Travis E. Oliphant wrote:
> Gabriel Gellner wrote:
> > Some colleagues noticed that var uses biased formula's by default in numpy,
> > searching for the reason only brought up:
> >
> > http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
> >
> > which I totally agree with, but there was no response? Any reason for this?
> I will try to respond to this as it was me who made the change.  I think 
> there have been responses, but I think I've preferred to stay quiet 
> rather than feed a flame war.   Ultimately, it is a matter of preference 
> and I don't think there would be equal weights given to all the 
> arguments surrounding the decision by everybody.
> 
> I will attempt to articulate my reasons:  dividing by n is the maximum 
> likelihood estimator of variance and I prefer that justification more 
> than the "un-biased" justification for a default (especially given that 
> bias is just one part of the "error" in an estimator).    Having every 
> package that computes the mean return the "un-biased" estimate gives it 
> more cultural weight than than the concept deserves, I think.  Any 
> surprise that is created by the different default should be mitigated by 
> the fact that it's an opportunity to learn something about what you are 
> doing.    Here is a paper I wrote on the subject that you might find 
> useful:
> 
> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
> (Hopefully, they will resolve a link problem at the above site soon, but 
> you can read the abstract).
> 
Thanks for the reply, I look forward to reading the paper when it is
available. The major issue in my mind is not the technical issue but the
surprise factor. I can't think of single other package that uses this as the
default, and since it is also a method of ndarray (which is a built in type
and can't be monkey patched) there is no way of taking a different view (that
is supplying my on function) without the confusion I am feeling in my own lab
. . . (less technical people need to understand that they shouldn't
use a method of the same name) 

I worry about having numpy take this unpopular stance (as far as packages go)
simply to fight the good fight, as a built in method/behaviour of any ndarray,
rather than as a built in function, which presents no such problem, as it
allows dissent over a clearly muddy issue.

Sorry for the noise, and I am happy to see their is a reason, but I can't help
but find this a wort for purely pedagogical reasons. 

Gabriel


From bsouthey at gmail.com  Wed Oct 15 12:26:18 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 15 Oct 2008 11:26:18 -0500
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <48F60213.2050204@enthought.com>
References: <20081015135626.GA12837@encolpuis> <48F60213.2050204@enthought.com>
Message-ID: <48F619AA.3040800@gmail.com>

Hi,
While I disagree, I really do not care because this is documented. But 
perhaps a clear warning is need at the start so it clear what the 
default ddof means instead of it being buried in the Notes section.

Also I am surprised that you did not directly reference the Stein 
estimator (your minimum mean-squared estimator) and known effects in 
your paper:
http://en.wikipedia.org/wiki/James-Stein_estimator
So I did not find thiss any different from what is already known about 
the Stein estimator.

Bruce

PS While I may have gotten access via my University, I did get it from 
the link *Access this item. 
<https://contentdm.lib.byu.edu/cgi-bin/showfile.exe?CISOROOT=/EER&CISOPTR=134&filename=135.pdf>
https://contentdm.lib.byu.edu/cgi-bin/showfile.exe?CISOROOT=/EER&CISOPTR=134&filename=135.pdf
*
Travis E. Oliphant wrote:
> Gabriel Gellner wrote:
>   
>> Some colleagues noticed that var uses biased formula's by default in numpy,
>> searching for the reason only brought up:
>>
>> http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
>>
>> which I totally agree with, but there was no response? Any reason for this?
>>     
> I will try to respond to this as it was me who made the change.  I think 
> there have been responses, but I think I've preferred to stay quiet 
> rather than feed a flame war.   Ultimately, it is a matter of preference 
> and I don't think there would be equal weights given to all the 
> arguments surrounding the decision by everybody.
>
> I will attempt to articulate my reasons:  dividing by n is the maximum 
> likelihood estimator of variance and I prefer that justification more 
> than the "un-biased" justification for a default (especially given that 
> bias is just one part of the "error" in an estimator).    Having every 
> package that computes the mean return the "un-biased" estimate gives it 
> more cultural weight than than the concept deserves, I think.  Any 
> surprise that is created by the different default should be mitigated by 
> the fact that it's an opportunity to learn something about what you are 
> doing.    Here is a paper I wrote on the subject that you might find 
> useful:
>
> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
> (Hopefully, they will resolve a link problem at the above site soon, but 
> you can read the abstract).
>
> I'm not trying to persuade anybody with this email (although if you can 
> download the paper at the above link, then I am trying to persuade with 
> that).  In this email I'm just trying to give context to the poster as I 
> think the question is legitimate.
>
> With that said, there is the ddof parameter so that you can change what 
> the divisor is.  I think that is a useful compromise.
>
> I'm unhappy with the internal inconsistency of cov, as I think it was an 
> oversight. I'd be happy to see cov changed as well to use the ddof 
> argument instead of the bias keyword, but that is an API change and 
> requires some transition discussion and work.
>
> The only other argument I've heard against the current situation is 
> "unit testing" with MATLAB or R code.   Just use ddof=1 when comparing 
> against MATLAB and R code is my suggestion.
>
> Best regards,
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/c975e925/attachment.html>

From charlesr.harris at gmail.com  Wed Oct 15 13:09:55 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 11:09:55 -0600
Subject: [Numpy-discussion] var bias reason?
In-Reply-To: <5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>
References: <20081015135626.GA12837@encolpuis> <48F60213.2050204@enthought.com>
	<5b8d13220810150819s1115a7c0h1eff75b03dff86e3@mail.gmail.com>
Message-ID: <e06186140810151009g27bfc49es85f95562afa93d6e@mail.gmail.com>

On Wed, Oct 15, 2008 at 9:19 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Wed, Oct 15, 2008 at 11:45 PM, Travis E. Oliphant
> <oliphant at enthought.com> wrote:
> > Gabriel Gellner wrote:
> >> Some colleagues noticed that var uses biased formula's by default in
> numpy,
> >> searching for the reason only brought up:
> >>
> >>
> http://article.gmane.org/gmane.comp.python.numeric.general/12438/match=var+bias
> >>
> >> which I totally agree with, but there was no response? Any reason for
> this?
> > I will try to respond to this as it was me who made the change.  I think
> > there have been responses, but I think I've preferred to stay quiet
> > rather than feed a flame war.   Ultimately, it is a matter of preference
> > and I don't think there would be equal weights given to all the
> > arguments surrounding the decision by everybody.
> >
> > I will attempt to articulate my reasons:  dividing by n is the maximum
> > likelihood estimator of variance and I prefer that justification more
> > than the "un-biased" justification for a default (especially given that
> > bias is just one part of the "error" in an estimator).    Having every
> > package that computes the mean return the "un-biased" estimate gives it
> > more cultural weight than than the concept deserves, I think.  Any
> > surprise that is created by the different default should be mitigated by
> > the fact that it's an opportunity to learn something about what you are
> > doing.    Here is a paper I wrote on the subject that you might find
> > useful:
> >
> >
> https://contentdm.lib.byu.edu/cdm4/item_viewer.php?CISOROOT=/EER&CISOPTR=134&CISOBOX=1&REC=1
> > (Hopefully, they will resolve a link problem at the above site soon, but
> > you can read the abstract).
>
> Yes, I hope too, I would be happy to read the article.
>
> On the limit of unbiasdness, the following document mentions an
> example (in a different context than variance estimation):
>
>
> http://www.stat.columbia.edu/~gelman/research/published/badbayesresponsemain.pdf<http://www.stat.columbia.edu/%7Egelman/research/published/badbayesresponsemain.pdf>
>
> AFAIK, even statisticians who consider themselves as "mostly
> frequentist" (if that makes any sense) do not advocate unbiasdness as


Frequently frequentist?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/37854604/attachment.html>

From kbasye1 at jhu.edu  Wed Oct 15 12:52:56 2008
From: kbasye1 at jhu.edu (Ken Basye)
Date: Wed, 15 Oct 2008 12:52:56 -0400
Subject: [Numpy-discussion] Array printing differences between 64- and
	32-bit platforms
Message-ID: <48F61FE8.9030906@jhu.edu>

Hi Folks,
  In porting some code to a 64-bit machine, I ran across the following 
issue. 
On the 64-bit machine, an array with dtype=int32 prints the dtype 
explicitly, whereas on
a 32 bit machine it doesn't.  The same is true for dtype=intc (since 
'intc is int32' --> True),
and the converse is true for dtype=int64 and dtype=longlong.  Arrays 
with dtype of plain int
both print without the dtype, but then you're not using the same 
underlying size anymore.
  I think this is handled in core/numeric.py in array_repr and just 
above, where certain types
are added to _typelessdata if they are subclasses of 'int'.  The 
issubclass test returns different values
on platforms with different word lengths.
  This difference can make doctests fail and although there are doctest 
tricks to prevent the failure,
they're a bit annoying to use.  I was wondering about introducing a new 
print setting that forced
dtypes to be printed always.   Is there any support for that? 
  Any other ideas would also be most welcome.

    Thanks,
        Ken Basye


From sidky at uchicago.edu  Wed Oct 15 14:31:04 2008
From: sidky at uchicago.edu (emil)
Date: Wed, 15 Oct 2008 13:31:04 -0500
Subject: [Numpy-discussion] memory usage (Emil Sidky)
In-Reply-To: <mailman.850.1224081957.2878.numpy-discussion@scipy.org>
References: <mailman.850.1224081957.2878.numpy-discussion@scipy.org>
Message-ID: <48F636E8.4050001@uchicago.edu>


> Huang-Wen Chen wrote:
>> Robert Kern wrote:
>>>> from numpy import *
>>>> for i in range(1000):
>>>>   a = random.randn(512**2)
>>>>   b = a.argsort(kind='quick')
>>> Can you try upgrading to numpy 1.2.0? On my machine with numpy 1.2.0
>>> on OS X, the memory usage is stable.
>>>   
>> I tried the code fragment on two platforms and the memory usage is also 
>> normal.
>>
>> 1. numpy 1.1.1, python 2.5.1 on Vista 32bit
>> 2. numpy 1.2.0, python 2.6 on RedHat 64bit
> 
> If I recall correctly, there were some major improvements in python's 
> memory management/garbage collection from version 2.4 to 2.5. If you 
> could try to upgrade your python to 2.5 (and possibly also your numpy to 
> 1.2.0), you'd probably see some better behaviour.
> 
> Regards,
> Vincent.
> 

Problem fixed. Thanks.

But it turns out there were two things going on:
(1) Upgrading to numpy 1.2 (even with python 2.4) fixed the memory usage
for the loop with argsort in it.
(2) Unfortunately, when I went back to my original program and ran it
with the upgraded numpy, it still was chewing up tons of memory. I
finally found the problem:
Consider the following two code snippets (extension of my previous example).
from numpy import *
d = []
for i in range(1000):
   a = random.randn(512**2)
   b = a.argsort(kind= 'quick')
   c = b[-100:]
   d.append(c)

and

from numpy import *
d = []
for i in range(1000):
   a = random.randn(512**2)
   b = a.argsort(kind= 'quick')
   c = b[-100:].copy()
   d.append(c)

The difference being that c is a reference to the last 100 elements of b
in the first example, while c is a copy of the last 100 in the second
example.
Both examples yield identical results (provide randn is run with the
same seed value). But the former chews up tons of memory, and the latter
doesn't.
I don't know if this explanation makes any sense, but it is as if python
has to keep all the generated b's around in the first example because c
is only a reference.

Anyway, bottom line is that my problem is solved.
Thanks,
Emil


From robert.kern at gmail.com  Wed Oct 15 15:06:21 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 14:06:21 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
Message-ID: <3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>

On Wed, Oct 15, 2008 at 00:23, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Hi All,
>
> numpy.linalg has qr and cholesky factorizations, but LU factorization is
> only available in scipy. That doesn't seem quite right. I think is would
> make sense to include the LU factorization in numpy among the basic linalg
> operations, and probably LU_solve also. Thoughts?

-1. As far as I am concerned, numpy.linalg exists because Numeric had
LinearAlgebra, and we had to provide it to allow people to upgrade. I
do not want to see an expansion of functionality to maintain.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From perry at stsci.edu  Wed Oct 15 15:09:27 2008
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 15 Oct 2008 15:09:27 -0400
Subject: [Numpy-discussion] memory usage (Emil Sidky)
In-Reply-To: <48F636E8.4050001@uchicago.edu>
References: <mailman.850.1224081957.2878.numpy-discussion@scipy.org>
	<48F636E8.4050001@uchicago.edu>
Message-ID: <10DCE169-DC14-4019-8990-BBD392C939A4@stsci.edu>

When you slice an array, you keep the original array in memory until  
the slice is deleted. The slice uses the original array memory and is  
not a copy. The second example explicitly makes a copy.

Perry


On Oct 15, 2008, at 2:31 PM, emil wrote:

>
>> Huang-Wen Chen wrote:
>>> Robert Kern wrote:
>>>>> from numpy import *
>>>>> for i in range(1000):
>>>>>   a = random.randn(512**2)
>>>>>   b = a.argsort(kind='quick')
>>>> Can you try upgrading to numpy 1.2.0? On my machine with numpy  
>>>> 1.2.0
>>>> on OS X, the memory usage is stable.
>>>>
>>> I tried the code fragment on two platforms and the memory usage  
>>> is also
>>> normal.
>>>
>>> 1. numpy 1.1.1, python 2.5.1 on Vista 32bit
>>> 2. numpy 1.2.0, python 2.6 on RedHat 64bit
>>
>> If I recall correctly, there were some major improvements in python's
>> memory management/garbage collection from version 2.4 to 2.5. If you
>> could try to upgrade your python to 2.5 (and possibly also your  
>> numpy to
>> 1.2.0), you'd probably see some better behaviour.
>>
>> Regards,
>> Vincent.
>>
>
> Problem fixed. Thanks.
>
> But it turns out there were two things going on:
> (1) Upgrading to numpy 1.2 (even with python 2.4) fixed the memory  
> usage
> for the loop with argsort in it.
> (2) Unfortunately, when I went back to my original program and ran it
> with the upgraded numpy, it still was chewing up tons of memory. I
> finally found the problem:
> Consider the following two code snippets (extension of my previous  
> example).
> from numpy import *
> d = []
> for i in range(1000):
>    a = random.randn(512**2)
>    b = a.argsort(kind= 'quick')
>    c = b[-100:]
>    d.append(c)
>
> and
>
> from numpy import *
> d = []
> for i in range(1000):
>    a = random.randn(512**2)
>    b = a.argsort(kind= 'quick')
>    c = b[-100:].copy()
>    d.append(c)
>
> The difference being that c is a reference to the last 100 elements  
> of b
> in the first example, while c is a copy of the last 100 in the second
> example.
> Both examples yield identical results (provide randn is run with the
> same seed value). But the former chews up tons of memory, and the  
> latter
> doesn't.
> I don't know if this explanation makes any sense, but it is as if  
> python
> has to keep all the generated b's around in the first example  
> because c
> is only a reference.
>
> Anyway, bottom line is that my problem is solved.
> Thanks,
> Emil
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From stefan at sun.ac.za  Wed Oct 15 15:43:34 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 15 Oct 2008 21:43:34 +0200
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
Message-ID: <9457e7c80810151243u614b7211o35de5f29acade57e@mail.gmail.com>

2008/10/15 Robert Kern <robert.kern at gmail.com>:
>> numpy.linalg has qr and cholesky factorizations, but LU factorization is
>> only available in scipy. That doesn't seem quite right. I think is would
>> make sense to include the LU factorization in numpy among the basic linalg
>> operations, and probably LU_solve also. Thoughts?
>
> -1. As far as I am concerned, numpy.linalg exists because Numeric had
> LinearAlgebra, and we had to provide it to allow people to upgrade. I
> do not want to see an expansion of functionality to maintain.

It's silly to have a crippled linear algebra module in NumPy.  Either
take it out, or finish it.

NumPy without the linear algebra module would make it much less useful
to many of us.

Regards
Stefan


From charlesr.harris at gmail.com  Wed Oct 15 15:49:54 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 13:49:54 -0600
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
Message-ID: <e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>

On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > Hi All,
> >
> > numpy.linalg has qr and cholesky factorizations, but LU factorization is
> > only available in scipy. That doesn't seem quite right. I think is would
> > make sense to include the LU factorization in numpy among the basic
> linalg
> > operations, and probably LU_solve also. Thoughts?
>
> -1. As far as I am concerned, numpy.linalg exists because Numeric had
> LinearAlgebra, and we had to provide it to allow people to upgrade. I
> do not want to see an expansion of functionality to maintain.
>

I would be happier with that argument if scipy was broken into separately
downloadable modules and released on a regular schedule.  As is, I think
that exposing already existing (and used) functions in numpy lapack_lite is
unlikely to increase the maitainence burden and will add to the usefulness
of barebones numpy out of the box.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/5bb7bd03/attachment.html>

From robert.kern at gmail.com  Wed Oct 15 15:56:52 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 14:56:52 -0500
Subject: [Numpy-discussion] dtype comparison and hashing
In-Reply-To: <7f9d599f0810150020m489b54day4c909f6cbc8c1203@mail.gmail.com>
References: <7f9d599f0810150020m489b54day4c909f6cbc8c1203@mail.gmail.com>
Message-ID: <3d375d730810151256u6c135679l61e9483a14553938@mail.gmail.com>

On Wed, Oct 15, 2008 at 02:20, Geoffrey Irving <irving at naml.us> wrote:
> Hello,
>
> Currently in numpy comparing dtypes for equality with == does an
> internal PyArray_EquivTypes check, which means that the dtypes NPY_INT
> and NPY_LONG compare as equal in python.  However, the hash function
> for dtypes reduces id(), which is therefore inconsistent with ==.
> Unfortunately I can't produce a python snippet showing this since I
> don't know how to create a NPY_INT dtype in pure python.
>
> Based on the source it looks like hash should raise a type error,
> since tp_hash is null but tp_richcompare is not.  Does the following
> snippet through an exception for others?
>
>>>> import numpy
>>>> hash(numpy.dtype('int'))
> 5708736
>
> This might be the problem:
>
> /* Macro to get the tp_richcompare field of a type if defined */
> #define RICHCOMPARE(t) (PyType_HasFeature((t), Py_TPFLAGS_HAVE_RICHCOMPARE) \
>                         ? (t)->tp_richcompare : NULL)
>
> I'm using the default Mac OS X 10.5 installation of python 2.5 and
> numpy, so maybe those weren't compiled correctly.  Has anyone else
> seen this issue?

Actually, the problem is that we provide a hash function explicitly.
In multiarraymodule.c:

    PyArrayDescr_Type.tp_hash = (hashfunc)_Py_HashPointer;

That is a violation of the hashing protocol (objects which compare
equal and are hashable need to hash equal), and should be fixed.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Oct 15 16:03:36 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 15:03:36 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <9457e7c80810151243u614b7211o35de5f29acade57e@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<9457e7c80810151243u614b7211o35de5f29acade57e@mail.gmail.com>
Message-ID: <3d375d730810151303t5f19057cq4fcf32962841178d@mail.gmail.com>

On Wed, Oct 15, 2008 at 14:43, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/10/15 Robert Kern <robert.kern at gmail.com>:
>>> numpy.linalg has qr and cholesky factorizations, but LU factorization is
>>> only available in scipy. That doesn't seem quite right. I think is would
>>> make sense to include the LU factorization in numpy among the basic linalg
>>> operations, and probably LU_solve also. Thoughts?
>>
>> -1. As far as I am concerned, numpy.linalg exists because Numeric had
>> LinearAlgebra, and we had to provide it to allow people to upgrade. I
>> do not want to see an expansion of functionality to maintain.
>
> It's silly to have a crippled linear algebra module in NumPy.  Either
> take it out, or finish it.

And what exactly would constitute "finishing it"? Considering that it
has not had LU decompositions since the days it was LinearAlgebra
without anyone stepping up to add it, I hardly consider it crippled
without it.

It has a clear purpose, replacing LinearAlgebra so that people could
upgrade from Numeric. It's not crippled because it doesn't serve some
other purpose.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From robert.kern at gmail.com  Wed Oct 15 16:04:47 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 15:04:47 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
Message-ID: <3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>

On Wed, Oct 15, 2008 at 14:49, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> > Hi All,
>> >
>> > numpy.linalg has qr and cholesky factorizations, but LU factorization is
>> > only available in scipy. That doesn't seem quite right. I think is would
>> > make sense to include the LU factorization in numpy among the basic
>> > linalg
>> > operations, and probably LU_solve also. Thoughts?
>>
>> -1. As far as I am concerned, numpy.linalg exists because Numeric had
>> LinearAlgebra, and we had to provide it to allow people to upgrade. I
>> do not want to see an expansion of functionality to maintain.
>
> I would be happier with that argument if scipy was broken into separately
> downloadable modules and released on a regular schedule.

Then that is the deficiency that we should spend time on, not
duplicating the functionality again.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Oct 15 16:08:02 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 14:08:02 -0600
Subject: [Numpy-discussion] Array printing differences between 64- and
	32-bit platforms
In-Reply-To: <48F61FE8.9030906@jhu.edu>
References: <48F61FE8.9030906@jhu.edu>
Message-ID: <e06186140810151308n2b087e76ue5f6e9203d668a9c@mail.gmail.com>

On Wed, Oct 15, 2008 at 10:52 AM, Ken Basye <kbasye1 at jhu.edu> wrote:

> Hi Folks,
>  In porting some code to a 64-bit machine, I ran across the following
> issue.
> On the 64-bit machine, an array with dtype=int32 prints the dtype
> explicitly, whereas on
> a 32 bit machine it doesn't.  The same is true for dtype=intc (since
> 'intc is int32' --> True),
> and the converse is true for dtype=int64 and dtype=longlong.  Arrays
> with dtype of plain int
> both print without the dtype, but then you're not using the same
> underlying size anymore.
>  I think this is handled in core/numeric.py in array_repr and just
> above, where certain types
> are added to _typelessdata if they are subclasses of 'int'.  The
> issubclass test returns different values
> on platforms with different word lengths.
>  This difference can make doctests fail and although there are doctest
> tricks to prevent the failure,
> they're a bit annoying to use.  I was wondering about introducing a new
> print setting that forced
> dtypes to be printed always.   Is there any support for that?
>  Any other ideas would also be most welcome.
>

I'm inclined to say always print the type, but that is a behavior change and
might break some current code. I'm not sure how we should handle small
fixups like that.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/292c6c3f/attachment.html>

From charlesr.harris at gmail.com  Wed Oct 15 16:21:55 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 14:21:55 -0600
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
Message-ID: <e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>

On Wed, Oct 15, 2008 at 2:04 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Oct 15, 2008 at 14:49, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> > On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >> > Hi All,
> >> >
> >> > numpy.linalg has qr and cholesky factorizations, but LU factorization
> is
> >> > only available in scipy. That doesn't seem quite right. I think is
> would
> >> > make sense to include the LU factorization in numpy among the basic
> >> > linalg
> >> > operations, and probably LU_solve also. Thoughts?
> >>
> >> -1. As far as I am concerned, numpy.linalg exists because Numeric had
> >> LinearAlgebra, and we had to provide it to allow people to upgrade. I
> >> do not want to see an expansion of functionality to maintain.
> >
> > I would be happier with that argument if scipy was broken into separately
> > downloadable modules and released on a regular schedule.
>
> Then that is the deficiency that we should spend time on, not
> duplicating the functionality again.
>

Should we break out the linear algebra part of scipy and make it a separate
package? I suspect that would add to the build burden, because we would then
have a new package to maintain and release binaries for. I don't see the
problem with having a bit of linear algebra as part of the numpy base
package. My own feeling is that numpy isn't the bare core of array
functionality, rather it is the elementary or student version with enough
functionality to be useful while scipy adds  advanced features that
commercial packages would charge extra for. To some extent this is also a
matter of hierarchy, as numpy includes functions used by packages further up
the food chain.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/9ef2e59c/attachment.html>

From robert.kern at gmail.com  Wed Oct 15 16:26:59 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 15:26:59 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
Message-ID: <3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>

On Wed, Oct 15, 2008 at 15:21, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Wed, Oct 15, 2008 at 2:04 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Wed, Oct 15, 2008 at 14:49, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> > On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com>
>> > wrote:
>> >>
>> >> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> > Hi All,
>> >> >
>> >> > numpy.linalg has qr and cholesky factorizations, but LU factorization
>> >> > is
>> >> > only available in scipy. That doesn't seem quite right. I think is
>> >> > would
>> >> > make sense to include the LU factorization in numpy among the basic
>> >> > linalg
>> >> > operations, and probably LU_solve also. Thoughts?
>> >>
>> >> -1. As far as I am concerned, numpy.linalg exists because Numeric had
>> >> LinearAlgebra, and we had to provide it to allow people to upgrade. I
>> >> do not want to see an expansion of functionality to maintain.
>> >
>> > I would be happier with that argument if scipy was broken into
>> > separately
>> > downloadable modules and released on a regular schedule.
>>
>> Then that is the deficiency that we should spend time on, not
>> duplicating the functionality again.
>
> Should we break out the linear algebra part of scipy and make it a separate
> package? I suspect that would add to the build burden, because we would then
> have a new package to maintain and release binaries for. I don't see the
> problem with having a bit of linear algebra as part of the numpy base
> package.

Which bits? The current set has worked fine for more than 10 years.
Where do we stop? There will always be someone who wants just one more
function. And a case can always be made that adding just that function
is reasonable.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Oct 15 16:33:47 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 14:33:47 -0600
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
Message-ID: <e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>

On Wed, Oct 15, 2008 at 2:26 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Oct 15, 2008 at 15:21, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> > On Wed, Oct 15, 2008 at 2:04 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Wed, Oct 15, 2008 at 14:49, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >> >
> >> > On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
> >> >> <charlesr.harris at gmail.com> wrote:
> >> >> > Hi All,
> >> >> >
> >> >> > numpy.linalg has qr and cholesky factorizations, but LU
> factorization
> >> >> > is
> >> >> > only available in scipy. That doesn't seem quite right. I think is
> >> >> > would
> >> >> > make sense to include the LU factorization in numpy among the basic
> >> >> > linalg
> >> >> > operations, and probably LU_solve also. Thoughts?
> >> >>
> >> >> -1. As far as I am concerned, numpy.linalg exists because Numeric had
> >> >> LinearAlgebra, and we had to provide it to allow people to upgrade. I
> >> >> do not want to see an expansion of functionality to maintain.
> >> >
> >> > I would be happier with that argument if scipy was broken into
> >> > separately
> >> > downloadable modules and released on a regular schedule.
> >>
> >> Then that is the deficiency that we should spend time on, not
> >> duplicating the functionality again.
> >
> > Should we break out the linear algebra part of scipy and make it a
> separate
> > package? I suspect that would add to the build burden, because we would
> then
> > have a new package to maintain and release binaries for. I don't see the
> > problem with having a bit of linear algebra as part of the numpy base
> > package.
>
> Which bits? The current set has worked fine for more than 10 years.
> Where do we stop? There will always be someone who wants just one more
> function. And a case can always be made that adding just that function
> is reasonable.
>

I would just add the bits that are already there and don't add any extra
dependencies, i.e., they are there when numpy is built without ATLAS or
other external packages. The determinant function in linalg uses the LU
decomposition, so I don't see why that shouldn't be available to the general
user.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/16b1a4fe/attachment.html>

From aisaac at american.edu  Wed Oct 15 16:38:36 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 15 Oct 2008 16:38:36 -0400
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
Message-ID: <48F654CC.9050501@american.edu>

On 10/15/2008 4:26 PM Robert Kern apparently wrote:
> Which bits?

Those in lapack_lite?
Alan Isaac



From robert.kern at gmail.com  Wed Oct 15 16:49:31 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 15 Oct 2008 15:49:31 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
	<e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
Message-ID: <3d375d730810151349p50d19c80v7dc8677315ae522b@mail.gmail.com>

On Wed, Oct 15, 2008 at 15:33, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Wed, Oct 15, 2008 at 2:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Wed, Oct 15, 2008 at 15:21, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> > On Wed, Oct 15, 2008 at 2:04 PM, Robert Kern <robert.kern at gmail.com>
>> > wrote:
>> >>
>> >> On Wed, Oct 15, 2008 at 14:49, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> >
>> >> > On Wed, Oct 15, 2008 at 1:06 PM, Robert Kern <robert.kern at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> On Wed, Oct 15, 2008 at 00:23, Charles R Harris
>> >> >> <charlesr.harris at gmail.com> wrote:
>> >> >> > Hi All,
>> >> >> >
>> >> >> > numpy.linalg has qr and cholesky factorizations, but LU
>> >> >> > factorization
>> >> >> > is
>> >> >> > only available in scipy. That doesn't seem quite right. I think is
>> >> >> > would
>> >> >> > make sense to include the LU factorization in numpy among the
>> >> >> > basic
>> >> >> > linalg
>> >> >> > operations, and probably LU_solve also. Thoughts?
>> >> >>
>> >> >> -1. As far as I am concerned, numpy.linalg exists because Numeric
>> >> >> had
>> >> >> LinearAlgebra, and we had to provide it to allow people to upgrade.
>> >> >> I
>> >> >> do not want to see an expansion of functionality to maintain.
>> >> >
>> >> > I would be happier with that argument if scipy was broken into
>> >> > separately
>> >> > downloadable modules and released on a regular schedule.
>> >>
>> >> Then that is the deficiency that we should spend time on, not
>> >> duplicating the functionality again.
>> >
>> > Should we break out the linear algebra part of scipy and make it a
>> > separate
>> > package? I suspect that would add to the build burden, because we would
>> > then
>> > have a new package to maintain and release binaries for. I don't see the
>> > problem with having a bit of linear algebra as part of the numpy base
>> > package.
>>
>> Which bits? The current set has worked fine for more than 10 years.
>> Where do we stop? There will always be someone who wants just one more
>> function. And a case can always be made that adding just that function
>> is reasonable.
>
> I would just add the bits that are already there and don't add any extra
> dependencies, i.e., they are there when numpy is built without ATLAS or
> other external packages. The determinant function in linalg uses the LU
> decomposition, so I don't see why that shouldn't be available to the general
> user.

I'm softening to this argument. But mostly because it will be you who
will have to defend this arbitrary line in the sand in the future
rather than me.  :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Wed Oct 15 16:50:49 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 15 Oct 2008 22:50:49 +0200
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
Message-ID: <9457e7c80810151350x7e8e1dd7nc05796b6733a4aa6@mail.gmail.com>

2008/10/15 Robert Kern <robert.kern at gmail.com>:
> Which bits? The current set has worked fine for more than 10 years.

I'm surprised no-one has requested the LU decomposition in NumPy
before -- it is a fundamental building block in linear algebra.  I
think it is going too far, stating that NumPy's linear algebra module
serves simply as an upgrade path for those coming from Numeric.  Its
use has developed far beyond that.

> Where do we stop? There will always be someone who wants just one more
> function. And a case can always be made that adding just that function
> is reasonable.

I'd rather we examine each function on a case-by-case basis, than to
put a solid freeze on NumPy.

Regards
St?fan


From oliphant at enthought.com  Wed Oct 15 17:22:25 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 15 Oct 2008 16:22:25 -0500
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
	<e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
Message-ID: <48F65F11.5050506@enthought.com>

Charles R Harris wrote:
>
>
> I would just add the bits that are already there and don't add any 
> extra dependencies, i.e., they are there when numpy is built without 
> ATLAS or other external packages. The determinant function in linalg 
> uses the LU decomposition, so I don't see why that shouldn't be 
> available to the general user.
If LU is already part of lapack_lite and somebody is willing to put in 
the work to expose the functionality to the end user in a reasonable 
way, then I think it should be added.

-Travis



From ellisonbg.net at gmail.com  Wed Oct 15 22:58:34 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 15 Oct 2008 19:58:34 -0700
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <48F65F11.5050506@enthought.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
	<e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
	<48F65F11.5050506@enthought.com>
Message-ID: <6ce0ac130810151958q200e555cl1d6ece2f4921ffff@mail.gmail.com>

> If LU is already part of lapack_lite and somebody is willing to put in
> the work to expose the functionality to the end user in a reasonable
> way, then I think it should be added.

+1


From charlesr.harris at gmail.com  Thu Oct 16 00:41:28 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 22:41:28 -0600
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <6ce0ac130810151958q200e555cl1d6ece2f4921ffff@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
	<e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
	<48F65F11.5050506@enthought.com>
	<6ce0ac130810151958q200e555cl1d6ece2f4921ffff@mail.gmail.com>
Message-ID: <e06186140810152141i564762aam79f8dc2e1d3a8668@mail.gmail.com>

OK, I take this as a go ahead with the proviso that it's my problem. The big
question is naming. Scipy has

lu
lu_factor
lu_solve

cholesky
cho_factor
cho_solve

The code for lu and lu_factor isn't the same, although they both look to
call the same underlying function; the same is true of the cholesky code. I
also see various functions with the same names as their numpy counterparts.
So my inclination would be to use lu and lu_solve. Likewise, maybe add
cho_solve to compliment cholesky. I don't have strong preferences one way or
the other. Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/b886dc17/attachment.html>

From charlesr.harris at gmail.com  Thu Oct 16 01:59:40 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 15 Oct 2008 23:59:40 -0600
Subject: [Numpy-discussion] LU factorization?
In-Reply-To: <e06186140810152141i564762aam79f8dc2e1d3a8668@mail.gmail.com>
References: <e06186140810142223t7d6cddaewfeb8d7f21a6be3eb@mail.gmail.com>
	<3d375d730810151206j702ea914tea5105c7c286a158@mail.gmail.com>
	<e06186140810151249r69feb6a1m3a8e640b2e712a2e@mail.gmail.com>
	<3d375d730810151304j42560063mee994bf4f21cea4c@mail.gmail.com>
	<e06186140810151321o2805fed3rf9b4f1b0e233100b@mail.gmail.com>
	<3d375d730810151326r2ef3aabas4089a541cd32eb9@mail.gmail.com>
	<e06186140810151333v45959d8as8c9976373a8c8fb@mail.gmail.com>
	<48F65F11.5050506@enthought.com>
	<6ce0ac130810151958q200e555cl1d6ece2f4921ffff@mail.gmail.com>
	<e06186140810152141i564762aam79f8dc2e1d3a8668@mail.gmail.com>
Message-ID: <e06186140810152259s3e4afbf8w65eb552b653b0752@mail.gmail.com>

On Wed, Oct 15, 2008 at 10:41 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

> OK, I take this as a go ahead with the proviso that it's my problem. The
> big question is naming. Scipy has
>
> lu
> lu_factor
> lu_solve
>
> cholesky
>
> cho_factor
> cho_solve
>
> The code for lu and lu_factor isn't the same, although they both look to
> call the same underlying function; the same is true of the cholesky code. I
> also see various functions with the same names as their numpy counterparts.
> So my inclination would be to use lu and lu_solve. Likewise, maybe add
> cho_solve to compliment cholesky. I don't have strong preferences one way or
> the other. Thoughts?
>

Looks like dpotrs isn't in lapack_lite (cho_solve), but dgetrs (lu_solve)
is, although currently missing an interface to python.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081015/46fa358c/attachment.html>

From animator333 at yahoo.com  Thu Oct 16 07:12:06 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Thu, 16 Oct 2008 16:42:06 +0530 (IST)
Subject: [Numpy-discussion] numpy fileIO
Message-ID: <365066.54989.qm@web94913.mail.in2.yahoo.com>

Hi,

I have never used numpy in my python applications until now. I am writing a python/openGL based tool to manipulate 3d geometrical data(meshes, curve, points etc.) I would be using numpy to store these objects(vertices, edges, faces, index etc.) at run time. One of my major concern is numpy's file IO capabilities. Geometrical objects would be stored in a structures, for example a logical structure to store a mesh goes like this:

struct(
name(string)
vertex(numpy.float16)[x, y, z]
normal(numpy.float16)[x, y, z]
edges(numpy.int)
st(2d numpy.float16)[u, v]
index(numpy.int)
)

There would be different structures for curve, patch, points and rest of the primitives. I am sure numpy users must have encounered similar scenerio where
you need to dump this data to a file and read it back. In my case, a general assumption of nearly 50-150 megs of data would be considered as normal size.
Before I go deep into coding It would be great if numpy user can share their expreience for the task.

I am also open for unconventional or off the route methods, provided they can do the job.(C/c++, 3rd party modules etc.)
Here is the summery of IO operations I would be working on:

1. Write different structures to a file.
2. Read data back from file.
3. if structure can be tagged(int or string) then read a particular structure using tag, from file.

Hope to here from numpy users soon :-)

Regards

Prashant



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/90eb5651/attachment.html>

From nicolas.roux at st.com  Thu Oct 16 08:52:41 2008
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Thu, 16 Oct 2008 14:52:41 +0200
Subject: [Numpy-discussion] Missing numpy.i
In-Reply-To: <001501c92f8a$8261a740$04000100@gnb.st.com>
Message-ID: <001a01c92f8e$147ede10$04000100@gnb.st.com>



Hello,

Under windows, using the latest numpy 1.20 and Swig release, I can't find anymore the numpy.i file.
I used to find it in numpy/doc/ in the release 1.04.

I tried to look some info the web without any success.

Thanks for you help,


Cheers,
Nicolas.





From stefan at sun.ac.za  Thu Oct 16 09:06:18 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 16 Oct 2008 15:06:18 +0200
Subject: [Numpy-discussion] Missing numpy.i
In-Reply-To: <001a01c92f8e$147ede10$04000100@gnb.st.com>
References: <001501c92f8a$8261a740$04000100@gnb.st.com>
	<001a01c92f8e$147ede10$04000100@gnb.st.com>
Message-ID: <9457e7c80810160606k4f35a5dmd06a39b096f0c138@mail.gmail.com>

2008/10/16 Nicolas ROUX <nicolas.roux at st.com>:
> Under windows, using the latest numpy 1.20 and Swig release, I can't find anymore the numpy.i file.
> I used to find it in numpy/doc/ in the release 1.04.

The docs dir has moved:

http://projects.scipy.org/scipy/numpy/browser/trunk/doc/swig

Cheers
Stefan


From nicolas.roux at st.com  Thu Oct 16 09:10:08 2008
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Thu, 16 Oct 2008 15:10:08 +0200
Subject: [Numpy-discussion] Missing numpy.i
In-Reply-To: <9457e7c80810160606k4f35a5dmd06a39b096f0c138@mail.gmail.com>
Message-ID: <001b01c92f90$83c9db10$04000100@gnb.st.com>

Thanks for your reply ;-)

In fact, I was talking about the 1.20 release installer, which is not
including the numpy.i any more.
Is this numpy.i not any more part of the binary delivery/installer ?
Should I take it manually/directly from the SVN repository, instead of
beeing installed automatically  ?


Cheers,
Nicolas. 



-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Stefan van der Walt
Sent: Thursday, October 16, 2008 3:06 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Missing numpy.i

2008/10/16 Nicolas ROUX <nicolas.roux at st.com>:
> Under windows, using the latest numpy 1.20 and Swig release, I can't find
anymore the numpy.i file.
> I used to find it in numpy/doc/ in the release 1.04.

The docs dir has moved:

http://projects.scipy.org/scipy/numpy/browser/trunk/doc/swig

Cheers
Stefan
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From stefan at sun.ac.za  Thu Oct 16 09:20:14 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 16 Oct 2008 15:20:14 +0200
Subject: [Numpy-discussion] Missing numpy.i
In-Reply-To: <001b01c92f90$83c9db10$04000100@gnb.st.com>
References: <9457e7c80810160606k4f35a5dmd06a39b096f0c138@mail.gmail.com>
	<001b01c92f90$83c9db10$04000100@gnb.st.com>
Message-ID: <9457e7c80810160620i2aeec4e3o4df1ae82906a1490@mail.gmail.com>

2008/10/16 Nicolas ROUX <nicolas.roux at st.com>:
> Thanks for your reply ;-)
>
> In fact, I was talking about the 1.20 release installer, which is not
> including the numpy.i any more.

This may have been an oversight.  The docs directory moved out of the
source tree, so it needs to be added to the installer separately.
David, could we install the docs dir as well?

You should be able to use the numpy.i from the URL I provided.

Cheers
Stefan


From nadavh at visionsense.com  Thu Oct 16 10:43:38 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 16 Oct 2008 16:43:38 +0200
Subject: [Numpy-discussion] numpy fileIO
References: <365066.54989.qm@web94913.mail.in2.yahoo.com>
Message-ID: <710F2847B0018641891D9A216027636029C2D6@ex3.envision.co.il>

Did you consider VTK?
I've used it a *little*: Probably it contains all the structures you need, along with c++ routines for I/O, manipulation and
(OpenGL) display, and a python interface.

  Nadav.

-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Prashant Saxena
????: ? 16-???????-08 13:12
??: numpy-discussion at scipy.org
????: [Numpy-discussion] numpy fileIO
 
Hi,

I have never used numpy in my python applications until now. I am writing a python/openGL based tool to manipulate 3d geometrical data(meshes, curve, points etc.) I would be using numpy to store these objects(vertices, edges, faces, index etc.) at run time. One of my major concern is numpy's file IO capabilities. Geometrical objects would be stored in a structures, for example a logical structure to store a mesh goes like this:

struct(
name(string)
vertex(numpy.float16)[x, y, z]
normal(numpy.float16)[x, y, z]
edges(numpy.int)
st(2d numpy.float16)[u, v]
index(numpy.int)
)

There would be different structures for curve, patch, points and rest of the primitives. I am sure numpy users must have encounered similar scenerio where
you need to dump this data to a file and read it back. In my case, a general assumption of nearly 50-150 megs of data would be considered as normal size.
Before I go deep into coding It would be great if numpy user can share their expreience for the task.

I am also open for unconventional or off the route methods, provided they can do the job.(C/c++, 3rd party modules etc.)
Here is the summery of IO operations I would be working on:

1. Write different structures to a file.
2. Read data back from file.
3. if structure can be tagged(int or string) then read a particular structure using tag, from file.

Hope to here from numpy users soon :-)

Regards

Prashant



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3716 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/d8022d9c/attachment.bin>

From animator333 at yahoo.com  Thu Oct 16 11:25:46 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Thu, 16 Oct 2008 20:55:46 +0530 (IST)
Subject: [Numpy-discussion] numpy fileIO
Message-ID: <321698.44581.qm@web94907.mail.in2.yahoo.com>

I am seeing all the OSS for this purpose but I would stick to use pure python and the scene graph I am developing for the application. I already did some test using pyOpenGL/python/wx.GLcanvas and a large data set of roughly 4000+ objects consisting nearly 1 million polygons using display lists and results were more then satisfactory. Although it was done quickly and preformence can be improved further by code optimizations. numpy can play a crucial role
in my application and I have to make sure before pinning down any decisions regarding tools and techniques I would be using.   

Prashant



----- Original Message ----
From: Nadav Horesh <nadavh at visionsense.com>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Sent: Thursday, 16 October, 2008 8:13:38 PM
Subject: Re: [Numpy-discussion] numpy fileIO

Did you consider VTK?
I've used it a *little*: Probably it contains all the structures you need, along with c++ routines for I/O, manipulation and
(OpenGL) display, and a python interface.

  Nadav.

-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Prashant Saxena
????: ? 16-???????-08 13:12
??: numpy-discussion at scipy.org
????: [Numpy-discussion] numpy fileIO

Hi,

I have never used numpy in my python applications until now. I am writing a python/openGL based tool to manipulate 3d geometrical data(meshes, curve, points etc.) I would be using numpy to store these objects(vertices, edges, faces, index etc.) at run time. One of my major concern is numpy's file IO capabilities. Geometrical objects would be stored in a structures, for example a logical structure to store a mesh goes like this:

struct(
name(string)
vertex(numpy.float16)[x, y, z]
normal(numpy.float16)[x, y, z]
edges(numpy.int)
st(2d numpy.float16)[u, v]
index(numpy.int)
)

There would be different structures for curve, patch, points and rest of the primitives. I am sure numpy users must have encounered similar scenerio where
you need to dump this data to a file and read it back. In my case, a general assumption of nearly 50-150 megs of data would be considered as normal size.
Before I go deep into coding It would be great if numpy user can share their expreience for the task.

I am also open for unconventional or off the route methods, provided they can do the job.(C/c++, 3rd party modules etc.)
Here is the summery of IO operations I would be working on:

1. Write different structures to a file.
2. Read data back from file.
3. if structure can be tagged(int or string) then read a particular structure using tag, from file.

Hope to here from numpy users soon :-)

Regards

Prashant



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/


      Unlimited freedom, unlimited storage. Get it now, on http://help.yahoo.com/l/in/yahoo/mail/yahoomail/tools/tools-08.html/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/a80eb1a8/attachment.html>

From lists_ravi at lavabit.com  Thu Oct 16 12:25:55 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Thu, 16 Oct 2008 12:25:55 -0400
Subject: [Numpy-discussion] numpy fileIO
In-Reply-To: <365066.54989.qm@web94913.mail.in2.yahoo.com>
References: <365066.54989.qm@web94913.mail.in2.yahoo.com>
Message-ID: <200810161225.55969.lists_ravi@lavabit.com>

On Thursday 16 October 2008 07:12:06 Prashant Saxena wrote:
> Here is the summery of IO operations I would be working on:
>
> 1. Write different structures to a file.
> 2. Read data back from file.
> 3. if structure can be tagged(int or string) then read a particular
> structure using tag, from file.

Use HDF5. Can be read/written using C, C++, Python, Matlab, Octave, Fortran 
and practically any other language you can think of. Python interface is at 
http://h5py.alfven.org and is also hosted on Google code. Supports tags, 
properties, etc. This is the format used by NASA for storing their images and 
will likely be supported for decades to come.

Regards,
Ravi




From Chris.Barker at noaa.gov  Thu Oct 16 12:33:43 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 16 Oct 2008 09:33:43 -0700
Subject: [Numpy-discussion] numpy fileIO
In-Reply-To: <200810161225.55969.lists_ravi@lavabit.com>
References: <365066.54989.qm@web94913.mail.in2.yahoo.com>
	<200810161225.55969.lists_ravi@lavabit.com>
Message-ID: <48F76CE7.9090508@noaa.gov>

Ravi wrote:
> Use HDF5. Can be read/written using C, C++, Python, Matlab, Octave, Fortran 
> and practically any other language you can think of. Python interface is at 
> http://h5py.alfven.org

there is also pytables, which providers a nice numpy/HDF interface.

http://www.pytables.org/

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From faltet at pytables.org  Thu Oct 16 12:33:22 2008
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 16 Oct 2008 18:33:22 +0200
Subject: [Numpy-discussion] numpy fileIO
In-Reply-To: <365066.54989.qm@web94913.mail.in2.yahoo.com>
References: <365066.54989.qm@web94913.mail.in2.yahoo.com>
Message-ID: <200810161833.22552.faltet@pytables.org>

A Thursday 16 October 2008, Prashant Saxena escrigu?:
> Hi,
>
> I have never used numpy in my python applications until now. I am
> writing a python/openGL based tool to manipulate 3d geometrical
> data(meshes, curve, points etc.) I would be using numpy to store
> these objects(vertices, edges, faces, index etc.) at run time. One of
> my major concern is numpy's file IO capabilities. Geometrical objects
> would be stored in a structures, for example a logical structure to
> store a mesh goes like this:
>
> struct(
> name(string)
> vertex(numpy.float16)[x, y, z]
> normal(numpy.float16)[x, y, z]
> edges(numpy.int)
> st(2d numpy.float16)[u, v]
> index(numpy.int)
> )
>
> There would be different structures for curve, patch, points and rest
> of the primitives. I am sure numpy users must have encounered similar
> scenerio where you need to dump this data to a file and read it back.
> In my case, a general assumption of nearly 50-150 megs of data would
> be considered as normal size. Before I go deep into coding It would
> be great if numpy user can share their expreience for the task.
>
> I am also open for unconventional or off the route methods, provided
> they can do the job.(C/c++, 3rd party modules etc.) Here is the
> summery of IO operations I would be working on:
>
> 1. Write different structures to a file.
> 2. Read data back from file.
> 3. if structure can be tagged(int or string) then read a particular
> structure using tag, from file.

Your structure seems like can be expressed through a record array [1].  
If that is the case, just try the included tofile()/fromfile() 
utilities included in numpy for saving/retrieving them.

If you need more functionality for recarray I/O, you can have a look at 
pytables [2], which allows you to access directly the recarrays on disk 
row by row (even if compressed) or use complex expressions to quickly 
select interesting parts, among other niceties.

If your data structures can't be fit in recarray structures directly, 
you may want to try to decompose them in smaller constructs and relate 
them in file through pointers (indices to other constructs).

[1] http://www.scipy.org/RecordArrays
[2] http://www.pytables.org

Hope that helps,

-- 
Francesc Alted


From cournape at gmail.com  Thu Oct 16 13:04:33 2008
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 17 Oct 2008 02:04:33 +0900
Subject: [Numpy-discussion] Missing numpy.i
In-Reply-To: <9457e7c80810160620i2aeec4e3o4df1ae82906a1490@mail.gmail.com>
References: <9457e7c80810160606k4f35a5dmd06a39b096f0c138@mail.gmail.com>
	<001b01c92f90$83c9db10$04000100@gnb.st.com>
	<9457e7c80810160620i2aeec4e3o4df1ae82906a1490@mail.gmail.com>
Message-ID: <5b8d13220810161004k60b43c0fyec9ed0462b9b0386@mail.gmail.com>

On Thu, Oct 16, 2008 at 10:20 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:

> This may have been an oversight.  The docs directory moved out of the
> source tree, so it needs to be added to the installer separately.
> David, could we install the docs dir as well?

Yes, as long as it is handled by distutils.

David

From animator333 at yahoo.com  Thu Oct 16 13:05:16 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Thu, 16 Oct 2008 22:35:16 +0530 (IST)
Subject: [Numpy-discussion] numpy fileIO
Message-ID: <915696.58152.qm@web94911.mail.in2.yahoo.com>

I have just started doing my tests using numpy and hdpy( http://h5py.alfven..org/ ).
Hope either of these would give me what I need.

Prashant



----- Original Message ----
From: Francesc Alted <faltet at pytables.org>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
Sent: Thursday, 16 October, 2008 10:03:22 PM
Subject: Re: [Numpy-discussion] numpy fileIO

A Thursday 16 October 2008, Prashant Saxena escrigu?:
> Hi,
>
> I have never used numpy in my python applications until now. I am
> writing a python/openGL based tool to manipulate 3d geometrical
> data(meshes, curve, points etc.) I would be using numpy to store
> these objects(vertices, edges, faces, index etc.) at run time. One of
> my major concern is numpy's file IO capabilities. Geometrical objects
> would be stored in a structures, for example a logical structure to
> store a mesh goes like this:
>
> struct(
> name(string)
> vertex(numpy.float16)[x, y, z]
> normal(numpy.float16)[x, y, z]
> edges(numpy.int)
> st(2d numpy.float16)[u, v]
> index(numpy.int)
> )
>
> There would be different structures for curve, patch, points and rest
> of the primitives. I am sure numpy users must have encounered similar
> scenerio where you need to dump this data to a file and read it back.
> In my case, a general assumption of nearly 50-150 megs of data would
> be considered as normal size. Before I go deep into coding It would
> be great if numpy user can share their expreience for the task.
>
> I am also open for unconventional or off the route methods, provided
> they can do the job.(C/c++, 3rd party modules etc.) Here is the
> summery of IO operations I would be working on:
>
> 1. Write different structures to a file.
> 2. Read data back from file.
> 3. if structure can be tagged(int or string) then read a particular
> structure using tag, from file.

Your structure seems like can be expressed through a record array [1].  
If that is the case, just try the included tofile()/fromfile() 
utilities included in numpy for saving/retrieving them.

If you need more functionality for recarray I/O, you can have a look at 
pytables [2], which allows you to access directly the recarrays on disk 
row by row (even if compressed) or use complex expressions to quickly 
select interesting parts, among other niceties.

If your data structures can't be fit in recarray structures directly, 
you may want to try to decompose them in smaller constructs and relate 
them in file through pointers (indices to other constructs).

[1] http://www.scipy.org/RecordArrays
[2] http://www.pytables.org

Hope that helps,

-- 
Francesc Alted
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/74cd7e91/attachment.html>

From hetland at tamu.edu  Thu Oct 16 14:25:41 2008
From: hetland at tamu.edu (Rob Hetland)
Date: Thu, 16 Oct 2008 13:25:41 -0500
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
Message-ID: <76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>


On Oct 14, 2008, at 12:56 AM, St?fan van der Walt wrote:

> Here is an implementation in Python, ctypes and in weave:
>
> http://mentat.za.net/source/pnpoly.tar.bz2
>
> Regards
> St?fan

This question gets asked about once a month on the mailing list.  
Perhaps pnpoly could find a permanent home in scipy?  (or somewhere?)   
Obviously, many would find it useful.

-Rob


----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331





From jdh2358 at gmail.com  Thu Oct 16 14:54:48 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 16 Oct 2008 13:54:48 -0500
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
	<76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
Message-ID: <88e473830810161154n23bce8fax8a881c86fa24c416@mail.gmail.com>

On Thu, Oct 16, 2008 at 1:25 PM, Rob Hetland <hetland at tamu.edu> wrote:
> This question gets asked about once a month on the mailing list.
> Perhaps pnpoly could find a permanent home in scipy?  (or somewhere?)
> Obviously, many would find it useful.

It is already in matplotlib

In [1]: import matplotlib.nxutils as nx

In [2]: nx.pnpoly
Out[2]: <built-in function pnpoly>

In [3]: nx.points_inside_poly
Out[3]: <built-in function points_inside_poly>

but one of should add it to the FAQ!


From berthe.loic at gmail.com  Thu Oct 16 15:22:13 2008
From: berthe.loic at gmail.com (LB)
Date: Thu, 16 Oct 2008 12:22:13 -0700 (PDT)
Subject: [Numpy-discussion] how to vectorize a row of eigenvalue
	computations ?
Message-ID: <d86b77cf-f436-4904-a0a1-94f541cb6d2f@l62g2000hse.googlegroups.com>

I've got an array S of shape (N, 6) with N >> 100000 containing the
six components of a stress field given in N points.

I need to make a lot of computation of principal stresses, which are
in fact the eigenvalues of the stress tensors.

I'm using the basic code described below :

import numpy as np

def calc_principal_stresses(S):
    """ Return the principal stress corresponding to the tensor S.
    S is interpreted as an array containing [Sx, Sy, Sz, Sxy, Syz,
Szx]

    Return array([S3, S2, S1])
    """
    p_stresses = np.linalg.eigvalsh( np.array(
	[  [ S[0], S[3], S[5]],
           [ S[3], S[1], S[4]],
           [ S[5], S[4], S[2]],
	]))

    return p_stresses.sort()

p_stresses = array([ calc_principal_stresses(s) for s in S])

Aside putting the sort function outside the loop, is there any way to
optimize or vectorize this kind of operation ?

  Regards,

--
LB


From hetland at tamu.edu  Thu Oct 16 15:28:05 2008
From: hetland at tamu.edu (Rob Hetland)
Date: Thu, 16 Oct 2008 14:28:05 -0500
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <88e473830810161154n23bce8fax8a881c86fa24c416@mail.gmail.com>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
	<76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
	<88e473830810161154n23bce8fax8a881c86fa24c416@mail.gmail.com>
Message-ID: <E31723B7-E636-4ED5-AE0F-A82FA3D6E8FB@tamu.edu>


On Oct 16, 2008, at 1:54 PM, John Hunter wrote:

> It is already in matplotlib
>
> In [1]: import matplotlib.nxutils as nx
>
> In [2]: nx.pnpoly
> Out[2]: <built-in function pnpoly>
>
> In [3]: nx.points_inside_poly
> Out[3]: <built-in function points_inside_poly>
>
> but one of should add it to the FAQ!


I did not know that very useful thing.  But now I do.  This is solid  
proof that lurking on the mailing lists makes you smarter.

Thanks!

-Rob

----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331





From gruben at bigpond.net.au  Thu Oct 16 18:06:25 2008
From: gruben at bigpond.net.au (Gary Ruben)
Date: Fri, 17 Oct 2008 09:06:25 +1100
Subject: [Numpy-discussion] numpy fileIO
In-Reply-To: <321698.44581.qm@web94907.mail.in2.yahoo.com>
References: <321698.44581.qm@web94907.mail.in2.yahoo.com>
Message-ID: <48F7BAE1.8060905@bigpond.net.au>

Prashant Saxena wrote:
> I am seeing all the OSS for this purpose but I would stick to use pure 
> python and the scene graph I am developing for the application. I 
> already did some test using pyOpenGL/python/wx.GLcanvas and a large data 
> set of roughly 4000+ objects consisting nearly 1 million polygons using 
> display lists and results were more then satisfactory. Although it was 
> done quickly and preformence can be improved further by code 
> optimizations. numpy can play a crucial role
> in my application and I have to make sure before pinning down any 
> decisions regarding tools and techniques I would be using.  
> 
> Prashant

Are you also aware of the OSG wrapper efforts for Python, which are here?:
<http://code.astraw.com/projects/pyosg/>
See also the links at the bottom of that page.

Gary R.


From charlesr.harris at gmail.com  Thu Oct 16 22:54:44 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 16 Oct 2008 20:54:44 -0600
Subject: [Numpy-discussion] how to vectorize a row of eigenvalue
	computations ?
In-Reply-To: <d86b77cf-f436-4904-a0a1-94f541cb6d2f@l62g2000hse.googlegroups.com>
References: <d86b77cf-f436-4904-a0a1-94f541cb6d2f@l62g2000hse.googlegroups.com>
Message-ID: <e06186140810161954w16f6e456lea421fd13c557471@mail.gmail.com>

On Thu, Oct 16, 2008 at 1:22 PM, LB <berthe.loic at gmail.com> wrote:

> I've got an array S of shape (N, 6) with N >> 100000 containing the
> six components of a stress field given in N points.
>
> I need to make a lot of computation of principal stresses, which are
> in fact the eigenvalues of the stress tensors.
>
> I'm using the basic code described below :
>
> import numpy as np
>
> def calc_principal_stresses(S):
>    """ Return the principal stress corresponding to the tensor S.
>    S is interpreted as an array containing [Sx, Sy, Sz, Sxy, Syz,
> Szx]
>
>    Return array([S3, S2, S1])
>    """
>    p_stresses = np.linalg.eigvalsh( np.array(
>        [  [ S[0], S[3], S[5]],
>           [ S[3], S[1], S[4]],
>           [ S[5], S[4], S[2]],
>        ]))
>
>    return p_stresses.sort()
>
> p_stresses = array([ calc_principal_stresses(s) for s in S])
>

Something like this?

In [1]: m = np.zeros((2,6)) + np.arange(6)

In [2]: m = m[:,[0,3,5,3,1,4,5,4,2]].reshape(-1,3,3)

In [3]: p_stresses = np.sort([np.linalg.eigvalsh(i) for i in m], axis = -1)

In [4]: p_stresses
Out[4]:
array([[-4.11645769, -2.04234215,  9.15879984],
       [-4.11645769, -2.04234215,  9.15879984]])


You might want to break this into more statements for clarity.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081016/22879d26/attachment.html>

From matthieu.brucher at gmail.com  Fri Oct 17 03:03:27 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 17 Oct 2008 09:03:27 +0200
Subject: [Numpy-discussion] ANN: NumPy 1.2.0
In-Reply-To: <e76aa17f0809290307j61c20884k1108f60e962151b3@mail.gmail.com>
References: <c7009a550809252307h2f99f742lcd321493997789a6@mail.gmail.com>
	<e76aa17f0809290100n73086679x56825e261932f15c@mail.gmail.com>
	<1222676350.21860.13.camel@bbc8> <gbq8tl$t0s$1@ger.gmane.org>
	<e76aa17f0809290307j61c20884k1108f60e962151b3@mail.gmail.com>
Message-ID: <e76aa17f0810170003u309db610wa8296a0f7f308e88@mail.gmail.com>

> BTW, I can confirm that the latest official MKL does not work with
> numpy, as it is explained on the Intel forum
> (http://software.intel.com/en-us/forums/intel-math-kernel-library/topic/60460).
> I get the i_free not defined issue.

For those who run into this issue, you have to use MKL 10.0.2 which
does nto have the issue.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From stefan at sun.ac.za  Fri Oct 17 04:24:19 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 17 Oct 2008 10:24:19 +0200
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
	<76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
Message-ID: <9457e7c80810170124g79a11145o9f367f697384e669@mail.gmail.com>

2008/10/16 Rob Hetland <hetland at tamu.edu>:
>
> On Oct 14, 2008, at 12:56 AM, St?fan van der Walt wrote:
>
>> Here is an implementation in Python, ctypes and in weave:
>>
>> http://mentat.za.net/source/pnpoly.tar.bz2
>>
>> Regards
>> St?fan
>
> This question gets asked about once a month on the mailing list.
> Perhaps pnpoly could find a permanent home in scipy?  (or somewhere?)
> Obviously, many would find it useful.

Maybe scipy.spatial could accommodate it?

St?fan


From affoster at gmail.com  Fri Oct 17 05:00:51 2008
From: affoster at gmail.com (Adam Foster)
Date: Fri, 17 Oct 2008 10:00:51 +0100
Subject: [Numpy-discussion] Request for Python 2.6 win32 package
Message-ID: <f5ba66d0810170200i4ab387b8nbd884a1b61ff2dd5@mail.gmail.com>

Please add numpy 1.2.0 win32 package for python 2.6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081017/0288dc00/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 17 05:04:24 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 17 Oct 2008 18:04:24 +0900
Subject: [Numpy-discussion] Request for Python 2.6 win32 package
In-Reply-To: <f5ba66d0810170200i4ab387b8nbd884a1b61ff2dd5@mail.gmail.com>
References: <f5ba66d0810170200i4ab387b8nbd884a1b61ff2dd5@mail.gmail.com>
Message-ID: <48F85518.6040307@ar.media.kyoto-u.ac.jp>

Adam Foster wrote:
> Please add numpy 1.2.0 win32 package for python 2.6

Hi,

    numpy 1.2 is not buildable with python 2.6. You will have to wait
for a later version, most probably 1.3,

cheers,

David


From affoster at gmail.com  Fri Oct 17 09:01:44 2008
From: affoster at gmail.com (Adam Foster)
Date: Fri, 17 Oct 2008 14:01:44 +0100
Subject: [Numpy-discussion] Request for Python 2.6 win32 package
Message-ID: <f5ba66d0810170601v45db9339v5d37b774bc149f15@mail.gmail.com>

numpy 1.2 is not buildable with python 2.6. You will have to wait
> for a later version, most probably 1.3,
>

Ok thanks David, guess I will have to wait till I can leverage the new
IEEE 754 support in python 2.6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081017/ad32822f/attachment.html>

From bertle at smoerz.org  Fri Oct 17 09:58:56 2008
From: bertle at smoerz.org (Roman Bertle)
Date: Fri, 17 Oct 2008 15:58:56 +0200
Subject: [Numpy-discussion] choose() broadcasting
In-Reply-To: <48E6319D.7050702@enthought.com>
References: <20081003111152.GA7228@smoerz.org>
	<20081003131402.GA7652@smoerz.org> <48E6319D.7050702@enthought.com>
Message-ID: <20081017135856.GA6141@smoerz.org>

* Travis E. Oliphant <oliphant at enthought.com> [081003 22:20]:
> Roman Bertle wrote:
> > Hello,
> >
> > I have found something I call a bug in the numpy choose() method and
> > wanted to report it in trac.
> >   
> Thanks for your report.  I'm not sure why you are having trouble with 
> Trac, but I've created a ticket for this problem.

Hello,

trac works for me know. And thank you for fixing the bug, the svn numpy
version works now for me. But there remains an issue I want to report.
choose is much slower in numpy than in numarray, and even more if an
output array is specified, as these tests show:

import timeit

setups = {
    'numarray': """
import numarray as N

n1, n2 = 4, 1000000
a1 = N.arange(n1*n2, type='Float64', shape=(n1,n2))
a2 = -N.arange(n1*n2, type='Float64', shape=(n1,n2))
a3 = -N.arange(n1*n2, type='Float64', shape=(n1,n2))
b1 = N.arange(n2, type='Float64')
b2 = -N.arange(n2, type='Float64')
b3 = -N.arange(n2, type='Float64')
c = N.remainder(N.arange(n2, type='Int32'),2)
""",
    'numpy': """
import numpy as N

n1, n2 = 4, 1000000
a1 = N.arange(n1*n2, dtype='Float64').reshape((n1,n2))
a2 = -N.arange(n1*n2, dtype='Float64').reshape((n1,n2))
a3 = -N.arange(n1*n2, dtype='Float64').reshape((n1,n2))
b1 = N.arange(n2, dtype='Float64')
b2 = -N.arange(n2, dtype='Float64')
b3 = -N.arange(n2, dtype='Float64')
c = N.remainder(N.arange(n2, dtype='Int32'),2)
"""
    }

stmta = "N.choose(c, (a1, a2))"
stmtao = "N.choose(c, (a1, a2), a3)"
stmtb = "N.choose(c, (b1, b2))"
stmtbo = "N.choose(c, (b1, b2), b3)"

timeit.Timer(setup=setups['numarray'], stmt=stmta).repeat(3,100)
[3.3187780380249023, 3.2966721057891846, 3.3234250545501709]

timeit.Timer(setup=setups['numpy'], stmt=stmta).repeat(3,100)
[14.842453002929688, 14.833296060562134, 14.836632966995239]

timeit.Timer(setup=setups['numarray'], stmt=stmtao).repeat(3,100)
[3.1973719596862793, 3.2031948566436768, 3.2093629837036133]

timeit.Timer(setup=setups['numpy'], stmt=stmtao).repeat(3,100)
[17.546916007995605, 17.548220157623291, 17.536314010620117]

timeit.Timer(setup=setups['numarray'], stmt=stmtb).repeat(3,100)
[0.6694338321685791, 0.66939401626586914, 0.67307686805725098]

timeit.Timer(setup=setups['numpy'], stmt=stmtb).repeat(3,100)
[3.7615809440612793, 3.7627589702606201, 3.7547731399536133]

timeit.Timer(setup=setups['numarray'], stmt=stmtbo).repeat(3,100)
[0.67037606239318848, 0.67186903953552246, 0.66994881629943848]

timeit.Timer(setup=setups['numpy'], stmt=stmtbo).repeat(3,100)
[4.4750981330871582, 4.4650890827178955, 4.4679431915283203]


Best Regards,

Roman


From jdh2358 at gmail.com  Fri Oct 17 10:40:06 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Fri, 17 Oct 2008 09:40:06 -0500
Subject: [Numpy-discussion] how to tell if a point is inside a polygon
In-Reply-To: <E31723B7-E636-4ED5-AE0F-A82FA3D6E8FB@tamu.edu>
References: <48F3D99C.1050301@jpl.nasa.gov>
	<9457e7c80810132256r3e4706dapf0285a63218d2e1c@mail.gmail.com>
	<76B9FA0B-BEA9-48F8-AEF7-EDD4610760A6@tamu.edu>
	<88e473830810161154n23bce8fax8a881c86fa24c416@mail.gmail.com>
	<E31723B7-E636-4ED5-AE0F-A82FA3D6E8FB@tamu.edu>
Message-ID: <88e473830810170740h7c30b858ue4e3b72f5b6cffc6@mail.gmail.com>

On Thu, Oct 16, 2008 at 2:28 PM, Rob Hetland <hetland at tamu.edu> wrote:

> I did not know that very useful thing.  But now I do.  This is solid
> proof that lurking on the mailing lists makes you smarter.

and that our documentation effort still has a long way to go !

FAQ added at
http://matplotlib.sourceforge.net/faq/howto_faq.html?#how-do-i-test-whether-a-point-is-inside-a-polygon

though I am having trouble getting the module functions pnpoly and
points_inside_poly to show up in the sphinx automodule documentation
for nxutils.  These functions are defined in extension code and I have
a post in to the sphinx mailing list

  http://groups.google.com/group/sphinx-dev/t/7ad1631d3117e4eb

but if anyone on this list has seen problems with automodule and
extension code functions, and knows how to fix them, let me know.


From tonyyu at MIT.EDU  Fri Oct 17 15:27:58 2008
From: tonyyu at MIT.EDU (Tony S Yu)
Date: Fri, 17 Oct 2008 15:27:58 -0400
Subject: [Numpy-discussion] Weird clipping when astype(int) used on large
	numbers
Message-ID: <C2A8F11D-567D-4E25-B178-359BEE680E3F@mit.edu>

I ran into this weird behavior with astype(int)

In [57]: a = np.array(1E13)

In [58]: a.astype(int)

Out[58]: array(-2147483648)

I understand why large numbers need to be clipped when converting to  
int (although I would have expected some sort of warning), but I'm  
puzzled by the negative value. Shouldn't the above code clip the value  
to the max positive int (2147483647)... and maybe issue a warning?

Thanks,
-Tony

P.S. In case this is a problem with my install:
OS X 10.5.5
Python 2.5.1
Numpy 1.2.0


From matthieu.brucher at gmail.com  Fri Oct 17 15:34:39 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 17 Oct 2008 21:34:39 +0200
Subject: [Numpy-discussion] Weird clipping when astype(int) used on
	large numbers
In-Reply-To: <C2A8F11D-567D-4E25-B178-359BEE680E3F@mit.edu>
References: <C2A8F11D-567D-4E25-B178-359BEE680E3F@mit.edu>
Message-ID: <e76aa17f0810171234m4583f263u8064d86200bb8995@mail.gmail.com>

Hi,

This a usual thing in integers conversions. If you transform an
integer like 0x0000FFFF from 16 bits to 8bits, you get 0xFFFF, thus a
negative number. As there are no processor instructions that do
saturations (DSP instructions), the behavior is to be expected.

Matthieu

2008/10/17 Tony S Yu <tonyyu at mit.edu>:
> I ran into this weird behavior with astype(int)
>
> In [57]: a = np.array(1E13)
>
> In [58]: a.astype(int)
>
> Out[58]: array(-2147483648)
>
> I understand why large numbers need to be clipped when converting to
> int (although I would have expected some sort of warning), but I'm
> puzzled by the negative value. Shouldn't the above code clip the value
> to the max positive int (2147483647)... and maybe issue a warning?
>
> Thanks,
> -Tony
>
> P.S. In case this is a problem with my install:
> OS X 10.5.5
> Python 2.5.1
> Numpy 1.2.0
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From charlesr.harris at gmail.com  Fri Oct 17 17:00:30 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 17 Oct 2008 15:00:30 -0600
Subject: [Numpy-discussion] Weird clipping when astype(int) used on
	large numbers
In-Reply-To: <C2A8F11D-567D-4E25-B178-359BEE680E3F@mit.edu>
References: <C2A8F11D-567D-4E25-B178-359BEE680E3F@mit.edu>
Message-ID: <e06186140810171400w2e14d6bbx7859a5496b214ec3@mail.gmail.com>

On Fri, Oct 17, 2008 at 1:27 PM, Tony S Yu <tonyyu at mit.edu> wrote:

> I ran into this weird behavior with astype(int)
>
> In [57]: a = np.array(1E13)
>
> In [58]: a.astype(int)
>
> Out[58]: array(-2147483648)
>
> I understand why large numbers need to be clipped when converting to
> int (although I would have expected some sort of warning), but I'm
> puzzled by the negative value. Shouldn't the above code clip the value
> to the max positive int (2147483647)... and maybe issue a warning?
>

Try more precision:

In [2]: a = np.array(1e13)

In [3]: a.astype(np.int64)
Out[3]: array(10000000000000)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081017/28321a1d/attachment.html>

From oliphant at enthought.com  Fri Oct 17 17:31:31 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 17 Oct 2008 16:31:31 -0500
Subject: [Numpy-discussion] choose() broadcasting
In-Reply-To: <20081017135856.GA6141@smoerz.org>
References: <20081003111152.GA7228@smoerz.org>	<20081003131402.GA7652@smoerz.org>
	<48E6319D.7050702@enthought.com> <20081017135856.GA6141@smoerz.org>
Message-ID: <48F90433.5080209@enthought.com>

Roman Bertle wrote:
> * Travis E. Oliphant <oliphant at enthought.com> [081003 22:20]:
>   
>> Roman Bertle wrote:
>>     
>>> Hello,
>>>
>>> I have found something I call a bug in the numpy choose() method and
>>> wanted to report it in trac.
>>>   
>>>       
>> Thanks for your report.  I'm not sure why you are having trouble with 
>> Trac, but I've created a ticket for this problem.
>>     
>
> Hello,
>
> trac works for me know. And thank you for fixing the bug, the svn numpy
> version works now for me. But there remains an issue I want to report.
> choose is much slower in numpy than in numarray, and even more if an
> output array is specified, as these tests show:
>   
Thanks for the report.   You should add another ticket for this case.  I 
suspect it might be a result of the extra copies that are done in the 
PyArray_Choose routine because the algorithm assumes contiguous arrays.

It deserves a look.   It probably wouldn't be too difficult to avoid the 
copy.

-Travis



From dpeterson at enthought.com  Fri Oct 17 19:01:17 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Fri, 17 Oct 2008 18:01:17 -0500
Subject: [Numpy-discussion] ANNOUNCE: EPD with Py2.5 version 4.0.30002 RC2
	available for testing
Message-ID: <48F9193D.3050100@enthought.com>

Hello,

We've recently posted the RC2 build of EPD (the Enthought Python 
Distribution) with Python 2.5 version 4.0.30002 to the EPD website.  You 
may download the RC from here:
	http://www.enthought.com/products/epdbeta.php

You can check out the release notes here:
	https://svn.enthought.com/epd/wiki/Python2.5.2/4.0.300/RC2

Please help us test it out and provide feedback on the EPD Trac
instance: https://svn.enthought.com/epd or via e-mail to 
epd-support at enthought.com.   If everything goes well, we are planning a 
final release for this coming Tuesday, October 21st.


About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included"
distribution of the Python? Programming Language, including over 60
additional tools and libraries. The EPD bundle includes NumPy, SciPy,
IPython, 2D and 3D visualization, database adapters, and a lot of
other tools right out of the box.

	http://www.enthought.com/products/epd.php

It is currently available as a single-click installer for Windows XP
(x86), Mac OS X (a universal binary for OS X 10.4 and above), and
RedHat 3 and 4 (x86 and amd64).

EPD is free for academic use.  An annual subscription and installation
support are available for individual commercial use.  An enterprise
subscription with support for particular deployment environments is also
available for commercial purchase.


-- Dave



From irving at naml.us  Sat Oct 18 18:43:38 2008
From: irving at naml.us (Geoffrey Irving)
Date: Sat, 18 Oct 2008 15:43:38 -0700
Subject: [Numpy-discussion] dtype comparison and hashing
In-Reply-To: <3d375d730810151256u6c135679l61e9483a14553938@mail.gmail.com>
References: <7f9d599f0810150020m489b54day4c909f6cbc8c1203@mail.gmail.com>
	<3d375d730810151256u6c135679l61e9483a14553938@mail.gmail.com>
Message-ID: <7f9d599f0810181543q168a4483tab087ac446050e7f@mail.gmail.com>

On Wed, Oct 15, 2008 at 12:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Oct 15, 2008 at 02:20, Geoffrey Irving <irving at naml.us> wrote:
>> Hello,
>>
>> Currently in numpy comparing dtypes for equality with == does an
>> internal PyArray_EquivTypes check, which means that the dtypes NPY_INT
>> and NPY_LONG compare as equal in python.  However, the hash function
>> for dtypes reduces id(), which is therefore inconsistent with ==.
>> Unfortunately I can't produce a python snippet showing this since I
>> don't know how to create a NPY_INT dtype in pure python.
>>
>> Based on the source it looks like hash should raise a type error,
>> since tp_hash is null but tp_richcompare is not.  Does the following
>> snippet through an exception for others?
>>
>>>>> import numpy
>>>>> hash(numpy.dtype('int'))
>> 5708736
>>
>> This might be the problem:
>>
>> /* Macro to get the tp_richcompare field of a type if defined */
>> #define RICHCOMPARE(t) (PyType_HasFeature((t), Py_TPFLAGS_HAVE_RICHCOMPARE) \
>>                         ? (t)->tp_richcompare : NULL)
>>
>> I'm using the default Mac OS X 10.5 installation of python 2.5 and
>> numpy, so maybe those weren't compiled correctly.  Has anyone else
>> seen this issue?
>
> Actually, the problem is that we provide a hash function explicitly.
> In multiarraymodule.c:
>
>    PyArrayDescr_Type.tp_hash = (hashfunc)_Py_HashPointer;
>
> That is a violation of the hashing protocol (objects which compare
> equal and are hashable need to hash equal), and should be fixed.

Thanks for finding that.

Geoffrey


From hoytak at cs.ubc.ca  Sat Oct 18 19:06:14 2008
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Sat, 18 Oct 2008 16:06:14 -0700
Subject: [Numpy-discussion] problem installing numpy using scons
Message-ID: <4db580fd0810181606q4b89fdday514a10617409d350@mail.gmail.com>

Hello,

I'm trying to install the latest numpy using setupscons.py, but it
gives me an error that I can't track down.  It's in the latest numpy
release (5946). with the latest numpyscons from the 0.9.3 branch.

Here's the last part of output:

is bootstrapping ? True
Executing scons command (pkg is numpy.core): /usr/local2/bin/python
"/usr/local2/lib/python2.5/site-packages/numscons-0.9.3dev-py2.5.egg/numscons/scons-local/scons.py"
-f numpy/core/SConstruct -I. scons_tool_path="" src_dir="numpy/core"
pkg_name="numpy.core" log_level=50
distutils_libdir="../../../../build/lib.linux-x86_64-2.5" cc_opt=gcc
cc_opt_path="/usr/bin" f77_opt=gfortran f77_opt_path="/usr/bin"
cxx_opt=g++ cxx_opt_path="/usr/bin"
include_bootstrap=../../../../numpy/core/include silent=0
bootstrapping=1
scons: Reading SConscript files ...
Checking for C header file Python.h... yes
Checking size of short ... yes
Checking size of int ... yes
Checking size of long ... yes
Checking size of float ... yes
Checking size of double ... yes
Checking size of long double ... yes
Checking size of Py_intptr_t ... yes
Checking whether PY_LONG_LONG is declared... yes
Checking size of PY_LONG_LONG ... yes
Checking whether CHAR_BIT is declared... yes
Checking whether PRIdPTR is declared... yes
Checking if math lib [] is usable for numpy ...  No !
Checking for C library m... yes
Checking if math lib ['m'] is usable for numpy ...  Yes !
AttributeError: SConfBase instance has no attribute 'CheckFuncsAtOnce':
  File "/usr/local2/src/numpy/numpy/core/SConstruct", line 2:
    GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
  File "/usr/local2/lib/python2.5/site-packages/numscons-0.9.3dev-py2.5.egg/numscons/core/numpyenv.py",
line 108:
    build_dir = '$build_dir', src_dir = '$src_dir')
  File "/usr/local2/lib/python2.5/site-packages/numscons-0.9.3dev-py2.5.egg/numscons/scons-local/scons-local-1.0.1/SCons/Script/SConscript.py",
line 533:
    return apply(_SConscript, [self.fs,] + files, subst_kw)
  File "/usr/local2/lib/python2.5/site-packages/numscons-0.9.3dev-py2.5.egg/numscons/scons-local/scons-local-1.0.1/SCons/Script/SConscript.py",
line 256:
    exec _file_ in call_stack[-1].globals
  File "/usr/local2/src/numpy/build/scons/numpy/core/SConscript", line 165:
    check_funcs(optional_stdfuncs)
  File "/usr/local2/src/numpy/build/scons/numpy/core/SConscript", line 154:
    st = config.CheckFuncsAtOnce(funcs)
error: Error while executing scons command. See above for more information.
If you think it is a problem in numscons, you can also try executing the scons
command with --log-level option for more detailed output of what numscons is
doing, for example --log-level=0; the lowest the level is, the more detailed
the output it.

I'm trying to install it on a 8 processor 64 bit Intel machine.  I
could post more details of the installation if needed, but they don't
seem relevant here (except that I've had a lot of issues getting the
regular install script to work with the correct compile parameters,
and so I thought I'd try scons).

If there's something I'm doing wrong, let me know; otherwise I'd be
happy to post a bug report.

--Hoyt


From mnandris at blueyonder.co.uk  Sat Oct 18 22:52:20 2008
From: mnandris at blueyonder.co.uk (Michael)
Date: Sun, 19 Oct 2008 03:52:20 +0100
Subject: [Numpy-discussion] Getting an array's indices when a given
	condition is true
Message-ID: <1224384740.3615.14.camel@mik>

Hi list, 

been playing around with stride_tricks and find it terrifically
productive; thankyou to everyone who has worked on this.  

I need to filter some data, getting the indices of all entries which are
less than or equal to 'limit'. How do i best go about that? 

Can you enumerate an array using broadcasting?

dat=np.array([  0.           1.61803399   3.23606798   4.85410197
6.47213595
   8.09016994   9.70820393  11.32623792  12.94427191  14.5623059    1.
   0.61803399   2.23606798   3.85410197   5.47213595   7.09016994
   8.70820393  10.32623792  11.94427191  13.5623059    2.
0.38196601
   1.23606798   2.85410197   4.47213595   6.09016994   7.70820393
   9.32623792  10.94427191  12.5623059    3.           1.38196601
   0.23606798   1.85410197   3.47213595   5.09016994   6.70820393
   8.32623792   9.94427191  11.5623059    4.           2.38196601
   0.76393202   0.85410197   2.47213595   4.09016994   5.70820393
   7.32623792   8.94427191  10.5623059    5.           3.38196601
   1.76393202   0.14589803   1.47213595   3.09016994   4.70820393
   6.32623792   7.94427191   9.5623059    6.           4.38196601
   2.76393202   1.14589803   0.47213595   2.09016994   3.70820393
   5.32623792   6.94427191   8.5623059    7.           5.38196601
   3.76393202   2.14589803   0.52786405   1.09016994   2.70820393
   4.32623792   5.94427191   7.5623059    8.           6.38196601
   4.76393202   3.14589803   1.52786405   0.09016994   1.70820393
   3.32623792   4.94427191   6.5623059    9.           7.38196601
   5.76393202   4.14589803   2.52786405   0.90983006   0.70820393
   2.32623792   3.94427191   5.5623059 ])

limit=1.30901699437

for i in dat: 
    if i <=limit: 
	print i 	

Michael

-- 
"When you think of the long and gloomy history of man, you will find far
more hideous crimes have been committed in the name of obedience than
have been committed in the name of rebellion". C.P.Snow,
"Either-Or" (1961)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081019/ae119f8c/attachment.sig>

From lbrooks at MIT.EDU  Sat Oct 18 23:00:42 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sat, 18 Oct 2008 21:00:42 -0600
Subject: [Numpy-discussion] Getting an array's indices when a
 given	condition is true
In-Reply-To: <1224384740.3615.14.camel@mik>
References: <1224384740.3615.14.camel@mik>
Message-ID: <48FAA2DA.6080800@mit.edu>

If you want the indexes, check out the np.where command, e.g.

idx = np.where(dat <= limit)

If you want the values, use:
val = dat[dat <= limit]

Lane


Michael wrote:
> Hi list, 
>
> been playing around with stride_tricks and find it terrifically
> productive; thankyou to everyone who has worked on this.  
>
> I need to filter some data, getting the indices of all entries which are
> less than or equal to 'limit'. How do i best go about that? 
>
> Can you enumerate an array using broadcasting?
>
> dat=np.array([  0.           1.61803399   3.23606798   4.85410197
> 6.47213595
>    8.09016994   9.70820393  11.32623792  12.94427191  14.5623059    1.
>    0.61803399   2.23606798   3.85410197   5.47213595   7.09016994
>    8.70820393  10.32623792  11.94427191  13.5623059    2.
> 0.38196601
>    1.23606798   2.85410197   4.47213595   6.09016994   7.70820393
>    9.32623792  10.94427191  12.5623059    3.           1.38196601
>    0.23606798   1.85410197   3.47213595   5.09016994   6.70820393
>    8.32623792   9.94427191  11.5623059    4.           2.38196601
>    0.76393202   0.85410197   2.47213595   4.09016994   5.70820393
>    7.32623792   8.94427191  10.5623059    5.           3.38196601
>    1.76393202   0.14589803   1.47213595   3.09016994   4.70820393
>    6.32623792   7.94427191   9.5623059    6.           4.38196601
>    2.76393202   1.14589803   0.47213595   2.09016994   3.70820393
>    5.32623792   6.94427191   8.5623059    7.           5.38196601
>    3.76393202   2.14589803   0.52786405   1.09016994   2.70820393
>    4.32623792   5.94427191   7.5623059    8.           6.38196601
>    4.76393202   3.14589803   1.52786405   0.09016994   1.70820393
>    3.32623792   4.94427191   6.5623059    9.           7.38196601
>    5.76393202   4.14589803   2.52786405   0.90983006   0.70820393
>    2.32623792   3.94427191   5.5623059 ])
>
> limit=1.30901699437
>
> for i in dat: 
>     if i <=limit: 
> 	print i 	
>
> Michael
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081018/cefbc9ce/attachment.html>

From robert.kern at gmail.com  Sat Oct 18 23:13:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 18 Oct 2008 22:13:16 -0500
Subject: [Numpy-discussion] Getting an array's indices when a given
	condition is true
In-Reply-To: <48FAA2DA.6080800@mit.edu>
References: <1224384740.3615.14.camel@mik> <48FAA2DA.6080800@mit.edu>
Message-ID: <3d375d730810182013k5e6010adn4a9db139614f439d@mail.gmail.com>

On Sat, Oct 18, 2008 at 22:00, Lane Brooks <lbrooks at mit.edu> wrote:
> If you want the indexes, check out the np.where command, e.g.
>
> idx = np.where(dat <= limit)

It's worth noting that where() confusingly has two modes of
functionality. This particular bit of functionality is also exposed
under the better-named nonzero() function. I prefer to use nonzero()
for this, and only use where() for its where(condition, value_if_true,
value_if_false) functionality.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lbrooks at MIT.EDU  Sun Oct 19 00:07:44 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sat, 18 Oct 2008 22:07:44 -0600
Subject: [Numpy-discussion] Images and numpy
Message-ID: <48FAB290.70004@mit.edu>

What are the preferred ways to get images, like jpgs and pngs, from disk 
into a numpy array and from a numpy array to disk? 

I did some google searches and found a PEP thread where Travis was 
proposing an extended buffer protocol that would make for easier 
interoperability with libraries such as PIL.  Did that materialize?

Lane


From robert.kern at gmail.com  Sun Oct 19 00:23:13 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 18 Oct 2008 23:23:13 -0500
Subject: [Numpy-discussion] Images and numpy
In-Reply-To: <48FAB290.70004@mit.edu>
References: <48FAB290.70004@mit.edu>
Message-ID: <3d375d730810182123q6825a422t62f3c6b916f33ffc@mail.gmail.com>

On Sat, Oct 18, 2008 at 23:07, Lane Brooks <lbrooks at mit.edu> wrote:
> What are the preferred ways to get images, like jpgs and pngs, from disk
> into a numpy array and from a numpy array to disk?
>
> I did some google searches and found a PEP thread where Travis was
> proposing an extended buffer protocol that would make for easier
> interoperability with libraries such as PIL.  Did that materialize?

Yes. There are two protocols, a Python level one, and a C level one.
Python 2.6 is the first release where the C one is standard. Recent
PILs support the Python level one.


In [9]: import Image

In [10]: img = Image.open('icon.png')

In [11]: import numpy

In [12]: numpy.asarray(img)
Out[12]:
array([[[  0,   0,   0,   0],
        [  0,   0,   0,   0],
        [  0,   0,   0,   0],
        ...,
        [  0,   0,   0,   0],
        [  0,   0,   0,   0],
        [  0,   0,   0,   0]],
...

In [14]: img2 = Image.fromarray(Out[12])

In [15]: img2
Out[15]: <Image.Image instance at 0x1c0baa8>

In [16]: img2.size
Out[16]: (48, 48)


You may also want to look at scipy's scipy.misc.pilutil module. It
doesn't use these newer APIs, but it does have extra functionality
like scaling float arrays to [0..255] to write out to PNGs and such.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lbrooks at MIT.EDU  Sun Oct 19 00:45:31 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sat, 18 Oct 2008 22:45:31 -0600
Subject: [Numpy-discussion] Images and numpy
In-Reply-To: <3d375d730810182123q6825a422t62f3c6b916f33ffc@mail.gmail.com>
References: <48FAB290.70004@mit.edu>
	<3d375d730810182123q6825a422t62f3c6b916f33ffc@mail.gmail.com>
Message-ID: <48FABB6B.1010304@mit.edu>

Robert Kern wrote:
> On Sat, Oct 18, 2008 at 23:07, Lane Brooks <lbrooks at mit.edu> wrote:
>   
>> What are the preferred ways to get images, like jpgs and pngs, from disk
>> into a numpy array and from a numpy array to disk?
>>
>> I did some google searches and found a PEP thread where Travis was
>> proposing an extended buffer protocol that would make for easier
>> interoperability with libraries such as PIL.  Did that materialize?
>>     
>
> Yes. There are two protocols, a Python level one, and a C level one.
> Python 2.6 is the first release where the C one is standard. Recent
> PILs support the Python level one.
>
>
> In [9]: import Image
>
> In [10]: img = Image.open('icon.png')
>
> In [11]: import numpy
>
> In [12]: numpy.asarray(img)
> Out[12]:
> array([[[  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         ...,
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0]],
> ...
>
> In [14]: img2 = Image.fromarray(Out[12])
>
> In [15]: img2
> Out[15]: <Image.Image instance at 0x1c0baa8>
>
> In [16]: img2.size
> Out[16]: (48, 48)
>
>
> You may also want to look at scipy's scipy.misc.pilutil module. It
> doesn't use these newer APIs, but it does have extra functionality
> like scaling float arrays to [0..255] to write out to PNGs and such.
>
>   
Great!  Thanks for the info.  Just what I need!

Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081018/ed06be71/attachment.html>

From nadavh at visionsense.com  Sun Oct 19 02:08:46 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sun, 19 Oct 2008 08:08:46 +0200
Subject: [Numpy-discussion] Images and numpy
References: <48FAB290.70004@mit.edu><3d375d730810182123q6825a422t62f3c6b916f33ffc@mail.gmail.com>
	<48FABB6B.1010304@mit.edu>
Message-ID: <710F2847B0018641891D9A216027636029C2D9@ex3.envision.co.il>

I encountered an exception: "nonstandard" format like 16 bits tiff I/O works only via the tostring/fromstring interface.

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Lane Brooks
????: ? 19-???????-08 06:45
??: Discussion of Numerical Python
????: Re: [Numpy-discussion] Images and numpy
 
Robert Kern wrote:
> On Sat, Oct 18, 2008 at 23:07, Lane Brooks <lbrooks at mit.edu> wrote:
>   
>> What are the preferred ways to get images, like jpgs and pngs, from disk
>> into a numpy array and from a numpy array to disk?
>>
>> I did some google searches and found a PEP thread where Travis was
>> proposing an extended buffer protocol that would make for easier
>> interoperability with libraries such as PIL.  Did that materialize?
>>     
>
> Yes. There are two protocols, a Python level one, and a C level one.
> Python 2.6 is the first release where the C one is standard. Recent
> PILs support the Python level one.
>
>
> In [9]: import Image
>
> In [10]: img = Image.open('icon.png')
>
> In [11]: import numpy
>
> In [12]: numpy.asarray(img)
> Out[12]:
> array([[[  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         ...,
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0],
>         [  0,   0,   0,   0]],
> ...
>
> In [14]: img2 = Image.fromarray(Out[12])
>
> In [15]: img2
> Out[15]: <Image.Image instance at 0x1c0baa8>
>
> In [16]: img2.size
> Out[16]: (48, 48)
>
>
> You may also want to look at scipy's scipy.misc.pilutil module. It
> doesn't use these newer APIs, but it does have extra functionality
> like scaling float arrays to [0..255] to write out to PNGs and such.
>
>   
Great!  Thanks for the info.  Just what I need!

Lane

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3620 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081019/106b6664/attachment.bin>

From robert.kern at gmail.com  Sun Oct 19 02:21:08 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 19 Oct 2008 01:21:08 -0500
Subject: [Numpy-discussion] Images and numpy
In-Reply-To: <710F2847B0018641891D9A216027636029C2D9@ex3.envision.co.il>
References: <48FAB290.70004@mit.edu>
	<3d375d730810182123q6825a422t62f3c6b916f33ffc@mail.gmail.com>
	<48FABB6B.1010304@mit.edu>
	<710F2847B0018641891D9A216027636029C2D9@ex3.envision.co.il>
Message-ID: <3d375d730810182321j2e6980b7i524d2e1c4498c85b@mail.gmail.com>

On Sun, Oct 19, 2008 at 01:08, Nadav Horesh <nadavh at visionsense.com> wrote:
> I encountered an exception: "nonstandard" format like 16 bits tiff I/O works only via the tostring/fromstring interface.

Yup. Internally, PIL doesn't necessarily represent the data in the
same way that numpy does, so you have to go through another interface
rather than the fast one.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Sun Oct 19 06:57:50 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 19 Oct 2008 19:57:50 +0900
Subject: [Numpy-discussion] problem installing numpy using scons
In-Reply-To: <4db580fd0810181606q4b89fdday514a10617409d350@mail.gmail.com>
References: <4db580fd0810181606q4b89fdday514a10617409d350@mail.gmail.com>
Message-ID: <5b8d13220810190357i618c1ec4t689d701518f5fb15@mail.gmail.com>

On Sun, Oct 19, 2008 at 8:06 AM, Hoyt Koepke <hoytak at cs.ubc.ca> wrote:

>
> If there's something I'm doing wrong, let me know; otherwise I'd be
> happy to post a bug report.

The trunk requires some changes which are not released yet. It will be
in the next release of numscons (0.9.3).

cheers,

David


From lbrooks at MIT.EDU  Sun Oct 19 15:28:49 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sun, 19 Oct 2008 13:28:49 -0600
Subject: [Numpy-discussion] numpy CAPI questions
Message-ID: <48FB8A71.3010508@mit.edu>

I am using the numpy CAPI to write an extension module that returns a 
numpy Array from an imaging data source.  I collect the image into a 
buffer that I allocate. I then create numpy Array using the 
PyArray_New(..) function and pass it the buffer.  I then set the 
NPY_OWNDATA flag on the Array because I want the Array to deallocate the 
buffer when it is deleted.  Is that the correct way to do it?  The code 
snippet below is what I wrote, and it seems to be working just fine, but 
I wanted to verify that I am doing things correctly.



uint16_t *data;

// malloc data and fill it with some image data...

img = PyArray_New(&PyArray_Type, 2, dims, NPY_UINT16, NULL, data, 2, 
NPY_C_CONTIGUOUS | NPY_WRITEABLE | NPY_ALIGNED, NULL);
PyArray_UpdateFlags((PyArrayObject *) img, NPY_OWNDATA | 
NPY_C_CONTIGUOUS | NPY_WRITEABLE | NPY_ALIGNED);
return Py_BuildValue("O", img);



Here are my questions:
1. Does NPY_OWNDATA mean that the object will deallocate the memory when 
the object is deleted?  The manual seems to indicate that as such but it 
is not explicitly stated.

2. Is my reference counting correct?  Do I need to call the 
PyArray_INCREF() on img?

Thanks,
Lane Brooks


From robert.kern at gmail.com  Sun Oct 19 16:09:46 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 19 Oct 2008 15:09:46 -0500
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <48FB8A71.3010508@mit.edu>
References: <48FB8A71.3010508@mit.edu>
Message-ID: <3d375d730810191309u4b4a5e9dl41350bdbb123e032@mail.gmail.com>

On Sun, Oct 19, 2008 at 14:28, Lane Brooks <lbrooks at mit.edu> wrote:
> I am using the numpy CAPI to write an extension module that returns a
> numpy Array from an imaging data source.  I collect the image into a
> buffer that I allocate. I then create numpy Array using the
> PyArray_New(..) function and pass it the buffer.  I then set the
> NPY_OWNDATA flag on the Array because I want the Array to deallocate the
> buffer when it is deleted.  Is that the correct way to do it?  The code
> snippet below is what I wrote, and it seems to be working just fine, but
> I wanted to verify that I am doing things correctly.

Preferably, you should create the array and then use the memory it
allocated as the buffer. Generally, malloc()/free() implementations
are paired, and if you mix-and-match, you're begging for trouble.

> Here are my questions:
> 1. Does NPY_OWNDATA mean that the object will deallocate the memory when
> the object is deleted?  The manual seems to indicate that as such but it
> is not explicitly stated.

Yes.

> 2. Is my reference counting correct?  Do I need to call the
> PyArray_INCREF() on img?

Personally, I always need to double-check my refcounting with
sys.getrefcount() (which, it should be noted, adds its own reference,
so the correct result for sys.getrefcount(foo()) should be 1). In your
actual code, do you have anything else in the Py_BuildValue()? If you
don't, then you don't need to use Py_BuildValue(); you can just return
img.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lbrooks at MIT.EDU  Sun Oct 19 17:52:50 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Sun, 19 Oct 2008 15:52:50 -0600
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <3d375d730810191309u4b4a5e9dl41350bdbb123e032@mail.gmail.com>
References: <48FB8A71.3010508@mit.edu>
	<3d375d730810191309u4b4a5e9dl41350bdbb123e032@mail.gmail.com>
Message-ID: <48FBAC32.1000003@mit.edu>

Robert Kern wrote:
> On Sun, Oct 19, 2008 at 14:28, Lane Brooks <lbrooks at mit.edu> wrote:
>   
>> 2. Is my reference counting correct?  Do I need to call the
>> PyArray_INCREF() on img?
>>     
>
> Personally, I always need to double-check my refcounting with
> sys.getrefcount() (which, it should be noted, adds its own reference,
> so the correct result for sys.getrefcount(foo()) should be 1). In your
> actual code, do you have anything else in the Py_BuildValue()? If you
> don't, then you don't need to use Py_BuildValue(); you can just return
> img.
>   


Thanks for the great tip on sys.getrefcount().  I understand reference 
counting a whole lot better now after playing with sys.getrefcount() 
function.

Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081019/e3925172/attachment.html>

From cournape at gmail.com  Sun Oct 19 22:12:40 2008
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 20 Oct 2008 11:12:40 +0900
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <48FBAC32.1000003@mit.edu>
References: <48FB8A71.3010508@mit.edu>
	<3d375d730810191309u4b4a5e9dl41350bdbb123e032@mail.gmail.com>
	<48FBAC32.1000003@mit.edu>
Message-ID: <5b8d13220810191912i1845f2bax6d83619bb75fb023@mail.gmail.com>

On Mon, Oct 20, 2008 at 6:52 AM, Lane Brooks <lbrooks at mit.edu> wrote:
>
>
> Thanks for the great tip on sys.getrefcount().  I understand reference
> counting a whole lot better now after playing with sys.getrefcount()
> function.

For debugging purpose, and if you need it in C, you can use the member
a->ob_refcount where a is any valid pointer to a PyObject. This is
more or less the equivalent of sys.getrefcount, but at the C level
(without the +1). The struct member ob_refcount is private and should
not be depended on in "real" code, but I personally like using it when
I am not sure about ref counting.

cheers,

David


From andrea.gavana at gmail.com  Mon Oct 20 05:20:05 2008
From: andrea.gavana at gmail.com (Andrea Gavana)
Date: Mon, 20 Oct 2008 10:20:05 +0100
Subject: [Numpy-discussion] Combinations of objects (?)
Message-ID: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>

Hi All,

    this is probably a very silly question, but combinatorial math is
not exactly my strength and I am not even sure on how to formulate the
question. I apologize if it is a very elementary problem.
Let's suppose that I have 60 oil wells and 3 surface facilities. Every
well must be tied-in (linked, attached) to one and only one of these
surface facilities. No orphan well is allowed (i.e., a well not
attached to any surface unit) and no multi-links are allowed (i.e.,
one well attached to 2 or more units). Is there any simple way in
numpy (scipy?) in which I can get the number of possible combinations
of wells attached to the different 3 units, without repetitions? For
example, I could have all 60 wells attached to unit number 1, then 59
on unit 1 and 1 on unit 3 and so on...

Thank you for your suggestions.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/


From tjhnson at gmail.com  Mon Oct 20 06:31:07 2008
From: tjhnson at gmail.com (T J)
Date: Mon, 20 Oct 2008 03:31:07 -0700
Subject: [Numpy-discussion] Combinations of objects (?)
In-Reply-To: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>
References: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>
Message-ID: <b02c60870810200331l32e62562y62565501728ae1e3@mail.gmail.com>

On Mon, Oct 20, 2008 at 2:20 AM, A. G. wrote:
> one well attached to 2 or more units). Is there any simple way in
> numpy (scipy?) in which I can get the number of possible combinations
> of wells attached to the different 3 units, without repetitions? For
> example, I could have all 60 wells attached to unit number 1, then 59
> on unit 1 and 1 on unit 3 and so on...


From:  http://tinyurl.com/6oeyx8

def boxings(n, k):
    seq, i = [n]*k + [0], k
    while i:
        yield tuple(seq[i] - seq[i+1] for i in xrange(k))
        i = seq.index(0) - 1
        seq[i:k] = [seq[i] - 1] * (k-i)


Example:

>>> from scipy import factorial as f
>>> x = list(boxings(60,3))
>>> len(x)
1891
>>> f(60+3-1) / f(60) / f(3-1)
1891.0000000000002


That thread contains another solution using itertools.


From aisaac at american.edu  Mon Oct 20 07:48:13 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 20 Oct 2008 07:48:13 -0400
Subject: [Numpy-discussion] Combinations of objects (?)
In-Reply-To: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>
References: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>
Message-ID: <48FC6FFD.3080509@american.edu>

On 10/20/2008 5:20 AM Andrea Gavana apparently wrote:
>     this is probably a very silly question, but combinatorial math is
> not exactly my strength and I am not even sure on how to formulate the
> question. I apologize if it is a very elementary problem.
> Let's suppose that I have 60 oil wells and 3 surface facilities. Every
> well must be tied-in (linked, attached) to one and only one of these
> surface facilities. No orphan well is allowed (i.e., a well not
> attached to any surface unit) and no multi-links are allowed (i.e.,
> one well attached to 2 or more units). Is there any simple way in
> numpy (scipy?) in which I can get the number of possible combinations
> of wells attached to the different 3 units, without repetitions? For
> example, I could have all 60 wells attached to unit number 1, then 59
> on unit 1 and 1 on unit 3 and so on...

I think you are just asking how many ways you
can put N marbles into K boxes, with no restrictions.
Since there are K possibilities for each marble,
wouldn't the answer to your problem be K**N?

Alan Isaac



From andrea.gavana at gmail.com  Mon Oct 20 11:02:47 2008
From: andrea.gavana at gmail.com (Andrea Gavana)
Date: Mon, 20 Oct 2008 16:02:47 +0100
Subject: [Numpy-discussion] Combinations of objects (?)
In-Reply-To: <48FC6FFD.3080509@american.edu>
References: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com>
	<48FC6FFD.3080509@american.edu>
Message-ID: <d5ff27200810200802q671e4b9du7298f5d1b74eb864@mail.gmail.com>

Hi All,

On Mon, Oct 20, 2008 at 12:48 PM, Alan G Isaac wrote:
> On 10/20/2008 5:20 AM Andrea Gavana apparently wrote:
>>     this is probably a very silly question, but combinatorial math is
>> not exactly my strength and I am not even sure on how to formulate the
>> question. I apologize if it is a very elementary problem.
>> Let's suppose that I have 60 oil wells and 3 surface facilities. Every
>> well must be tied-in (linked, attached) to one and only one of these
>> surface facilities. No orphan well is allowed (i.e., a well not
>> attached to any surface unit) and no multi-links are allowed (i.e.,
>> one well attached to 2 or more units). Is there any simple way in
>> numpy (scipy?) in which I can get the number of possible combinations
>> of wells attached to the different 3 units, without repetitions? For
>> example, I could have all 60 wells attached to unit number 1, then 59
>> on unit 1 and 1 on unit 3 and so on...
>
> I think you are just asking how many ways you
> can put N marbles into K boxes, with no restrictions.
> Since there are K possibilities for each marble,
> wouldn't the answer to your problem be K**N?

Uhm, I am not sure, I imagined TJ answer is correct by reading the
Python thread he kindly posted. As there shouldn't be repetitions
(i.e., each combination is unique) and every single well must be
attached to one and only one unit, it seems to me that 3**60 is a very
big number :-D
But combinatorial things are pretty obscure to me...

In any case, thanks to you all for your suggestions.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/


From lists_ravi at lavabit.com  Mon Oct 20 11:32:22 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Mon, 20 Oct 2008 11:32:22 -0400
Subject: [Numpy-discussion] Octave-numpy bridge?
Message-ID: <200810201132.23195.lists_ravi@lavabit.com>

Hi all,
  Is anyone aware of a bridge between octave & numpy? As I port stuff from 
Matlab to numpy, I noticed that most of my Matlab code has workarounds that 
allow the code to be used from octave. My current methodology for porting is 
to use octave to generate inputs/outputs for a function, then write the 
results out in HDF5, read them into python+numpy, and compare the results. 
This would be much faster, especially when dealing with mex files, if I could 
use octave from my ipython shell, a la mlabwrap.
  I searched the web but did not find anything on this topic that was recent.

Regards,
Ravi




From josef.pktd at gmail.com  Mon Oct 20 12:05:11 2008
From: josef.pktd at gmail.com (joep)
Date: Mon, 20 Oct 2008 09:05:11 -0700 (PDT)
Subject: [Numpy-discussion] Combinations of objects (?)
In-Reply-To: <d5ff27200810200802q671e4b9du7298f5d1b74eb864@mail.gmail.com>
References: <d5ff27200810200220u2d46017es58065fd019e71426@mail.gmail.com> 
	<48FC6FFD.3080509@american.edu>
	<d5ff27200810200802q671e4b9du7298f5d1b74eb864@mail.gmail.com>
Message-ID: <0383fcea-aa8c-4797-bebd-ed60eab85d03@v13g2000pro.googlegroups.com>



On Oct 20, 11:02?am, "Andrea Gavana" <andrea.gav... at gmail.com> wrote:
> Hi All,
>
>
>
> On Mon, Oct 20, 2008 at 12:48 PM, Alan G Isaac wrote:
> > On 10/20/2008 5:20 AM Andrea Gavana apparently wrote:
> >> ? ? this is probably a very silly question, but combinatorial math is
> >> not exactly my strength and I am not even sure on how to formulate the
> >> question. I apologize if it is a very elementary problem.
> >> Let's suppose that I have 60 oil wells and 3 surface facilities. Every
> >> well must be tied-in (linked, attached) to one and only one of these
> >> surface facilities. No orphan well is allowed (i.e., a well not
> >> attached to any surface unit) and no multi-links are allowed (i.e.,
> >> one well attached to 2 or more units). Is there any simple way in
> >> numpy (scipy?) in which I can get the number of possible combinations
> >> of wells attached to the different 3 units, without repetitions? For
> >> example, I could have all 60 wells attached to unit number 1, then 59
> >> on unit 1 and 1 on unit 3 and so on...
>
> > I think you are just asking how many ways you
> > can put N marbles into K boxes, with no restrictions.
> > Since there are K possibilities for each marble,
> > wouldn't the answer to your problem be K**N?
>
> Uhm, I am not sure, I imagined TJ answer is correct by reading the
> Python thread he kindly posted. As there shouldn't be repetitions
> (i.e., each combination is unique) and every single well must be
> attached to one and only one unit, it seems to me that 3**60 is a very
> big number :-D
> But combinatorial things are pretty obscure to me...
>
> In any case, thanks to you all for your suggestions.
>
> Andrea.
>
> "Imagination Is The Only Weapon In The War Against Reality."http://xoomer.alice.it/infinity77/
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion


I briefly looked at the thread by TJ and the discussion is on
indistinguishable object, i.e. in your case if you only care about how
many but not which wells are connected to a given facility.

Think of Alans answer in terms of numbers in base 3, you have 60
digits (wells), and each digit can be either 1,2 or 3 (facility to
which well is assigned to). The total number of numbers that can be
represented this way is 3**60.

e.g with 4 wells: 3**4=81 possibilities
1111
1112
1113
1121
1122
1123
1131
...

easy to check for 2 wells: 3**2=9 possibilities

Josef


From oliphant at enthought.com  Mon Oct 20 13:15:56 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 20 Oct 2008 12:15:56 -0500
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <48FB8A71.3010508@mit.edu>
References: <48FB8A71.3010508@mit.edu>
Message-ID: <48FCBCCC.4040909@enthought.com>

Lane Brooks wrote:
> I am using the numpy CAPI to write an extension module that returns a 
> numpy Array from an imaging data source.  I collect the image into a 
> buffer that I allocate. I then create numpy Array using the 
> PyArray_New(..) function and pass it the buffer.  I then set the 
> NPY_OWNDATA flag on the Array because I want the Array to deallocate the 
> buffer when it is deleted.  Is that the correct way to do it?  The code 
> snippet below is what I wrote, and it seems to be working just fine, but 
> I wanted to verify that I am doing things correctly.
>
>   
NPY_OWNDATA means the object will try to deallocate the memory (make 
sure it was allocated with the same allocator as NumPy uses).  
Otherwise, you will need to set up another approach as I showed in my 
blog posting several weeks ago.

Also, don't use Py_BuildValue with "O" as it will create another 
reference so that img will have an extra reference to it when it is 
returned to Python.  Use "N" instead.

However, in this case you don't need to use Py_BuildValue at all because 
you are returning only one array.

The PyArray_UpdateFlags call is not used for changing NPY_OWNDATA.  It 
is only useful for changing FORTRAN, CONTIGUOUS, ALIGNED, and WRITEABLE 
flags which are convenience flags.  This call does the check first and 
then sets the state of the flag to reflect the actual situation for the 
array.

Instead use

PyArray_FLAGS(arr) |= NPY_OWNDATA;

-Travis



From ryanlists at gmail.com  Mon Oct 20 14:50:54 2008
From: ryanlists at gmail.com (Ryan Krauss)
Date: Mon, 20 Oct 2008 13:50:54 -0500
Subject: [Numpy-discussion] windows install problem for 1.2
Message-ID: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>

I need to upgrade my Python installation.  I just uninstalled everything,
deleted C:\Python25, installed Python 2.5.2 from the msi, and am now trying
to install numpy 1.2 from numpy-1.2.0-win32-superpack-python2.5.exe.  It
flashes something up for a second and then goes away giving me no
information.  I tried running it from a cmd.exe prompt to see if some
helpful message spits out, but it says the program is too big to fit in
memory.

I appreciate that this is probably not enough information for anyone to fix
my problem, but I don't know how to get more info.

Are there prereqs besides Python itself to install numpy 1.2?

Thanks,

Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081020/21962b01/attachment.html>

From gael.varoquaux at normalesup.org  Mon Oct 20 15:20:56 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 20 Oct 2008 21:20:56 +0200
Subject: [Numpy-discussion] Memmapping .npy
Message-ID: <20081020192056.GB21981@phare.normalesup.org>

Currently memmapping a .npy file is not implemented. Is there a reason
than lack of time for this? Are there any major difficulties?

Cheers,

Ga?l


From gael.varoquaux at normalesup.org  Mon Oct 20 15:25:31 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 20 Oct 2008 21:25:31 +0200
Subject: [Numpy-discussion] Memmapping .npy
In-Reply-To: <20081020192056.GB21981@phare.normalesup.org>
References: <20081020192056.GB21981@phare.normalesup.org>
Message-ID: <20081020192531.GC21981@phare.normalesup.org>

On Mon, Oct 20, 2008 at 09:20:56PM +0200, Gael Varoquaux wrote:
> Currently memmapping a .npy file is not implemented. Is there a reason
> than lack of time for this? Are there any major difficulties?

Actually, let me rephrase this question to be more clear:

Is the functionnality needed in the numpy.lib.format.open_memmap? If yes
why isn't it exposed in numpy.load? If not, the questions above apply.

Thanks,

Ga?l


From robert.kern at gmail.com  Mon Oct 20 15:27:54 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 20 Oct 2008 14:27:54 -0500
Subject: [Numpy-discussion] Memmapping .npy
In-Reply-To: <20081020192531.GC21981@phare.normalesup.org>
References: <20081020192056.GB21981@phare.normalesup.org>
	<20081020192531.GC21981@phare.normalesup.org>
Message-ID: <3d375d730810201227s7ae8f7f4jab684cdc0bde03da@mail.gmail.com>

On Mon, Oct 20, 2008 at 14:25, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Oct 20, 2008 at 09:20:56PM +0200, Gael Varoquaux wrote:
>> Currently memmapping a .npy file is not implemented. Is there a reason
>> than lack of time for this? Are there any major difficulties?
>
> Actually, let me rephrase this question to be more clear:
>
> Is the functionnality needed in the numpy.lib.format.open_memmap?

Yes.

> If yes
> why isn't it exposed in numpy.load? If not, the questions above apply.

I have no idea. I didn't write that code.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Mon Oct 20 15:30:28 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 20 Oct 2008 21:30:28 +0200
Subject: [Numpy-discussion] Memmapping .npy
In-Reply-To: <3d375d730810201227s7ae8f7f4jab684cdc0bde03da@mail.gmail.com>
References: <20081020192056.GB21981@phare.normalesup.org>
	<20081020192531.GC21981@phare.normalesup.org>
	<3d375d730810201227s7ae8f7f4jab684cdc0bde03da@mail.gmail.com>
Message-ID: <20081020193028.GD21981@phare.normalesup.org>

On Mon, Oct 20, 2008 at 02:27:54PM -0500, Robert Kern wrote:
> > If yes why isn't it exposed in numpy.load? If not, the questions
> > above apply.

> I have no idea. I didn't write that code.

Robert, you know these things way better than me, so please pardon my
stupid question:

  Would it be desirable to expose memmapping in numpy.load? Or is this
  going to drive us into problems?

If it would be desireable, I could try to find time for a patch. I could
use this in my work, and if I am going to implement it, I might as well
do it for everybody.

Ga?l


From robert.kern at gmail.com  Mon Oct 20 15:45:57 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 20 Oct 2008 14:45:57 -0500
Subject: [Numpy-discussion] Memmapping .npy
In-Reply-To: <20081020193028.GD21981@phare.normalesup.org>
References: <20081020192056.GB21981@phare.normalesup.org>
	<20081020192531.GC21981@phare.normalesup.org>
	<3d375d730810201227s7ae8f7f4jab684cdc0bde03da@mail.gmail.com>
	<20081020193028.GD21981@phare.normalesup.org>
Message-ID: <3d375d730810201245l6b235c9dmd7a7b732e0a386a5@mail.gmail.com>

On Mon, Oct 20, 2008 at 14:30, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Oct 20, 2008 at 02:27:54PM -0500, Robert Kern wrote:
>> > If yes why isn't it exposed in numpy.load? If not, the questions
>> > above apply.
>
>> I have no idea. I didn't write that code.
>
> Robert, you know these things way better than me, so please pardon my
> stupid question:
>
>  Would it be desirable to expose memmapping in numpy.load? Or is this
>  going to drive us into problems?
>
> If it would be desireable, I could try to find time for a patch. I could
> use this in my work, and if I am going to implement it, I might as well
> do it for everybody.

load() would need to grow a mode= keyword argument to properly support
memory-mapping. Possibly, we could change it to

def load(filename, mmap_mode=None):
    ...

With mmap_mode=None, just do a plain read; otherwise mmap with that
particular mode. We can introduce that immediately in 1.3 since
memmap=True never worked.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From lbrooks at MIT.EDU  Mon Oct 20 17:13:51 2008
From: lbrooks at MIT.EDU (Lane Brooks)
Date: Mon, 20 Oct 2008 15:13:51 -0600
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <48FCBCCC.4040909@enthought.com>
References: <48FB8A71.3010508@mit.edu> <48FCBCCC.4040909@enthought.com>
Message-ID: <48FCF48F.3020601@mit.edu>

Travis E. Oliphant wrote:
> Lane Brooks wrote:
>   
>> I am using the numpy CAPI to write an extension module that returns a 
>> numpy Array from an imaging data source.  I collect the image into a 
>> buffer that I allocate. I then create numpy Array using the 
>> PyArray_New(..) function and pass it the buffer.  I then set the 
>> NPY_OWNDATA flag on the Array because I want the Array to deallocate the 
>> buffer when it is deleted.  Is that the correct way to do it?  The code 
>> snippet below is what I wrote, and it seems to be working just fine, but 
>> I wanted to verify that I am doing things correctly.
>>
>>   
>>     
> NPY_OWNDATA means the object will try to deallocate the memory (make 
> sure it was allocated with the same allocator as NumPy uses).  
> Otherwise, you will need to set up another approach as I showed in my 
> blog posting several weeks ago.
>
> Also, don't use Py_BuildValue with "O" as it will create another 
> reference so that img will have an extra reference to it when it is 
> returned to Python.  Use "N" instead.
>
> However, in this case you don't need to use Py_BuildValue at all because 
> you are returning only one array.
>
> The PyArray_UpdateFlags call is not used for changing NPY_OWNDATA.  It 
> is only useful for changing FORTRAN, CONTIGUOUS, ALIGNED, and WRITEABLE 
> flags which are convenience flags.  This call does the check first and 
> then sets the state of the flag to reflect the actual situation for the 
> array.
>
> Instead use
>
> PyArray_FLAGS(arr) |= NPY_OWNDATA;
>
> -Travis
>   
Thanks for all this valuable feedback.  I read your blog post and like 
the idea but not the overhead.  I guess my initial approach of doing a 
memory handoff to the numpy Array was a bit naive.  It seems to be 
working, but I guess that is because numpy uses free to deallocate memory?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081020/fbb0e33a/attachment.html>

From robert.kern at gmail.com  Mon Oct 20 17:22:07 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 20 Oct 2008 16:22:07 -0500
Subject: [Numpy-discussion] numpy CAPI questions
In-Reply-To: <48FCF48F.3020601@mit.edu>
References: <48FB8A71.3010508@mit.edu> <48FCBCCC.4040909@enthought.com>
	<48FCF48F.3020601@mit.edu>
Message-ID: <3d375d730810201422u6bb55eb1s4a00b7b2a7ea834c@mail.gmail.com>

On Mon, Oct 20, 2008 at 16:13, Lane Brooks <lbrooks at mit.edu> wrote:
> Thanks for all this valuable feedback.  I read your blog post and like the
> idea but not the overhead.  I guess my initial approach of doing a memory
> handoff to the numpy Array was a bit naive.  It seems to be working, but I
> guess that is because numpy uses free to deallocate memory?

It uses PyMem_Free() (except for a few places that I've just noticed)
which itself uses free() *except* if Python is compiled with
-DPYMALLOC_DEBUG, in which case it uses its own debugging
implementations.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Joris.DeRidder at ster.kuleuven.be  Mon Oct 20 19:05:40 2008
From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder)
Date: Tue, 21 Oct 2008 01:05:40 +0200
Subject: [Numpy-discussion] statistical model fitting & comparison
Message-ID: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>

Hi,

I'm interested in developing some general-use Python/Numpy code for  
linear model fitting and comparison. The fitting is easy enough with  
Numpy, but the automated comparison of the submodels to identify which  
model describes best the data, requires some work. Before I embark on  
this, I would like to find out if anyone actually already wrote  
something along these lines, in order not to reinvent the wheel.

Cheers,
Joris



Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From david at ar.media.kyoto-u.ac.jp  Mon Oct 20 23:30:44 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 12:30:44 +0900
Subject: [Numpy-discussion] Octave-numpy bridge?
In-Reply-To: <200810201132.23195.lists_ravi@lavabit.com>
References: <200810201132.23195.lists_ravi@lavabit.com>
Message-ID: <48FD4CE4.6040602@ar.media.kyoto-u.ac.jp>

Ravi wrote:
> Hi all,
>   Is anyone aware of a bridge between octave & numpy? As I port stuff from 
> Matlab to numpy, I noticed that most of my Matlab code has workarounds that 
> allow the code to be used from octave. My current methodology for porting is 
> to use octave to generate inputs/outputs for a function, then write the 
> results out in HDF5, read them into python+numpy, and compare the results. 
> This would be much faster, especially when dealing with mex files, if I could 
> use octave from my ipython shell, a la mlabwrap.
>   I searched the web but did not find anything on this topic that was recent.
>   

Hi Ravi,

    I am not aware of such a bridge, but if octave has a C "engine" like
matlab, it should be relatively straightfoward to port it from matlab to
octave. I don't know how matlab and octave are compatible at the C API
level, but only a few functions are needed for the low-level wrapper,

David


From david at ar.media.kyoto-u.ac.jp  Mon Oct 20 23:38:34 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 12:38:34 +0900
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
Message-ID: <48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>

Ryan Krauss wrote:
> I need to upgrade my Python installation.  I just uninstalled
> everything, deleted C:\Python25, installed Python 2.5.2 from the msi,
> and am now trying to install numpy 1.2 from
> numpy-1.2.0-win32-superpack-python2.5.exe.  It flashes something up
> for a second and then goes away giving me no information.  I tried
> running it from a cmd.exe prompt to see if some helpful message spits
> out, but it says the program is too big to fit in memory. 

Hi Ryan,

    This is the first time I heard of this problem. I guess you checked
the obvious (do you have enough ram). What is your platform exactly ?
Did you check your installer is ok ? Something I should have done is to
provide for a checksum: here is the checksum for the time being:

ad603ad13cf403fbf394d7e3eba8b996  numpy-1.2.0-win32-superpack-python2.5.exe

> Are there prereqs besides Python itself to install numpy 1.2?

No.

cheers,

David


From millman at berkeley.edu  Tue Oct 21 05:12:19 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 21 Oct 2008 02:12:19 -0700
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>

On Mon, Oct 20, 2008 at 8:38 PM, David Cournapeau :
>    This is the first time I heard of this problem. I guess you checked
> the obvious (do you have enough ram). What is your platform exactly ?
> Did you check your installer is ok ? Something I should have done is to
> provide for a checksum: here is the checksum for the time being:
>
> ad603ad13cf403fbf394d7e3eba8b996  numpy-1.2.0-win32-superpack-python2.5.exe

The md5 checksums are also in the release notes (which you can find by
clicking on the little notepad by the release on the download page).
Here are the release notes for the 1.2.0 release:
http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=628858

Of course, that is terribly difficult to find.  I haven't been able to
figure out a better way to have sourceforge display checksum
information.  Ideally, they could display the checksum information
like they do for the filesize and architecture information:
http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103&release_id=628858

If anyone knows how to make sourceforge display this information,
please let me know.

I would also like it if you could somehow embed the checksum in the
URL.  Maybe something like this:
https://wiki.mozilla.org/Firefox/Feature_Brainstorming:Downloads#MD5_Checksum

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From faltet at pytables.org  Tue Oct 21 05:23:13 2008
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 21 Oct 2008 11:23:13 +0200
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
	<c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>
Message-ID: <200810211123.13228.faltet@pytables.org>

A Tuesday 21 October 2008, Jarrod Millman escrigu?:
> On Mon, Oct 20, 2008 at 8:38 PM, David Cournapeau :
> >    This is the first time I heard of this problem. I guess you
> > checked the obvious (do you have enough ram). What is your platform
> > exactly ? Did you check your installer is ok ? Something I should
> > have done is to provide for a checksum: here is the checksum for
> > the time being:
> >
> > ad603ad13cf403fbf394d7e3eba8b996 
> > numpy-1.2.0-win32-superpack-python2.5.exe
>
> The md5 checksums are also in the release notes (which you can find
> by clicking on the little notepad by the release on the download
> page). Here are the release notes for the 1.2.0 release:
> http://sourceforge.net/project/shownotes.php?group_id=1369&release_id
>=628858
>
> Of course, that is terribly difficult to find.  I haven't been able
> to figure out a better way to have sourceforge display checksum
> information.  Ideally, they could display the checksum information
> like they do for the filesize and architecture information:
> http://sourceforge.net/project/showfiles.php?group_id=1369&package_id
>=175103&release_id=628858
>
> If anyone knows how to make sourceforge display this information,
> please let me know.
>
> I would also like it if you could somehow embed the checksum in the
> URL.  Maybe something like this:
> https://wiki.mozilla.org/Firefox/Feature_Brainstorming:Downloads#MD5_
>Checksum

I normally put the MD5 information in a separate file in the same 
repository.  Something like:

http://www.pytables.org/download/stable/
Parent Directory	 	 -
pytables-2.0.4.md5	04-Jul-2008 14:23 	249
pytables-2.0.4.tar.gz	04-Jul-2008 14:22 	3.3M
pytablesmanual-2.0.4-html.tar.gz	04-Jul-2008 14:22 	1.7M
pytablesmanual-2.0.4.pdf	04-Jul-2008 15:27 	1.2M
tables-2.0.4.tar.gz	04-Jul-2008 14:22 	3.3M
tables-2.0.4.win32-py2.4.exe	04-Jul-2008 14:22 	1.0M
tables-2.0.4.win32-py2.5.exe	04-Jul-2008 14:22 	1.0M

where the MD5 hashes are in pytables-2.0.4.md5.  The MD5 signatures can 
be computed easily by using the 'cfv' utility:

cfv -C -t md5 -f pytables-$VERSION.md5 *

HTH,

-- 
Francesc Alted


From millman at berkeley.edu  Tue Oct 21 05:39:30 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 21 Oct 2008 02:39:30 -0700
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <200810211123.13228.faltet@pytables.org>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
	<c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>
	<200810211123.13228.faltet@pytables.org>
Message-ID: <c7009a550810210239j5db0f214q9d10998a94332cbc@mail.gmail.com>

On Tue, Oct 21, 2008 at 2:23 AM, Francesc Alted <faltet at pytables.org> wrote:
> I normally put the MD5 information in a separate file in the same
> repository.

I was thinking about doing something like that, but I have been trying
to minimize the number of files that I upload to sourceforge, because
each file requires a ridiculous number of mouse clicks.  The whole
process of making a file release on sourceforge is annoyingly manual.

Of course, I should actually just figure out how to script making a
release on sourceforge.  I have seen a couple of tools that look like
they might be useful:

Releaseforge:
http://releaseforge.sourceforge.net/

Sourceforge Utilities:
http://sfutils.sourceforge.net/

Has anyone found a good way of scripting sourceforge?

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david at ar.media.kyoto-u.ac.jp  Tue Oct 21 05:28:27 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 18:28:27 +0900
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <c7009a550810210239j5db0f214q9d10998a94332cbc@mail.gmail.com>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>	<c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>	<200810211123.13228.faltet@pytables.org>
	<c7009a550810210239j5db0f214q9d10998a94332cbc@mail.gmail.com>
Message-ID: <48FDA0BB.6040706@ar.media.kyoto-u.ac.jp>

Jarrod Millman wrote:
>
> Has anyone found a good way of scripting sourceforge?
>   

What do we use sourceforge for now ? It seems that we only use it for
the source/installers archives, now, right ? We use sourceforge neither
for ML, Forums, bug tracking or source code control, so why using it at
all ?

cheers,

David


From millman at berkeley.edu  Tue Oct 21 06:11:55 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 21 Oct 2008 03:11:55 -0700
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <48FDA0BB.6040706@ar.media.kyoto-u.ac.jp>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
	<c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>
	<200810211123.13228.faltet@pytables.org>
	<c7009a550810210239j5db0f214q9d10998a94332cbc@mail.gmail.com>
	<48FDA0BB.6040706@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550810210311l6773dda8k219d51cc14108d9b@mail.gmail.com>

On Tue, Oct 21, 2008 at 2:28 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> What do we use sourceforge for now ? It seems that we only use it for
> the source/installers archives, now, right ? We use sourceforge neither
> for ML, Forums, bug tracking or source code control, so why using it at
> all ?

I don't mind hosting them somewhere else.  Do you have some place in mind?

Google Code looks like it has a 10MB file size limit:
http://code.google.com/support/bin/answer.py?answer=56621&topic=10456

Launchpad seems to be considering upping their limit to 200MB:
https://bugs.launchpad.net/launchpad-foundations/+bug/254052

The scipy server can't handle the load or bandwidth needed.

My first inclination would be to look at Launchpad:
https://launchpad.net/numpy
https://launchpad.net/scipy

How do you like releasing files on launchpad?  Can you write a script
to upload release files and notes?  Can you script making the
announcement?

It looks like they md5 support builtin:
https://code.launchpad.net/numpy.scons.support/+download

Ideas?  Comments?

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david at ar.media.kyoto-u.ac.jp  Tue Oct 21 06:12:37 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 19:12:37 +0900
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <c7009a550810210311l6773dda8k219d51cc14108d9b@mail.gmail.com>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>	<c7009a550810210212qc231d89wfea6b56d6e1e8c16@mail.gmail.com>	<200810211123.13228.faltet@pytables.org>	<c7009a550810210239j5db0f214q9d10998a94332cbc@mail.gmail.com>	<48FDA0BB.6040706@ar.media.kyoto-u.ac.jp>
	<c7009a550810210311l6773dda8k219d51cc14108d9b@mail.gmail.com>
Message-ID: <48FDAB15.3020000@ar.media.kyoto-u.ac.jp>

Jarrod Millman wrote:
>
> I don't mind hosting them somewhere else.  Do you have some place in mind?
>   

I was wondering why scipy.org could not be used :)

> Google Code looks like it has a 10MB file size limit:
> http://code.google.com/support/bin/answer.py?answer=56621&topic=10456
>
> Launchpad seems to be considering upping their limit to 200MB:
> https://bugs.launchpad.net/launchpad-foundations/+bug/254052
>
> The scipy server can't handle the load or bandwidth needed.
>
> My first inclination would be to look at Launchpad:
> https://launchpad.net/numpy
> https://launchpad.net/scipy
>
> How do you like releasing files on launchpad? 

I found it cumbersome enough so that I stopped using it, and only
uploaded it through pypi (which is straightfoward to use, but has a
limit of 5 Mb I believe) for numscons.

Now, since they have started having a public API since this summer,
there is hope for a scriptable process:

http://news.launchpad.net/api/recipe-for-uploading-files-via-the-api

But the beta for the public API is not public yet (it is not difficult
to apply, though, it takes an email + a couple of days, at least that
was the case a few weeks ago). Also, there are still no real tools
around it yet.

cheers,

David


From josef.pktd at gmail.com  Tue Oct 21 09:27:13 2008
From: josef.pktd at gmail.com (joep)
Date: Tue, 21 Oct 2008 06:27:13 -0700 (PDT)
Subject: [Numpy-discussion] Octave-numpy bridge?
In-Reply-To: <48FD4CE4.6040602@ar.media.kyoto-u.ac.jp>
References: <200810201132.23195.lists_ravi@lavabit.com>
	<48FD4CE4.6040602@ar.media.kyoto-u.ac.jp>
Message-ID: <43384ba4-5daa-4754-8f3c-dfaa991cb39e@b1g2000hsg.googlegroups.com>



On Oct 20, 11:30?pm, David Cournapeau <da... at ar.media.kyoto-u.ac.jp>
wrote:
> Ravi wrote:
> > Hi all,
> > ? Is anyone aware of a bridge between octave & numpy? As I port stuff from
> > Matlab to numpy, I noticed that most of my Matlab code has workarounds that
> > allow the code to be used from octave. My current methodology for porting is
> > to use octave to generate inputs/outputs for a function, then write the
> > results out in HDF5, read them into python+numpy, and compare the results.
> > This would be much faster, especially when dealing with mex files, if I could
> > use octave from my ipython shell, a la mlabwrap.
> > ? I searched the web but did not find anything on this topic that was recent.
>
> Hi Ravi,
>

Hi,
I just saw this recently, but I have not yet looked at it more
closely
(since I am currently using matlab.)
The install explanation in http://bazaar.launchpad.net/~individ/pytave/trunk/files
looks very concentrated on Posix and I'm a windows user.

However, this seems what you are looking for.


see http://www.nabble.com/Python-to-Octave-bridge-td20031139.html

Announcing Pytave - Python to Octave extension

Embeds the Octave language interpreter as an extension to Python,
enabling existing m-files to be used from Python.

Features:
 * Implicit type conversions between Python and Octave. Supports all
Numeric integer, real double (and possibly real float) matrices
 * Architecture independent - no assumption on endianess or integer
sizes

Call Octave code in Python:
 >>> import pytave
 >>> pytave.feval(1, "cos", 0)
(1.0,)

Project homepage:
https://launchpad.net/pytave




From fadhley.salim at uk.calyon.com  Tue Oct 21 09:41:03 2008
From: fadhley.salim at uk.calyon.com (Fadhley Salim)
Date: Tue, 21 Oct 2008 14:41:03 +0100
Subject: [Numpy-discussion] Building a Win32/Python2.4 multi-version egg for
	Numpy
Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204483DF1@MXCU10MX1.MSX.CIB>

Is there a way to make a multi-version egg from the Numpy source code
(supplied from sourceforge). 

Ideally I'd like to create an automated process that allows me to
quickly make a Win32 egg any time a new release of Numpy comes out so
that my team can test it and adjust our project's dependancies as they
see fit.

Initially I'd like to make multiversion eggs for both Numpy 1.1 and 1.2
so that my team can get started on testing this library. Is there
anybody here who has managed to build an egg from Numpy?

Thanks,

Sal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/cb5e4d5a/attachment.html>
-------------- next part --------------
This email does not create a legal relationship between any member of the 
Cr?dit Agricole group and the recipient or constitute investment advice. 
The content of this email should not be copied or disclosed (in whole or 
part) to any other person. It may contain information which is 
confidential, privileged or otherwise protected from disclosure. If you 
are not the intended recipient, you should notify us and delete it from 
your system. Emails may be monitored, are not secure and may be amended, 
destroyed or contain viruses and in communicating with us such conditions 
are accepted. Any content which does not relate to business matters is not 
endorsed by us.

Calyon is authorised by the Commission Bancaire in France and regulated by 
the Financial Services Authority for the conduct of business in the 
United Kingdom. Calyon is incorporated in France with limited liability 
and registered in England & Wales. Registration number: FC008194. 
Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA.


From robert.kern at gmail.com  Tue Oct 21 09:58:24 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 21 Oct 2008 08:58:24 -0500
Subject: [Numpy-discussion] Building a Win32/Python2.4 multi-version egg
	for Numpy
In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204483DF1@MXCU10MX1.MSX.CIB>
References: <7F347D91614EBC48AA7540A2A76D3BB204483DF1@MXCU10MX1.MSX.CIB>
Message-ID: <3d375d730810210658l4ca6e4e6s57b4eb64cd2e41@mail.gmail.com>

On Tue, Oct 21, 2008 at 08:41, Fadhley Salim
<fadhley.salim at uk.calyon.com> wrote:
> Is there a way to make a multi-version egg from the Numpy source code
> (supplied from sourceforge).
>
> Ideally I'd like to create an automated process that allows me to quickly
> make a Win32 egg any time a new release of Numpy comes out so that my team
> can test it and adjust our project's dependancies as they see fit.
>
> Initially I'd like to make multiversion eggs for both Numpy 1.1 and 1.2 so
> that my team can get started on testing this library. Is there anybody here
> who has managed to build an egg from Numpy?

python setupegg.py bdist_egg

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Tue Oct 21 10:23:06 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 21 Oct 2008 16:23:06 +0200
Subject: [Numpy-discussion] statistical model fitting & comparison
In-Reply-To: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>
References: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>
Message-ID: <9457e7c80810210723j2b1d78ebhd0010934a90a1925@mail.gmail.com>

Hi Joris

2008/10/21 Joris De Ridder <Joris.DeRidder at ster.kuleuven.be>:
> I'm interested in developing some general-use Python/Numpy code for
> linear model fitting and comparison. The fitting is easy enough with
> Numpy, but the automated comparison of the submodels to identify which
> model describes best the data, requires some work. Before I embark on
> this, I would like to find out if anyone actually already wrote
> something along these lines, in order not to reinvent the wheel.

I would guess that this is part of Jonathan Taylor's "models" package,
but I may be wrong.  It is available at the Neuroimaging in Python
homepage:

http://neuroimaging.scipy.org/

You may also want to look at the optimisation scikit and the RANSAC
cookbook example.

Regards
St?fan


From ryanlists at gmail.com  Tue Oct 21 10:25:58 2008
From: ryanlists at gmail.com (Ryan Krauss)
Date: Tue, 21 Oct 2008 09:25:58 -0500
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
Message-ID: <c5b438120810210725j40b094a8w9c3fa2376d3b81f4@mail.gmail.com>

OK, I was sure this couldn't be the problem since I downloaded it twice and
also downloaded the 1.1 superpack and had the same issue, but here is my
md5sum:

In [9]: md5sum.md5sum('numpy-1.2.0-win32-superpack-python2.5.exe')
Out[9]: '76ebe1fb78694e31e81e3e696be1e310'

which is different.  I will try turning off all virus and firewall stuff and
try again.  How does this happen?

Ryan

On 10/20/08, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>
> Ryan Krauss wrote:
> > I need to upgrade my Python installation.  I just uninstalled
> > everything, deleted C:\Python25, installed Python 2.5.2 from the msi,
> > and am now trying to install numpy 1.2 from
> > numpy-1.2.0-win32-superpack-python2.5.exe.  It flashes something up
> > for a second and then goes away giving me no information.  I tried
> > running it from a cmd.exe prompt to see if some helpful message spits
> > out, but it says the program is too big to fit in memory.
>
>
> Hi Ryan,
>
>     This is the 1.1 first time I heard of this problem. I guess you checked
> the obvious (do you have enough ram). What is your platform exactly ?
> Did you check your installer is ok ? Something I should have done is to
> provide for a checksum: here is the checksum for the time being:
>
> ad603ad13cf403fbf394d7e3eba8b996  numpy-1.2.0-win32-superpack-python2.5.exe
>
>
> > Are there prereqs besides Python itself to install numpy 1.2?
>
>
> No.
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/5273c620/attachment.html>

From fadhley.salim at uk.calyon.com  Tue Oct 21 10:28:21 2008
From: fadhley.salim at uk.calyon.com (Fadhley Salim)
Date: Tue, 21 Oct 2008 15:28:21 +0100
Subject: [Numpy-discussion] Numpy Compile error on Win32 with Visual
	Studio.Net 2003
Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204483DF2@MXCU10MX1.MSX.CIB>

Can one of the Numpy experts explain what this means and how I might be
able to solve it? I'm trying to compile Numpy 1.2 - I just zipped the
tar file from Sourceforge.
 
Thanks!
 
compiling C sources
creating build\temp.win32-2.4\Release\numpy\random
creating build\temp.win32-2.4\Release\numpy\random\mtrand
C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c
/nologo /Ox /MD /W3 /GX /DNDEBUG -Inumpy\core\inclu
de -Ibuild\src.win32-2.4\numpy\core\include/numpy -Inumpy\core\src
-Inumpy\core\include -ID:\Python24\include -ID:\Pytho
n24\PC /Tcnumpy\random\mtrand\mtrand.c
/Fobuild\temp.win32-2.4\Release\numpy\random\mtrand\mtrand.obj
mtrand.c
numpy\random\mtrand\mtrand.c(2829) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(3182) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(5043) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(5192) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(5324) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(5739) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(5739) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(6041) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(6041) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(7485) : error C2026: string too big,
trailing characters truncated
numpy\random\mtrand\mtrand.c(8390) : error C2026: string too big,
trailing characters truncated
error: Command "C:\Program Files\Microsoft Visual Studio .NET
2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -I
numpy\core\include -Ibuild\src.win32-2.4\numpy\core\include/numpy
-Inumpy\core\src -Inumpy\core\include -ID:\Python24\in
clude -ID:\Python24\PC /Tcnumpy\random\mtrand\mtrand.c
/Fobuild\temp.win32-2.4\Release\numpy\random\mtrand\mtrand.obj" f
ailed with exit status 2
------------------------------------------------------------------------
Press ENTER to close the interactive session..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/75b00364/attachment.html>
-------------- next part --------------
This email does not create a legal relationship between any member of the 
Cr?dit Agricole group and the recipient or constitute investment advice. 
The content of this email should not be copied or disclosed (in whole or 
part) to any other person. It may contain information which is 
confidential, privileged or otherwise protected from disclosure. If you 
are not the intended recipient, you should notify us and delete it from 
your system. Emails may be monitored, are not secure and may be amended, 
destroyed or contain viruses and in communicating with us such conditions 
are accepted. Any content which does not relate to business matters is not 
endorsed by us.

Calyon is authorised by the Commission Bancaire in France and regulated by 
the Financial Services Authority for the conduct of business in the 
United Kingdom. Calyon is incorporated in France with limited liability 
and registered in England & Wales. Registration number: FC008194. 
Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA.


From david at ar.media.kyoto-u.ac.jp  Tue Oct 21 10:15:41 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 23:15:41 +0900
Subject: [Numpy-discussion] Numpy Compile error on Win32 with
 Visual	Studio.Net 2003
In-Reply-To: <7F347D91614EBC48AA7540A2A76D3BB204483DF2@MXCU10MX1.MSX.CIB>
References: <7F347D91614EBC48AA7540A2A76D3BB204483DF2@MXCU10MX1.MSX.CIB>
Message-ID: <48FDE40D.2000408@ar.media.kyoto-u.ac.jp>

Fadhley Salim wrote:
> Can one of the Numpy experts explain what this means and how I might
> be able to solve it? I'm trying to compile Numpy 1.2 - I just zipped
> the tar file from Sourceforge.

numpy 1.2 is not buildable with visual studio 2003, because of a
compiler bug.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Oct 21 10:16:25 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 21 Oct 2008 23:16:25 +0900
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <c5b438120810210725j40b094a8w9c3fa2376d3b81f4@mail.gmail.com>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
	<c5b438120810210725j40b094a8w9c3fa2376d3b81f4@mail.gmail.com>
Message-ID: <48FDE439.9020902@ar.media.kyoto-u.ac.jp>

Ryan Krauss wrote:
> OK, I was sure this couldn't be the problem since I downloaded it
> twice and also downloaded the 1.1 superpack and had the same issue,
> but here is my md5sum:
>
> In [9]: md5sum.md5sum('numpy-1.2.0-win32-superpack-python2.5.exe')
> Out[9]: '76ebe1fb78694e31e81e3e696be1e310'
>
> which is different.  I will try turning off all virus and firewall
> stuff and try again.  How does this happen?

What do you use to download the executable ?

cheers,

David


From lists_ravi at lavabit.com  Tue Oct 21 10:39:00 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Tue, 21 Oct 2008 10:39:00 -0400
Subject: [Numpy-discussion] Octave-numpy bridge?
In-Reply-To: <43384ba4-5daa-4754-8f3c-dfaa991cb39e@b1g2000hsg.googlegroups.com>
References: <200810201132.23195.lists_ravi@lavabit.com>
	<48FD4CE4.6040602@ar.media.kyoto-u.ac.jp>
	<43384ba4-5daa-4754-8f3c-dfaa991cb39e@b1g2000hsg.googlegroups.com>
Message-ID: <200810211039.01639.lists_ravi@lavabit.com>

On Tuesday 21 October 2008 09:27:13 joep wrote:
> However, this seems what you are looking for.
>
> see http://www.nabble.com/Python-to-Octave-bridge-td20031139.html
>
> Announcing Pytave - Python to Octave extension
>
> Embeds the Octave language interpreter as an extension to Python,
> enabling existing m-files to be used from Python.

Looks interesting; this is very much the same approach I used in a few 
postings on c++-sig (now cplusplus-sig) two weeks ago to build a transparent 
ublas-numpy bridge. I am not familiar with the Octave engine, but this would 
probably be the simplest way to make it work.

Thank you very much for the pointer.

Regards,
Ravi
 



From ryanlists at gmail.com  Tue Oct 21 11:27:30 2008
From: ryanlists at gmail.com (Ryan Krauss)
Date: Tue, 21 Oct 2008 10:27:30 -0500
Subject: [Numpy-discussion] windows install problem for 1.2
In-Reply-To: <48FDE439.9020902@ar.media.kyoto-u.ac.jp>
References: <c5b438120810201150r3b11d937m3c9c1d89a8b121ae@mail.gmail.com>
	<48FD4EBA.7090507@ar.media.kyoto-u.ac.jp>
	<c5b438120810210725j40b094a8w9c3fa2376d3b81f4@mail.gmail.com>
	<48FDE439.9020902@ar.media.kyoto-u.ac.jp>
Message-ID: <c5b438120810210827o13ba9e31mc9f3c51950c0c0ca@mail.gmail.com>

Firefox.

After turning off my firewall and antivirus software, I re-downloaded, got
the correct md5sum, and everything is fine.

Thanks for your help.

Ryan

On 10/21/08, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>
> Ryan Krauss wrote:
> > OK, I was sure this couldn't be the problem since I downloaded it
> > twice and also downloaded the 1.1 superpack and had the same issue,
> > but here is my md5sum:
> >
> > In [9]: md5sum.md5sum('numpy-1.2.0-win32-superpack-python2.5.exe')
> > Out[9]: '76ebe1fb78694e31e81e3e696be1e310'
> >
> > which is different.  I will try turning off all virus and firewall
> > stuff and try again.  How does this happen?
>
>
> What do you use to download the executable ?
>
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/e425f8bf/attachment.html>

From v.gkinis at gfy.ku.dk  Tue Oct 21 11:30:23 2008
From: v.gkinis at gfy.ku.dk (Vasileios Gkinis)
Date: Tue, 21 Oct 2008 17:30:23 +0200
Subject: [Numpy-discussion] detecting a nan with all() or any()
Message-ID: <48FDF58F.6090207@gfy.ku.dk>

Hi,

I would like to detect if my array for example a = array((1,2,3,nan)) 
contains any nans
So when i use:
In [31]: all(a!=nan)
I get
Out[31]: True
And when i use:
In [35]: any(a==nan)
I get
Out[35]: False
Which looks rather wrong to me...or i am simply missing something
Can somebody comment on that...?

Regards
Vasili


-- 

------------------------------------------------------------

Vasileios Gkinis, PhD Student

Centre for Ice and Climate

Niels Bohr Institute

Juliane Maries Vej 30, room 321

DK-2100 Copenhagen

Denmark

Office: +45 35325913

v.gkinis at gfy.ku.dk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/fea172b1/attachment.html>

From robert.kern at gmail.com  Tue Oct 21 12:07:35 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 21 Oct 2008 11:07:35 -0500
Subject: [Numpy-discussion] detecting a nan with all() or any()
In-Reply-To: <48FDF58F.6090207@gfy.ku.dk>
References: <48FDF58F.6090207@gfy.ku.dk>
Message-ID: <3d375d730810210907v406e4142tb2970addb0b7ae69@mail.gmail.com>

On Tue, Oct 21, 2008 at 10:30, Vasileios Gkinis <v.gkinis at gfy.ku.dk> wrote:
> Hi,
>
> I would like to detect if my array for example a = array((1,2,3,nan))
> contains any nans
> So when i use:
> In [31]: all(a!=nan)
> I get
> Out[31]: True
> And when i use:
> In [35]: any(a==nan)
> I get
> Out[35]: False
> Which looks rather wrong to me...or i am simply missing something
> Can somebody comment on that...?

One of the properties of nans is that nan!=nan. Instead, use the
function isnan().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rmay31 at gmail.com  Tue Oct 21 15:30:35 2008
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 21 Oct 2008 14:30:35 -0500
Subject: [Numpy-discussion] Loadtxt .bz2 support
Message-ID: <48FE2DDB.8070808@gmail.com>

Hi,

I noticed numpy.loadtxt has support for gzipped text files, but not for
bz2'd files.  Here's a 3 line patch to add bzip2 support to loadtxt.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: loadtxt_bzip2_support.diff
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/65b0c2a3/attachment.ksh>

From bsouthey at gmail.com  Tue Oct 21 17:01:10 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 21 Oct 2008 16:01:10 -0500
Subject: [Numpy-discussion] statistical model fitting & comparison
In-Reply-To: <9457e7c80810210723j2b1d78ebhd0010934a90a1925@mail.gmail.com>
References: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>
	<9457e7c80810210723j2b1d78ebhd0010934a90a1925@mail.gmail.com>
Message-ID: <48FE4316.6020905@gmail.com>

St?fan van der Walt wrote:
> Hi Joris
>
> 2008/10/21 Joris De Ridder <Joris.DeRidder at ster.kuleuven.be>:
>   
>> I'm interested in developing some general-use Python/Numpy code for
>> linear model fitting and comparison. The fitting is easy enough with
>> Numpy, but the automated comparison of the submodels to identify which
>> model describes best the data, requires some work. Before I embark on
>> this, I would like to find out if anyone actually already wrote
>> something along these lines, in order not to reinvent the wheel.
>>     
>
> I would guess that this is part of Jonathan Taylor's "models" package,
> but I may be wrong.  It is available at the Neuroimaging in Python
> homepage:
>
> http://neuroimaging.scipy.org/
>
> You may also want to look at the optimisation scikit and the RANSAC
> cookbook example.
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   
Hi,
I think you are on your own here as it is a huge chunk to chew! 
Depending on what you really mean by linear models is also part of that 
(the Wikipedia entry is amusing). Most people probably to stats 
applications like lm in R and glm in SAS.

I do have an interest and the code I have (for logistic regression) is 
probably just as easy to rewrite.

Bruce



From bioinformed at gmail.com  Tue Oct 21 17:38:20 2008
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Tue, 21 Oct 2008 17:38:20 -0400
Subject: [Numpy-discussion] statistical model fitting & comparison
In-Reply-To: <48FE4316.6020905@gmail.com>
References: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>
	<9457e7c80810210723j2b1d78ebhd0010934a90a1925@mail.gmail.com>
	<48FE4316.6020905@gmail.com>
Message-ID: <2e1434c10810211438o1737b44udcb05dd6d3594f71@mail.gmail.com>

On Tue, Oct 21, 2008 at 5:01 PM, Bruce Southey <bsouthey at gmail.com> wrote:

> I think you are on your own here as it is a huge chunk to chew!
> Depending on what you really mean by linear models is also part of that
> (the Wikipedia entry is amusing). Most people probably to stats
> applications like lm in R and glm in SAS.
>
> I do have an interest and the code I have (for logistic regression) is
> probably just as easy to rewrite.
>


I also have some fairly robust code for logistic and polytomous regression
based on NumPy and SciPy.

-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/17276ae5/attachment.html>

From travis at enthought.com  Tue Oct 21 17:53:51 2008
From: travis at enthought.com (Travis Vaught)
Date: Tue, 21 Oct 2008 16:53:51 -0500
Subject: [Numpy-discussion] ANN: Enthought Python Distribution - New Release
Message-ID: <02DFC911-B86B-4F92-A83F-1FB7AADAB98B@enthought.com>

Greetings,

Enthought, Inc. is very pleased to announce the newest release of the  
Enthought Python Distribution (EPD) Py2.5 v4.0.30002:

http://www.enthought.com/epd

This release contains updates to many of EPD's packages, including  
NumPy, IPython, matplotlib, VTK, etc.  This is also the first release  
to include a 3.x version of the Enthought Tool Suite (http://code.enthought.com/ 
).

The release notes for this release, including the list of included  
packages, may be found here:
https://svn.enthought.com/epd/wiki/Python2.5.2/4.0.300/GA

Many thanks to the EPD team for putting this release together, and to  
the community of folks who have provided all of the valuable tools  
bundled here.

Best Regards,

Travis


---------
About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included"  
distribution of the Python? Programming Language, including over 80  
additional tools and libraries. The EPD bundle includes NumPy, SciPy,  
IPython, 2D and 3D visualization, database adapters, and a lot of  
other tools right out of the box.

http://www.enthought.com/products/epd.php

It is currently available as an easy, single-click installer for  
Windows XP (x86), Mac OS X (a universal binary for Intel 10.4 and  
above) and RedHat EL3 (x86 and amd64).

EPD is free for 30-day trial use and for use in degree-granting  
academic institutions.  An annual Subscription and installation  
support are available for commercial use (http://www.enthought.com/products/epddownload.php 
) including an Enterprise Subscription with support for particular  
deployment environments (http://www.enthought.com/products/enterprise.php 
).



From paul at rudin.co.uk  Tue Oct 21 19:12:59 2008
From: paul at rudin.co.uk (Paul Rudin)
Date: Wed, 22 Oct 2008 00:12:59 +0100
Subject: [Numpy-discussion] spherical harmonic decomposition of 3dmodels
Message-ID: <877i81h79g.fsf@rudin.co.uk>



I'm trying to learn to use numpy/scipy, and at the moment I'm looking at
implementing a spherical harmonic shape descriptor for a 3d
model. Something along the lines described on page 8 of
<http://www.cs.princeton.edu/~funk/tog03.pdf>.

I've made code that does the voxelization bit from some geometry. That
is I've got something that calulates an NxNxN array with 1s at the
points near the meshes of the original model, and zeros elsewhere.

Using this grid I've got a function taking polar coordinates to 1 or 0,
but I've so far failed to use that function to make the 2D grid giving
the descriptor.

scipy.special.sph_harm gives me the spherical harmonic for a given
degree and coordinates, and my function can obviously be sampled at any
points, but putting them together so far eludes me. 

Any hints gratefully received...




From bevan07 at gmail.com  Tue Oct 21 21:51:13 2008
From: bevan07 at gmail.com (Bevan Jenkins)
Date: Wed, 22 Oct 2008 01:51:13 +0000 (UTC)
Subject: [Numpy-discussion] Timeseries scikit - dates prior to 1900
Message-ID: <loom.20081022T013653-2@post.gmane.org>

Hello,

I have just stumbled into a slight issue with scikits.timeseries and dates 
prior to 1900.  Timeseries requires dates to be >= 1900.  It took me a little 
while to discover this because I was trying to create dates at a frequency 
greater than daily (e.g. hourly) which leads to python crashing.  Once I 
reduced the frequency to daily, the error let me know about the 1900 issue as 
below:
In [1]: import scikits.timeseries as ts

In [2]: D = ts.Date(freq='D', year = 1850, month = 1, day =1)

In [3]: print (D)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

ValueError: year=1850 is before 1900; the datetime strftime() methods require 
year >= 1900

This is what i was trying initially:

In [4]: E = ts.Date(freq='H', year = 1850, month = 1, day =1, hour =0)

In [5]: print (E)

<<python.exe crashes>>


I was wondering if there is anyway to use timeseries with dates <1900?
If not I will forgo timeseries for the select few datasets I have with data 
<1900.

Thanks for the v. useful scikit,

Bevan Jenkins





From charlesr.harris at gmail.com  Tue Oct 21 23:59:40 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 21 Oct 2008 21:59:40 -0600
Subject: [Numpy-discussion] Loadtxt .bz2 support
In-Reply-To: <48FE2DDB.8070808@gmail.com>
References: <48FE2DDB.8070808@gmail.com>
Message-ID: <e06186140810212059t529efe0eoa478f2cfa98b8fb4@mail.gmail.com>

On Tue, Oct 21, 2008 at 1:30 PM, Ryan May <rmay31 at gmail.com> wrote:

> Hi,
>
> I noticed numpy.loadtxt has support for gzipped text files, but not for
> bz2'd files.  Here's a 3 line patch to add bzip2 support to loadtxt.
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
>
> Index: numpy/lib/io.py
> ===================================================================
> --- numpy/lib/io.py     (revision 5953)
> +++ numpy/lib/io.py     (working copy)
> @@ -320,6 +320,9 @@
>         if fname.endswith('.gz'):
>             import gzip
>             fh = gzip.open(fname)
> +        elif fname.endswith('.bz2'):
> +            import bz2
> +            fh = bz2.BZ2File(fname)
>         else:
>             fh = file(fname)
>     elif hasattr(fname, 'seek'):
>

Could you open a ticket for this? Mark it as an enhancement.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081021/77be375d/attachment.html>

From opossumnano at gmail.com  Wed Oct 22 05:04:23 2008
From: opossumnano at gmail.com (Tiziano Zito)
Date: Wed, 22 Oct 2008 11:04:23 +0200
Subject: [Numpy-discussion] Modular toolkit for Data Processing 2.4 released!
Message-ID: <20081022090423.GA18890@localhost>

We are glad to announce release 2.4 of the Modular toolkit for Data
Processing (MDP).

MDP is a Python library of widely used data processing algorithms that
can be combined according to a pipeline analogy to build more complex
data processing software. The base of available algorithms includes,
to name but the most common, Principal Component Analysis (PCA and
NIPALS), several Independent Component Analysis algorithms (CuBICA,
FastICA, TDSEP, and JADE), Slow Feature Analysis, Restricted Boltzmann
Machine, and Locally Linear Embedding.

What's new in version 2.4?
--------------------------------------

- The new version introduces a new parallel package to execute the MDP
  algorithms on multiple processors or machines. The package also offers
  an interface to develop customized schedulers and parallel algorithms.
  Old MDP scripts can be turned into their parallelized equivalent with
  one simple command.

- The number of available algorithms is increased with the Locally
  Linear Embedding and Hessian eigenmaps algorithms to perform
  dimensionality reduction and manifold learning (many thanks to Jake
  VanderPlas for his contribution!)

- Some more bug fixes, useful features, and code migration towards
  Python 3.0

Resources
---------
Download: http://sourceforge.net/project/showfiles.php?group_id=116959
Homepage: http://mdp-toolkit.sourceforge.net
Mailing list: http://sourceforge.net/mail/?group_id=116959

--

 Pietro Berkes
 Volen Center for Complex Systems
 Brandeis University
 Waltham, MA, USA

 Niko Wilbert
 Institute for Theoretical Biology
 Humboldt-University
 Berlin, Germany

 Tiziano Zito
 Bernstein Center for Computational Neuroscience
 Humboldt-University
 Berlin, Germany



From david at ar.media.kyoto-u.ac.jp  Wed Oct 22 06:36:04 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 22 Oct 2008 19:36:04 +0900
Subject: [Numpy-discussion] scipy super pack installer for win32: please test
Message-ID: <48FF0214.4080302@ar.media.kyoto-u.ac.jp>

Hi,

    A quick note to mention I have generated a "superpack" installer for
scipy, for testing purposes. This is similar to numpy superpack
installer: the installer detects your CPU at installation time and
install the right scipy:

http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0.dev4826-win32-superpack-python2.5.exe

How to test:
------------

Inside python:

import scipy
scipy.test()

If python does not crash, this is OK. Test failures are OK, since those
will be fixed within the 0.7.0 release. Please report any problem on
scipy-dev ML,

cheers,

David


From Joris.DeRidder at ster.kuleuven.be  Wed Oct 22 06:56:29 2008
From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder)
Date: Wed, 22 Oct 2008 12:56:29 +0200
Subject: [Numpy-discussion] statistical model fitting & comparison
In-Reply-To: <2e1434c10810211438o1737b44udcb05dd6d3594f71@mail.gmail.com>
References: <00509E0B-1D22-4F32-883B-4A58025016A2@ster.kuleuven.be>
	<9457e7c80810210723j2b1d78ebhd0010934a90a1925@mail.gmail.com>
	<48FE4316.6020905@gmail.com>
	<2e1434c10810211438o1737b44udcb05dd6d3594f71@mail.gmail.com>
Message-ID: <9628BBDB-1B65-4A6D-88C0-9B03E8B046B0@ster.kuleuven.be>


Thanks for the pointers. I'll produce some code to show what I have in  
mind, and then come back to the list.

Cheers,
Joris


Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm



From rocksportrocker at googlemail.com  Wed Oct 22 08:20:05 2008
From: rocksportrocker at googlemail.com (Uwe Schmitt)
Date: Wed, 22 Oct 2008 14:20:05 +0200
Subject: [Numpy-discussion] Google Groups
Message-ID: <c9bc81620810220520r4fb7c0d2la38ce78c456dc39a@mail.gmail.com>

Hi,

this mailing list disappeared from google groups.
Is there a reason for this ? Did I miss something ?

Greetings, Uwe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081022/8155c604/attachment.html>

From wbaxter at gmail.com  Wed Oct 22 08:41:43 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 22 Oct 2008 21:41:43 +0900
Subject: [Numpy-discussion] scipy super pack installer for win32: please
	test
In-Reply-To: <48FF0214.4080302@ar.media.kyoto-u.ac.jp>
References: <48FF0214.4080302@ar.media.kyoto-u.ac.jp>
Message-ID: <e86a5fd00810220541k22b463c9t2e858eb21cb9ebb8@mail.gmail.com>

I'm getting a 404 on that url.

--bb

On Wed, Oct 22, 2008 at 7:36 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    A quick note to mention I have generated a "superpack" installer for
> scipy, for testing purposes. This is similar to numpy superpack
> installer: the installer detects your CPU at installation time and
> install the right scipy:
>
> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/scipy-0.7.0.dev4826-win32-superpack-python2.5.exe
>
> How to test:
> ------------
>
> Inside python:
>
> import scipy
> scipy.test()
>
> If python does not crash, this is OK. Test failures are OK, since those
> will be fixed within the 0.7.0 release. Please report any problem on
> scipy-dev ML,
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From david at ar.media.kyoto-u.ac.jp  Wed Oct 22 08:36:41 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 22 Oct 2008 21:36:41 +0900
Subject: [Numpy-discussion] scipy super pack installer for win32: please
 test
In-Reply-To: <e86a5fd00810220541k22b463c9t2e858eb21cb9ebb8@mail.gmail.com>
References: <48FF0214.4080302@ar.media.kyoto-u.ac.jp>
	<e86a5fd00810220541k22b463c9t2e858eb21cb9ebb8@mail.gmail.com>
Message-ID: <48FF1E59.2090800@ar.media.kyoto-u.ac.jp>

Bill Baxter wrote:
> I'm getting a 404 on that url.
>   

It should work now.

David


From robert.kern at gmail.com  Wed Oct 22 09:07:28 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 08:07:28 -0500
Subject: [Numpy-discussion] Google Groups
In-Reply-To: <c9bc81620810220520r4fb7c0d2la38ce78c456dc39a@mail.gmail.com>
References: <c9bc81620810220520r4fb7c0d2la38ce78c456dc39a@mail.gmail.com>
Message-ID: <3d375d730810220607r339cfacaja77bec5c57bdffb1@mail.gmail.com>

On Wed, Oct 22, 2008 at 07:20, Uwe Schmitt
<rocksportrocker at googlemail.com> wrote:
> Hi,
>
> this mailing list disappeared from google groups.
> Is there a reason for this ? Did I miss something ?

I don't know. That gateway was not managed by us. IIRC Mike McLay was
the one who started it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From yger at unic.cnrs-gif.fr  Wed Oct 22 12:21:48 2008
From: yger at unic.cnrs-gif.fr (Pierre Yger)
Date: Wed, 22 Oct 2008 18:21:48 +0200
Subject: [Numpy-discussion] Optimization of loops
Message-ID: <200810221821.48143.yger@unic.cnrs-gif.fr>

Hi all, 

This is my first mail to the mailing list, and I would like to know if anybody 
has a great idea about the use or not of Numpy and loops in Python.

So here is my problem : I've a large list of tuple (id, time), 
id being integer between [0, ..., N] and time float values.
I want to have a mysort() function that will be able to explode this list into 
N lists of differents sizes, that will contained the times associated to each 
id.

Example:

>> spikes = [(0, 2.3),(1, 5.6),(3, 2.5),(0, 5.2),(3, 10.2),(2, 16.2)]

mysort(spikes) 

should return:

[[2.3, 5.2], [5.6], [16.2], [2.5, 10.2]]

Intuitively, the simplest way to do that is to append elements while going 
through all the tuples of the main list (called spikes) to empty lists:

res = [[] for i in xrange(N)]

for id, time in my_list:
????????res[id].append(time)

But this loop seems to be incredibly slow for large lists ! 

A faster way (after having performed some profiling) seems to do:
spikes = numpy.array(spikes) # Convert the list into a numpy array
res = []
for id in xrange(N):
????????res.append(spikes[spikes[:,0] == id, 1]) # Use Numpy indexes

Nevertheless, this is still rather slow. Does anybody have any idea about a 
faster way to do this ? Is there a Numpy function that could be used ?

Thanks in advance, 

Pierre


From pgmdevlist at gmail.com  Wed Oct 22 13:30:00 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 22 Oct 2008 13:30:00 -0400
Subject: [Numpy-discussion] Timeseries scikit - dates prior to 1900
In-Reply-To: <loom.20081022T013653-2@post.gmane.org>
References: <loom.20081022T013653-2@post.gmane.org>
Message-ID: <200810221330.00306.pgmdevlist@gmail.com>

bevan,
Thanks for reporting. I'll try to find to correct that and will keep you 
posted.
Cheers
P.


From pgmdevlist at gmail.com  Wed Oct 22 14:57:17 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 22 Oct 2008 14:57:17 -0400
Subject: [Numpy-discussion] Timeseries scikit - dates prior to 1900
In-Reply-To: <loom.20081022T013653-2@post.gmane.org>
References: <loom.20081022T013653-2@post.gmane.org>
Message-ID: <200810221457.17446.pgmdevlist@gmail.com>

Bevan,
After digging a bit, I don't think it's gonna be possible any time soon to 
print a date w/ a year prior 1900. The reason is that we use the 
datetime.strftime under the hood, and this method is intrinsicly limited to 
years after 1900.
Note that this limitation affects only printing and date representation. You 
should be able to safely use the modulewith dates prior the XXth century.
We'll put your comment on the TODO list, though. Still, it might not be 
implemented before a while.
Sorry for any inconvenience,
Sincerely,
P.



From yakov.keselman at gmail.com  Wed Oct 22 16:09:53 2008
From: yakov.keselman at gmail.com (Yakov Keselman)
Date: Wed, 22 Oct 2008 13:09:53 -0700
Subject: [Numpy-discussion] Sparse N-dim arrays support?
Message-ID: <dc3116ef0810221309r5e3b77f2r132cf0b974b8efc2@mail.gmail.com>

Hello everyone!

I wonder if support for generic N-dimensional arrays has been added to
NumPy since January 2003 (the last time this question was asked in
this newsgroup). If not, is there some interest in trying to add this
data structure to NumPy? It may not seem very useful for scientific
computations, but it definitely makes sense for analytical (a.k.a. BI)
applications.

Any pointers to code (Python or otherwise) are much appreciated. Thanks!

= Yakov

2003 posting: http://article.gmane.org/gmane.comp.python.numeric.general/11570/


From robert.kern at gmail.com  Wed Oct 22 16:17:21 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 15:17:21 -0500
Subject: [Numpy-discussion] Sparse N-dim arrays support?
In-Reply-To: <dc3116ef0810221309r5e3b77f2r132cf0b974b8efc2@mail.gmail.com>
References: <dc3116ef0810221309r5e3b77f2r132cf0b974b8efc2@mail.gmail.com>
Message-ID: <3d375d730810221317xd257eefhbbd85a998dfbde28@mail.gmail.com>

On Wed, Oct 22, 2008 at 15:09, Yakov Keselman <yakov.keselman at gmail.com> wrote:
> Hello everyone!
>
> I wonder if support for generic N-dimensional arrays has been added to
> NumPy since January 2003 (the last time this question was asked in
> this newsgroup).

No.

> If not, is there some interest in trying to add this
> data structure to NumPy?

There is certainly interest in seeing it done. None of that has
materialized into actions, though. If you're willing to spearhead the
implementation effort, I'm sure many people will be delighted to help,
though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rmay31 at gmail.com  Wed Oct 22 16:59:30 2008
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 22 Oct 2008 15:59:30 -0500
Subject: [Numpy-discussion] Loadtxt .bz2 support
In-Reply-To: <e06186140810212059t529efe0eoa478f2cfa98b8fb4@mail.gmail.com>
References: <48FE2DDB.8070808@gmail.com>
	<e06186140810212059t529efe0eoa478f2cfa98b8fb4@mail.gmail.com>
Message-ID: <48FF9432.1070104@gmail.com>

Charles R Harris wrote:
> On Tue, Oct 21, 2008 at 1:30 PM, Ryan May <rmay31 at gmail.com> wrote:
> 
>> Hi,
>>
>> I noticed numpy.loadtxt has support for gzipped text files, but not for
>> bz2'd files.  Here's a 3 line patch to add bzip2 support to loadtxt.
>>
>> Ryan
>>
>> --
>> Ryan May
>> Graduate Research Assistant
>> School of Meteorology
>> University of Oklahoma
>>
>> Index: numpy/lib/io.py
>> ===================================================================
>> --- numpy/lib/io.py     (revision 5953)
>> +++ numpy/lib/io.py     (working copy)
>> @@ -320,6 +320,9 @@
>>         if fname.endswith('.gz'):
>>             import gzip
>>             fh = gzip.open(fname)
>> +        elif fname.endswith('.bz2'):
>> +            import bz2
>> +            fh = bz2.BZ2File(fname)
>>         else:
>>             fh = file(fname)
>>     elif hasattr(fname, 'seek'):
>>
> 
> Could you open a ticket for this? Mark it as an enhancement.
> 

Done. #940

http://scipy.org/scipy/numpy/ticket/940

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From fperez.net at gmail.com  Wed Oct 22 17:00:20 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 22 Oct 2008 14:00:20 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
Message-ID: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>

Hi all,

much delayed, but here it is, finally.  The doc regarding our
discussion about PEP 225 is attached, and I'm keeping a public copy
for viewing convenience (with html) here:

https://cirl.berkeley.edu/fperez/static/numpy-pep225/

Note that, as indicated in the link above, the real doc is in bzr, so
you can check it out to generate patches or a  branch (the preferred
formats for big changes).

This is just a first cut, going from memory and notes.  I'd appreciate
any feedback, corrections, etc.  I'm giving a talk at the BayPiggies
group Nov 13 about SciPy and will take the opportunity to bring this
document up, since that user group has a lot of python people,
including some core developers.

Since this is from memory and written by me, it's likely pretty
biased.  But I really want to clarify that I'm trying to act as a
scribe here, not to push my own agenda (I do have one :).  So please
bring up anything you feel is missing/misstated here; I'd really like
to have a doc that people feel is a fair representation of the
community's views to bring to the core python team.  That way we
either get our operators or not once and for all, but this issue can
be put to rest in the language for good (as of 2.7/3.1, of course).

So let's have Nov 11 or so as a wrapup deadline on this discussion, so
I can have the summary ready for the Nov 13 talk.

Best,

f
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: numpy-pep225.txt
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081022/c1ca3e78/attachment.txt>

From robert.kern at gmail.com  Wed Oct 22 17:15:52 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 16:15:52 -0500
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
Message-ID: <3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>

On Wed, Oct 22, 2008 at 16:00, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi all,
>
> much delayed, but here it is, finally.  The doc regarding our
> discussion about PEP 225 is attached, and I'm keeping a public copy
> for viewing convenience (with html) here:
>
> https://cirl.berkeley.edu/fperez/static/numpy-pep225/
>
> Note that, as indicated in the link above, the real doc is in bzr, so
> you can check it out to generate patches or a  branch (the preferred
> formats for big changes).
>
> This is just a first cut, going from memory and notes.  I'd appreciate
> any feedback, corrections, etc.  I'm giving a talk at the BayPiggies
> group Nov 13 about SciPy and will take the opportunity to bring this
> document up, since that user group has a lot of python people,
> including some core developers.
>
> Since this is from memory and written by me, it's likely pretty
> biased.  But I really want to clarify that I'm trying to act as a
> scribe here, not to push my own agenda (I do have one :).  So please
> bring up anything you feel is missing/misstated here; I'd really like
> to have a doc that people feel is a fair representation of the
> community's views to bring to the core python team.  That way we
> either get our operators or not once and for all, but this issue can
> be put to rest in the language for good (as of 2.7/3.1, of course).

You sentence ending "Operator form for logical_X functions" is not finished.

I would add another position (my own) to the "Arguments neutral
towards or against the PEP":

* I really only care about having just *one* extra operator, one that
I can (ab)use for matrix multiplication. It's the only operation that
is common enough and with one obvious implementation (I'm looking at
you, Matlab's "\") to warrant it, IMO. Doubling the number of
operators and special methods is not a price that I'm willing to pay
to get it, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From perry at stsci.edu  Wed Oct 22 17:24:20 2008
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 22 Oct 2008 17:24:20 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
Message-ID: <6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>


On Oct 22, 2008, at 5:15 PM, Robert Kern wrote:
>
> I would add another position (my own) to the "Arguments neutral
> towards or against the PEP":
>
> * I really only care about having just *one* extra operator, one that
> I can (ab)use for matrix multiplication. It's the only operation that
> is common enough and with one obvious implementation (I'm looking at
> you, Matlab's "\") to warrant it, IMO. Doubling the number of
> operators and special methods is not a price that I'm willing to pay
> to get it, though.
>
Note that although some of the proposals allow for a doubling of the  
number of possible operators, numpy doesn't have to use them all and  
thus doesn't need to double the number of special methods. I'd agree  
that only a few are really necessary (to matrix multiply I'd add the  
logical operators as well).

Perry



From ggellner at uoguelph.ca  Wed Oct 22 17:29:12 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Wed, 22 Oct 2008 17:29:12 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at	Scipy
In-Reply-To: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
Message-ID: <20081022212912.GA15388@encolpuis>

On Wed, Oct 22, 2008 at 02:00:20PM -0700, Fernando Perez wrote:
> Hi all,
> 
> much delayed, but here it is, finally.  The doc regarding our
> discussion about PEP 225 is attached, and I'm keeping a public copy
> for viewing convenience (with html) here:
> 
> https://cirl.berkeley.edu/fperez/static/numpy-pep225/
> 
> Note that, as indicated in the link above, the real doc is in bzr, so
> you can check it out to generate patches or a  branch (the preferred
> formats for big changes).
> 
> This is just a first cut, going from memory and notes.  I'd appreciate
> any feedback, corrections, etc.  I'm giving a talk at the BayPiggies
> group Nov 13 about SciPy and will take the opportunity to bring this
> document up, since that user group has a lot of python people,
> including some core developers.
> 
> Since this is from memory and written by me, it's likely pretty
> biased.  But I really want to clarify that I'm trying to act as a
> scribe here, not to push my own agenda (I do have one :).  So please
> bring up anything you feel is missing/misstated here; I'd really like
> to have a doc that people feel is a fair representation of the
> community's views to bring to the core python team.  That way we
> either get our operators or not once and for all, but this issue can
> be put to rest in the language for good (as of 2.7/3.1, of course).
> 
> So let's have Nov 11 or so as a wrapup deadline on this discussion, so
> I can have the summary ready for the Nov 13 talk.
> 
> Best,
> 
> f
Thanks for taking the lead on this. Just in case you want more examples Fortran 90/95
allows for operator definition like R see:
http://h21007.www2.hp.com/portal/download/files/unprot/fortran/docs/lrm/lrm0184.htm#generic_inter_op

I think this is ideal as I am used to it, but at the least I vote for a
multiplication operator.

Gabriel



From robert.kern at gmail.com  Wed Oct 22 18:01:34 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 17:01:34 -0500
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
Message-ID: <3d375d730810221501i4e5dd318v315e580967b707cf@mail.gmail.com>

On Wed, Oct 22, 2008 at 16:24, Perry Greenfield <perry at stsci.edu> wrote:
>
> On Oct 22, 2008, at 5:15 PM, Robert Kern wrote:
>>
>> I would add another position (my own) to the "Arguments neutral
>> towards or against the PEP":
>>
>> * I really only care about having just *one* extra operator, one that
>> I can (ab)use for matrix multiplication. It's the only operation that
>> is common enough and with one obvious implementation (I'm looking at
>> you, Matlab's "\") to warrant it, IMO. Doubling the number of
>> operators and special methods is not a price that I'm willing to pay
>> to get it, though.
>>
> Note that although some of the proposals allow for a doubling of the
> number of possible operators, numpy doesn't have to use them all and
> thus doesn't need to double the number of special methods. I'd agree
> that only a few are really necessary (to matrix multiply I'd add the
> logical operators as well).

It doesn't double the number of special methods that numpy uses, but
it does double the number of special methods defined in the language.
*That's* what I'm worried about.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Wed Oct 22 19:04:17 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 22 Oct 2008 16:04:17 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
Message-ID: <48FFB171.8010800@noaa.gov>

Fernando Perez wrote:
> much delayed, but here it is, finally.  The doc regarding our
> discussion about PEP 225 is attached, and I'm keeping a public copy
> for viewing convenience (with html) here:
> 
> https://cirl.berkeley.edu/fperez/static/numpy-pep225/

Is there a use for more operators outside of numpy? I imagine there is, 
but I don't know what they might be. However, we can make a much 
stronger case for adding it to the language if there is broader 
applicability.

I-never-write-a-python-script-without-numpy-ly yours,

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From fperez.net at gmail.com  Wed Oct 22 19:14:56 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 22 Oct 2008 16:14:56 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <200810221537.00378.binet@cern.ch>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<200810221537.00378.binet@cern.ch>
Message-ID: <db6b5ecc0810221614u5727984dh320812f93ca6b7eb@mail.gmail.com>

Hi

Please keep your replies on list.  It's important that this thread in
particular is publicly archived, since it may serve as the reference
for further discussions.

On Wed, Oct 22, 2008 at 3:37 PM, Sebastien Binet <binet at cern.ch> wrote:
> Fernando,
>
>> much delayed, but here it is, finally.  The doc regarding our
>> discussion about PEP 225 is attached, and I'm keeping a public copy
>> for viewing convenience (with html) here:
>>
>> https://cirl.berkeley.edu/fperez/static/numpy-pep225/
>>
>> Note that, as indicated in the link above, the real doc is in bzr, so
>> you can check it out to generate patches or a  branch (the preferred
>> formats for big changes).
>>
>> This is just a first cut, going from memory and notes.  I'd appreciate
>> any feedback, corrections, etc.
>
> what about the D-language-like syntax I proposed which has the advantage of not
> introducing gazillions of new operators (but just stating that the operation
> will be broadcasted/vectorized) ?
>
> # create array of doubles
> a = ...
> # add 42 to all elements
> a[] += 42.
> # apply a function
> b = sin(a[])
> # or this?
> b = sin[](a)
>
> c = a[] + b[]

Regardless of its merits in D, I'd bet this has little chance of
flying in python.  The pep is simply about adding some special methods
with matching operator form, inline with existing python syntactic
tradition.  This suggestion is a pretty radical departure from
Python's existing style.

Having said that, I'm not Guido and I could be 100% wrong here, so
I'll most certainly put your comments in when I update the document
later.

>> I'm giving a talk at the BayPiggies
>> group Nov 13 about SciPy
> count me in ;)
>
> cheers,
> sebastien.
> --
> ###################################
> # Dr. Sebastien Binet
> # Lawrence Berkeley National Lab.
> # 1 Cyclotron Road
> # Berkeley, CA 94720
> ###################################
>
>

Cheers,

f


From efiring at hawaii.edu  Wed Oct 22 20:17:18 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 22 Oct 2008 14:17:18 -1000
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
Message-ID: <48FFC28E.5070207@hawaii.edu>

Perry Greenfield wrote:
> On Oct 22, 2008, at 5:15 PM, Robert Kern wrote:
>> I would add another position (my own) to the "Arguments neutral
>> towards or against the PEP":
>>
>> * I really only care about having just *one* extra operator, one that
>> I can (ab)use for matrix multiplication. It's the only operation that
>> is common enough and with one obvious implementation (I'm looking at
>> you, Matlab's "\") to warrant it, IMO. Doubling the number of
>> operators and special methods is not a price that I'm willing to pay
>> to get it, though.
>>
> Note that although some of the proposals allow for a doubling of the  
> number of possible operators, numpy doesn't have to use them all and  
> thus doesn't need to double the number of special methods. I'd agree  
> that only a few are really necessary (to matrix multiply I'd add the  
> logical operators as well).

I think a stronger general numpy case might be made for the logical 
operators than for matrix multiplication.  An alternative approach, and 
I think preferable to introducing new logical operators, is PEP 335: 
http://www.python.org/dev/peps/pep-0335/.  It is still on the "under 
consideration" list, but it has been there since 2004.  Perhaps it needs 
a strong push?

Eric

> 
> Perry
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From Chris.Barker at noaa.gov  Thu Oct 23 12:32:06 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 23 Oct 2008 09:32:06 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <48FFC28E.5070207@hawaii.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu>
Message-ID: <4900A706.70806@noaa.gov>

Eric Firing wrote:
> I think a stronger general numpy case might be made for the logical 
> operators than for matrix multiplication.  An alternative approach, and 
> I think preferable to introducing new logical operators, is PEP 335: 
> http://www.python.org/dev/peps/pep-0335/.

I like that!

However, it is orthogonal to the additional operators question.

Some thoughts:

I do like the idea of having additional operators, but it seems that the 
strongest use case is matrix operations. While the current matrix 
implementation is clearly inadequate, we had a good discussion a while 
back about how it could be improved, and maybe if we could find a way to 
implement that well, we'd be OK. I always did find Matlab's "dot' 
operators to be kind of annoying (though mostly because what I most 
often need was the "dot" version)


I guess what I'm saying is that adding a bunch of operators just for a 
few matrix operations may not be worth it.

If we can see substantial other uses, then it may be worth it -- are 
there any?

An example is the PEP referenced above -- it would not be nearly as 
strong if numpy were the only motivator -- I think the database example 
is a very strong addition -- is there something similar for additional 
operators?

-Chris







-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From bryan at cole.uklinux.net  Thu Oct 23 12:34:52 2008
From: bryan at cole.uklinux.net (Bryan Cole)
Date: Thu, 23 Oct 2008 17:34:52 +0100
Subject: [Numpy-discussion] Optimization of loops
In-Reply-To: <200810221821.48143.yger@unic.cnrs-gif.fr>
References: <200810221821.48143.yger@unic.cnrs-gif.fr>
Message-ID: <1224779692.7661.64.camel@pc2.cole.uklinux.net>


> >> spikes = [(0, 2.3),(1, 5.6),(3, 2.5),(0, 5.2),(3, 10.2),(2, 16.2)]
> 
> mysort(spikes) 
> 
> should return:
> 
> [[2.3, 5.2], [5.6], [16.2], [2.5, 10.2]]
> 
> Intuitively, the simplest way to do that is to append elements while going 
> through all the tuples of the main list (called spikes) to empty lists:
> 
> res = [[] for i in xrange(N)]
> 
> for id, time in my_list:
>         res[id].append(time)
> 
> But this loop seems to be incredibly slow for large lists ! 
> 
> A faster way (after having performed some profiling) seems to do:
> spikes = numpy.array(spikes) # Convert the list into a numpy array
> res = []
> for id in xrange(N):
>         res.append(spikes[spikes[:,0] == id, 1]) # Use Numpy indexes
> 
> Nevertheless, this is still rather slow. Does anybody have any idea about a 
> faster way to do this ? Is there a Numpy function that could be used ?

I'm surprised you find using numpy indexing (your second method) is
faster. In my simple test (N=10, 100000 points in the list), your method
1 (using python lists) took 0.15s whereas your method 2 (numpy indexing)
took 1.8s. Big difference. 

I wonder how you are profiling this. I expect the numpy indexing to be
quite fast (and you only do it N times) but the overhead of copying the
entire list into an array is probably the bottleneck. If you need to
iterate over the final output arrays, there's a further penalty in using
a numpy array as iteration is not as fast as for a python list. 

There might be some benefit in using numpy arrays if you can create your
input directly as an array without creating the list-of-tuples first.

If you really need more speed you could try Cython (a means of compiling
python to C-code, while adding type-declarations to permit
optimisation). There may be a worthwhile improvement running the
python-list method through Cython (I'll try this tonight to satisfy my
own curiosity). If you can arrange for the inputs and outputs to be
numpy arrays, then you can do the iteration over the data and copy into
output array using pure C (but coded in Cython; much easier than C).
This will be as fast as it's possible to go. You need one pass to figure
out how big the output arrays need to be then a second to copy the data.

Finally, an even better approach may be to avoid creating such a large
data-structure in the first place. If you can replace your list with a
generator, you save time by avoiding the need to allocate the memory to
hold a large list. Similarly, if you don't need to store the output
lists for repeated use, then outputing a list of generators may be more
efficient. Whether this is viable depends on the context.

HTH
BC

> 
> Thanks in advance, 
> 
> Pierre




From charlesr.harris at gmail.com  Thu Oct 23 13:52:16 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 23 Oct 2008 11:52:16 -0600
Subject: [Numpy-discussion] Optimization of loops
In-Reply-To: <200810221821.48143.yger@unic.cnrs-gif.fr>
References: <200810221821.48143.yger@unic.cnrs-gif.fr>
Message-ID: <e06186140810231052g50e12616k4ca18e82311d3921@mail.gmail.com>

On Wed, Oct 22, 2008 at 10:21 AM, Pierre Yger <yger at unic.cnrs-gif.fr> wrote:

> Hi all,
>
> This is my first mail to the mailing list, and I would like to know if
> anybody
> has a great idea about the use or not of Numpy and loops in Python.
>
> So here is my problem : I've a large list of tuple (id, time),
> id being integer between [0, ..., N] and time float values.
> I want to have a mysort() function that will be able to explode this list
> into
> N lists of differents sizes, that will contained the times associated to
> each
> id.
>
> Example:
>
> >> spikes = [(0, 2.3),(1, 5.6),(3, 2.5),(0, 5.2),(3, 10.2),(2, 16.2)]
>
> mysort(spikes)
>
> should return:
>
> [[2.3, 5.2], [5.6], [16.2], [2.5, 10.2]]
>
> Intuitively, the simplest way to do that is to append elements while going
> through all the tuples of the main list (called spikes) to empty lists:
>
> res = [[] for i in xrange(N)]
>
> for id, time in my_list:
>         res[id].append(time)
>
> But this loop seems to be incredibly slow for large lists !
>
> A faster way (after having performed some profiling) seems to do:
> spikes = numpy.array(spikes) # Convert the list into a numpy array
> res = []
> for id in xrange(N):
>         res.append(spikes[spikes[:,0] == id, 1]) # Use Numpy indexes
>
> Nevertheless, this is still rather slow. Does anybody have any idea about a
> faster way to do this ? Is there a Numpy function that could be used ?
>

If you want to stick to lists you can try something like

In [1]: import bisect

In [2]: spikes = [(0, 2.3),(1, 5.6),(3, 2.5),(0, 5.2),(3, 10.2),(2, 16.2)]

In [3]: spikes.sort()

In [4]: ind = [i for i,j in spikes]

In [5]: tim = [j for i,j in spikes]

In [6]: mylist = []

In [7]: for i in xrange(4) :
   ...:     beg = bisect.bisect_left(ind,i)
   ...:     end = bisect.bisect_right(ind,i)
   ...:     mylist.append(tim[beg:end])
   ...:

In [8]: mylist
Out[8]:
[[2.2999999999999998, 5.2000000000000002],
 [5.5999999999999996],
 [16.199999999999999],
 [2.5, 10.199999999999999]]


There is a numpy version of this also which might be faster if your lists
are really huge.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081023/233bb527/attachment.html>

From stefan at sun.ac.za  Thu Oct 23 17:04:37 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 23 Oct 2008 23:04:37 +0200
Subject: [Numpy-discussion] Optimization of loops
In-Reply-To: <e06186140810231052g50e12616k4ca18e82311d3921@mail.gmail.com>
References: <200810221821.48143.yger@unic.cnrs-gif.fr>
	<e06186140810231052g50e12616k4ca18e82311d3921@mail.gmail.com>
Message-ID: <9457e7c80810231404s354e025h849711ac6bb0ca60@mail.gmail.com>

2008/10/23 Charles R Harris <charlesr.harris at gmail.com>:
> In [4]: ind = [i for i,j in spikes]
> In [5]: tim = [j for i,j in spikes]

Just for interest's sake, 'zip' is an interesting function in that it
works both ways around:

In [26]: zip(*zip([1,2,3],[3,4,5]))
Out[26]: [(1, 2, 3), (3, 4, 5)]

So a person can do

In [27]: ind, tim = zip(*spikes)

Cheers
St?fan


From pierre.yger at gmail.com  Thu Oct 23 18:55:50 2008
From: pierre.yger at gmail.com (Pierre Yger)
Date: Fri, 24 Oct 2008 00:55:50 +0200
Subject: [Numpy-discussion] Optimization of loops
In-Reply-To: <1224779692.7661.64.camel@pc2.cole.uklinux.net>
References: <200810221821.48143.yger@unic.cnrs-gif.fr>
	<1224779692.7661.64.camel@pc2.cole.uklinux.net>
Message-ID: <490100F6.6030806@unic.cnrs-gif.fr>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks everybody for all your feedback !!!

Ok, so first, I just must have to precise what was my problem and why I
used Numpy. Because I want to extract only some ids in the lists. In
fact, I've a test in my list implementation:

for id, time in my_list:
    if id in id_list:
        res[id].append(time)

So this test is what slows down the whole stuff. The implementation
proposed with bisect is very nice and has the advantaged of not spending
time allocating numpy arrays, and also to avoid the previous test. The
list where id will not be in id_list will return empty lists, and that's
what I wanted.

import bisect
my_list.sort()
time, ind = zip(*my_list)
for i in id_list :
    beg = bisect.bisect_left(ind,i)
    end = bisect.bisect_right(ind,i)
    mylist.append(tim[beg:end])

But thanks a lot for all you precious comments, you solve a big part of
my problem. I'll see how to use generator and how it speeds up or not
the whole process. I would also be very interested to know what the code
looks like with Cython and what are the results....

Cheers,

Pierre Yger

>>>> spikes = [(0, 2.3),(1, 5.6),(3, 2.5),(0, 5.2),(3, 10.2),(2, 16.2)]
>> mysort(spikes) 
>>
>> should return:
>>
>> [[2.3, 5.2], [5.6], [16.2], [2.5, 10.2]]
>>
>> Intuitively, the simplest way to do that is to append elements while going 
>> through all the tuples of the main list (called spikes) to empty lists:
>>
>> res = [[] for i in xrange(N)]
>>
>> for id, time in my_list:
>>         res[id].append(time)
>>
>> But this loop seems to be incredibly slow for large lists ! 
>>
>> A faster way (after having performed some profiling) seems to do:
>> spikes = numpy.array(spikes) # Convert the list into a numpy array
>> res = []
>> for id in xrange(N):
>>         res.append(spikes[spikes[:,0] == id, 1]) # Use Numpy indexes
>>
>> Nevertheless, this is still rather slow. Does anybody have any idea about a 
>> faster way to do this ? Is there a Numpy function that could be used ?
> 
> I'm surprised you find using numpy indexing (your second method) is
> faster. In my simple test (N=10, 100000 points in the list), your method
> 1 (using python lists) took 0.15s whereas your method 2 (numpy indexing)
> took 1.8s. Big difference. 
> 
> I wonder how you are profiling this. I expect the numpy indexing to be
> quite fast (and you only do it N times) but the overhead of copying the
> entire list into an array is probably the bottleneck. If you need to
> iterate over the final output arrays, there's a further penalty in using
> a numpy array as iteration is not as fast as for a python list. 
> 
> There might be some benefit in using numpy arrays if you can create your
> input directly as an array without creating the list-of-tuples first.
> 
> If you really need more speed you could try Cython (a means of compiling
> python to C-code, while adding type-declarations to permit
> optimisation). There may be a worthwhile improvement running the
> python-list method through Cython (I'll try this tonight to satisfy my
> own curiosity). If you can arrange for the inputs and outputs to be
> numpy arrays, then you can do the iteration over the data and copy into
> output array using pure C (but coded in Cython; much easier than C).
> This will be as fast as it's possible to go. You need one pass to figure
> out how big the output arrays need to be then a second to copy the data.
> 
> Finally, an even better approach may be to avoid creating such a large
> data-structure in the first place. If you can replace your list with a
> generator, you save time by avoiding the need to allocate the memory to
> hold a large list. Similarly, if you don't need to store the output
> lists for repeated use, then outputing a list of generators may be more
> efficient. Whether this is viable depends on the context.
> 
> HTH
> BC
> 
>> Thanks in advance, 
>>
>> Pierre
> 
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkkBAPUACgkQTyWWLSJwVgLX1wCfYtO89V5nicsevkMm1CWShP/j
QqsAnRdqqkqsoy7rwfT2dQWCgj/Jx+/X
=qDBo
-----END PGP SIGNATURE-----


From christoph_goebl at hotmail.com  Fri Oct 24 05:49:39 2008
From: christoph_goebl at hotmail.com (=?iso-8859-1?Q?Christoph_G=F6bl?=)
Date: Fri, 24 Oct 2008 11:49:39 +0200
Subject: [Numpy-discussion] installation problem on Red Hat
Message-ID: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>


dear members,
 
I'm very sorry to bother you with a (hopefully) simple problem...
I need pyhton and the numerical package to run another program.
I installed Python, it works fine. But I can't install the numpy package. To install the oder Numeric package was no problem, but I need the newer numpy...
 
after 
 
"python setup.py install"
 
I get an error message after sime while:
 ...
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
 
compile options: '-DNO_ATLAS_INFO=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c'
/usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: cannot find -lgcc_s
collect2: ld returned 1 exit status
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: cannot find -lgcc_s
collect2: ld returned 1 exit status
error: Command "/usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" failed with exit status 1
 ...
I don't know, what I can do... On a Suse 10.2 it ran easily, but on the other computer, Red Hat 3.4.3_9, X86_64, gcc 3.4.3 there is always this error message.
I read in another forum, that a person solved a similar problem using
 
unsetenv ldflags
 
But - sorry I'm a Newbie in Linux and Python - there the installation was on another platform I think. 
Anyway, maybe it's a linking problem?
 
thank you very much for any thoughts you may waste on my problems...
best regards,
Christoph

_________________________________________________________________
Hol dir 30 kostenlose Emoticons f?r deinen Windows Live Messenger
http://www.livemessenger-emoticons.com/funfamily/de-at/

From nadavh at visionsense.com  Fri Oct 24 10:41:06 2008
From: nadavh at visionsense.com (Nadav Horesh)
Date: Fri, 24 Oct 2008 16:41:06 +0200
Subject: [Numpy-discussion] installation problem on Red Hat
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
Message-ID: <710F2847B0018641891D9A216027636029C2E2@ex3.envision.co.il>

It could be a version mismatch between two gcc (and the corresponding libraries) versions: you surly have gcc at /usr/bin, but the fortran compiler you use (g77) is as /usr/local/bin.

  Nadav


-----????? ??????-----
???: numpy-discussion-bounces at scipy.org ??? Christoph G?bl
????: ? 24-???????-08 11:49
??: numpy-discussion at scipy.org
????: [Numpy-discussion] installation problem on Red Hat
 

dear members,
 
I'm very sorry to bother you with a (hopefully) simple problem...
I need pyhton and the numerical package to run another program.
I installed Python, it works fine. But I can't install the numpy package. To install the oder Numeric package was no problem, but I need the newer numpy...
 
after 
 
"python setup.py install"
 
I get an error message after sime while:
 ...
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
 
compile options: '-DNO_ATLAS_INFO=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c'
/usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: cannot find -lgcc_s
collect2: ld returned 1 exit status
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack
/usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c
/usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c
/usr/bin/ld: cannot find -lgcc_s
collect2: ld returned 1 exit status
error: Command "/usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" failed with exit status 1
 ...
I don't know, what I can do... On a Suse 10.2 it ran easily, but on the other computer, Red Hat 3.4.3_9, X86_64, gcc 3.4.3 there is always this error message.
I read in another forum, that a person solved a similar problem using
 
unsetenv ldflags
 
But - sorry I'm a Newbie in Linux and Python - there the installation was on another platform I think. 
Anyway, maybe it's a linking problem?
 
thank you very much for any thoughts you may waste on my problems...
best regards,
Christoph

_________________________________________________________________
Hol dir 30 kostenlose Emoticons f?r deinen Windows Live Messenger
http://www.livemessenger-emoticons.com/funfamily/de-at/
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4339 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/cf0d4b3e/attachment.bin>

From f.yw at hotmail.com  Fri Oct 24 12:30:15 2008
From: f.yw at hotmail.com (frank wang)
Date: Fri, 24 Oct 2008 10:30:15 -0600
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
Message-ID: <BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>


Hi,
 
I have to send this request second time since my first message contains the attached data file which is too big and was blocked by the system. So this time I will not attach the data file.
 
I have converted a matlab function to python using numpy.  both matlab and python run slow. I know that numpy has a lot of features, so I hope some experts can help me to speed up the code.
 
Here is how I run the code:
 
 upsample.upsample(cdata,4*10240000*401.0/812.0,25600000.0,'r')
 
Where cdata is about 70000 complex data.
 
Thanks
 
Frank
 
 
from numpy import zeros,ceil,pi,arange,concatenate,sincfrom pylab import plot,clf,show,figure, psd, grid,xlabel, figureimport pdbdef upsample(input,Fs_old,Fsamp,filt_type):
 """Perform resampling the input data from rate Fs to Fsamp Note:y=zeros((N)) shape is (N,). y=zeros((N,1)) shape is (N,1). Example of how to read a two columns floating data file created by  Matlab. d=fromfile("filename",dtype='float',count=-1,sep=' ') x=len(d) data=d.reshape([x/2,2]) """
 Ts=1.0/Fs_old Tsamp=1.0/Fsamp Fw=6000000.0
 L=len(input) N=ceil(Fsamp/Fs_old*L) y=zeros((N),dtype='float64') C=pi*Fw t0=arange(0,Ts,Tsamp) #print t0
 P = 16 input=concatenate((zeros((P)),input,zeros((P))),1) #print input
 out = 0 for mm in arange(P+1):  tt=t0-mm*Ts  out=out+input[P+mm]*sinc(Fw*tt) #print tt #print "out\n" #print out
 y[0:len(t0)]=out #print y
 B=len(t0) for m in arange(P+2,L+P+1):  delta=Tsamp-(Ts-t0[-1])  t1=arange(delta,Ts,Tsamp)  out=0  for mm in arange(-P,P+1):   tt=(m-1-P)*Ts+t1-(mm+m-(P+2)+1)*Ts   out=out+input[m+mm-1]*sinc(Fw*tt)
  y[B:B+len(t1)]=out  t0=t1  B=B+len(t1)   clf() figure(4)     psd(y,256,Fs=25.6)     #show()
_________________________________________________________________
You live life beyond your PC. So now Windows goes beyond your PC.
http://clk.atdmt.com/MRT/go/115298556/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/e43a9a83/attachment.html>

From borreguero at gmail.com  Fri Oct 24 13:43:43 2008
From: borreguero at gmail.com (Jose Borreguero)
Date: Fri, 24 Oct 2008 13:43:43 -0400
Subject: [Numpy-discussion] creating a Numeric array from a numpy array
	LEAKS memory
In-Reply-To: <7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>
	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
Message-ID: <7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>

Dear numpy users,

I need to pass a Numeric array to some oldie code from a numpy array. I
decided to go like this:

for i in range(BIGNUMER):
    my_numpy_array=grabArray(i)
    na=Numeric.array( my_numpy_array, Numeric.Float)
    oldie_code(na)

The constructor line:
na=Numeric.array( my_numpy_array, Numeric.Float)
does leak memory.

Is there a way to pass the Numeric array to oldie_code without the leaks?

Regards,
-- 
Jose M. Borreguero
Postdoctoral Associate
Oak Ridge National Laboratory
P.O. Box 2008, M.S. 6164
Oak Ridge, TN 37831
phone: 865-241-3071 fax: 865-576-5491
Email: borreguerojm at ornl.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/4c0aa604/attachment.html>

From oliphant at enthought.com  Fri Oct 24 13:54:03 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 24 Oct 2008 12:54:03 -0500
Subject: [Numpy-discussion] creating a Numeric array from a numpy array
 LEAKS memory
In-Reply-To: <7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
	<7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
Message-ID: <49020BBB.5000302@enthought.com>

Jose Borreguero wrote:
> Dear numpy users,
>
> I need to pass a Numeric array to some oldie code from a numpy array. 
> I decided to go like this:
>
> for i in range(BIGNUMER):
>     my_numpy_array=grabArray(i)
>     na=Numeric.array( my_numpy_array, Numeric.Float)
>     oldie_code(na)
>
> The constructor line:
> na=Numeric.array( my_numpy_array, Numeric.Float)
> does leak memory.
>
> Is there a way to pass the Numeric array to oldie_code without the leaks?
This should work without memory leaks, but there may be a bug in NumPy 
or Numeric.

Which version of Numeric and NumPy do you have?

-Travis



From borreguero at gmail.com  Fri Oct 24 14:16:03 2008
From: borreguero at gmail.com (Jose Borreguero)
Date: Fri, 24 Oct 2008 14:16:03 -0400
Subject: [Numpy-discussion] creating a Numeric array from a numpy array
	LEAKS memory
In-Reply-To: <49020BBB.5000302@enthought.com>
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>
	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
	<7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
	<49020BBB.5000302@enthought.com>
Message-ID: <7cced4ed0810241116k77bf996ew586f8c327c7e2c2d@mail.gmail.com>

numpy 1.1.0 (from /usr/lib/python2.4/site-packages/numpy/version.py)
Numeric 24.2 (from
/usr/lib/python2.4/site-packages/Numeric/numeric_version.py)

I also tried with an intermediate list, but got the same result:
*mylist=list(my_numpy_array)
na=Numeric.array( mylist, Numeric.Float)*
I don't have memory leaks if I use something like:
*mylist=[0.0]*BIGNUMBER*
*na=Numeric.array( mylist, Numeric.Float)*

-Jose

On Fri, Oct 24, 2008 at 1:54 PM, Travis E. Oliphant
<oliphant at enthought.com>wrote:

> Jose Borreguero wrote:
> > Dear numpy users,
> >
> > I need to pass a Numeric array to some oldie code from a numpy array.
> > I decided to go like this:
> >
> > for i in range(BIGNUMER):
> >     my_numpy_array=grabArray(i)
> >     na=Numeric.array( my_numpy_array, Numeric.Float)
> >     oldie_code(na)
> >
> > The constructor line:
> > na=Numeric.array( my_numpy_array, Numeric.Float)
> > does leak memory.
> >
> > Is there a way to pass the Numeric array to oldie_code without the leaks?
> This should work without memory leaks, but there may be a bug in NumPy
> or Numeric.
>
> Which version of Numeric and NumPy do you have?
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Jose M. Borreguero
Postdoctoral Associate
Oak Ridge National Laboratory
P.O. Box 2008, M.S. 6164
Oak Ridge, TN 37831
phone: 865-241-3071 fax: 865-576-5491
Email: borreguerojm at ornl.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/2f06e6c7/attachment.html>

From borreguero at gmail.com  Fri Oct 24 14:39:59 2008
From: borreguero at gmail.com (Jose Borreguero)
Date: Fri, 24 Oct 2008 14:39:59 -0400
Subject: [Numpy-discussion] creating a Numeric array from a numpy array
	LEAKS memory
In-Reply-To: <7cced4ed0810241116k77bf996ew586f8c327c7e2c2d@mail.gmail.com>
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>
	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
	<7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
	<49020BBB.5000302@enthought.com>
	<7cced4ed0810241116k77bf996ew586f8c327c7e2c2d@mail.gmail.com>
Message-ID: <7cced4ed0810241139r1ebb887ajf05fec86226bb42b@mail.gmail.com>

My bad. Using the intermediate list does *not* leak.
Still, the original problems stays. Can anyone run the following code in
their machine and see if they have leaks?
Maybe it only happens to me :(*

import numpy,Numeric
big=10000000
na=numpy.array([0.0,])
for i in range(big):
    Na=Numeric.array(na,Numeric.Float)*

-Jose

On Fri, Oct 24, 2008 at 2:16 PM, Jose Borreguero <borreguero at gmail.com>wrote:

> numpy 1.1.0 (from /usr/lib/python2.4/site-packages/numpy/version.py)
> Numeric 24.2 (from
> /usr/lib/python2.4/site-packages/Numeric/numeric_version.py)
>
> I also tried with an intermediate list, but got the same result:
> *mylist=list(my_numpy_array)
> na=Numeric.array( mylist, Numeric.Float)*
> I don't have memory leaks if I use something like:
> *mylist=[0.0]*BIGNUMBER*
> *na=Numeric.array( mylist, Numeric.Float)*
>
> -Jose
>
>
> On Fri, Oct 24, 2008 at 1:54 PM, Travis E. Oliphant <
> oliphant at enthought.com> wrote:
>
>> Jose Borreguero wrote:
>> > Dear numpy users,
>> >
>> > I need to pass a Numeric array to some oldie code from a numpy array.
>> > I decided to go like this:
>> >
>> > for i in range(BIGNUMER):
>> >     my_numpy_array=grabArray(i)
>> >     na=Numeric.array( my_numpy_array, Numeric.Float)
>> >     oldie_code(na)
>> >
>> > The constructor line:
>> > na=Numeric.array( my_numpy_array, Numeric.Float)
>> > does leak memory.
>> >
>> > Is there a way to pass the Numeric array to oldie_code without the
>> leaks?
>> This should work without memory leaks, but there may be a bug in NumPy
>> or Numeric.
>>
>> Which version of Numeric and NumPy do you have?
>>
>> -Travis
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Jose M. Borreguero
> Postdoctoral Associate
> Oak Ridge National Laboratory
> P.O. Box 2008, M.S. 6164
> Oak Ridge, TN 37831
> phone: 865-241-3071 fax: 865-576-5491
> Email: borreguerojm at ornl.gov
>



-- 
Jose M. Borreguero
Postdoctoral Associate
Oak Ridge National Laboratory
P.O. Box 2008, M.S. 6164
Oak Ridge, TN 37831
phone: 865-241-3071 fax: 865-576-5491
Email: borreguerojm at ornl.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/b0087aa2/attachment.html>

From pav at iki.fi  Fri Oct 24 14:52:06 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 24 Oct 2008 18:52:06 +0000 (UTC)
Subject: [Numpy-discussion] creating a Numeric array from a numpy
	array	LEAKS memory
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>
	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
	<7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
	<49020BBB.5000302@enthought.com>
	<7cced4ed0810241116k77bf996ew586f8c327c7e2c2d@mail.gmail.com>
	<7cced4ed0810241139r1ebb887ajf05fec86226bb42b@mail.gmail.com>
Message-ID: <gdt5gm$fkg$1@ger.gmane.org>

Fri, 24 Oct 2008 14:39:59 -0400, Jose Borreguero wrote:

> My bad. Using the intermediate list does *not* leak. Still, the original
> problems stays. Can anyone run the following code in their machine and
> see if they have leaks? Maybe it only happens to me :(*
> 
> import numpy,Numeric
> big=10000000
> na=numpy.array([0.0,])
> for i in range(big):
>     Na=Numeric.array(na,Numeric.Float)*

Yep, leaks also here: (Numeric 24.2, numpy 1.2.0)

	import sys, numpy, Numeric
	na = numpy.array([0.0])
	for i in xrange(1000000):
	    foo = Numeric.array(na, Numeric.Float)
	print sys.getrefcount(na)

The getrefcount prints 1000002, so it seems like there's a refcount error 
somewhere. But

	na = numpy.array([0.0])
	for i in xrange(1000000):
	    foo = numpy.array(na, numpy.float_)
	print sys.getrefcount(na)

refcounts correctly.

-- 
Pauli Virtanen



From numpy-discussion at maubp.freeserve.co.uk  Fri Oct 24 15:05:01 2008
From: numpy-discussion at maubp.freeserve.co.uk (Peter)
Date: Fri, 24 Oct 2008 20:05:01 +0100
Subject: [Numpy-discussion] creating a Numeric array from a numpy array
	LEAKS memory
In-Reply-To: <gdt5gm$fkg$1@ger.gmane.org>
References: <mailman.97723.1224869620.3608.numpy-discussion@lists.sourceforge.net>
	<7cced4ed0810241036o22bfbc7dk4a052810f0bb3f43@mail.gmail.com>
	<7cced4ed0810241043h40c7779fnf6053a619c7c1d3a@mail.gmail.com>
	<49020BBB.5000302@enthought.com>
	<7cced4ed0810241116k77bf996ew586f8c327c7e2c2d@mail.gmail.com>
	<7cced4ed0810241139r1ebb887ajf05fec86226bb42b@mail.gmail.com>
	<gdt5gm$fkg$1@ger.gmane.org>
Message-ID: <320fb6e00810241205lcf48b61xfa89ce5784961d7a@mail.gmail.com>

On Fri, Oct 24, 2008 at 7:52 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Yep, leaks also here: (Numeric 24.2, numpy 1.2.0)
>
>        import sys, numpy, Numeric
>        na = numpy.array([0.0])
>        for i in xrange(1000000):
>            foo = Numeric.array(na, Numeric.Float)
>        print sys.getrefcount(na)
>
> The getrefcount prints 1000002, so it seems like there's a refcount error
> somewhere.

Same leak here using Numeric 24.2 and numpy 1.0.1 on Linux.

> But
>
>        na = numpy.array([0.0])
>        for i in xrange(1000000):
>            foo = numpy.array(na, numpy.float_)
>        print sys.getrefcount(na)
>
> refcounts correctly.

Also fine.  And for the record using the intermediate list also works for me:

import sys, numpy, Numeric
na = numpy.array([0.0])
na_list = list(na)
for i in xrange(1000000):
     foo = Numeric.array(na_list, Numeric.Float)
print sys.getrefcount(na)

Peter


From mat.yeates at gmail.com  Fri Oct 24 17:48:55 2008
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Fri, 24 Oct 2008 14:48:55 -0700
Subject: [Numpy-discussion] help vectorizing something
Message-ID: <7d0c05ac0810241448h6aceae0djded79aa0004238fd@mail.gmail.com>

Hi
I  have 2 vectors A and B. For each value in A I want to find the location
in B of the same value. Both A and B have unique elements.

Of course I could something like
For each index of A:
       v =A[index]
       location = numpy.where(B == v)

But I have very large lists and it will take too long.

Thanks to any one of you  vectorization gurus that has any ideas.

Mathew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/bcff5eb2/attachment.html>

From charlesr.harris at gmail.com  Fri Oct 24 18:12:46 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 24 Oct 2008 16:12:46 -0600
Subject: [Numpy-discussion] help vectorizing something
In-Reply-To: <7d0c05ac0810241448h6aceae0djded79aa0004238fd@mail.gmail.com>
References: <7d0c05ac0810241448h6aceae0djded79aa0004238fd@mail.gmail.com>
Message-ID: <e06186140810241512p773b4de6ieb02ad26f027969f@mail.gmail.com>

On Fri, Oct 24, 2008 at 3:48 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:

> Hi
> I  have 2 vectors A and B. For each value in A I want to find the location
> in B of the same value. Both A and B have unique elements.
>
> Of course I could something like
> For each index of A:
>        v =A[index]
>        location = numpy.where(B == v)
>
> But I have very large lists and it will take too long.
>

In [1]: A = array([1,2,3])

In [2]: B = array([5,1,3,0,2,4])

In [3]: i = B.argsort()

In [4]: Bsorted = B[i]

In [5]: indices = i[searchsorted(Bsorted,A)]

In [6]: indices
Out[6]: array([1, 4, 2])

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/0569ad64/attachment.html>

From mat.yeates at gmail.com  Fri Oct 24 18:23:16 2008
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Fri, 24 Oct 2008 15:23:16 -0700
Subject: [Numpy-discussion] help vectorizing something
In-Reply-To: <e06186140810241512p773b4de6ieb02ad26f027969f@mail.gmail.com>
References: <7d0c05ac0810241448h6aceae0djded79aa0004238fd@mail.gmail.com>
	<e06186140810241512p773b4de6ieb02ad26f027969f@mail.gmail.com>
Message-ID: <7d0c05ac0810241523k3ee518c4s815bb316a52df979@mail.gmail.com>

hmmmm. I don't understand the result.

If
a=array([ 1,  2,  3,  7, 10]) and b=array([ 1,  2,  3,  8, 10])

I want to get the result [0,1,2,4] but[searchsorted(a,b) produces
[0,1,2,4,4] ?? and searchsorted(b,a) produces [0,1,2,3,4]

??
Mathew


On Fri, Oct 24, 2008 at 3:12 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Fri, Oct 24, 2008 at 3:48 PM, Mathew Yeates <mat.yeates at gmail.com>wrote:
>
>> Hi
>> I  have 2 vectors A and B. For each value in A I want to find the location
>> in B of the same value. Both A and B have unique elements.
>>
>> Of course I could something like
>> For each index of A:
>>        v =A[index]
>>        location = numpy.where(B == v)
>>
>> But I have very large lists and it will take too long.
>>
>
> In [1]: A = array([1,2,3])
>
> In [2]: B = array([5,1,3,0,2,4])
>
> In [3]: i = B.argsort()
>
> In [4]: Bsorted = B[i]
>
> In [5]: indices = i[searchsorted(Bsorted,A)]
>
> In [6]: indices
> Out[6]: array([1, 4, 2])
>
> Chuck
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/b146610d/attachment.html>

From charlesr.harris at gmail.com  Fri Oct 24 19:03:33 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 24 Oct 2008 17:03:33 -0600
Subject: [Numpy-discussion] help vectorizing something
In-Reply-To: <7d0c05ac0810241523k3ee518c4s815bb316a52df979@mail.gmail.com>
References: <7d0c05ac0810241448h6aceae0djded79aa0004238fd@mail.gmail.com>
	<e06186140810241512p773b4de6ieb02ad26f027969f@mail.gmail.com>
	<7d0c05ac0810241523k3ee518c4s815bb316a52df979@mail.gmail.com>
Message-ID: <e06186140810241603y69eec343uf10173983984e51f@mail.gmail.com>

On Fri, Oct 24, 2008 at 4:23 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:

> hmmmm. I don't understand the result.
>
> If
> a=array([ 1,  2,  3,  7, 10]) and b=array([ 1,  2,  3,  8, 10])
>
> I want to get the result [0,1,2,4] but[searchsorted(a,b) produces
> [0,1,2,4,4] ?? and searchsorted(b,a) produces [0,1,2,3,4]
>

Because b isn't a subset of a. You can get around this by counting the
number, i.e.,

cnt = searchsorted(a,b, side='right') - seachsorted(a, b, side='left')

so that

In [1]: a = array([ 1,  2,  3,  7, 10])

In [2]: b = array([ 1,  2,  3,  8, 10])

In [3]: il = searchsorted(a, b, side='left')

In [4]: ir = searchsorted(a, b, side='right')

In [5]: compress(ir - il, il)
Out[5]: array([0, 1, 2, 4])

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081024/7adbc540/attachment.html>

From cournape at gmail.com  Sat Oct 25 05:57:23 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 25 Oct 2008 18:57:23 +0900
Subject: [Numpy-discussion] problem installing numpy using scons
In-Reply-To: <5b8d13220810190357i618c1ec4t689d701518f5fb15@mail.gmail.com>
References: <4db580fd0810181606q4b89fdday514a10617409d350@mail.gmail.com>
	<5b8d13220810190357i618c1ec4t689d701518f5fb15@mail.gmail.com>
Message-ID: <5b8d13220810250257q5c1c3397l255f787c5fd49c96@mail.gmail.com>

On Sun, Oct 19, 2008 at 7:57 PM, David Cournapeau <cournape at gmail.com> wrote:
>
> The trunk requires some changes which are not released yet. It will be
> in the next release of numscons (0.9.3).

Which has just been released.

David


From hoytak at cs.ubc.ca  Sat Oct 25 12:09:38 2008
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Sat, 25 Oct 2008 09:09:38 -0700
Subject: [Numpy-discussion] problem installing numpy using scons
In-Reply-To: <5b8d13220810250257q5c1c3397l255f787c5fd49c96@mail.gmail.com>
References: <4db580fd0810181606q4b89fdday514a10617409d350@mail.gmail.com>
	<5b8d13220810190357i618c1ec4t689d701518f5fb15@mail.gmail.com>
	<5b8d13220810250257q5c1c3397l255f787c5fd49c96@mail.gmail.com>
Message-ID: <4db580fd0810250909x1a014ae9s976fd1cb52fb4464@mail.gmail.com>

Excellent; thank you -- I will test it out soon.

-- Hoyt

On Sat, Oct 25, 2008 at 2:57 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sun, Oct 19, 2008 at 7:57 PM, David Cournapeau <cournape at gmail.com> wrote:
>>
>> The trunk requires some changes which are not released yet. It will be
>> in the next release of numscons (0.9.3).
>
> Which has just been released.
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From nono.231 at gmail.com  Sat Oct 25 16:14:46 2008
From: nono.231 at gmail.com (I. Soumpasis)
Date: Sat, 25 Oct 2008 21:14:46 +0100
Subject: [Numpy-discussion] ANN: Python programs for epidemic modelling
Message-ID: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>

Dear lists,

DeductiveThinking.com now provides the Python programs for the book of M.
Keeling & P. Rohani "Modeling Infectious Diseases in Humans and Animals",
Princeton University Press, 2008. The book has on-line material which
includes programs for different models in various programming languages and
mathematical tools such as, "C++, FORTRAN and Matlab, while some are also
coded in the web-based Java programming language to allow readers to quickly
experiment with these types of models", as it is stated at the website. The
Python version of the programs were written long ago and submitted to the
book's on line material website (available soon). The Python programs with
the basic equations modelled and the results in figures were now uploaded on
a special wiki page of DeductiveThinking.com.

Since, the programs are heavily using numpy, scipy and matplotlib libraries,
I send this announcement to all the three lists and the main python-list;
sorry for double-posting. The announcement with the related links is
uploaded here http://blog.deductivethinking.com/?p=29. The programs are at
http://wiki.deductivethinking.com/wiki/Python_Programs_for_Modelling_Infectious_Diseases_book.
For those who are interested on modelling and epidemiology, they can take a
look at the main site (http://deductivethinking.com) or the main page of the
wiki (http://wiki.deductivethinking.com) and follow the epidemiology links.
The website is in its beginning, so limited information is uploaded up to
now.

Thanks for your time and I hope it will be useful for some people,
Best Regards,
Ilias Soumpasis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081025/2ffb172e/attachment.html>

From ggellner at uoguelph.ca  Sat Oct 25 17:04:12 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Sat, 25 Oct 2008 17:04:12 -0400
Subject: [Numpy-discussion] ANN: Python programs for epidemic modelling
In-Reply-To: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>
References: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>
Message-ID: <20081025210412.GA6250@encolpuis>

>    Since, the programs are heavily using numpy, scipy and matplotlib
>    libraries, I send this announcement to all the three lists and the main
>    python-list; sorry for double-posting. The announcement with the related
>    links is uploaded here [1]http://blog.deductivethinking.com/?p=29. The
>    programs are at
>    [2]http://wiki.deductivethinking.com/wiki/Python_Programs_for_Modelling_Infectious_Diseases_book.
Thanks for this, it is great!

Gabriel


From aisaac at american.edu  Sat Oct 25 17:46:25 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 25 Oct 2008 17:46:25 -0400
Subject: [Numpy-discussion] [SciPy-user] ANN: Python programs for
	epidemic modelling
In-Reply-To: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>
References: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>
Message-ID: <490393B1.7030904@american.edu>

On 10/25/2008 4:14 PM I. Soumpasis apparently wrote:
> http://blog.deductivethinking.com/?p=29

This is cool.
But I do not see a license.
May I hope this is released under the new BSD license,
like the packages it depends on?

Thanks,
Alan Isaac



From nono.231 at gmail.com  Sat Oct 25 18:07:53 2008
From: nono.231 at gmail.com (I. Soumpasis)
Date: Sat, 25 Oct 2008 23:07:53 +0100
Subject: [Numpy-discussion] [SciPy-user] ANN: Python programs for
	epidemic modelling
In-Reply-To: <490393B1.7030904@american.edu>
References: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>
	<490393B1.7030904@american.edu>
Message-ID: <3ff92a550810251507q3696aa15x46f5b96684ff7f3c@mail.gmail.com>

2008/10/25 Alan G Isaac <aisaac at american.edu>

> On 10/25/2008 4:14 PM I. Soumpasis apparently wrote:
> > http://blog.deductivethinking.com/?p=29
>
> This is cool.
> But I do not see a license.
> May I hope this is released under the new BSD license,
> like the packages it depends on?
>
> The programs are GPL licensed. More info on the section of copyrights
http://wiki.deductivethinking.com/wiki/Deductive_Thinking:Copyrights.

I hope it is ok,
Ilias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081025/202067fa/attachment.html>

From aisaac at american.edu  Sat Oct 25 23:55:43 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 25 Oct 2008 23:55:43 -0400
Subject: [Numpy-discussion] [SciPy-user] ANN: Python programs
 for	epidemic modelling
In-Reply-To: <3ff92a550810251507q3696aa15x46f5b96684ff7f3c@mail.gmail.com>
References: <3ff92a550810251314m18d0596fxffb0a09658a21260@mail.gmail.com>	<490393B1.7030904@american.edu>
	<3ff92a550810251507q3696aa15x46f5b96684ff7f3c@mail.gmail.com>
Message-ID: <4903EA3F.2070504@american.edu>

On 10/25/2008 6:07 PM I. Soumpasis wrote:
> The programs are GPL licensed. More info on the section of copyrights
 > http://wiki.deductivethinking.com/wiki/Deductive_Thinking:Copyrights.
> I hope it is ok,

Well, that depends what you mean by "ok".

Obviously, the author picks the license s/he prefers.
But a GPL license means that some people will avoid
your code, so you make wish to make sure you thought
the licensing issue for this code carefully.

As a point of comparison,
note that all your package dependencies have
a new BSD license.

Alan Isaac



From millman at berkeley.edu  Sun Oct 26 04:58:57 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 01:58:57 -0700
Subject: [Numpy-discussion] NumPy 1.2.1 to be tagged 10/28/08
Message-ID: <c7009a550810260158g8bdfc08xda5cf44c2612a01e@mail.gmail.com>

Hey,

I plan to release NumPy 1.2.1 before the SciPy 0.7 sprint that Stefan
is organizing for Nov. 1-2.  So I will be tagging the release on
Wednesday, Oct. 29th.

Is anyone planning to back port anymore fixes to the 1.2.x branch?  If
so, please do so by Tuesday, Oct. 28th.

Here is what has been back ported
(http://projects.scipy.org/scipy/numpy/log/branches/1.2.x):

bug fix for subclassing object arrays:
http://projects.scipy.org/scipy/numpy/changeset/5891

MaskedArray fixes:
http://projects.scipy.org/scipy/numpy/changeset/5936
http://projects.scipy.org/scipy/numpy/changeset/5948

Python 2.4 compatible lookfor:
http://projects.scipy.org/scipy/numpy/changeset/5945

Setuptools fix:
http://projects.scipy.org/scipy/numpy/changeset/5956

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From philbinj at gmail.com  Sun Oct 26 09:43:49 2008
From: philbinj at gmail.com (James Philbin)
Date: Sun, 26 Oct 2008 13:43:49 +0000
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <4900A706.70806@noaa.gov>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
Message-ID: <2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>

This hack for defining infix operators might be relevant:
http://code.activestate.com/recipes/384122/

James


From pav at iki.fi  Sun Oct 26 10:13:37 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 26 Oct 2008 16:13:37 +0200
Subject: [Numpy-discussion] docs.scipy.org -- new site for the documentation
	marathon
Message-ID: <1225030418.8147.86.camel@idol>

Hi all,

Ga?l and I finished moving the Scipy documentation editor web app to a
new site (at Enthought, thanks!). The Numpy documentation marathon can
now be found at

	http://docs.scipy.org/numpy/

instead of the old location. Everything should work as previously, or
even better, but if something is broken, please notify me.

    ***

The domain name was, however, very tempting for hosting also the
Sphinx-generated documentation, so we put those there too. You can
browse to

	http://docs.scipy.org/

to see how they currently look like. Suggestions how to improve this
are welcome!

Currently, the sources for the Sphinx stuff can be found here:

	https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide
	http://www.iki.fi/pav/tmp/scipy-refguide.tar.gz
	http://www.iki.fi/pav/tmp/docsscipy-frontpage.tar.gz
	(The latter two are tarballs because of problems with
	launchpad.net and apparently Bzr 1.6.1.)

But I think I will move all of them to Scipy's SVN -- I believe the
advantages of using the same revision control system as numpy and scipy
themselves will outweigh those of using Bzr.

Eventually, we can also make all this documentation editable on the web
via the Documentation marathon app. (Actually, we already could, but
creating new files is not possible using the app yet, and there are some
performance issues remaining in rendering long documentation files.)

   ***

Now, the role of docs.scipy.org warrants discussion, because on the one
hand, the domain "docs.scipy.org" looks very official, and on the other
hand, "scipy.org/Documentation" claims to be the place for official
documentation. What do you think: should we use the current front page
of docs.scipy.org, shifting the focus and entry point of documentation
to the Sphinx-generated pages, or still keep the focus on the Moin wiki
at scipy.org?

I'd like to see a move towards a full dedicated documentation site -- I
believe using Sphinx is the way to go in the future, and documentation
generated by it does have a reassuring and clear visual appearance.
Also, making the Sphinx documentation more prominent could help people
to focus documentation efforts, and write more of it :)

Cheers,

-- 
Pauli Virtanen




From wnbell at gmail.com  Sun Oct 26 12:41:28 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 26 Oct 2008 12:41:28 -0400
Subject: [Numpy-discussion] docs.scipy.org -- new site for the
	documentation marathon
In-Reply-To: <1225030418.8147.86.camel@idol>
References: <1225030418.8147.86.camel@idol>
Message-ID: <d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>

On Sun, Oct 26, 2008 at 10:13 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
> Now, the role of docs.scipy.org warrants discussion, because on the one
> hand, the domain "docs.scipy.org" looks very official, and on the other
> hand, "scipy.org/Documentation" claims to be the place for official
> documentation. What do you think: should we use the current front page
> of docs.scipy.org, shifting the focus and entry point of documentation
> to the Sphinx-generated pages, or still keep the focus on the Moin wiki
> at scipy.org?
>
> I'd like to see a move towards a full dedicated documentation site -- I
> believe using Sphinx is the way to go in the future, and documentation
> generated by it does have a reassuring and clear visual appearance.
> Also, making the Sphinx documentation more prominent could help people
> to focus documentation efforts, and write more of it :)
>

Great work, I definitely like the look of the Sphinx documentation.

As a concrete example of your question, I've been working on some
scipy.sparse documentation [1], and it seems like it will be fairly
lengthy when completed.  Would it be appropriate to merge it into the
Sphinx documentation for scipy.sparse [2], or should the Sphinx docs
be more concise?

[1] http://www.scipy.org/SciPyPackages/Sparse
[2] http://docs.scipy.org/doc/scipy/reference/sparse.html

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From millman at berkeley.edu  Sun Oct 26 13:25:07 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 10:25:07 -0700
Subject: [Numpy-discussion] docs.scipy.org -- new site for the
	documentation marathon
In-Reply-To: <1225030418.8147.86.camel@idol>
References: <1225030418.8147.86.camel@idol>
Message-ID: <c7009a550810261025g5d9415b7w1fef2d9059f5c644@mail.gmail.com>

Thanks so much for doing this.  It looks great.

On Sun, Oct 26, 2008 at 7:13 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Now, the role of docs.scipy.org warrants discussion, because on the one
> hand, the domain "docs.scipy.org" looks very official, and on the other
> hand, "scipy.org/Documentation" claims to be the place for official
> documentation. What do you think: should we use the current front page
> of docs.scipy.org, shifting the focus and entry point of documentation
> to the Sphinx-generated pages, or still keep the focus on the Moin wiki
> at scipy.org?

docs.scipy.org should be the official documentation.  The Moin wiki
documentation served its purpose, but now there is something much
better.

I propose that the 'Documentation' sidebar on http://www.scipy.org/
point to http://docs.scipy.org.  Eventually, I would like to see all
the content from http://www.scipy.org/Documentation move into the
Sphinx-based system (and then we can just delete the Moin page).
Until that happens we can just leave the link on http://docs.scipy.org
to http://www.scipy.org/.

Also I have cross-posted my response to the numpy list, but let's have
this discussion on the scipy developer's list going forward.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Sun Oct 26 13:37:00 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 10:37:00 -0700
Subject: [Numpy-discussion] docs.scipy.org -- new site for the
	documentation marathon
In-Reply-To: <d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
References: <1225030418.8147.86.camel@idol>
	<d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
Message-ID: <c7009a550810261037x7dc44769k23da3c41d6dc8f2f@mail.gmail.com>

On Sun, Oct 26, 2008 at 9:41 AM, Nathan Bell <wnbell at gmail.com> wrote:
> As a concrete example of your question, I've been working on some
> scipy.sparse documentation [1], and it seems like it will be fairly
> lengthy when completed.  Would it be appropriate to merge it into the
> Sphinx documentation for scipy.sparse [2], or should the Sphinx docs
> be more concise?
>
> [1] http://www.scipy.org/SciPyPackages/Sparse
> [2] http://docs.scipy.org/doc/scipy/reference/sparse.html

Thanks Nathan for working on the sparse documentation!  I would like
to see all your documentation in the Sphinx docs:
http://docs.scipy.org/doc/scipy/reference/sparse.htm

All the documentation should be in the Sphinx system.  One of the main
advantages of using the Sphinx system is that it is much easier to
make living documentation if it is more closely integrated with the
code.  First, when changes are made to the code it is much easier to
just use a tool like grep or grin to find all the places where you
need to update the documentation.  Since we can use matplotlib to
generate figures when the documentation is created, we can better
ensure that the figures in our documentation can be generated by the
code in our documentation.  We can also use our testing infrastructure
to run any code in the documentation.  This has the advantages of
providing us with more tests and helps keep code in our documentation
runnable.

I would like to see http://www.scipy.org/SciPyPackages/Sparse removed
and all the content merged into
http://docs.scipy.org/doc/scipy/reference/sparse.html

I have cross-posted my response to the numpy list, but let's have this
discussion on the scipy developer's list going forward.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From dwf at cs.toronto.edu  Sun Oct 26 21:45:12 2008
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Sun, 26 Oct 2008 21:45:12 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
Message-ID: <9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>

On 26-Oct-08, at 9:43 AM, James Philbin wrote:

> This hack for defining infix operators might be relevant:
> http://code.activestate.com/recipes/384122/

I think someone mentioned this at the doc BOF, but it was raised that  
this has problems with associativity, etc.

David


From robert.kern at gmail.com  Sun Oct 26 23:45:42 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 26 Oct 2008 22:45:42 -0500
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
Message-ID: <3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>

On Fri, Oct 24, 2008 at 11:30, frank wang <f.yw at hotmail.com> wrote:
> Hi,
>
> I have to send this request second time since my first message contains the
> attached data file which is too big and was blocked by the system. So this
> time I will not attach the data file.
>
> I have converted a matlab function to python using numpy.  both matlab and
> python run slow. I know that numpy has a lot of features, so I hope some
> experts can help me to speed up the code.

Can you describe in higher level terms what you are trying to do? I'm
having trouble following the code.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From philbinj at gmail.com  Mon Oct 27 07:22:58 2008
From: philbinj at gmail.com (James Philbin)
Date: Mon, 27 Oct 2008 11:22:58 +0000
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
Message-ID: <2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>

One operator which could be used is '%'. We could keep the current
behaviour for ARRAY%SCALAR but have ARRAY%ARRAY as being matrix
multiplication. It has the same precedence as *,/.

James


From nicolas.roux at st.com  Mon Oct 27 07:41:06 2008
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Mon, 27 Oct 2008 12:41:06 +0100
Subject: [Numpy-discussion] How to do:   y[y<T] = y+T
Message-ID: <005501c93828$e65e3230$e7ad810a@gnb.st.com>

Hello,

I hope this is not a silly question ;-)
I have a Numpy array, and I want to process it with :
  "if the value is lower than Threshold, then increase by Threshold"


I would like to translate it as:
 y[y<Treshold] = y + Treshold

To benefit from the Numpy speed.
But this doesn't work, any idea ?

Thanks,
Cheers,
Nicolas.



From ggellner at uoguelph.ca  Mon Oct 27 07:47:48 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Mon, 27 Oct 2008 07:47:48 -0400
Subject: [Numpy-discussion] How to do:   y[y<T] = y+T
In-Reply-To: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
References: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
Message-ID: <20081027114748.GA18556@encolpuis>

On Mon, Oct 27, 2008 at 12:41:06PM +0100, Nicolas ROUX wrote:
> Hello,
> 
> I hope this is not a silly question ;-)
> I have a Numpy array, and I want to process it with :
>   "if the value is lower than Threshold, then increase by Threshold"
> 
> 
> I would like to translate it as:
>  y[y<Treshold] = y + Treshold
> 
You are close, you just need to have the right hand side also use vector
indexing (since otherwise you are trying to add something of length y to a
subset of y):

    y[y < Threshold] = y[y < Threshold] + Threshold

Gabriel


From uschmitt at mineway.de  Mon Oct 27 07:45:44 2008
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Mon, 27 Oct 2008 12:45:44 +0100
Subject: [Numpy-discussion] [mailinglist]  How to do:   y[y<T] = y+T
In-Reply-To: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
References: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
Message-ID: <4905A9E8.9090100@mineway.de>

Nicolas ROUX schrieb:
> Hello,
>
> I hope this is not a silly question ;-)
> I have a Numpy array, and I want to process it with :
>   "if the value is lower than Threshold, then increase by Threshold"
>
>
> I would like to translate it as:
>  y[y<Treshold] = y + Treshold
>   
Hi,

your solution does not work, becaus the arrays on both
side do not have the same size in generall.

You can do it in place:

    y[y<T] += T

or explicitely (slower/more memory):

    y[y<T] = y[y<T] + T

Greetings, Uwe
> To benefit from the Numpy speed.
> But this doesn't work, any idea ?
>
> Thanks,
> Cheers,
> Nicolas.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   


-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik

mineway GmbH
Science Park 2
D-66123 Saarbr?cken

Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376

uschmitt at mineway.de
www.mineway.de

Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339






From ggellner at uoguelph.ca  Mon Oct 27 08:06:34 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Mon, 27 Oct 2008 08:06:34 -0400
Subject: [Numpy-discussion] [mailinglist]  How to do:   y[y<T] = y+T
In-Reply-To: <4905A9E8.9090100@mineway.de>
References: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
	<4905A9E8.9090100@mineway.de>
Message-ID: <20081027120634.GA18975@encolpuis>

On Mon, Oct 27, 2008 at 12:45:44PM +0100, Uwe Schmitt wrote:
> Nicolas ROUX schrieb:
> > Hello,
> >
> > I hope this is not a silly question ;-)
> > I have a Numpy array, and I want to process it with :
> >   "if the value is lower than Threshold, then increase by Threshold"
> >
> >
> > I would like to translate it as:
> >  y[y<Treshold] = y + Treshold
> >   
> Hi,
> 
> your solution does not work, becaus the arrays on both
> side do not have the same size in generall.
> 
> You can do it in place:
> 
>     y[y<T] += T
> 
Nice, I didn't know this :-) Thanks.

Gabriel


From david.douard at logilab.fr  Mon Oct 27 07:51:00 2008
From: david.douard at logilab.fr (David Douard)
Date: Mon, 27 Oct 2008 12:51:00 +0100
Subject: [Numpy-discussion] How to do:   y[y<T] = y+T
In-Reply-To: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
References: <005501c93828$e65e3230$e7ad810a@gnb.st.com>
Message-ID: <200810271251.02212.david.douard@logilab.fr>

Le Monday 27 October 2008 12:41:06 Nicolas ROUX, vous avez ?crit?:
> Hello,
>
> I hope this is not a silly question ;-)
> I have a Numpy array, and I want to process it with :
>   "if the value is lower than Threshold, then increase by Threshold"
>
>
> I would like to translate it as:
>  y[y<Treshold] = y + Treshold

let's see :

y[y<T] += T

Is it what you want ?

>
> To benefit from the Numpy speed.
> But this doesn't work, any idea ?
>
> Thanks,
> Cheers,
> Nicolas.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



-- 
David Douard                        LOGILAB, Paris (France), +33 1 45 32 03 12
Formations Python, Zope, Debian :   http://www.logilab.fr/formations
D?veloppement logiciel sur mesure : http://www.logilab.fr/services
Informatique scientifique :         http://www.logilab.fr/science
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081027/7d1937d7/attachment.sig>

From nicolas.roux at st.com  Mon Oct 27 08:51:46 2008
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Mon, 27 Oct 2008 13:51:46 +0100
Subject: [Numpy-discussion] How to do:   y[y<T] = y+T
In-Reply-To: <200810271251.02212.david.douard@logilab.fr>
Message-ID: <005c01c93832$c56942e0$e7ad810a@gnb.st.com>

Thanks for all of you,
for this fast and good reply ;-)


Nicolas.

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Douard
Sent: Monday, October 27, 2008 12:51 PM
To: numpy-discussion at scipy.org
Subject: Re: [Numpy-discussion] How to do: y[y<T] = y+T

Le Monday 27 October 2008 12:41:06 Nicolas ROUX, vous avez ?crit?:
> Hello,
>
> I hope this is not a silly question ;-)
> I have a Numpy array, and I want to process it with :
>   "if the value is lower than Threshold, then increase by Threshold"
>
>
> I would like to translate it as:
>  y[y<Treshold] = y + Treshold

let's see :

y[y<T] += T

Is it what you want ?

>
> To benefit from the Numpy speed.
> But this doesn't work, any idea ?
>
> Thanks,
> Cheers,
> Nicolas.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



-- 
David Douard                        LOGILAB, Paris (France), +33 1 45 32 03
12
Formations Python, Zope, Debian :   http://www.logilab.fr/formations
D?veloppement logiciel sur mesure : http://www.logilab.fr/services
Informatique scientifique :         http://www.logilab.fr/science



From dwf at cs.toronto.edu  Mon Oct 27 09:09:44 2008
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Mon, 27 Oct 2008 09:09:44 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
Message-ID: <BB12C035-BE89-4C20-9A83-684B61653243@cs.toronto.edu>

On 27-Oct-08, at 7:22 AM, James Philbin wrote:

> One operator which could be used is '%'. We could keep the current
> behaviour for ARRAY%SCALAR but have ARRAY%ARRAY as being matrix
> multiplication. It has the same precedence as *,/.

The problem is that it would monkey with existing semantics for  
broadcasting, and break with all the other arithmetic operators in  
this regard. I can't see it ever being accepted for that reason.


Example:

In [514]: x
Out[514]:
array([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])

In [515]: y
Out[515]: array([0, 1, 2])

In [516]: x % y
Out[516]:
array([[0, 0, 0],
        [0, 0, 1],
        [0, 0, 0]])


David


From aarchiba at physics.mcgill.ca  Mon Oct 27 09:11:16 2008
From: aarchiba at physics.mcgill.ca (Anne Archibald)
Date: Mon, 27 Oct 2008 09:11:16 -0400
Subject: [Numpy-discussion] How to do: y[y<T] = y+T
In-Reply-To: <005c01c93832$c56942e0$e7ad810a@gnb.st.com>
References: <200810271251.02212.david.douard@logilab.fr>
	<005c01c93832$c56942e0$e7ad810a@gnb.st.com>
Message-ID: <ce557a360810270611s55a6ee17yaa33c645605129df@mail.gmail.com>

If what you are trying to do is actually ensure all data is within the
range [a,b], you may be interested to know that python's % operator
works on floating-point numbers:

In [1]: -0.1 % 1
Out[1]: 0.90000000000000002

So if you want all samples in the range (0,1) you can just do y%=1.

Anne

2008/10/27 Nicolas ROUX <nicolas.roux at st.com>:
> Thanks for all of you,
> for this fast and good reply ;-)
>
>
> Nicolas.
>
> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org
> [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of David Douard
> Sent: Monday, October 27, 2008 12:51 PM
> To: numpy-discussion at scipy.org
> Subject: Re: [Numpy-discussion] How to do: y[y<T] = y+T
>
> Le Monday 27 October 2008 12:41:06 Nicolas ROUX, vous avez ?crit :
>> Hello,
>>
>> I hope this is not a silly question ;-)
>> I have a Numpy array, and I want to process it with :
>>   "if the value is lower than Threshold, then increase by Threshold"
>>
>>
>> I would like to translate it as:
>>  y[y<Treshold] = y + Treshold
>
> let's see :
>
> y[y<T] += T
>
> Is it what you want ?
>
>>
>> To benefit from the Numpy speed.
>> But this doesn't work, any idea ?
>>
>> Thanks,
>> Cheers,
>> Nicolas.
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> David Douard                        LOGILAB, Paris (France), +33 1 45 32 03
> 12
> Formations Python, Zope, Debian :   http://www.logilab.fr/formations
> D?veloppement logiciel sur mesure : http://www.logilab.fr/services
> Informatique scientifique :         http://www.logilab.fr/science
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

From kwmsmith at gmail.com  Mon Oct 27 10:45:31 2008
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 27 Oct 2008 09:45:31 -0500
Subject: [Numpy-discussion] Optimization of loops
In-Reply-To: <490100F6.6030806@unic.cnrs-gif.fr>
References: <200810221821.48143.yger@unic.cnrs-gif.fr>
	<1224779692.7661.64.camel@pc2.cole.uklinux.net>
	<490100F6.6030806@unic.cnrs-gif.fr>
Message-ID: <c266a7930810270745q79bc18b8yae71fe599e0ad786@mail.gmail.com>

On Thu, Oct 23, 2008 at 5:55 PM, Pierre Yger <pierre.yger at gmail.com> wrote:

>
> import bisect
> my_list.sort()
> time, ind = zip(*my_list)
> for i in id_list :
>     beg = bisect.bisect_left(ind,i)
>    end = bisect.bisect_right(ind,i)
>    mylist.append(tim[beg:end])
>

I've always found itertools.groupby to be the ideal function for such
groupings.  From what I can tell, it saves about 30% in time over the bisect
version.

[code]
from random import randrange, random
import bisect
from itertools import groupby

N = 1000
nn = 100000

dta = []
for i in xrange(nn):
    dta.append((randrange(0,N), random()))

def bsct(dta):
    dta.sort()
    ind, time = zip(*dta)
    ret_list = []
    for i in xrange(N):
        beg = bisect.bisect_left(ind, i)
        end = bisect.bisect_right(ind, i)
        ret_list.append((i,time[beg:end]))
    ret_list = [subl for subl in ret_list if len(subl) == 2]
    return ret_list

def gpby(dta):
    dta.sort()
    keyfunc = lambda x: x[0]
    res = []
    for key, subit in groupby(dta, keyfunc):
        res.append((key, tuple(time for ind, time in subit)))
    return res

if __name__ == '__main__':
    from time import clock
    dta1 = dta
    dta2 = dta[:]
    c1 = clock()
    bd = bsct(dta1)
    print "bisect: %f" % (clock() - c1)
    c1 = clock()
    gb = gpby(dta2)
    print "groupby: %f" % (clock() - c1)
    assert bd == gb
[/code]

bisect: 1.430000
groupby: 0.970000

Hope this helps,

Kurt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081027/44ff563d/attachment.html>

From nicolas.roux at st.com  Mon Oct 27 11:38:30 2008
From: nicolas.roux at st.com (Nicolas ROUX)
Date: Mon, 27 Oct 2008 16:38:30 +0100
Subject: [Numpy-discussion] How to do :    x[i,j] = y[k, j] with k = i+sj
In-Reply-To: <4905A9E8.9090100@mineway.de>
Message-ID: <006f01c9384a$109ab700$e7ad810a@gnb.st.com>

Hi,

Me again ;-)
I have now a new question to ask (I hope not too silly).

How to do :

for j in range(yHeight):
   for i in range(xWidth):
      x[j,i] = y[k,i] with k = numpy.mod(i+sj,yHeight)

With efficient numpy code, without the double "for" ?

Thanks, 


Cheers,
Nicolas.



-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Uwe Schmitt
Sent: Monday, October 27, 2008 12:46 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] [mailinglist] How to do: y[y<T] = y+T

Nicolas ROUX schrieb:
> Hello,
>
> I hope this is not a silly question ;-)
> I have a Numpy array, and I want to process it with :
>   "if the value is lower than Threshold, then increase by Threshold"
>
>
> I would like to translate it as:
>  y[y<Treshold] = y + Treshold
>   
Hi,

your solution does not work, becaus the arrays on both
side do not have the same size in generall.

You can do it in place:

    y[y<T] += T

or explicitely (slower/more memory):

    y[y<T] = y[y<T] + T

Greetings, Uwe
> To benefit from the Numpy speed.
> But this doesn't work, any idea ?
>
> Thanks,
> Cheers,
> Nicolas.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   


-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik

mineway GmbH
Science Park 2
D-66123 Saarbr?cken

Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376

uschmitt at mineway.de
www.mineway.de

Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339




_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From HAWRYLA at novachem.com  Mon Oct 27 11:29:41 2008
From: HAWRYLA at novachem.com (Andrew Hawryluk)
Date: Mon, 27 Oct 2008 09:29:41 -0600
Subject: [Numpy-discussion] any interest in including a second-order
	gradient?
Message-ID: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>

We wrote a simple variation on the gradient() function to calculate the
second derivatives. Would there be any interest in including a
gradient2() in numpy?

Andrew



def gradient2(f, *varargs):
    """Calculate the second-order gradient of an N-dimensional scalar
function.

    Uses central differences on the interior and first differences on
boundaries
    to give the same shape.

    Inputs:

      f -- An N-dimensional array giving samples of a scalar function

      varargs -- 0, 1, or N scalars giving the sample distances in each
direction

    Outputs:

      N arrays of the same shape as f giving the derivative of f with
respect
      to each dimension.

    """
    N = len(f.shape)  # number of dimensions
    n = len(varargs)
    if n == 0:
        dx = [1.0]*N
    elif n == 1:
        dx = [varargs[0]]*N
    elif n == N:
        dx = list(varargs)
    else:
        raise SyntaxError, "invalid number of arguments"

    # use central differences on interior and first differences on
endpoints

    outvals = []

    # create slice objects --- initially all are [:, :, ..., :]
    slice1 = [slice(None)]*N
    slice2 = [slice(None)]*N
    slice3 = [slice(None)]*N
    

    otype = f.dtype.char
    if otype not in ['f', 'd', 'F', 'D']:
        otype = 'd'

    for axis in range(N):
        # select out appropriate parts for this dimension
        out = zeros(f.shape, f.dtype.char)
        slice1[axis] = slice(1, -1)
        slice2[axis] = slice(2, None)
        slice3[axis] = slice(None, -2)
        # 1D equivalent -- out[1:-1] = (f[2:] - 2*f[1:-1] + f[:-2])
        out[slice1] = (f[slice2] - 2*f[slice1] + f[slice3])
        slice1[axis] = 0
        slice2[axis] = 1
        slice3[axis] = 2
        # 1D equivalent -- out[0] = (f[2] - 2*f[1] + f[0])
        out[slice1] = (f[slice3] - 2*f[slice2] + f[slice1])
        slice1[axis] = -1
        slice2[axis] = -2
        slice3[axis] = -3
        # 1D equivalent -- out[-1] = (f[-1] - 2*f{-2] + f[-3])
        out[slice1] = (f[slice1] - 2*f[slice2] + f[slice3])

        # divide by the squared step size
        outvals.append(out / dx[axis] / dx[axis])

        # reset the slice object in this dimension to ":"
        slice1[axis] = slice(None)
        slice2[axis] = slice(None)
        slice3[axis] = slice(None)

    if N == 1:
        return outvals[0]
    else:
        return outvals


From robert.kern at gmail.com  Mon Oct 27 13:51:39 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 27 Oct 2008 12:51:39 -0500
Subject: [Numpy-discussion] How to do : x[i,j] = y[k, j] with k = i+sj
In-Reply-To: <006f01c9384a$109ab700$e7ad810a@gnb.st.com>
References: <4905A9E8.9090100@mineway.de>
	<006f01c9384a$109ab700$e7ad810a@gnb.st.com>
Message-ID: <3d375d730810271051w5280bd80v94c60b280c87b523@mail.gmail.com>

On Mon, Oct 27, 2008 at 10:38, Nicolas ROUX <nicolas.roux at st.com> wrote:
> Hi,
>
> Me again ;-)
> I have now a new question to ask (I hope not too silly).
>
> How to do :
>
> for j in range(yHeight):
>   for i in range(xWidth):
>      x[j,i] = y[k,i] with k = numpy.mod(i+sj,yHeight)
>
> With efficient numpy code, without the double "for" ?

j, i = numpy.ogrid[:yHeight, :xWidth]
k = (i+j) % yHeight
x = y[k,i]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From efiring at hawaii.edu  Mon Oct 27 14:31:44 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Mon, 27 Oct 2008 08:31:44 -1000
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
Message-ID: <49060910.60903@hawaii.edu>

James Philbin wrote:
> One operator which could be used is '%'. We could keep the current
> behaviour for ARRAY%SCALAR but have ARRAY%ARRAY as being matrix
> multiplication. It has the same precedence as *,/.
> 

No, having completely unrelated meanings for the same operator symbol 
sounds like a recipe for chaos.  The whole point is to make the code 
more readable, not less.

Eric


From fperez.net at gmail.com  Mon Oct 27 15:27:46 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 27 Oct 2008 12:27:46 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <49060910.60903@hawaii.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
Message-ID: <db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>

On Mon, Oct 27, 2008 at 11:31 AM, Eric Firing <efiring at hawaii.edu> wrote:
> James Philbin wrote:
>> One operator which could be used is '%'. We could keep the current
>> behaviour for ARRAY%SCALAR but have ARRAY%ARRAY as being matrix
>> multiplication. It has the same precedence as *,/.
>>
>
> No, having completely unrelated meanings for the same operator symbol
> sounds like a recipe for chaos.  The whole point is to make the code
> more readable, not less.

Big -1 from me on this one too, for the same reasons Eric states.

I just got back from some travel and will try to update the doc later
this evening with all the feedback and will post again, so that we can
converge on a final doc, which I'll then pitch over the fence to the
python-dev list.

Thanks to all who have written, and keep it coming!

Cheers,

f


From robince at gmail.com  Mon Oct 27 15:56:56 2008
From: robince at gmail.com (Robin)
Date: Mon, 27 Oct 2008 19:56:56 +0000
Subject: [Numpy-discussion] ParallelProgramming wiki page
Message-ID: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>

Hi,

I made some changes to the ParallelProgramming wiki page to outline
use of the (multi)processing module as well as the threading module.

I'm very much not an expert on this - just researched it for myself,
so please feel free to correct/ extend/ delete as appropriate.

Robin


From markbak at gmail.com  Mon Oct 27 16:01:47 2008
From: markbak at gmail.com (Mark Bakker)
Date: Mon, 27 Oct 2008 21:01:47 +0100
Subject: [Numpy-discussion] problems multiplying poly1d with number from
	array
Message-ID: <6946b9500810271301t3120274cy34642c8748cee3cf@mail.gmail.com>

Hello list -

I can multiply a poly1d instance with a number, but when I multiply with a
number from an array, the order matters. That seems a bug:

>>> a = array([2])
>>> p = poly1d([1,2])
>>> print 2*p  # Works

2 x + 4
>>> print a[0]*p  # Doesn't work, returns an array
[2 4]
>>> print p*a[0]  # Works

2 x + 4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081027/5cec47a9/attachment.html>

From markbak at gmail.com  Mon Oct 27 16:07:38 2008
From: markbak at gmail.com (Mark Bakker)
Date: Mon, 27 Oct 2008 21:07:38 +0100
Subject: [Numpy-discussion] poly1d with complex coefficients doesn't
	integrate
Message-ID: <6946b9500810271307p6102df37l89a2df08a50ee6ce@mail.gmail.com>

I seem to be able to create a poly1d instance with complex coefficients, and
it works correctly. But I cannot integrate it. Does poly1d not support
complex coefficients? Any reason why not, that shouldn't be too difficult,
should it?
Thanks, Mark

>>> p = poly1d([1+1j,2+2j])
>>> p(2)
(4+4j)
>>> q=p.integ()
>>> print q # This is obviously the wrong answer
     2
0.5 x + 2 x
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081027/112d3e47/attachment.html>

From robert.kern at gmail.com  Mon Oct 27 16:20:03 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 27 Oct 2008 15:20:03 -0500
Subject: [Numpy-discussion] poly1d with complex coefficients doesn't
	integrate
In-Reply-To: <6946b9500810271307p6102df37l89a2df08a50ee6ce@mail.gmail.com>
References: <6946b9500810271307p6102df37l89a2df08a50ee6ce@mail.gmail.com>
Message-ID: <3d375d730810271320m354f704ard792be2e2efefc39@mail.gmail.com>

On Mon, Oct 27, 2008 at 15:07, Mark Bakker <markbak at gmail.com> wrote:
> I seem to be able to create a poly1d instance with complex coefficients, and
> it works correctly. But I cannot integrate it. Does poly1d not support
> complex coefficients? Any reason why not, that shouldn't be too difficult,
> should it?

It's a bug in polyint(). It explicitly constructs an array for the
result with the dtype float. This is something we should fix.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From hep.sebastien.binet at gmail.com  Mon Oct 27 16:20:46 2008
From: hep.sebastien.binet at gmail.com (Sebastien Binet)
Date: Mon, 27 Oct 2008 13:20:46 -0700
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
Message-ID: <200810271320.55078.binet@cern.ch>

On Monday 27 October 2008 12:56:56 Robin wrote:
> Hi,
>
> I made some changes to the ParallelProgramming wiki page to outline
> use of the (multi)processing module as well as the threading module.
>
> I'm very much not an expert on this - just researched it for myself,
> so please feel free to correct/ extend/ delete as appropriate.

I would mention the backport of multiprocessing for python-2.{4,5}:
 http://code.google.com/p/python-multiprocessing
so the amount of editing when one switches from 2.{4,5} to 2.6 is minimal :)

cheers,
sebastien.
-- 
###################################
# Dr. Sebastien Binet
# Lawrence Berkeley National Lab.
# 1 Cyclotron Road
# Berkeley, CA 94720
###################################

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081027/fe10ef06/attachment.sig>

From robert.kern at gmail.com  Mon Oct 27 16:24:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 27 Oct 2008 15:24:32 -0500
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <200810271320.55078.binet@cern.ch>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
	<200810271320.55078.binet@cern.ch>
Message-ID: <3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>

On Mon, Oct 27, 2008 at 15:20, Sebastien Binet
<hep.sebastien.binet at gmail.com> wrote:
> On Monday 27 October 2008 12:56:56 Robin wrote:
>> Hi,
>>
>> I made some changes to the ParallelProgramming wiki page to outline
>> use of the (multi)processing module as well as the threading module.
>>
>> I'm very much not an expert on this - just researched it for myself,
>> so please feel free to correct/ extend/ delete as appropriate.
>
> I would mention the backport of multiprocessing for python-2.{4,5}:
>  http://code.google.com/p/python-multiprocessing
> so the amount of editing when one switches from 2.{4,5} to 2.6 is minimal :)

Go for it. The wiki is open to editing.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From erik.tollerud at gmail.com  Mon Oct 27 16:54:09 2008
From: erik.tollerud at gmail.com (Erik Tollerud)
Date: Mon, 27 Oct 2008 13:54:09 -0700
Subject: [Numpy-discussion] numpy array change notifier?
Message-ID: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>

Is there any straightforward way of notifying on change of a numpy
array that leaves the numpy arrays still efficient?

That is, I would like to do the following:

class C:
    def __init__(self,arr):
         self.arr = arr
         #what code do put here?

    def notify(self):
        print 'do something fun and exciting'


>>>o = C(array([1,2,3,4]))
>>>print o.arr #just print the array
[1 2 3 4]
>>>edarr = o.arr
>>>edarr[2] = 10 #this should now call the notify method
do something fun and exciting
>>>print o.arr
[1 2 10 4]



So is there a means of registering the array or, failing that, setting
up the class so that all the numpy tricks work with o.arr[whatever]
while allowing me to implement a property that calls the notifier?


From nouiz at nouiz.org  Mon Oct 27 16:54:12 2008
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 27 Oct 2008 16:54:12 -0400
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
	<200810271320.55078.binet@cern.ch>
	<3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>
Message-ID: <2d1d7fe70810271354t215d6ad5yb2b54ca1fd013f85@mail.gmail.com>

Hi,

Their exist open source version of parallel BLAS library. I modified
the section "Use parallel primitives" to tell it. But my English is
bad, so if someone can check it, it would be nice.

Fred

On Mon, Oct 27, 2008 at 4:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Oct 27, 2008 at 15:20, Sebastien Binet
> <hep.sebastien.binet at gmail.com> wrote:
>> On Monday 27 October 2008 12:56:56 Robin wrote:
>>> Hi,
>>>
>>> I made some changes to the ParallelProgramming wiki page to outline
>>> use of the (multi)processing module as well as the threading module.
>>>
>>> I'm very much not an expert on this - just researched it for myself,
>>> so please feel free to correct/ extend/ delete as appropriate.
>>
>> I would mention the backport of multiprocessing for python-2.{4,5}:
>>  http://code.google.com/p/python-multiprocessing
>> so the amount of editing when one switches from 2.{4,5} to 2.6 is minimal :)
>
> Go for it. The wiki is open to editing.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Mon Oct 27 16:56:53 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 27 Oct 2008 15:56:53 -0500
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
Message-ID: <3d375d730810271356u67b65a0fmb8d8cdaee035aad9@mail.gmail.com>

On Mon, Oct 27, 2008 at 15:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
> Is there any straightforward way of notifying on change of a numpy
> array that leaves the numpy arrays still efficient?

Not currently, no.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Mon Oct 27 16:43:15 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 27 Oct 2008 16:43:15 -0400
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
Message-ID: <200810271643.15984.pgmdevlist@gmail.com>

On Monday 27 October 2008 16:54:09 Erik Tollerud wrote:
> Is there any straightforward way of notifying on change of a numpy
> array that leaves the numpy arrays still efficient?

Erik, may be you could try the trick presented here : 
http://www.scipy.org/Subclasses
in the __array_wrap__ section.



From bpederse at gmail.com  Mon Oct 27 17:10:38 2008
From: bpederse at gmail.com (Brent Pedersen)
Date: Mon, 27 Oct 2008 14:10:38 -0700
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <3d375d730810271356u67b65a0fmb8d8cdaee035aad9@mail.gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
	<3d375d730810271356u67b65a0fmb8d8cdaee035aad9@mail.gmail.com>
Message-ID: <e183a99d0810271410p55d99d55ld2556f51ef2cf155@mail.gmail.com>

On Mon, Oct 27, 2008 at 1:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Oct 27, 2008 at 15:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
>> Is there any straightforward way of notifying on change of a numpy
>> array that leaves the numpy arrays still efficient?
>
> Not currently, no.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

out of curiosity,
would something like this affect efficiency (and/or work):

class Notify(numpy.ndarray):
    def __setitem__(self, *args):
        self.notify(*args)
        return super(Notify, self).__setitem__(*args)

    def notify(self, *args):
        print 'notify:', args


with also overriding setslice?


From dmitrey.kroshko at scipy.org  Mon Oct 27 17:17:11 2008
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Mon, 27 Oct 2008 23:17:11 +0200
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <2d1d7fe70810271354t215d6ad5yb2b54ca1fd013f85@mail.gmail.com>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>	<200810271320.55078.binet@cern.ch>	<3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>
	<2d1d7fe70810271354t215d6ad5yb2b54ca1fd013f85@mail.gmail.com>
Message-ID: <49062FD7.30803@scipy.org>

Did you mean this one
http://www.netlib.org/scalapack/pblas_qref.html
?

As for the ParallelProgramming wiki page, there are some words in 
section "Use parallel primitives" about numpy.dot still I can't 
understand from the section: if I get numpy from sources and compile it 
(via "python setup.py build") in my AMD X2, will numpy.dot use 2nd CPU 
or not?

Regards, D.

Fr?d?ric Bastien wrote:
> Hi,
>
> Their exist open source version of parallel BLAS library. I modified
> the section "Use parallel primitives" to tell it. But my English is
> bad, so if someone can check it, it would be nice.
>
> Fred
>
> On Mon, Oct 27, 2008 at 4:24 PM, Robert Kern <robert.kern at gmail.com> wrote:
>   
>> On Mon, Oct 27, 2008 at 15:20, Sebastien Binet
>> <hep.sebastien.binet at gmail.com> wrote:
>>     
>>> On Monday 27 October 2008 12:56:56 Robin wrote:
>>>       
>>>> Hi,
>>>>
>>>> I made some changes to the ParallelProgramming wiki page to outline
>>>> use of the (multi)processing module as well as the threading module.
>>>>
>>>> I'm very much not an expert on this - just researched it for myself,
>>>> so please feel free to correct/ extend/ delete as appropriate.
>>>>         
>>> I would mention the backport of multiprocessing for python-2.{4,5}:
>>>  http://code.google.com/p/python-multiprocessing
>>> so the amount of editing when one switches from 2.{4,5} to 2.6 is minimal :)
>>>       
>> Go for it. The wiki is open to editing.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>   



From robince at gmail.com  Mon Oct 27 17:28:49 2008
From: robince at gmail.com (Robin)
Date: Mon, 27 Oct 2008 21:28:49 +0000
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <49062FD7.30803@scipy.org>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
	<200810271320.55078.binet@cern.ch>
	<3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>
	<2d1d7fe70810271354t215d6ad5yb2b54ca1fd013f85@mail.gmail.com>
	<49062FD7.30803@scipy.org>
Message-ID: <b14b627d0810271428m3d194099lf3be9b864837566f@mail.gmail.com>

On Mon, Oct 27, 2008 at 9:17 PM, dmitrey <dmitrey.kroshko at scipy.org> wrote:
> As for the ParallelProgramming wiki page, there are some words in
> section "Use parallel primitives" about numpy.dot still I can't
> understand from the section: if I get numpy from sources and compile it
> (via "python setup.py build") in my AMD X2, will numpy.dot use 2nd CPU
> or not?

Not unless you build numpy against a paralell enabled BLAS, for
example Intel MKL, ATLAS etc.

I think if you compile ATLAS with threading enabled, and then build
numpy using the appropriate ptlapack libraries (I forget the exact
name) then the dot should use the second CPU. As Frederic added to the
wiki - the number of threads to use can only be provided to atlas at
compile time.

With MKL I think you can choose this at run time (I think through an
environment variable but I'm not sure).

Similarly with the GOTO blas, but I'm not sure if numpy builds with
that, so maybe we should take that reference out.

Robin


From rmay31 at gmail.com  Mon Oct 27 17:45:12 2008
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 27 Oct 2008 16:45:12 -0500
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <e183a99d0810271410p55d99d55ld2556f51ef2cf155@mail.gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>	<3d375d730810271356u67b65a0fmb8d8cdaee035aad9@mail.gmail.com>
	<e183a99d0810271410p55d99d55ld2556f51ef2cf155@mail.gmail.com>
Message-ID: <49063668.6010108@gmail.com>

Brent Pedersen wrote:
> On Mon, Oct 27, 2008 at 1:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Oct 27, 2008 at 15:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
>>> Is there any straightforward way of notifying on change of a numpy
>>> array that leaves the numpy arrays still efficient?
>> Not currently, no.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> 
> out of curiosity,
> would something like this affect efficiency (and/or work):
> 
> class Notify(numpy.ndarray):
>     def __setitem__(self, *args):
>         self.notify(*args)
>         return super(Notify, self).__setitem__(*args)
> 
>     def notify(self, *args):
>         print 'notify:', args
> 
> 
> with also overriding setslice?

I haven't given this much thought, but you'd also likely need to do this
for the infix operators (+=, etc.).

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From bpederse at gmail.com  Mon Oct 27 19:54:33 2008
From: bpederse at gmail.com (Brent Pedersen)
Date: Mon, 27 Oct 2008 16:54:33 -0700
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <49063668.6010108@gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
	<3d375d730810271356u67b65a0fmb8d8cdaee035aad9@mail.gmail.com>
	<e183a99d0810271410p55d99d55ld2556f51ef2cf155@mail.gmail.com>
	<49063668.6010108@gmail.com>
Message-ID: <e183a99d0810271654i51d38d0foead0a8d37d64b88b@mail.gmail.com>

On Mon, Oct 27, 2008 at 2:45 PM, Ryan May <rmay31 at gmail.com> wrote:
> Brent Pedersen wrote:
>> On Mon, Oct 27, 2008 at 1:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Mon, Oct 27, 2008 at 15:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
>>>> Is there any straightforward way of notifying on change of a numpy
>>>> array that leaves the numpy arrays still efficient?
>>> Not currently, no.
>>>
>>> --
>>> Robert Kern
>>>
>>> "I have come to believe that the whole world is an enigma, a harmless
>>> enigma that is made terrible by our own mad attempt to interpret it as
>>> though it had an underlying truth."
>>>  -- Umberto Eco
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> out of curiosity,
>> would something like this affect efficiency (and/or work):
>>
>> class Notify(numpy.ndarray):
>>     def __setitem__(self, *args):
>>         self.notify(*args)
>>         return super(Notify, self).__setitem__(*args)
>>
>>     def notify(self, *args):
>>         print 'notify:', args
>>
>>
>> with also overriding setslice?
>
> I haven't given this much thought, but you'd also likely need to do this
> for the infix operators (+=, etc.).
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

ah. i didnt think of that. i guess one could loop through dir(self) in
__init__ and apply a wrapper to a list of method names or to all
callables.


From stefan at sun.ac.za  Tue Oct 28 04:14:47 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 28 Oct 2008 10:14:47 +0200
Subject: [Numpy-discussion] any interest in including a second-order
	gradient?
In-Reply-To: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
Message-ID: <9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>

Hi Andrew

We should discuss different options for the implementation.  The
namespace is fairly cluttered, and it may be that we want to implement
gradient3 some time in the future as well.  Maybe something like

gradient(f, 1, 2, 3, order=2)

would work -- then we can combine gradient and gradient2 (and gradient3).

What do you think?

Regards
St?fan

2008/10/27 Andrew Hawryluk <HAWRYLA at novachem.com>:
> We wrote a simple variation on the gradient() function to calculate the
> second derivatives. Would there be any interest in including a
> gradient2() in numpy?
>
> Andrew
>
>
>
> def gradient2(f, *varargs):
>    """Calculate the second-order gradient of an N-dimensional scalar
> function.
>
>    Uses central differences on the interior and first differences on
> boundaries
>    to give the same shape.
>
>    Inputs:
>
>      f -- An N-dimensional array giving samples of a scalar function
>
>      varargs -- 0, 1, or N scalars giving the sample distances in each
> direction
>
>    Outputs:
>
>      N arrays of the same shape as f giving the derivative of f with
> respect
>      to each dimension.
>
>    """
>    N = len(f.shape)  # number of dimensions
>    n = len(varargs)
>    if n == 0:
>        dx = [1.0]*N
>    elif n == 1:
>        dx = [varargs[0]]*N
>    elif n == N:
>        dx = list(varargs)
>    else:
>        raise SyntaxError, "invalid number of arguments"
>
>    # use central differences on interior and first differences on
> endpoints
>
>    outvals = []
>
>    # create slice objects --- initially all are [:, :, ..., :]
>    slice1 = [slice(None)]*N
>    slice2 = [slice(None)]*N
>    slice3 = [slice(None)]*N
>
>
>    otype = f.dtype.char
>    if otype not in ['f', 'd', 'F', 'D']:
>        otype = 'd'
>
>    for axis in range(N):
>        # select out appropriate parts for this dimension
>        out = zeros(f.shape, f.dtype.char)
>        slice1[axis] = slice(1, -1)
>        slice2[axis] = slice(2, None)
>        slice3[axis] = slice(None, -2)
>        # 1D equivalent -- out[1:-1] = (f[2:] - 2*f[1:-1] + f[:-2])
>        out[slice1] = (f[slice2] - 2*f[slice1] + f[slice3])
>        slice1[axis] = 0
>        slice2[axis] = 1
>        slice3[axis] = 2
>        # 1D equivalent -- out[0] = (f[2] - 2*f[1] + f[0])
>        out[slice1] = (f[slice3] - 2*f[slice2] + f[slice1])
>        slice1[axis] = -1
>        slice2[axis] = -2
>        slice3[axis] = -3
>        # 1D equivalent -- out[-1] = (f[-1] - 2*f{-2] + f[-3])
>        out[slice1] = (f[slice1] - 2*f[slice2] + f[slice3])
>
>        # divide by the squared step size
>        outvals.append(out / dx[axis] / dx[axis])
>
>        # reset the slice object in this dimension to ":"
>        slice1[axis] = slice(None)
>        slice2[axis] = slice(None)
>        slice3[axis] = slice(None)
>
>    if N == 1:
>        return outvals[0]
>    else:
>        return outvals
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From zbyszek at in.waw.pl  Tue Oct 28 04:46:44 2008
From: zbyszek at in.waw.pl (Zbyszek Szmek)
Date: Tue, 28 Oct 2008 09:46:44 +0100
Subject: [Numpy-discussion] numpy ticket #852 (temporary files in savez)
Message-ID: <20081028084644.GG16444@szyszka.in.waw.pl>

Hi,
I looked at this ticket and whipped up two alternative patches, like mentioned in 
the description: in-memory or temporary dir on disk. On my computer
the second one is slightly faster.

I think it is important to merge some version of this fix. In it's current form,
someone using numpy.io.savez() allows any user on the same system to overwrite
arbitrary files with his permissions.

Thanks,
Zbyszek


From nouiz at nouiz.org  Tue Oct 28 09:14:05 2008
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 28 Oct 2008 09:14:05 -0400
Subject: [Numpy-discussion] ParallelProgramming wiki page
In-Reply-To: <b14b627d0810271428m3d194099lf3be9b864837566f@mail.gmail.com>
References: <b14b627d0810271256v510d26e6m3c84fdbb69dfefd8@mail.gmail.com>
	<200810271320.55078.binet@cern.ch>
	<3d375d730810271324m9cc66f3pb03f5a5355a6e75@mail.gmail.com>
	<2d1d7fe70810271354t215d6ad5yb2b54ca1fd013f85@mail.gmail.com>
	<49062FD7.30803@scipy.org>
	<b14b627d0810271428m3d194099lf3be9b864837566f@mail.gmail.com>
Message-ID: <2d1d7fe70810280614h47638ba0j729fb2426da08469@mail.gmail.com>

On Mon, Oct 27, 2008 at 5:28 PM, Robin <robince at gmail.com> wrote:
[...]
>
> Similarly with the GOTO blas, but I'm not sure if numpy builds with
> that, so maybe we should take that reference out.
>
I'd like to let it their, as it is at the same performance level then
MKL. ATLAS don't have the same performence level of MKL... If you
build GOTO BLAS and build lapack with it, it should link with numpy.
But I haven't the time to test this.


Fred


From HAWRYLA at novachem.com  Tue Oct 28 17:28:31 2008
From: HAWRYLA at novachem.com (Andrew Hawryluk)
Date: Tue, 28 Oct 2008 15:28:31 -0600
Subject: [Numpy-discussion] any interest in including a
	second-ordergradient?
In-Reply-To: <9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
Message-ID: <48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>

I agree that the gradient functions should be combined, especially considering how much redundant code would be added by keeping them separate. Here is one possible implementation, but I don't like the signature yet as it departs from the current behaviour. At the risk of demonstrating my ignorance, is there no way to place the named parameter (order) after the variable-length parameter (dx) in Python?

Andrew


def gradient(f,order=1,*varargs):
    """Calculate the first or second order gradient of an N-dimensional scalar function.

    Uses central differences on the interior and first differences on boundaries
    to give the same shape.

    Inputs:

      f -- An N-dimensional array giving samples of a scalar function

      varargs -- 0, 1, or N scalars giving the sample distances in each direction

    Outputs:

      N arrays of the same shape as f giving the derivative of f with respect
      to each dimension.

    """
    if order not in [1,2]:
        raise SyntaxError, "invalid value for order of differentiation"
    
    N = len(f.shape)  # number of dimensions
    n = len(varargs)
    if n == 0:
        dx = [1.0]*N
    elif n == 1:
        dx = [varargs[0]]*N
    elif n == N:
        dx = list(varargs)
    else:
        raise SyntaxError, "invalid number of arguments"

    # use central differences on interior and first differences on endpoints

    outvals = []

    # create slice objects --- initially all are [:, :, ..., :]
    slice1 = [slice(None)]*N
    slice2 = [slice(None)]*N
    slice3 = [slice(None)]*N
    
    if order == 1:
        # first order derivative weightings
        forward = [-1.0,1.0,0]
        backward = [0,-1.0,1.0]
        centered = [-0.5,0,0.5]
    elif order == 2:
        # second order derivative weightings
        forward = backward = centered = [1.0,-2.0,1.0]
    else:
        raise RuntimeError, "Programming error"
    
    otype = f.dtype.char
    if otype not in ['f', 'd', 'F', 'D']:
        otype = 'd'

    for axis in range(N):
        out = np.zeros(f.shape, f.dtype.char)
        # Centered Finite-Divided-Difference Formulas
        slice1[axis] = slice(None, -2)
        slice2[axis] = slice(1, -1)
        slice3[axis] = slice(2, None)
        out[slice2] = (f[slice1]*centered[0] + f[slice2]*centered[1] + f[slice3]*centered[2])
        # Forward Finite-Divided-Difference Formulas
        slice1[axis] = 0
        slice2[axis] = 1
        slice3[axis] = 2
        out[slice1] = (f[slice1]*forward[0] + f[slice2]*forward[1] + f[slice3]*forward[2])
        # Barkward Finite-Divided-Difference Formulas
        slice1[axis] = -3
        slice2[axis] = -2
        slice3[axis] = -1
        out[slice3] = (f[slice1]*backward[0] + f[slice2]*backward[1] + f[slice3]*backward[2])

        # divide by the step size
        outvals.append(out / dx[axis]**order)
        
        # reset the slice object in this dimension to ":"
        slice1[axis] = slice(None)
        slice2[axis] = slice(None)
        slice3[axis] = slice(None)

    if N == 1:
        return outvals[0]
    else:
        return outvals




> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-
> bounces at scipy.org] On Behalf Of St?fan van der Walt
> Sent: 28 Oct 2008 2:15 AM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] any interest in including a second-
> ordergradient?
> 
> Hi Andrew
> 
> We should discuss different options for the implementation.  The
> namespace is fairly cluttered, and it may be that we want to implement
> gradient3 some time in the future as well.  Maybe something like
> 
> gradient(f, 1, 2, 3, order=2)
> 
> would work -- then we can combine gradient and gradient2 (and
> gradient3).
> 
> What do you think?
> 
> Regards
> St?fan
> 
> 2008/10/27 Andrew Hawryluk <HAWRYLA at novachem.com>:
> > We wrote a simple variation on the gradient() function to calculate
> > the second derivatives. Would there be any interest in including a
> > gradient2() in numpy?
> >
> > Andrew
> >
> >
> >
> > def gradient2(f, *varargs):
> >    """Calculate the second-order gradient of an N-dimensional scalar
> > function.
> >
> >    Uses central differences on the interior and first differences on
> > boundaries
> >    to give the same shape.
> >
> >    Inputs:
> >
> >      f -- An N-dimensional array giving samples of a scalar function
> >
> >      varargs -- 0, 1, or N scalars giving the sample distances in
> each
> > direction
> >
> >    Outputs:
> >
> >      N arrays of the same shape as f giving the derivative of f with
> > respect
> >      to each dimension.
> >
> >    """
> >    N = len(f.shape)  # number of dimensions
> >    n = len(varargs)
> >    if n == 0:
> >        dx = [1.0]*N
> >    elif n == 1:
> >        dx = [varargs[0]]*N
> >    elif n == N:
> >        dx = list(varargs)
> >    else:
> >        raise SyntaxError, "invalid number of arguments"
> >
> >    # use central differences on interior and first differences on
> > endpoints
> >
> >    outvals = []
> >
> >    # create slice objects --- initially all are [:, :, ..., :]
> >    slice1 = [slice(None)]*N
> >    slice2 = [slice(None)]*N
> >    slice3 = [slice(None)]*N
> >
> >
> >    otype = f.dtype.char
> >    if otype not in ['f', 'd', 'F', 'D']:
> >        otype = 'd'
> >
> >    for axis in range(N):
> >        # select out appropriate parts for this dimension
> >        out = zeros(f.shape, f.dtype.char)
> >        slice1[axis] = slice(1, -1)
> >        slice2[axis] = slice(2, None)
> >        slice3[axis] = slice(None, -2)
> >        # 1D equivalent -- out[1:-1] = (f[2:] - 2*f[1:-1] + f[:-2])
> >        out[slice1] = (f[slice2] - 2*f[slice1] + f[slice3])
> >        slice1[axis] = 0
> >        slice2[axis] = 1
> >        slice3[axis] = 2
> >        # 1D equivalent -- out[0] = (f[2] - 2*f[1] + f[0])
> >        out[slice1] = (f[slice3] - 2*f[slice2] + f[slice1])
> >        slice1[axis] = -1
> >        slice2[axis] = -2
> >        slice3[axis] = -3
> >        # 1D equivalent -- out[-1] = (f[-1] - 2*f{-2] + f[-3])
> >        out[slice1] = (f[slice1] - 2*f[slice2] + f[slice3])
> >
> >        # divide by the squared step size
> >        outvals.append(out / dx[axis] / dx[axis])
> >
> >        # reset the slice object in this dimension to ":"
> >        slice1[axis] = slice(None)
> >        slice2[axis] = slice(None)
> >        slice3[axis] = slice(None)
> >
> >    if N == 1:
> >        return outvals[0]
> >    else:
> >        return outvals
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


From robert.kern at gmail.com  Tue Oct 28 17:36:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 16:36:16 -0500
Subject: [Numpy-discussion] any interest in including a
	second-ordergradient?
In-Reply-To: <48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
Message-ID: <3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>

On Tue, Oct 28, 2008 at 16:28, Andrew Hawryluk <HAWRYLA at novachem.com> wrote:
> I agree that the gradient functions should be combined, especially considering how much redundant code would be added by keeping them separate. Here is one possible implementation, but I don't like the signature yet as it departs from the current behaviour. At the risk of demonstrating my ignorance, is there no way to place the named parameter (order) after the variable-length parameter (dx) in Python?

There's no good way to do it. It's unfortunate that the current
function uses *args for this. It's just ... not good for many reasons.
A better one would be like so:

def gradient(f, dx=None, order=1):
    N = len(f.shape)
    if dx is None:
        dx = np.ones([N])
    elif np.isscalar(dx):
        dx = np.array([dx] * N)
    elif len(dx) == N:
        dx = np.asarray(dx)
    else:
        raise ValueError("dx must be blah blah blah")
    ...

Personally, I would make a new function with this API, and deprecate
the current gradient(). Dunno what the name should be, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From silva at lma.cnrs-mrs.fr  Tue Oct 28 17:57:25 2008
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Tue, 28 Oct 2008 22:57:25 +0100
Subject: [Numpy-discussion] any interest in including
	a	second-ordergradient?
In-Reply-To: <48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
Message-ID: <1225231045.20998.12.camel@localhost.localdomain>

Le mardi 28 octobre 2008 ? 15:28 -0600, Andrew Hawryluk a ?crit :
> I agree that the gradient functions should be combined, especially
> considering how much redundant code would be added by keeping them
> separate. Here is one possible implementation, but I don't like the
> signature yet as it departs from the current behaviour. At the risk of
> demonstrating my ignorance, is there no way to place the named
> parameter (order) after the variable-length parameter (dx) in Python?

> > St?fan van der Walt
> > We should discuss different options for the implementation.  The
> > namespace is fairly cluttered, and it may be that we want to implement
> > gradient3 some time in the future as well.  Maybe something like
> > 	gradient(f, 1, 2, 3, order=2)
> > would work -- then we can combine gradient and gradient2 (and
> > gradient3). What do you think?

> > Andrew Hawryluk:
> > > We wrote a simple variation on the gradient() function to calculate
> > > the second derivatives. Would there be any interest in including a
> > > gradient2() in numpy?

Are there some parts of the code that may be used only once to calculate
both the gradient and the second derivative (isn't it called the
hessian, at least in the N-d case) ?
If a common function would fasten the computation of the gradient and
the hessian with a single call to a new function gradients(), it is
worth...
If the intent is just a reduction of the total length of the file
containing the gradient and gradient2 functions, I do not understand why
modifying the existent code. Why not creating a new function hessian()
having the same signature than gradient?
-- 
Fabrice Silva



From dwf at cs.toronto.edu  Wed Oct 29 00:14:34 2008
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 29 Oct 2008 00:14:34 -0400
Subject: [Numpy-discussion] any interest in including
	a	second-ordergradient?
In-Reply-To: <1225231045.20998.12.camel@localhost.localdomain>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<1225231045.20998.12.camel@localhost.localdomain>
Message-ID: <5A6DDC03-B222-46A4-A21A-7F83FFE6E199@cs.toronto.edu>

On 28-Oct-08, at 5:57 PM, Fabrice Silva wrote:

> Are there some parts of the code that may be used only once to  
> calculate
> both the gradient and the second derivative (isn't it called the
> hessian, at least in the N-d case) ?

Probably.  I'd imagine depends on your differencing scheme; central  
differences on the second derivative might imply a need to do two non- 
central differences to sample the gradient at +eps and -eps on each  
axis, respectively.

And yes, the square matrix of second derivatives of an N-d scalar  
function is typically called the Hessian, but some of the properties  
one typically expects from a "Hessian" (i.e. symmetry) are not  
necessarily true unless all of the the partial second derivatives are  
continuous.

> If a common function would fasten the computation of the gradient and
> the hessian with a single call to a new function gradients(), it is
> worth...
> If the intent is just a reduction of the total length of the file
> containing the gradient and gradient2 functions, I do not understand  
> why
> modifying the existent code. Why not creating a new function hessian()
> having the same signature than gradient?

As Stefan mentioned, the number of functions that get included when  
you type "from numpy import *" is already pretty large, and it'd be  
nice not to make it larger. Though you do raise a good point -- if  
it's known that the user wants both the gradient and the Hessian, and  
you can get the first for free while computing the second, then that's  
preferable to having to make two calls and duplicate work.

David


From keflavich at gmail.com  Wed Oct 29 01:44:06 2008
From: keflavich at gmail.com (Adam)
Date: Tue, 28 Oct 2008 23:44:06 -0600
Subject: [Numpy-discussion] "Advanced indexing" question - subset of data
	cube as a 2D array.
Message-ID: <fceb3d4a0810282244n3273cbc3uc418e74ca6b9a856@mail.gmail.com>

Hi numpy group,
    I have a problem I know there is an elegant solution to, but I can't
wrap my head around the right way to do the indexing.

The problem:
I have a 2D array that has been chopped up into 3 dimensions - it was [ time
X detectors ], it is now [ scans X time X detectors ].  During the chopping,
some of the time points and detector points have been removed, so the 3D
array contains only a subset of the data in the 2D array.  I'd like to
restore the 3D array back to the shape of the original 2D array b/c it's
being stored in a netCDF file that is not flexible.

My solution:
In [58]: array2d.shape
Out[58]: (11008, 144)

In [59]: array3d.shape
Out[59]: (23, 337, 107)

In [60]: whscan.shape
Out[60]: (7751,)

In [61]: 23*337
Out[61]: 7751

In [62]: temp2d = reshape(array3d,[23*337,107])

In [63]: temp2d2 = zeros([23*337,144])

In [64]: temp2d2[:,f.bolo_indices] = temp2d

In [65]: array2d[whscan,:] = temp2d2


This works, but it feels wrong to me: I think there should be a way to do
this by directly indexing array2d with two numpy arrays....

In the process of asking this question, I might have come up with the answer
(courtesy Stefan at http://mentat.za.net/):

In [85]: bi = (f.bolo_indices[np.newaxis,:]+ones([7751,1])).astype('int')

In [86]: whc = (whscan[:,np.newaxis] + ones([1,107])).astype('int')

In [87]: array2d[whc,bi] = temp2d

I thought this had worked, but the values didn't seem to be going to the
right places when I re-examined them.

Thanks,
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081028/f3ca02ff/attachment.html>

From fperez.net at gmail.com  Wed Oct 29 02:50:26 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 28 Oct 2008 23:50:26 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
Message-ID: <db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>

On Mon, Oct 27, 2008 at 12:27 PM, Fernando Perez <fperez.net at gmail.com> wrote:

> I just got back from some travel and will try to update the doc later
> this evening with all the feedback and will post again, so that we can
> converge on a final doc, which I'll then pitch over the fence to the
> python-dev list.

OK, thanks everyone for the feedback.  I've updated the bzr repo in
case anyone is using that, as well as the static copy:

https://cirl.berkeley.edu/fperez/static/numpy-pep225/

Please have a look (the html is in sync with the source reST doc) and
let me know if you have any more feedback, changes, etc.  I've tried
to put in all of your comments and suggestions, but please say so  if
I've missed something or you feel anything not to be accurate.

I'll leave it for a few days up, and if there are no objections by
next week, I'll  send it to the Python-dev list.  That will give them
some days to think about it, in case anyone from that  list is
interested in talking about it at the Nov 13 baypiggies meeting.

Cheers,

f


From dwf at cs.toronto.edu  Wed Oct 29 05:00:33 2008
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 29 Oct 2008 05:00:33 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
Message-ID: <E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>

Hi Fernando,

In Robert's comment, I think the reST processor somehow got rid of a  
backslash. In my browser I see

	(I'm looking at you, Matlab's "")

although this is an aside that will be lost on anyone who hasn't used  
the Matlab backslash operator anyway.

In fact, one other thing just came to mind that isn't obvious and (so  
far as I can remember) hasn't been brought up: even though Robert and  
others (myself included) really only care about an operator to  
facilitate matrix multiplication, I think most of us would support the  
~*=  (mentioned in the PEP) as a natural companion to ~*.

Am I right about this? I realize it can't buy you the same in-place  
semantics as +-= and friends currently enjoy with numpy arrays, but I  
think most folks would just *expect* it to work,  i.e. if foo is 3x4  
and bar is 4x1 (or maybe a length 4 rank 1 array) then

	foo ~*= bar

results in foo now pointing to an array that (however you choose to  
handle rank) contains exactly 3 elements.


Cheers,

David


On 29-Oct-08, at 2:50 AM, Fernando Perez wrote:

> On Mon, Oct 27, 2008 at 12:27 PM, Fernando Perez  
> <fperez.net at gmail.com> wrote:
>
>> I just got back from some travel and will try to update the doc later
>> this evening with all the feedback and will post again, so that we  
>> can
>> converge on a final doc, which I'll then pitch over the fence to the
>> python-dev list.
>
> OK, thanks everyone for the feedback.  I've updated the bzr repo in
> case anyone is using that, as well as the static copy:
>
> https://cirl.berkeley.edu/fperez/static/numpy-pep225/
>
> Please have a look (the html is in sync with the source reST doc) and
> let me know if you have any more feedback, changes, etc.  I've tried
> to put in all of your comments and suggestions, but please say so  if
> I've missed something or you feel anything not to be accurate.
>
> I'll leave it for a few days up, and if there are no objections by
> next week, I'll  send it to the Python-dev list.  That will give them
> some days to think about it, in case anyone from that  list is
> interested in talking about it at the Nov 13 baypiggies meeting.
>
> Cheers,
>
> f
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From millman at berkeley.edu  Wed Oct 29 10:51:56 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 29 Oct 2008 07:51:56 -0700
Subject: [Numpy-discussion] ANN: NumPy 1.2.1
Message-ID: <c7009a550810290751s1ded4869i3889ef8712fbec7@mail.gmail.com>

I'm pleased to announce the release of NumPy 1.2.1.

NumPy is the fundamental package needed for scientific computing with
Python.  It contains:

 * a powerful N-dimensional array object
 * sophisticated (broadcasting) functions
 * basic linear algebra functions
 * basic Fourier transforms
 * sophisticated random number capabilities
 * tools for integrating Fortran code.

Besides it's obvious scientific uses, NumPy can also be used as an
efficient multi-dimensional container of generic data. Arbitrary
data-types can be defined. This allows NumPy to seamlessly and
speedily integrate with a wide-variety of databases.

This bugfix release comes almost one month after the 1.2.0 release.
Please note that NumPy 1.2.1 requires Python 2.4 or greater.

For information, please see the release notes:
https://sourceforge.net/project/shownotes.php?release_id=636728&group_id=1369

You can download the release from here:
https://sourceforge.net/project/showfiles.php?group_id=1369

Thank you to everybody who contributed to this release.

Enjoy,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From adam.ginsburg at colorado.edu  Wed Oct 29 12:11:43 2008
From: adam.ginsburg at colorado.edu (Adam Ginsburg)
Date: Wed, 29 Oct 2008 10:11:43 -0600
Subject: [Numpy-discussion] "Advanced indexing" question - subset of
	data cube as a 2D array.
In-Reply-To: <fceb3d4a0810282244n3273cbc3uc418e74ca6b9a856@mail.gmail.com>
References: <fceb3d4a0810282244n3273cbc3uc418e74ca6b9a856@mail.gmail.com>
Message-ID: <fceb3d4a0810290911u7532d0c7x6c9aad30c62c72b2@mail.gmail.com>

> In [85]: bi = (f.bolo_indices[np.newaxis,:]+
>
> ones([7751,1])).astype('int')
>
> In [86]: whc = (whscan[:,np.newaxis] + ones([1,107])).astype('int')
>
> In [87]: array2d[whc,bi] = temp2d
>
> I thought this had worked, but the values didn't seem to be going to the right places when I re-examined them.

I think I answered my own question: if I'd used zeros() instead of
ones() it would have worked fine.  I don't know why I tried to use
ones().

Adam


From HAWRYLA at novachem.com  Wed Oct 29 12:23:10 2008
From: HAWRYLA at novachem.com (Andrew Hawryluk)
Date: Wed, 29 Oct 2008 10:23:10 -0600
Subject: [Numpy-discussion] any interest in
	includinga	second-ordergradient?
In-Reply-To: <5A6DDC03-B222-46A4-A21A-7F83FFE6E199@cs.toronto.edu>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com><9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com><48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com><1225231045.20998.12.camel@localhost.localdomain>
	<5A6DDC03-B222-46A4-A21A-7F83FFE6E199@cs.toronto.edu>
Message-ID: <48C01AE7354EC240A26F19CEB995E943033AEF90@CHMAILMBX01.novachem.com>

> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-
> bounces at scipy.org] On Behalf Of David Warde-Farley
> Sent: 28 Oct 2008 10:15 PM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] any interest in includinga second-
> ordergradient?
> 
> On 28-Oct-08, at 5:57 PM, Fabrice Silva wrote:
> 
> > Are there some parts of the code that may be used only once to
> > calculate both the gradient and the second derivative (isn't it
> called
> > the hessian, at least in the N-d case) ?
> 
> Probably.  I'd imagine depends on your differencing scheme; central
> differences on the second derivative might imply a need to do two non-
> central differences to sample the gradient at +eps and -eps on each
> axis, respectively.

That is true, at least in theory. The formula for the first derivative
can be interpreted as the average between the slopes on either side of a
point while the second derivative is proportional to the difference
between those slopes. In practice, however, both formulas are simplified
algebraically such that you don't get a 'free' first derivative while
calculating the second. The forward and backward differences at the
edges are also quite different between the first and second order.

> And yes, the square matrix of second derivatives of an N-d scalar
> function is typically called the Hessian, but some of the properties
> one typically expects from a "Hessian" (i.e. symmetry) are not
> necessarily true unless all of the the partial second derivatives are
> continuous.
> 
> > If a common function would fasten the computation of the gradient
and
> > the hessian with a single call to a new function gradients(), it is
> > worth...
> > If the intent is just a reduction of the total length of the file
> > containing the gradient and gradient2 functions, I do not understand
> > why modifying the existent code. Why not creating a new function
> > hessian() having the same signature than gradient?
> 
> As Stefan mentioned, the number of functions that get included when
you
> type "from numpy import *" is already pretty large, and it'd be nice
> not to make it larger. Though you do raise a good point -- if it's
> known that the user wants both the gradient and the Hessian, and you
> can get the first for free while computing the second, then that's
> preferable to having to make two calls and duplicate work.
> 
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

There could be a way to separate the first and second derivatives in a
way that lets the second take advantage of the work done by the first,
but it would yield only a small improvement on the second and it would
require a reduction in performance on the first. It would also make the
code much more confusing, and harder to extend in the future to higher
order derivatives.

On the whole it appears that Stefan's suggestion is the winner: one
function that returns one derivative of a specified order.

Andrew


From lists_ravi at lavabit.com  Wed Oct 29 12:26:21 2008
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 29 Oct 2008 12:26:21 -0400
Subject: [Numpy-discussion] "Advanced indexing" question - subset of
	data cube as a 2D array.
In-Reply-To: <fceb3d4a0810282244n3273cbc3uc418e74ca6b9a856@mail.gmail.com>
References: <fceb3d4a0810282244n3273cbc3uc418e74ca6b9a856@mail.gmail.com>
Message-ID: <200810291226.22629.lists_ravi@lavabit.com>

On Wednesday 29 October 2008 01:44:06 Adam wrote:
> In [62]: temp2d = reshape(array3d,[23*337,107])
>
> In [63]: temp2d2 = zeros([23*337,144])
>
> In [64]: temp2d2[:,f.bolo_indices] = temp2d
>
> In [65]: array2d[whscan,:] = temp2d2
>
>
> This works, but it feels wrong to me: I think there should be a way to do
> this by directly indexing array2d with two numpy arrays....

Does numpy.ix_ not fit your use case?

In [2]: y = arange(15).reshape((3,5))

In [3]: y
Out[3]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

In [4]: y[ ix_( array([1,2]), array([1,3,4]) ) ]
Out[4]:
array([[ 6,  8,  9],
       [11, 13, 14]])


Regards,
Ravi




From HAWRYLA at novachem.com  Wed Oct 29 12:32:12 2008
From: HAWRYLA at novachem.com (Andrew Hawryluk)
Date: Wed, 29 Oct 2008 10:32:12 -0600
Subject: [Numpy-discussion] any interest in including
	asecond-ordergradient?
In-Reply-To: <3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com><9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com><48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
Message-ID: <48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>

> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-
> bounces at scipy.org] On Behalf Of Robert Kern
> Sent: 28 Oct 2008 3:36 PM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] any interest in including asecond-
> ordergradient?
> 
> On Tue, Oct 28, 2008 at 16:28, Andrew Hawryluk <HAWRYLA at novachem.com>
> wrote:
> > I agree that the gradient functions should be combined, especially
> considering how much redundant code would be added by keeping them
> separate. Here is one possible implementation, but I don't like the
> signature yet as it departs from the current behaviour. At the risk of
> demonstrating my ignorance, is there no way to place the named
> parameter (order) after the variable-length parameter (dx) in Python?
> 
> There's no good way to do it. It's unfortunate that the current
> function uses *args for this. It's just ... not good for many reasons.
> A better one would be like so:
> 
> def gradient(f, dx=None, order=1):
>     N = len(f.shape)
>     if dx is None:
>         dx = np.ones([N])
>     elif np.isscalar(dx):
>         dx = np.array([dx] * N)
>     elif len(dx) == N:
>         dx = np.asarray(dx)
>     else:
>         raise ValueError("dx must be blah blah blah")
>     ...
> 
> Personally, I would make a new function with this API, and deprecate
> the current gradient(). Dunno what the name should be, though.
> 
> --
> Robert Kern

That would be a huge improvement, and only differs from the current
behaviour when it is called on a multidimensional array with different
step sizes in each direction (the least common application, I suspect).

Any chance of using the proposed API with the existing name? What is
deemed sufficient justification for modifying the API of an existing
NumPy function? It causes trouble for existing users, but the number of
future users exceeds the number of existing users, so maybe it is worth
the trouble. If not, you could just call it 'grad' and deprecate the old
one as you suggest.

Andrew


From fperez.net at gmail.com  Wed Oct 29 12:59:42 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 29 Oct 2008 09:59:42 -0700
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
Message-ID: <db6b5ecc0810290959r76a1974dw330512eb38d0f533@mail.gmail.com>

Hey David,

On Wed, Oct 29, 2008 at 2:00 AM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> Hi Fernando,
>
> In Robert's comment, I think the reST processor somehow got rid of a
> backslash. In my browser I see
>
>        (I'm looking at you, Matlab's "")
>
> although this is an aside that will be lost on anyone who hasn't used
> the Matlab backslash operator anyway.

Oops, thanks.  Fixed now.

> In fact, one other thing just came to mind that isn't obvious and (so
> far as I can remember) hasn't been brought up: even though Robert and
> others (myself included) really only care about an operator to
> facilitate matrix multiplication, I think most of us would support the
> ~*=  (mentioned in the PEP) as a natural companion to ~*.

[...]

Thanks.  I added your comment below Robert's, with attribution, and
pushed again (to launchpad and to the static page).

Thanks for the edits!

Cheers,

f


From stefan at sun.ac.za  Wed Oct 29 13:29:52 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 29 Oct 2008 19:29:52 +0200
Subject: [Numpy-discussion] any interest in including
	asecond-ordergradient?
In-Reply-To: <48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>
Message-ID: <9457e7c80810291029s6d0dff59s7752a2553955ce8b@mail.gmail.com>

2008/10/29 Andrew Hawryluk <HAWRYLA at novachem.com>:
> Any chance of using the proposed API with the existing name? What is
> deemed sufficient justification for modifying the API of an existing
> NumPy function? It causes trouble for existing users, but the number of
> future users exceeds the number of existing users, so maybe it is worth
> the trouble. If not, you could just call it 'grad' and deprecate the old
> one as you suggest.

Again, I would opt to change the API now and save us and our users the
pain of doing it later (looking at the API for a function like
gradient, it strikes me that the API freeze was premature and done
without a proper review -- this function would never have passed!).
My opinion is not a popular one, though, so I suspect we'll have to
create a new function.  If we do that, it is important that we rewrite
the old function in terms of the new one, i.e.

def gradient(x, *varargs): return new_gradient(x, varargs)

so that we don't have to maintain two functions.

Regards
St?fan


From fperez.net at gmail.com  Wed Oct 29 13:46:55 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 29 Oct 2008 10:46:55 -0700
Subject: [Numpy-discussion] any interest in including
	asecond-ordergradient?
In-Reply-To: <9457e7c80810291029s6d0dff59s7752a2553955ce8b@mail.gmail.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>
	<9457e7c80810291029s6d0dff59s7752a2553955ce8b@mail.gmail.com>
Message-ID: <db6b5ecc0810291046s3f5c7801pdf00b0a0f00a0941@mail.gmail.com>

Howdy,

Minor comment on all this, from the peanut gallery...

Without commenting on the original gradient API or changes, I'm a
strong -1000 on introducing the notion of 'order' into a gradient
function.  The gradient, from every definition I can remember, is a
first-order operation.  Matlab's gradient is written this way:

http://www.mathworks.com/access/helpdesk/help/techdoc/index.html?/access/helpdesk/help/techdoc/ref/gradient.html

and for good reason: nobody speaks of 'second order gradients'.  The
gradient is an operation defined for functions f:R^n -> R as the
vector of all first partial derivatives, end of story.  In numpy lingo
a function like this is represented by an N-d array, and we're all OK
with that analogy. But I really think that gradient() in numpy should
stick to what the math literature has defined up until now.

I think it's fine to ask for functions that compute higher order
derivatives of n-d arrays: we already have diff(), which operates on a
single direction, and a hessian could make sense (with the caveats
David points out).   But with higher order derivatives there are many
more combinations to worry about, and I really think it's a bad idea
to lump those issues into the definition of gradient, which was a
perfectly unambiguous object up until this point.

Just my devalued U$ 1e-2...

Cheers,

f


From efiring at hawaii.edu  Wed Oct 29 15:33:03 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 29 Oct 2008 09:33:03 -1000
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<3d375d730810221415m33acef65rd8b79db1c985e04d@mail.gmail.com>
	<6FF1176C-F185-4070-AC92-EE27D9347E27@stsci.edu>
	<48FFC28E.5070207@hawaii.edu> <4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
Message-ID: <4908BA6F.90909@hawaii.edu>

David Warde-Farley wrote:
> Hi Fernando,
> 
> In Robert's comment, I think the reST processor somehow got rid of a  
> backslash. In my browser I see
> 
> 	(I'm looking at you, Matlab's "")
> 
> although this is an aside that will be lost on anyone who hasn't used  
> the Matlab backslash operator anyway.
> 
> In fact, one other thing just came to mind that isn't obvious and (so  
> far as I can remember) hasn't been brought up: even though Robert and  
> others (myself included) really only care about an operator to  
> facilitate matrix multiplication, I think most of us would support the  
> ~*=  (mentioned in the PEP) as a natural companion to ~*.

This makes no sense to me at all.  Why use in-place notation for an 
operation that is not in-place?  The point of having a matrix 
multiplication operator is to facilitate code that corresponds more 
closely to mathematical notation.  The in-place operations live in the 
computer science world, not the mathematical world.  The proposal for 
~*= violates language consistency while doing nothing to improve 
mathematical readability.

Eric

> 
> Am I right about this? I realize it can't buy you the same in-place  
> semantics as +-= and friends currently enjoy with numpy arrays, but I  
> think most folks would just *expect* it to work,  i.e. if foo is 3x4  
> and bar is 4x1 (or maybe a length 4 rank 1 array) then
> 
> 	foo ~*= bar
> 
> results in foo now pointing to an array that (however you choose to  
> handle rank) contains exactly 3 elements.
> 
> 
> Cheers,
> 
> David
> 
> 
> On 29-Oct-08, at 2:50 AM, Fernando Perez wrote:
> 
>> On Mon, Oct 27, 2008 at 12:27 PM, Fernando Perez  
>> <fperez.net at gmail.com> wrote:
>>
>>> I just got back from some travel and will try to update the doc later
>>> this evening with all the feedback and will post again, so that we  
>>> can
>>> converge on a final doc, which I'll then pitch over the fence to the
>>> python-dev list.
>> OK, thanks everyone for the feedback.  I've updated the bzr repo in
>> case anyone is using that, as well as the static copy:
>>
>> https://cirl.berkeley.edu/fperez/static/numpy-pep225/
>>
>> Please have a look (the html is in sync with the source reST doc) and
>> let me know if you have any more feedback, changes, etc.  I've tried
>> to put in all of your comments and suggestions, but please say so  if
>> I've missed something or you feel anything not to be accurate.
>>
>> I'll leave it for a few days up, and if there are no objections by
>> next week, I'll  send it to the Python-dev list.  That will give them
>> some days to think about it, in case anyone from that  list is
>> interested in talking about it at the Nov 13 baypiggies meeting.
>>
>> Cheers,
>>
>> f
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Wed Oct 29 15:43:45 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 29 Oct 2008 14:43:45 -0500
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <4908BA6F.90909@hawaii.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
	<4908BA6F.90909@hawaii.edu>
Message-ID: <3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>

On Wed, Oct 29, 2008 at 14:33, Eric Firing <efiring at hawaii.edu> wrote:
> David Warde-Farley wrote:
>> Hi Fernando,
>>
>> In Robert's comment, I think the reST processor somehow got rid of a
>> backslash. In my browser I see
>>
>>       (I'm looking at you, Matlab's "")
>>
>> although this is an aside that will be lost on anyone who hasn't used
>> the Matlab backslash operator anyway.
>>
>> In fact, one other thing just came to mind that isn't obvious and (so
>> far as I can remember) hasn't been brought up: even though Robert and
>> others (myself included) really only care about an operator to
>> facilitate matrix multiplication, I think most of us would support the
>> ~*=  (mentioned in the PEP) as a natural companion to ~*.
>
> This makes no sense to me at all.  Why use in-place notation for an
> operation that is not in-place?  The point of having a matrix
> multiplication operator is to facilitate code that corresponds more
> closely to mathematical notation.  The in-place operations live in the
> computer science world, not the mathematical world.  The proposal for
> ~*= violates language consistency while doing nothing to improve
> mathematical readability.

Eh, that's not entirely true.

  x = 1
  x += 2

That's not in-place. They are called "augmented assignments", not
"in-place operations" for this reason. The defining characteristic is
that "x <op>= y" should be equivalent to "x = x <op> y" except
possibly for *optional* in-place semantics.

But this isn't a choice for us to make. If we add operators to the
Python language, their associated augmented assignments will also be
added (no matter what we think about it). If a class doesn't define
the appropriate __magic__ method, then Python will execute "x = x <op>
y" and give us the semantics that David was suggesting. This is a
non-controversy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wesmckinn at gmail.com  Wed Oct 29 18:23:29 2008
From: wesmckinn at gmail.com (Wes McKinney)
Date: Wed, 29 Oct 2008 18:23:29 -0400
Subject: [Numpy-discussion] Enabling NaN-usage in F77 code on Windows
Message-ID: <6c476c8a0810291523n2845000bxdc5486656e91967e@mail.gmail.com>

I'm having some trouble getting NaN's to return from f77 code running under
latest f2py in both g77 and gfortran. I would prefer to use gfortran but
whenever I set a result value = NAN, it comes back to Python as 0. Has
anyone tackled this issue? I am new to using f2py, have been moving along
fine with everything else but ran into this.

Here is a sample function for the rolling mean of a series having this
behavior:

      SUBROUTINE ROLLMEAN(DATA,WINDOW,N,AVE,T)
      INTEGER*8 N, WINDOW
      REAL*8 AVE,DATA(N),T(N)
      INTEGER*8 J
      REAL*8 P,S,S1
C
CF2PY REAL*8 INTENT(IN, COPY) DATA
CF2PY INTEGER*8 INTENT(IN) WINDOW
CF2PY INTEGER*8 INTENT(HIDE), DEPEND(DATA), CHECK(N>=2) :: N = SHAPE(DATA,
0)
CF2PY REAL*8 INTENT(OUT, COPY), DEPEND(N), DIMENSION(N) :: T
CF2PY REAL*8 INTENT(HIDE) :: AVE
C
      S=0.
      S1=0.
      DO J=1,WINDOW
          P=DATA(J)
          S1=S1+P
          T(J)=NAN
      ENDDO
      AVE=S1/WINDOW
      T(WINDOW)=AVE
      DO J=WINDOW+1,N
          P=DATA(J)
          S=DATA(J-WINDOW)
          S1=S1+P-S
          AVE=S1/WINDOW
          T(J)=AVE
      ENDDO
      RETURN
      END


Thanks,
Wes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081029/2b46d462/attachment.html>

From dwf at cs.toronto.edu  Wed Oct 29 19:37:56 2008
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 29 Oct 2008 19:37:56 -0400
Subject: [Numpy-discussion] (Late) summary of PEP-225 discussion at Scipy
In-Reply-To: <3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<4900A706.70806@noaa.gov>
	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
	<4908BA6F.90909@hawaii.edu>
	<3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>
Message-ID: <291F12FC-FCFB-4152-9A6D-762878D8AF8F@cs.toronto.edu>

On 29-Oct-08, at 3:43 PM, Robert Kern wrote:

> Eh, that's not entirely true.
>
>  x = 1
>  x += 2
>
> That's not in-place. They are called "augmented assignments", not
> "in-place operations" for this reason. The defining characteristic is
> that "x <op>= y" should be equivalent to "x = x <op> y" except
> possibly for *optional* in-place semantics.

Indeed.

x = "foo"
x += "bar"

This definitely isn't in-place, since strings are immutable, but it  
works, and people expect it to work.

Of course it's worth mentioning in the docs somewhere that only the  
augmented assignments for element-wise ops are in-place (which should  
be obvious to anyone who knows what an outer product is).

David


From oliphant at enthought.com  Wed Oct 29 20:05:50 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 29 Oct 2008 19:05:50 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields from
	a data-type
Message-ID: <4908FA5E.9090803@enthought.com>


Hi all,

I'd like to add to NumPy the ability to clone a data-type object so that 
only a view fields are copied over but that it retains the same total size.

This would allow, for example, the ability to "select out a few records" 
from a structured array using

subarr = arr.view(cloned_dtype)

Right now, it is hard to do this because you have to at least add a 
"dummy" field at the end.  A simple method on the dtype class 
(fromfields or something) would be easy to add.

It was thought in the past to do this with indexing

arr['field1', 'field2']

And that would still be possible (and mostly implemented) if this 
feature is added.

Thoughts?

-Travis



From robert.kern at gmail.com  Wed Oct 29 20:26:36 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 29 Oct 2008 19:26:36 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <4908FA5E.9090803@enthought.com>
References: <4908FA5E.9090803@enthought.com>
Message-ID: <3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>

On Wed, Oct 29, 2008 at 19:05, Travis E. Oliphant
<oliphant at enthought.com> wrote:
>
> Hi all,
>
> I'd like to add to NumPy the ability to clone a data-type object so that
> only a view fields are copied over but that it retains the same total size.
>
> This would allow, for example, the ability to "select out a few records"
> from a structured array using
>
> subarr = arr.view(cloned_dtype)
>
> Right now, it is hard to do this because you have to at least add a
> "dummy" field at the end.  A simple method on the dtype class
> (fromfields or something) would be easy to add.

I'm not sure what this accomplishes. Would the dummy fields that fill
in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
with no numpy.void scalars? That would be a novel feature, but I'm not
sure it fits the problem. On the contrary:

> It was thought in the past to do this with indexing
>
> arr['field1', 'field2']
>
> And that would still be possible (and mostly implemented) if this
> feature is added.

This appears more like the interface that people want. Except that I
think people were thinking that it would follow fancy indexing syntax:

  arr[['field1', 'field2']]

I guess there are two ways to implement this. One is to make a new
array that just contains the desired fields. Another is to make a view
that just points to the desired fields in the original array provided
that we have a new feature for inaccessible dummy fields. One point
for the former approach is that it is closer to fancy indexing which
must always make a copy. The latter approach breaks that connection.

OTOH, now that I think about it, I don't think there is really any
coherent way to mix field selection with any other indexing
operations. At least, not within the same brackets. Hmm. So maybe the
link to fancy indexing can be ignored as, ahem, fanciful.

Overall, I guess, I would present the feature slightly differently.
Provide a kind of inaccessible and invisible dtype for implementing
dummy fields. This is useful in other places like file parsing. At the
same time, implement a function that uses this capability to make
views with a subset of the fields of a structured array. I'm not sure
that people need an API for replacing the fields of a dtype like this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Wed Oct 29 23:37:33 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 29 Oct 2008 23:37:33 -0400
Subject: [Numpy-discussion] augmented assignment and in-place operations
In-Reply-To: <3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>	<4900A706.70806@noaa.gov>	<2b1c8c4f0810260643p581ecfc5xc4e3b62ba0c82019@mail.gmail.com>	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>	<49060910.60903@hawaii.edu>	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>	<4908BA6F.90909@hawaii.edu>
	<3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>
Message-ID: <49092BFD.3000306@american.edu>

On 10/29/2008 3:43 PM Robert Kern wrote:
> The defining characteristic is 
> that "x <op>= y" should be equivalent to "x = x <op> y" except 
> possibly for *optional* in-place semantics. 

This gets at a bit of the Language Reference that I've
never understood.

         when possible, the actual operation is performed
         in-place, meaning that rather than creating a new
         object and assigning that to the target, the old
         object is modified instead.

What does that mean?  I assume "when possible" means in part
"when mutable", ruling out e.g. Python integers or floats.
But the rest does not really seem part of the language, but
rather seems to be normative?  That is, I could define
__iadd__ anyway I want.  Is the above saying that I "should"
define __iadd__ to do an in-place operation "if possible".
If so, why is such a normative statement part of the
language reference?  Or is it a statement about the language
that I'm just not getting?

Confusedly,
Alan




From robert.kern at gmail.com  Thu Oct 30 00:18:26 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 29 Oct 2008 23:18:26 -0500
Subject: [Numpy-discussion] augmented assignment and in-place operations
In-Reply-To: <49092BFD.3000306@american.edu>
References: <db6b5ecc0810221400p315cbeeide3a62eda53677df@mail.gmail.com>
	<9EE8FF7F-AE36-4B08-9C7C-D4827E3E1706@cs.toronto.edu>
	<2b1c8c4f0810270422y620b9472mc1d5c40cfa3e48a8@mail.gmail.com>
	<49060910.60903@hawaii.edu>
	<db6b5ecc0810271227v77add4a5hc3a87355ae241283@mail.gmail.com>
	<db6b5ecc0810282350t6431b50ep99a6f276851f1809@mail.gmail.com>
	<E7B0360F-DFC1-4C9E-83F8-F4F48EBA1FA1@cs.toronto.edu>
	<4908BA6F.90909@hawaii.edu>
	<3d375d730810291243u565cc464u39d3b1adb788af7@mail.gmail.com>
	<49092BFD.3000306@american.edu>
Message-ID: <3d375d730810292118q7be42f96qdb8555b48546980d@mail.gmail.com>

On Wed, Oct 29, 2008 at 22:37, Alan G Isaac <aisaac at american.edu> wrote:
> On 10/29/2008 3:43 PM Robert Kern wrote:
>> The defining characteristic is
>> that "x <op>= y" should be equivalent to "x = x <op> y" except
>> possibly for *optional* in-place semantics.
>
> This gets at a bit of the Language Reference that I've
> never understood.
>
>         when possible, the actual operation is performed
>         in-place, meaning that rather than creating a new
>         object and assigning that to the target, the old
>         object is modified instead.
>
> What does that mean?  I assume "when possible" means in part
> "when mutable", ruling out e.g. Python integers or floats.

In part, yes. But also "when possible" excludes in-place matrix
multiplication on ndarrays since we don't, as numpy policy, resize
ndarrays implicitly.

> But the rest does not really seem part of the language, but
> rather seems to be normative?  That is, I could define
> __iadd__ anyway I want.  Is the above saying that I "should"
> define __iadd__ to do an in-place operation "if possible".
> If so, why is such a normative statement part of the
> language reference?  Or is it a statement about the language
> that I'm just not getting?

It is a guide to the intended usage of the feature. You can see other
such guides in the documentation for __repr__, for example. The use of
the word "should" is not rigidly consistent throughout the document.
In other places, it is closer to the meaning of MUST in RFC 2119:

  http://www.faqs.org/rfcs/rfc2119.html

The reason such guides are in the reference manual is because ...
well, where else are you going to put them? Describing the use case of
a feature is, in this case I think, essential to describing the
feature. You have to explain why one would bother defining __iadd__ if
one already had __add__. Similarly, you have to explain why one would
use __repr__ over __str__ or vice versa.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From crleblanc at gmail.com  Thu Oct 30 00:55:29 2008
From: crleblanc at gmail.com (Chris LeBlanc)
Date: Thu, 30 Oct 2008 17:55:29 +1300
Subject: [Numpy-discussion] passing a C array to embedded Python from C code
Message-ID: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>

Hello,

I'm working on seismic processing software called Globe Claritas.  The
core is written in C and a bit of Fortran.  I would like to embed
Python in this software, so a user can use Python code to manipulate
the seismic data.  This would give our users all the power of Python
and NumPy and any external modules they may want to use.

I was thinking I could use the Python C API to initialize Python and
run the code supplied by the user.  I'm thinking I could create a
NumPy array that uses the same C array (1d, 2d, or 3d) that our
program is using instead of copying the memory.  Perhaps using the
NumPy API function PyArray_SimpleNewFromData() or something similar
(similar to this posting: http://blog.enthought.com/?p=62).  I would
then pass this NumPy array as an object to the Python code, which it
could manipulate.  The C array is an array of floating point data
samples for each seismic trace, so hopefully well suited to this.

At this point, I'm thinking the python code wouldn't need to return
any objects because it would be modifying the seismic data (and trace
headers) in memory.

Does this approach make sense?  Is there a better way to go about it?
Maybe calling a custom module from the Python code that does the C
array to NumPy translation using Cython/pyrex/swig/etc.  Would it be
possible to use the same C arrays from here without copying them?

Is there a more obvious way to do it that I might be missing?  Does
anyone have examples of creating a NumPy array in C, and passing it to
a Python instance?  That would be a huge help, I have no experience
with the C or NumPy APIs.

Thanks,
Chris LeBlanc


From stefan at sun.ac.za  Thu Oct 30 02:33:23 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 30 Oct 2008 08:33:23 +0200
Subject: [Numpy-discussion] any interest in including
	asecond-ordergradient?
In-Reply-To: <db6b5ecc0810291046s3f5c7801pdf00b0a0f00a0941@mail.gmail.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>
	<9457e7c80810291029s6d0dff59s7752a2553955ce8b@mail.gmail.com>
	<db6b5ecc0810291046s3f5c7801pdf00b0a0f00a0941@mail.gmail.com>
Message-ID: <9457e7c80810292333y2b6a37d7h676055f81ee8faaa@mail.gmail.com>

Hi Fernando,

Thanks for your input.

2008/10/29 Fernando Perez <fperez.net at gmail.com>:
> I think it's fine to ask for functions that compute higher order
> derivatives of n-d arrays: we already have diff(), which operates on a
> single direction, and a hessian could make sense (with the caveats
> David points out).   But with higher order derivatives there are many
> more combinations to worry about, and I really think it's a bad idea
> to lump those issues into the definition of gradient, which was a
> perfectly unambiguous object up until this point.

Maybe we should focus on writing a decent 'deriv' function then.  I
know Konrad Hinsen's Scientific had a derivatives package
(Scientific.Functions.Derivatives) that implemented automatic
differentiation:

http://en.wikipedia.org/wiki/Automatic_differentiation

Cheers
St?fan


From stefan at sun.ac.za  Thu Oct 30 02:55:36 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 30 Oct 2008 08:55:36 +0200
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
References: <4908FA5E.9090803@enthought.com>
	<3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
Message-ID: <9457e7c80810292355o630fb220o16fc35d64277809a@mail.gmail.com>

2008/10/30 Robert Kern <robert.kern at gmail.com>:
> Provide a kind of inaccessible and invisible dtype for implementing
> dummy fields. This is useful in other places like file parsing. At the
> same time, implement a function that uses this capability to make
> views with a subset of the fields of a structured array. I'm not sure
> that people need an API for replacing the fields of a dtype like this.

I have thought about adding such an invisible dtype before
(incidentally, it was while parsing files!).  I would be interested in
exploring this route further.

Regards
St?fan


From matthieu.brucher at gmail.com  Thu Oct 30 04:08:11 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 30 Oct 2008 09:08:11 +0100
Subject: [Numpy-discussion] passing a C array to embedded Python from C
	code
In-Reply-To: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
References: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
Message-ID: <e76aa17f0810300108l60563810p56d73f9f5f278566@mail.gmail.com>

> Does this approach make sense?  Is there a better way to go about it?
> Maybe calling a custom module from the Python code that does the C
> array to NumPy translation using Cython/pyrex/swig/etc.  Would it be
> possible to use the same C arrays from here without copying them?

Hi,

Your case seems to fit the array interface. The goal is to create a C
structure with some additional information that Numpy can understand,
and then your array will be treated as a Numpy array. If you can
follow a French tutorial, you can go on
http://matthieu-brucher.developpez.com/tutoriels/python/swig-numpy/#LV
to have a skeletton for your issue.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From faltet at pytables.org  Thu Oct 30 05:33:00 2008
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 30 Oct 2008 10:33:00 +0100
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
References: <4908FA5E.9090803@enthought.com>
	<3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
Message-ID: <200810301033.01417.faltet@pytables.org>

A Thursday 30 October 2008, Robert Kern escrigu?:
> On Wed, Oct 29, 2008 at 19:05, Travis E. Oliphant
>
> <oliphant at enthought.com> wrote:
> > Hi all,
> >
> > I'd like to add to NumPy the ability to clone a data-type object so
> > that only a view fields are copied over but that it retains the
> > same total size.
> >
> > This would allow, for example, the ability to "select out a few
> > records" from a structured array using
> >
> > subarr = arr.view(cloned_dtype)
> >
> > Right now, it is hard to do this because you have to at least add a
> > "dummy" field at the end.  A simple method on the dtype class
> > (fromfields or something) would be easy to add.
>
> I'm not sure what this accomplishes. Would the dummy fields that fill
> in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
> with no numpy.void scalars? That would be a novel feature, but I'm
> not
>
> sure it fits the problem. On the contrary:
> > It was thought in the past to do this with indexing
> >
> > arr['field1', 'field2']
> >
> > And that would still be possible (and mostly implemented) if this
> > feature is added.
>
> This appears more like the interface that people want. Except that I
> think people were thinking that it would follow fancy indexing
> syntax:
>
>   arr[['field1', 'field2']]

I've thought about that too.  That would be a great thing to have, IMO.

>
> I guess there are two ways to implement this. One is to make a new
> array that just contains the desired fields. Another is to make a
> view that just points to the desired fields in the original array
> provided that we have a new feature for inaccessible dummy fields.
> One point for the former approach is that it is closer to fancy
> indexing which must always make a copy. The latter approach breaks
> that connection.

Yeah.  I'd vote for avoid the copy.

> OTOH, now that I think about it, I don't think there is really any
> coherent way to mix field selection with any other indexing
> operations. At least, not within the same brackets. Hmm. So maybe the
> link to fancy indexing can be ignored as, ahem, fanciful.

Well, one can always check that fields in the fancy list are either 
strings (map to name fields) or integers (map to positional fields).  
However, I'm not sure if this check would be too expensive.

> Overall, I guess, I would present the feature slightly differently.
> Provide a kind of inaccessible and invisible dtype for implementing
> dummy fields. This is useful in other places like file parsing. At
> the same time, implement a function that uses this capability to make
> views with a subset of the fields of a structured array. I'm not sure
> that people need an API for replacing the fields of a dtype like
> this.

Mmh, not sure on what you are proposing there.  You mean something like:

In [21]: t = numpy.dtype([('f0','i4'),('f1', 'f8'), ('f2', 'S20')])

In [22]: nt = t.astype(['f2', 'f0'])

In [23]: ra = numpy.zeros(10, dtype=t)

In [24]: nra = ra.view(nt)

In [25]: ra
Out[25]:
array([(0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
       (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
       (0, 0.0, ''), (0, 0.0, '')],
      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '|S20')])

In [26]: nra
Out[26]:
array([('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0),
       ('', 0), ('', 0), ('', 0)],
      dtype=[('f2', '|S20'), ('f0', '<i4')])

?

In that case, that would be a great feature to add.

-- 
Francesc Alted


From vandemeent at damtp.cam.ac.uk  Thu Oct 30 07:19:00 2008
From: vandemeent at damtp.cam.ac.uk (Jan-Willem van de Meent)
Date: Thu, 30 Oct 2008 11:19:00 +0000
Subject: [Numpy-discussion] Numpy matrix multiplication slow even though
	ATLAS linked
Message-ID: <200810301119.00700.vandemeent@damtp.cam.ac.uk>

Dear all,

This is my first post to this list. I am having perfomance issues with with 
numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes  about 1m40s, even 
though numpy is appears to link to my atlas libraries:

I did a quick benchmark by running the following script: 

    #! /usr/bin/env python
    import numpy
    import time

    try:
       import numpy.core._dotblas
       print 'Using ATLAS:'
    except ImportError:
       print 'No ATLAS:'

    t = time.time()
    x = numpy.random.random((1000,1000))
    y = numpy.random.random((1000,1000))
    z = numpy.dot(x, y)

    print time.time()-t

My laptop is a Dell D620 Core Duo T2300 1.66 Ghz, running Archlinux with GCC 
4.3.2, atlas 3.8.2, python 2.5.2 and numpy 1.2.1. Output of the script above 
is:

    Using ATLAS:
    7.99549412727

A department desktop PC, Pentium D 3.00 Ghz, running Scientific Linux, with 
GCC 4.1.2, atlas 3.7.30, python 2.5.1 and numpy 1.1.0, runs this test 24 
times faster:

    Using ATLAS:
    0.337520122528

So even though _dotblas.so exists, matrix multiplication appears to run at 
pretty much the same speed as if atlas were not  available. Running ldd on 
_dotblas.so suggests  that numpy is indeed linking to the atlas libs:

ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so
        linux-gate.so.1 =>  (0xb7fcf000)
        libatlas.so => /usr/lib/libatlas.so (0xb7cb5000)
        liblapack.so => /usr/lib/liblapack.so (0xb77ab000)
        libcblas.so => /usr/lib/libcblas.so (0xb778b000)
        libf77blas.so => /usr/lib/libf77blas.so (0xb776f000)
        libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7630000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb7618000)
        libc.so.6 => /lib/libc.so.6 (0xb74d6000)
        libm.so.6 => /lib/libm.so.6 (0xb74b0000)
        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb73ff000)
        libdl.so.2 => /lib/libdl.so.2 (0xb73fb000)
        libutil.so.1 => /lib/libutil.so.1 (0xb73f6000)
        /lib/ld-linux.so.2 (0xb7fd0000)

Something appears to be going wrong in the compilation process. In particular, 
the compile process appears unable to determine the version of the atlas libs 
installed (I get NO_ATLAS_INFO). I've pasted some snippets from the build 
output which may be relevant below.

Can anyone help me figure this out? 

Many thanks in advance,
Jan-Willem

-- 
Jan-Willem van de Meent
Research Student
Goldstein Lab, DAMTP
University of Cambridge

--
Snippets from setup.py output
...

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = 
['lapack', 'atlas', 'lapack', 'cblas', 'f77blas', 'ptcblas', 'ptf77blas']
    library_dirs = ['/usr/lib']
    language = f77
    include_dirs = ['/usr/include']

...

customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: 
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=i686 -mtune=generic -O2 -pipe -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread 
_configtest.o -L/usr/lib -llapack -latlas -llapack -lcblas -lf77blas -lptcblas -lptf77blas -o 
_configtest
/usr/bin/ld: _configtest: hidden symbol `__powidf2' 
in /usr/lib/gcc/i686-pc-linux-gnu/4.3.2/libgcc.a(_powidf2.o) is referenced by 
DSO
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
/usr/bin/ld: _configtest: hidden symbol `__powidf2' 
in /usr/lib/gcc/i686-pc-linux-gnu/4.3.2/libgcc.a(_powidf2.o) is referenced by 
DSO
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: 
  FOUND:
    libraries = 
['lapack', 'atlas', 'lapack', 'cblas', 'f77blas', 'ptcblas', 'ptf77blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 2)]
    include_dirs = ['/usr/include']


From ckkart at hoc.net  Thu Oct 30 08:51:36 2008
From: ckkart at hoc.net (Christan K.)
Date: Thu, 30 Oct 2008 12:51:36 +0000 (UTC)
Subject: [Numpy-discussion] numpy array change notifier?
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
Message-ID: <loom.20081030T124150-652@post.gmane.org>

Erik Tollerud <erik.tollerud <at> gmail.com> writes:

> 
> Is there any straightforward way of notifying on change of a numpy
> array that leaves the numpy arrays still efficient?
> 

You can instantiate the following ndarray derived class with a callable 
argument. Any change to the data will call the callback function.

In [5]: def notify(arg):
   ...:     print 'array changed'
   ...:
   ...:

In [6]: x = cbarray([1,2,3,4],cb=notify)

In [7]: x[2] = -1
array changed

alternatively I have used the pubsub module available as part of wxPython (see 
the commented line in _notify())

class cbarray(N.ndarray):
    def __new__(subtype, data, cb=None, dtype=None, copy=False):
        subtype.__defaultcb = cb

        if copy:
            data = N.array(data,dtype=dtype)
        else:
            data = N.asarray(data,dtype=dtype)

        data = data.view(subtype)
        return data

    def _notify(self):
        if self.cb is not None:
            self.cb()
        #Publisher().sendMessage(('changed'))
        
    def _get_shape(self):
        return super(cbarray, self).shape
    shape = property(_get_shape)

    def __setitem__(self, item, val):
        N.ndarray.__setitem__(self, item, val)
        self._notify()
        
    def __array_finalize__(self,obj):
        if not hasattr(self, "cb"):
            # The object does not already have a `.cb` attribute
            self.cb = getattr(obj,'cb',self.__defaultcb)

    def __reduce__(self):
        object_state = list(N.ndarray.__reduce__(self))
        subclass_state = (self.cb,)
        object_state[2] = (object_state[2],subclass_state)
        return tuple(object_state)

    def __setstate__(self,state):
        nd_state, own_state = state
        N.ndarray.__setstate__(self,nd_state)
        
        cb, = own_state
        self.cb = cb

Hope thqat helps, Christian




From oliphant at enthought.com  Thu Oct 30 09:27:56 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 30 Oct 2008 08:27:56 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
 from a data-type
In-Reply-To: <3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
References: <4908FA5E.9090803@enthought.com>
	<3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
Message-ID: <4909B65C.3070908@enthought.com>


> I'm not sure what this accomplishes. Would the dummy fields that fill
> in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
> with no numpy.void scalars? That would be a novel feature, but I'm not
> sure it fits the problem. On the contrary:
>   

Yes, that was the idea.   You can do it now, but only in C.   The real 
problem right now from my point of view is that there is no way to tell 
the dtype constructor to "pad the itemsize to x bytes".    If that were 
changed, then many things would be possible. 

> OTOH, now that I think about it, I don't think there is really any
> coherent way to mix field selection with any other indexing
> operations. At least, not within the same brackets. Hmm. So maybe the
> link to fancy indexing can be ignored as, ahem, fanciful.
>   
Yeah,  I was wondering how to do it well, myself, and couldn't come up 
with anything which is why I went the .view route with another dtype.   

By "inaccessible and invisible dtype" do you mean something like the 
basic built-in void data type, but which doesn't try to report itself 
when the dtype prints?

That sounds interesting but I'm not sure it's necessary because the 
field specification can already skip bytes (just not bytes at the end 
--- which is what I would like to fix).    Perhaps what is needed is a 
"pseudo-dtype" (something like 'c' compared to 'S1') which doesn't 
actually create a new dtype but which is handled differently when the 
dtype is created with the [('field1', type), ('field2', type2)] 
approach.   Specifically, it doesn't add an entry to the fields 
dictionary nor an entry to the names but does affect the itemsize of the 
element (and the offset of follow-on fields).

So, let's assume the character is 'v':

If we have an array with underlying dtype:

od = [('date', 'S10'), ('high', 'f4'), ('low', 'f4'), ('close', 'f4'), 
('volume', 'i4')]

Then, we could define a new dtype

nd = [('date', 'S10'), ('', 'v8'), ('close', 'f4'), ('', 'v4')]

and  arr.view(nd)   would provide a view of the array where element 
selection would be a tuple with just the date and close elements but the 
itemsize would be exactly the same but nd.names would be ['date', 'close']

I like this approach.  It impacts the API the very least but provides 
the desired functionality.

-Travis



-Travis



> Overall, I guess, I would present the feature slightly differently.
> Provide a kind of inaccessible and invisible dtype for implementing
> dummy fields. This is useful in other places like file parsing. At the
> same time, implement a function that uses this capability to make
> views with a subset of the fields of a structured array. I'm not sure
> that people need an API for replacing the fields of a dtype like this.
>
>   



From elcorto at gmx.net  Thu Oct 30 09:57:03 2008
From: elcorto at gmx.net (Steve Schmerler)
Date: Thu, 30 Oct 2008 14:57:03 +0100
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <200810271643.15984.pgmdevlist@gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
	<200810271643.15984.pgmdevlist@gmail.com>
Message-ID: <20081030135703.GA5551@ramrod.starsheriffs.de>

On Oct 27 16:43 -0400, Pierre GM  wrote:
> 
> Erik, may be you could try the trick presented here : 
> http://www.scipy.org/Subclasses
> in the __array_wrap__ section.

Stupid question: How do I find pages like
http://www.scipy.org/Subclasses on scipy.org?

I can find them with the search function, but only when I know that it's
there :)

best,
steve


From pgmdevlist at gmail.com  Thu Oct 30 10:06:48 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 30 Oct 2008 10:06:48 -0400
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <20081030135703.GA5551@ramrod.starsheriffs.de>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
	<200810271643.15984.pgmdevlist@gmail.com>
	<20081030135703.GA5551@ramrod.starsheriffs.de>
Message-ID: <200810301006.48243.pgmdevlist@gmail.com>

Steve,
Right, there's not a lot of visibility for this one. You could try 
google 'numpy subclass'. there's also an entry on subclassing in the numpy 
user guide (through docs.scipy.org).
Cheers
P.

On Thursday 30 October 2008 09:57:03 Steve Schmerler wrote:
> On Oct 27 16:43 -0400, Pierre GM  wrote:
> > Erik, may be you could try the trick presented here :
> > http://www.scipy.org/Subclasses
> > in the __array_wrap__ section.
>
> Stupid question: How do I find pages like
> http://www.scipy.org/Subclasses on scipy.org?
>
> I can find them with the search function, but only when I know that it's
> there :)
>
> best,
> steve
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion




From elcorto at gmx.net  Thu Oct 30 11:15:28 2008
From: elcorto at gmx.net (Steve Schmerler)
Date: Thu, 30 Oct 2008 16:15:28 +0100
Subject: [Numpy-discussion] numpy array change notifier?
In-Reply-To: <200810301006.48243.pgmdevlist@gmail.com>
References: <fbab3e0d0810271354m130bdacche5d01aadac6871cd@mail.gmail.com>
	<200810271643.15984.pgmdevlist@gmail.com>
	<20081030135703.GA5551@ramrod.starsheriffs.de>
	<200810301006.48243.pgmdevlist@gmail.com>
Message-ID: <20081030151528.GA5975@ramrod.starsheriffs.de>

On Oct 30 10:06 -0400, Pierre GM  wrote:
> Steve,
> Right, there's not a lot of visibility for this one. 

Yes, wouldn't it be better placed in the Cookbook section (or any other
suitable place that's visible to anyone entering scipy.org)?  Other
pages are unfortunately also not visible, like
http://www.scipy.org/EricsBroadcastingDoc .

best,
steve


From rob.clewley at gmail.com  Thu Oct 30 11:16:03 2008
From: rob.clewley at gmail.com (Rob Clewley)
Date: Thu, 30 Oct 2008 11:16:03 -0400
Subject: [Numpy-discussion] Automatic differentiation (was Re: second-order
	gradient)
Message-ID: <a749952d0810300816t237477fdmec077569060a0b3c@mail.gmail.com>

> Maybe we should focus on writing a decent 'deriv' function then.  I
> know Konrad Hinsen's Scientific had a derivatives package
> (Scientific.Functions.Derivatives) that implemented automatic
> differentiation:
>
> http://en.wikipedia.org/wiki/Automatic_differentiation

That would be great, but wouldn't that be best suited as a utility
requiring Sympy? You'll want to take advantage of all sorts of
symbolic classes, especially for any source code transformation
approach. IMO Hinsen's implementation isn't a very efficient or
attractive solution to AD given the great existing C/C++ codes out
there. Maybe we should be looking to provide a python interface to an
existing open source package such as ADOL-C, but I'm all in favour of
a new pure python approach too. What would be perfect is to have a
single interface to a python AD package that would support a faster
implementation if the user wished to install a C/C++ package,
otherwise would default to a pure python equivalent.

-Rob


From rob.clewley at gmail.com  Thu Oct 30 11:18:50 2008
From: rob.clewley at gmail.com (Rob Clewley)
Date: Thu, 30 Oct 2008 11:18:50 -0400
Subject: [Numpy-discussion] any interest in including
	asecond-ordergradient?
In-Reply-To: <9457e7c80810292333y2b6a37d7h676055f81ee8faaa@mail.gmail.com>
References: <48C01AE7354EC240A26F19CEB995E943033AEF84@CHMAILMBX01.novachem.com>
	<9457e7c80810280114s34d56a51j57841503979352b8@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF8D@CHMAILMBX01.novachem.com>
	<3d375d730810281436v698cdaf2tdd6f99461e123c91@mail.gmail.com>
	<48C01AE7354EC240A26F19CEB995E943033AEF91@CHMAILMBX01.novachem.com>
	<9457e7c80810291029s6d0dff59s7752a2553955ce8b@mail.gmail.com>
	<db6b5ecc0810291046s3f5c7801pdf00b0a0f00a0941@mail.gmail.com>
	<9457e7c80810292333y2b6a37d7h676055f81ee8faaa@mail.gmail.com>
Message-ID: <a749952d0810300818u31f193f2m10e6c8596d61c0bc@mail.gmail.com>

> 2008/10/29 Fernando Perez <fperez.net at gmail.com>:
>> I think it's fine to ask for functions that compute higher order
>> derivatives of n-d arrays: we already have diff(), which operates on a
>> single direction, and a hessian could make sense (with the caveats
>> David points out).   But with higher order derivatives there are many
>> more combinations to worry about, and I really think it's a bad idea
>> to lump those issues into the definition of gradient, which was a
>> perfectly unambiguous object up until this point.

I'm basically in favour of Fernando's suggestion to keep gradient
simple and add a hessian function. Higher numerical derivatives from
data aren't very reliable anyway. You're much better off interpolating
with a polynomial and then differentiating that.

> Maybe we should focus on writing a decent 'deriv' function then.  I
> know Konrad Hinsen's Scientific had a derivatives package
> (Scientific.Functions.Derivatives) that implemented automatic
> differentiation:

Improving the support for a gradient of array data is an entirely
independent project in my mind - but I like this idea and I replied in
a new thread.

-Rob


From robert.kern at gmail.com  Thu Oct 30 11:24:37 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 10:24:37 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <4909B65C.3070908@enthought.com>
References: <4908FA5E.9090803@enthought.com>
	<3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
	<4909B65C.3070908@enthought.com>
Message-ID: <3d375d730810300824j3048e163n5fb8a8e3eb467bcf@mail.gmail.com>

On Thu, Oct 30, 2008 at 08:27, Travis E. Oliphant
<oliphant at enthought.com> wrote:
>
>> I'm not sure what this accomplishes. Would the dummy fields that fill
>> in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
>> with no numpy.void scalars? That would be a novel feature, but I'm not
>> sure it fits the problem. On the contrary:
>>
>
> Yes, that was the idea.   You can do it now, but only in C.   The real
> problem right now from my point of view is that there is no way to tell
> the dtype constructor to "pad the itemsize to x bytes".    If that were
> changed, then many things would be possible.
>
>> OTOH, now that I think about it, I don't think there is really any
>> coherent way to mix field selection with any other indexing
>> operations. At least, not within the same brackets. Hmm. So maybe the
>> link to fancy indexing can be ignored as, ahem, fanciful.
>>
> Yeah,  I was wondering how to do it well, myself, and couldn't come up
> with anything which is why I went the .view route with another dtype.
>
> By "inaccessible and invisible dtype" do you mean something like the
> basic built-in void data type, but which doesn't try to report itself
> when the dtype prints?

The field doesn't report itself when the *values* print, is what I'm
concerned with. The dtype should display the dummy fields such that
repr() can accurately reconstruct the dtype.

> That sounds interesting but I'm not sure it's necessary because the
> field specification can already skip bytes (just not bytes at the end
> --- which is what I would like to fix).    Perhaps what is needed is a
> "pseudo-dtype" (something like 'c' compared to 'S1') which doesn't
> actually create a new dtype but which is handled differently when the
> dtype is created with the [('field1', type), ('field2', type2)]
> approach.   Specifically, it doesn't add an entry to the fields
> dictionary nor an entry to the names but does affect the itemsize of the
> element (and the offset of follow-on fields).
>
> So, let's assume the character is 'v':
>
> If we have an array with underlying dtype:
>
> od = [('date', 'S10'), ('high', 'f4'), ('low', 'f4'), ('close', 'f4'),
> ('volume', 'i4')]
>
> Then, we could define a new dtype
>
> nd = [('date', 'S10'), ('', 'v8'), ('close', 'f4'), ('', 'v4')]

To do this, we would also have to fix the current behavior of
converting ''s to 'f0', 'f1', etc., when these are passed to the
dtype() constructor.

> and  arr.view(nd)   would provide a view of the array where element
> selection would be a tuple with just the date and close elements but the
> itemsize would be exactly the same but nd.names would be ['date', 'close']
>
> I like this approach.  It impacts the API the very least but provides
> the desired functionality.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Oct 30 11:26:45 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 10:26:45 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <200810301033.01417.faltet@pytables.org>
References: <4908FA5E.9090803@enthought.com>
	<3d375d730810291726s197838f6o1e1ea69632ae9775@mail.gmail.com>
	<200810301033.01417.faltet@pytables.org>
Message-ID: <3d375d730810300826i2748369dx77579e85de96e0ac@mail.gmail.com>

On Thu, Oct 30, 2008 at 04:33, Francesc Alted <faltet at pytables.org> wrote:
> A Thursday 30 October 2008, Robert Kern escrigu?:
>> On Wed, Oct 29, 2008 at 19:05, Travis E. Oliphant
>>
>> <oliphant at enthought.com> wrote:
>> > Hi all,
>> >
>> > I'd like to add to NumPy the ability to clone a data-type object so
>> > that only a view fields are copied over but that it retains the
>> > same total size.
>> >
>> > This would allow, for example, the ability to "select out a few
>> > records" from a structured array using
>> >
>> > subarr = arr.view(cloned_dtype)
>> >
>> > Right now, it is hard to do this because you have to at least add a
>> > "dummy" field at the end.  A simple method on the dtype class
>> > (fromfields or something) would be easy to add.
>>
>> I'm not sure what this accomplishes. Would the dummy fields that fill
>> in the space be inaccessible? E.g. tuple(subarr[i,j,k]) gives a tuple
>> with no numpy.void scalars? That would be a novel feature, but I'm
>> not
>>
>> sure it fits the problem. On the contrary:
>> > It was thought in the past to do this with indexing
>> >
>> > arr['field1', 'field2']
>> >
>> > And that would still be possible (and mostly implemented) if this
>> > feature is added.
>>
>> This appears more like the interface that people want. Except that I
>> think people were thinking that it would follow fancy indexing
>> syntax:
>>
>>   arr[['field1', 'field2']]
>
> I've thought about that too.  That would be a great thing to have, IMO.
>
>> I guess there are two ways to implement this. One is to make a new
>> array that just contains the desired fields. Another is to make a
>> view that just points to the desired fields in the original array
>> provided that we have a new feature for inaccessible dummy fields.
>> One point for the former approach is that it is closer to fancy
>> indexing which must always make a copy. The latter approach breaks
>> that connection.
>
> Yeah.  I'd vote for avoid the copy.
>
>> OTOH, now that I think about it, I don't think there is really any
>> coherent way to mix field selection with any other indexing
>> operations. At least, not within the same brackets. Hmm. So maybe the
>> link to fancy indexing can be ignored as, ahem, fanciful.
>
> Well, one can always check that fields in the fancy list are either
> strings (map to name fields) or integers (map to positional fields).
> However, I'm not sure if this check would be too expensive.

That's not my concern. The problem is that the field-indexing applies
to the entire array, not just an axis. So what would the following
mean?

  a[['foo', 'bar'], [1,2,3]]

Compared to

  a[[5,8,10], [1,2,3]]

>> Overall, I guess, I would present the feature slightly differently.
>> Provide a kind of inaccessible and invisible dtype for implementing
>> dummy fields. This is useful in other places like file parsing. At
>> the same time, implement a function that uses this capability to make
>> views with a subset of the fields of a structured array. I'm not sure
>> that people need an API for replacing the fields of a dtype like
>> this.
>
> Mmh, not sure on what you are proposing there.  You mean something like:
>
> In [21]: t = numpy.dtype([('f0','i4'),('f1', 'f8'), ('f2', 'S20')])
>
> In [22]: nt = t.astype(['f2', 'f0'])
>
> In [23]: ra = numpy.zeros(10, dtype=t)
>
> In [24]: nra = ra.view(nt)
>
> In [25]: ra
> Out[25]:
> array([(0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
>       (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
>       (0, 0.0, ''), (0, 0.0, '')],
>      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '|S20')])
>
> In [26]: nra
> Out[26]:
> array([('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0),
>       ('', 0), ('', 0), ('', 0)],
>      dtype=[('f2', '|S20'), ('f0', '<i4')])
>
> ?
>
> In that case, that would be a great feature to add.

That's what Travis is proposing. I would like to see a function that
does this (however it is implemented under the covers):

  nra = subset_fields(ra, ['f0', 'f2'])

With the view, I don't think you can reorder the fields as in your example.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From stefan at sun.ac.za  Thu Oct 30 11:36:44 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 30 Oct 2008 17:36:44 +0200
Subject: [Numpy-discussion] Automatic differentiation (was Re:
	second-order gradient)
In-Reply-To: <a749952d0810300816t237477fdmec077569060a0b3c@mail.gmail.com>
References: <a749952d0810300816t237477fdmec077569060a0b3c@mail.gmail.com>
Message-ID: <9457e7c80810300836m2f5daebauef4de83f983a0999@mail.gmail.com>

2008/10/30 Rob Clewley <rob.clewley at gmail.com>:
>> http://en.wikipedia.org/wiki/Automatic_differentiation
>
> That would be great, but wouldn't that be best suited as a utility
> requiring Sympy? You'll want to take advantage of all sorts of
> symbolic classes, especially for any source code transformation
> approach. IMO Hinsen's implementation isn't a very efficient or
> attractive solution to AD given the great existing C/C++ codes out
> there. Maybe we should be looking to provide a python interface to an
> existing open source package such as ADOL-C, but I'm all in favour of
> a new pure python approach too. What would be perfect is to have a
> single interface to a python AD package that would support a faster
> implementation if the user wished to install a C/C++ package,
> otherwise would default to a pure python equivalent.

In your experience, is this functionality enough to start a separate
package, or should we try to include it somewhere else?  Otherwise we
could think of a new SciKit.

Regards
St?fan


From rob.clewley at gmail.com  Thu Oct 30 12:42:29 2008
From: rob.clewley at gmail.com (Rob Clewley)
Date: Thu, 30 Oct 2008 12:42:29 -0400
Subject: [Numpy-discussion] Automatic differentiation (was Re:
	second-order gradient)
In-Reply-To: <9457e7c80810300836m2f5daebauef4de83f983a0999@mail.gmail.com>
References: <a749952d0810300816t237477fdmec077569060a0b3c@mail.gmail.com>
	<9457e7c80810300836m2f5daebauef4de83f983a0999@mail.gmail.com>
Message-ID: <a749952d0810300942k56c85c7dtad53345706317cbe@mail.gmail.com>

> In your experience, is this functionality enough to start a separate
> package, or should we try to include it somewhere else?  Otherwise we
> could think of a new SciKit.

I confess to knowing no details about scikits so I don't know what the
difference really is between a "new package" and a scikit. To do this
properly you'd end up with a sizable body of code, and the potential
dependency on Sympy would also suggest making it somewhat separate.
I'd defer to others on that point, although I don't really see what
other package it would naturally fit with because AD has multiple
applications.

-Rob


From faltet at pytables.org  Thu Oct 30 12:59:52 2008
From: faltet at pytables.org (Francesc Alted)
Date: Thu, 30 Oct 2008 17:59:52 +0100
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
	from a data-type
In-Reply-To: <3d375d730810300826i2748369dx77579e85de96e0ac@mail.gmail.com>
References: <4908FA5E.9090803@enthought.com>
	<200810301033.01417.faltet@pytables.org>
	<3d375d730810300826i2748369dx77579e85de96e0ac@mail.gmail.com>
Message-ID: <200810301759.53344.faltet@pytables.org>

A Thursday 30 October 2008, Robert Kern escrigu?:
[clip]
> >> OTOH, now that I think about it, I don't think there is really any
> >> coherent way to mix field selection with any other indexing
> >> operations. At least, not within the same brackets. Hmm. So maybe
> >> the link to fancy indexing can be ignored as, ahem, fanciful.
> >
> > Well, one can always check that fields in the fancy list are either
> > strings (map to name fields) or integers (map to positional
> > fields). However, I'm not sure if this check would be too
> > expensive.
>
> That's not my concern. The problem is that the field-indexing applies
> to the entire array, not just an axis. So what would the following
> mean?
>
>   a[['foo', 'bar'], [1,2,3]]
>
> Compared to
>
>   a[[5,8,10], [1,2,3]]

Well, as I see them, fields are like another axis, just that it is 
always the leading one.  In order to cope with them we could use a 
generalization of what it works already:

In [15]: ra = numpy.zeros((3,4), "i4,f4")

In [16]: ra['f1'][[1,2],[0,3]]  # this already works
Out[16]: array([ 0.,  0.], dtype=float32)

In [17]: ra[['f1','f2']][[1,2],[0,3]]   # this could be make to work
Out[17]:
array([(0, 0.0), (0, 0.0)],
      dtype=[('f0', '<i4'), ('f1', '<f4')])

> >> Overall, I guess, I would present the feature slightly
> >> differently. Provide a kind of inaccessible and invisible dtype
> >> for implementing dummy fields. This is useful in other places like
> >> file parsing. At the same time, implement a function that uses
> >> this capability to make views with a subset of the fields of a
> >> structured array. I'm not sure that people need an API for
> >> replacing the fields of a dtype like this.
> >
> > Mmh, not sure on what you are proposing there.  You mean something
> > like:
> >
> > In [21]: t = numpy.dtype([('f0','i4'),('f1', 'f8'), ('f2', 'S20')])
> >
> > In [22]: nt = t.astype(['f2', 'f0'])
> >
> > In [23]: ra = numpy.zeros(10, dtype=t)
> >
> > In [24]: nra = ra.view(nt)
> >
> > In [25]: ra
> > Out[25]:
> > array([(0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
> >       (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
> >       (0, 0.0, ''), (0, 0.0, '')],
> >      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '|S20')])
> >
> > In [26]: nra
> > Out[26]:
> > array([('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('',
> > 0), ('', 0), ('', 0), ('', 0)],
> >      dtype=[('f2', '|S20'), ('f0', '<i4')])
> >
> > ?
> >
> > In that case, that would be a great feature to add.
>
> That's what Travis is proposing. I would like to see a function that
> does this (however it is implemented under the covers):
>
>   nra = subset_fields(ra, ['f0', 'f2'])

Interesting.

> With the view, I don't think you can reorder the fields as in your
> example.

That's a pity.  Providing a dtype with the notion of an internal reorder 
can be very powerful in some situations.  But I guess that implementing 
this would be complicated.

-- 
Francesc Alted


From bsouthey at gmail.com  Thu Oct 30 13:50:49 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 30 Oct 2008 12:50:49 -0500
Subject: [Numpy-discussion] Adding the ability to "clone" a few fields
 from a data-type
In-Reply-To: <200810301759.53344.faltet@pytables.org>
References: <4908FA5E.9090803@enthought.com>	<200810301033.01417.faltet@pytables.org>	<3d375d730810300826i2748369dx77579e85de96e0ac@mail.gmail.com>
	<200810301759.53344.faltet@pytables.org>
Message-ID: <4909F3F9.3030008@gmail.com>

Francesc Alted wrote:
> A Thursday 30 October 2008, Robert Kern escrigu?:
> [clip]
>   
>>>> OTOH, now that I think about it, I don't think there is really any
>>>> coherent way to mix field selection with any other indexing
>>>> operations. At least, not within the same brackets. Hmm. So maybe
>>>> the link to fancy indexing can be ignored as, ahem, fanciful.
>>>>         
>>> Well, one can always check that fields in the fancy list are either
>>> strings (map to name fields) or integers (map to positional
>>> fields). However, I'm not sure if this check would be too
>>> expensive.
>>>       
>> That's not my concern. The problem is that the field-indexing applies
>> to the entire array, not just an axis. So what would the following
>> mean?
>>
>>   a[['foo', 'bar'], [1,2,3]]
>>
>> Compared to
>>
>>   a[[5,8,10], [1,2,3]]
>>     
>
> Well, as I see them, fields are like another axis, just that it is 
> always the leading one.  In order to cope with them we could use a 
> generalization of what it works already:
>
> In [15]: ra = numpy.zeros((3,4), "i4,f4")
>
> In [16]: ra['f1'][[1,2],[0,3]]  # this already works
> Out[16]: array([ 0.,  0.], dtype=float32)
>
> In [17]: ra[['f1','f2']][[1,2],[0,3]]   # this could be make to work
> Out[17]:
> array([(0, 0.0), (0, 0.0)],
>       dtype=[('f0', '<i4'), ('f1', '<f4')])
>
>   
>>>> Overall, I guess, I would present the feature slightly
>>>> differently. Provide a kind of inaccessible and invisible dtype
>>>> for implementing dummy fields. This is useful in other places like
>>>> file parsing. At the same time, implement a function that uses
>>>> this capability to make views with a subset of the fields of a
>>>> structured array. I'm not sure that people need an API for
>>>> replacing the fields of a dtype like this.
>>>>         
>>> Mmh, not sure on what you are proposing there.  You mean something
>>> like:
>>>
>>> In [21]: t = numpy.dtype([('f0','i4'),('f1', 'f8'), ('f2', 'S20')])
>>>
>>> In [22]: nt = t.astype(['f2', 'f0'])
>>>
>>> In [23]: ra = numpy.zeros(10, dtype=t)
>>>
>>> In [24]: nra = ra.view(nt)
>>>
>>> In [25]: ra
>>> Out[25]:
>>> array([(0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
>>>       (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''), (0, 0.0, ''),
>>>       (0, 0.0, ''), (0, 0.0, '')],
>>>      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '|S20')])
>>>
>>> In [26]: nra
>>> Out[26]:
>>> array([('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('', 0), ('',
>>> 0), ('', 0), ('', 0), ('', 0)],
>>>      dtype=[('f2', '|S20'), ('f0', '<i4')])
>>>
>>> ?
>>>
>>> In that case, that would be a great feature to add.
>>>       
>> That's what Travis is proposing. I would like to see a function that
>> does this (however it is implemented under the covers):
>>
>>   nra = subset_fields(ra, ['f0', 'f2'])
>>     
>
> Interesting.
>
>   
>> With the view, I don't think you can reorder the fields as in your
>> example.
>>     
>
> That's a pity.  Providing a dtype with the notion of an internal reorder 
> can be very powerful in some situations.  But I guess that implementing 
> this would be complicated.
>
>   
In general I agree with the idea but this starts sounding like R's data 
frames. So, is part of the goal to replicate some of the function of R's 
data frames?
For example the extract function 
(http://rweb.stat.umn.edu/R/library/base/html/Extract.data.frame.html)
(there is also the cookbook example of setting a name to null to remove 
it, see http://www.r-cookbook.com/node/50).



Bruce



From Chris.Barker at noaa.gov  Thu Oct 30 14:00:31 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 30 Oct 2008 11:00:31 -0700
Subject: [Numpy-discussion] passing a C array to embedded Python from C
	code
In-Reply-To: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
References: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
Message-ID: <4909F63F.6000201@noaa.gov>


A few comments:

Chris LeBlanc wrote:
>  I'm thinking I could create a
> NumPy array that uses the same C array (1d, 2d, or 3d) that our
> program is using instead of copying the memory.

yes, you can do that.

> At this point, I'm thinking the python code wouldn't need to return
> any objects because it would be modifying the seismic data (and trace
> headers) in memory.

There are a number of operations in Python that would require copies of 
the data, and you may want to be able to create data in python, and 
process it with your C code, so you probably do want to support passing 
data arrays from Python to the C code as well.


> Maybe calling a custom module from the Python code that does the C
> array to NumPy translation using Cython/pyrex/swig/etc.  Would it be
> possible to use the same C arrays from here without copying them?

yes, and you probably do want to use one of Cython, SWIG, or Ctypes -- 
it's really easier not to have to handle the reference counting, etc, 
yourself.

There are a set of SWIG typemaps and docs in the numpy distribution (in 
the docs dir, I believe), and there are various Wiki pages describing 
how  to do it with Cython and ctypes as well.

How to choose?

My thoughts on SWIG: SWIG is a pretty complex system, so you need to 
learn what is essentially yet another language. However, the numpy.i 
typemaps work well for the simple cases, so it may be easy to get 
started. The big advantage of SWIG is that it auto-generates the 
wrapper, once you have defined the typemaps you need. This gives a real 
advantage if you need to :

- provide wrappers for more than one language (PERL, Java, etc..)

- are wrapping a substantial library, particularly one that is under 
active development.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Thu Oct 30 14:02:00 2008
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 30 Oct 2008 11:02:00 -0700
Subject: [Numpy-discussion] passing a C array to embedded Python from
	C	code
In-Reply-To: <e76aa17f0810300108l60563810p56d73f9f5f278566@mail.gmail.com>
References: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
	<e76aa17f0810300108l60563810p56d73f9f5f278566@mail.gmail.com>
Message-ID: <4909F698.8060609@noaa.gov>

Matthieu Brucher wrote:
>  If you can
> follow a French tutorial, you can go on
> http://matthieu-brucher.developpez.com/tutoriels/python/swig-numpy/#LV
> to have a skeletton for your issue.

That looks very useful -- any chance of an English translation? My one 
year of high school French is proving useless. Otherwise, the code 
itself is still quite helpful.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From matthieu.brucher at gmail.com  Thu Oct 30 14:17:52 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 30 Oct 2008 19:17:52 +0100
Subject: [Numpy-discussion] passing a C array to embedded Python from C
	code
In-Reply-To: <4909F698.8060609@noaa.gov>
References: <c342c1150810292155x102ec373od5768387be0808a6@mail.gmail.com>
	<e76aa17f0810300108l60563810p56d73f9f5f278566@mail.gmail.com>
	<4909F698.8060609@noaa.gov>
Message-ID: <e76aa17f0810301117k14de8804lac17daa83d4a76db@mail.gmail.com>

2008/10/30 Christopher Barker <Chris.Barker at noaa.gov>:
> Matthieu Brucher wrote:
>>  If you can
>> follow a French tutorial, you can go on
>> http://matthieu-brucher.developpez.com/tutoriels/python/swig-numpy/#LV
>> to have a skeletton for your issue.
>
> That looks very useful -- any chance of an English translation? My one
> year of high school French is proving useless. Otherwise, the code
> itself is still quite helpful.
>
> -Chris

I thought I put it on my blog, but no :|
I may find one day the time to translate it and to improve/enhance it.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From anthony.floyd at convergent.ca  Thu Oct 30 14:16:44 2008
From: anthony.floyd at convergent.ca (Anthony Floyd)
Date: Thu, 30 Oct 2008 11:16:44 -0700
Subject: [Numpy-discussion] passing a C array to embedded Python
	fromC	code
Message-ID: <7EFBEC7FA86C1141B59B59EEAEE3294FBC38AE@EMAIL2.exchange.electric.net>

Hi Chris,

> Matthieu Brucher wrote:
> >  If you can
> > follow a French tutorial, you can go on
> > 
> http://matthieu-brucher.developpez.com/tutoriels/python/swig-numpy/#LV
> > to have a skeletton for your issue.
> 
> That looks very useful -- any chance of an English 
> translation? My one 
> year of high school French is proving useless. Otherwise, the code 
> itself is still quite helpful.
> 

Google does a pretty good job on this one:

http://translate.google.com/translate?u=http%3A%2F%2Fmatthieu-brucher.developpez.com%2Ftutoriels%2Fpython%2Fswig-numpy%2F%23LV&sl=fr&tl=en&hl=en&ie=UTF-8

Anthony.

--
Anthony Floyd, PhD
Convergent Manufacturing Technologies Inc.
6190 Agronomy Rd, Suite 403
Vancouver BC  V6T 1Z3
CANADA

Email: Anthony.Floyd at convergent.ca | Tel:   604-822-9682 x102
WWW:   http://www.convergent.ca    | Fax:   604-822-9659  

CMT is hiring: See http://www.convergent.ca for details
 


From charlesr.harris at gmail.com  Thu Oct 30 14:41:51 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 30 Oct 2008 12:41:51 -0600
Subject: [Numpy-discussion] Numpy matrix multiplication slow even though
	ATLAS linked
In-Reply-To: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
Message-ID: <e06186140810301141q69a0d524v69e9aae8ec1ac71a@mail.gmail.com>

On Thu, Oct 30, 2008 at 5:19 AM, Jan-Willem van de Meent <
vandemeent at damtp.cam.ac.uk> wrote:

> Dear all,
>
> This is my first post to this list. I am having perfomance issues with with
> numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes  about 1m40s, even
> though numpy is appears to link to my atlas libraries:
>
> I did a quick benchmark by running the following script:
>
>    #! /usr/bin/env python
>    import numpy
>    import time
>
>    try:
>       import numpy.core._dotblas
>       print 'Using ATLAS:'
>    except ImportError:
>       print 'No ATLAS:'
>
>    t = time.time()
>    x = numpy.random.random((1000,1000))
>    y = numpy.random.random((1000,1000))
>    z = numpy.dot(x, y)
>
>    print time.time()-t
>
> My laptop is a Dell D620 Core Duo T2300 1.66 Ghz, running Archlinux with
> GCC
> 4.3.2, atlas 3.8.2, python 2.5.2 and numpy 1.2.1. Output of the script
> above
> is:
>
>    Using ATLAS:
>    7.99549412727
>
> A department desktop PC, Pentium D 3.00 Ghz, running Scientific Linux, with
> GCC 4.1.2, atlas 3.7.30, python 2.5.1 and numpy 1.1.0, runs this test 24
> times faster:
>
>    Using ATLAS:
>    0.337520122528
>

About .40 here with numpy from svn.


>
> So even though _dotblas.so exists, matrix multiplication appears to run at
> pretty much the same speed as if atlas were not  available. Running ldd on
> _dotblas.so suggests  that numpy is indeed linking to the atlas libs:
>
> ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so
>        linux-gate.so.1 =>  (0xb7fcf000)
>        libatlas.so => /usr/lib/libatlas.so (0xb7cb5000)
>        liblapack.so => /usr/lib/liblapack.so (0xb77ab000)
>        libcblas.so => /usr/lib/libcblas.so (0xb778b000)
>        libf77blas.so => /usr/lib/libf77blas.so (0xb776f000)
>        libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7630000)
>        libpthread.so.0 => /lib/libpthread.so.0 (0xb7618000)
>        libc.so.6 => /lib/libc.so.6 (0xb74d6000)
>        libm.so.6 => /lib/libm.so.6 (0xb74b0000)
>        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb73ff000)
>        libdl.so.2 => /lib/libdl.so.2 (0xb73fb000)
>        libutil.so.1 => /lib/libutil.so.1 (0xb73f6000)
>        /lib/ld-linux.so.2 (0xb7fd0000)
>

What's in /usr/local/lib? Do you have a 64 bit system? What does locate
libatlas return?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081030/3060b76a/attachment.html>

From dfranci at seas.upenn.edu  Thu Oct 30 18:16:04 2008
From: dfranci at seas.upenn.edu (Frank Lagor)
Date: Thu, 30 Oct 2008 17:16:04 -0500
Subject: [Numpy-discussion] Complete LAPACK needed
Message-ID: <9fddf64a0810301516r7561a724t77456faa3d0e35ca@mail.gmail.com>

Dear all,

I need to use functions in scipy which depend on having a complete lapack
library.  However, I am having a bit of trouble installing numpy and
referencing a complete lapack library that I built.  I have a few questions
that I am hoping someone can help me answer:

1) This machine is a cluster, so do I need to be sure that the lapack
library is a .so file?  (I think only if I want the processors to access the
file at the same time, which is my need...) If so, how do I make a shared
library file?
2) If I can tell numpy to use some default version of lapack, will it be a
complete version?
3) I was unable to getting numpy installed by referencing the lapack.a
library from the site.cfg file. I also tried to set the LAPACK environment
variable in my .bashrc file, but this did not seem to work either.  The
problem is that when I configure numpy, it just does not recognize this file
as a lapack library.  It ignores it and tries to use an lapack lite library
located somewhere else (which doesn't even work anyway).

Any ideas? suggestions?
Thanks in advance!
Frank
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081030/fbbe09a2/attachment.html>

From vandemeent at damtp.cam.ac.uk  Thu Oct 30 18:19:01 2008
From: vandemeent at damtp.cam.ac.uk (Jan-Willem van de Meent)
Date: Thu, 30 Oct 2008 22:19:01 +0000
Subject: [Numpy-discussion] Numpy matrix multiplication slow even though
	ATLAS linked
In-Reply-To: <e06186140810301141q69a0d524v69e9aae8ec1ac71a@mail.gmail.com>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<e06186140810301141q69a0d524v69e9aae8ec1ac71a@mail.gmail.com>
Message-ID: <200810302219.01782.vandemeent@damtp.cam.ac.uk>

On Thursday 30 October 2008 18:41:51 Charles R Harris wrote:
> On Thu, Oct 30, 2008 at 5:19 AM, Jan-Willem van de Meent <
>
> vandemeent at damtp.cam.ac.uk> wrote:
> > Dear all,
> >
> > This is my first post to this list. I am having perfomance issues with
> > with numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes  about
> > 1m40s, even though numpy is appears to link to my atlas libraries:
> >
> > I did a quick benchmark by running the following script:
> >
> >    #! /usr/bin/env python
> >    import numpy
> >    import time
> >
> >    try:
> >       import numpy.core._dotblas
> >       print 'Using ATLAS:'
> >    except ImportError:
> >       print 'No ATLAS:'
> >
> >    t = time.time()
> >    x = numpy.random.random((1000,1000))
> >    y = numpy.random.random((1000,1000))
> >    z = numpy.dot(x, y)
> >
> >    print time.time()-t
> >
> > My laptop is a Dell D620 Core Duo T2300 1.66 Ghz, running Archlinux with
> > GCC
> > 4.3.2, atlas 3.8.2, python 2.5.2 and numpy 1.2.1. Output of the script
> > above
> > is:
> >
> >    Using ATLAS:
> >    7.99549412727
> >
> > A department desktop PC, Pentium D 3.00 Ghz, running Scientific Linux,
> > with GCC 4.1.2, atlas 3.7.30, python 2.5.1 and numpy 1.1.0, runs this
> > test 24 times faster:
> >
> >    Using ATLAS:
> >    0.337520122528
>
> About .40 here with numpy from svn.
>
> > So even though _dotblas.so exists, matrix multiplication appears to run
> > at pretty much the same speed as if atlas were not  available. Running
> > ldd on _dotblas.so suggests  that numpy is indeed linking to the atlas
> > libs:
> >
> > ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so
> >        linux-gate.so.1 =>  (0xb7fcf000)
> >        libatlas.so => /usr/lib/libatlas.so (0xb7cb5000)
> >        liblapack.so => /usr/lib/liblapack.so (0xb77ab000)
> >        libcblas.so => /usr/lib/libcblas.so (0xb778b000)
> >        libf77blas.so => /usr/lib/libf77blas.so (0xb776f000)
> >        libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7630000)
> >        libpthread.so.0 => /lib/libpthread.so.0 (0xb7618000)
> >        libc.so.6 => /lib/libc.so.6 (0xb74d6000)
> >        libm.so.6 => /lib/libm.so.6 (0xb74b0000)
> >        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb73ff000)
> >        libdl.so.2 => /lib/libdl.so.2 (0xb73fb000)
> >        libutil.so.1 => /lib/libutil.so.1 (0xb73f6000)
> >        /lib/ld-linux.so.2 (0xb7fd0000)
>
> What's in /usr/local/lib? Do you have a 64 bit system? What does locate
> libatlas return?
>
> Chuck

Thanks for the response. I'm on 32 bit and all ATLAS files are installed 
in /usr/lib (headers are in /usr/include/atlas)

/usr/lib/libatlas.a
/usr/lib/libatlas.so
/usr/lib/libblas.so
/usr/lib/libblas.so.3
/usr/lib/libblas.so.3.0.3
/usr/lib/libcblas.a
/usr/lib/libcblas.so
/usr/lib/libf77blas.a
/usr/lib/libf77blas.so
/usr/lib/liblapack.a
/usr/lib/liblapack.so
/usr/lib/liblapack.so.3
/usr/lib/libptcblas.a
/usr/lib/libptf77blas.a

I posted a full build log earlier today, but I think it is still awaiting 
approval of moderators because of file size limitations.

JW


From robert.kern at gmail.com  Thu Oct 30 18:32:22 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 17:32:22 -0500
Subject: [Numpy-discussion] Numpy matrix multiplication slow even though
	ATLAS linked
In-Reply-To: <200810302219.01782.vandemeent@damtp.cam.ac.uk>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<e06186140810301141q69a0d524v69e9aae8ec1ac71a@mail.gmail.com>
	<200810302219.01782.vandemeent@damtp.cam.ac.uk>
Message-ID: <3d375d730810301532u2823218ak29f1540e90f3c15@mail.gmail.com>

On Thu, Oct 30, 2008 at 17:19, Jan-Willem van de Meent
<vandemeent at damtp.cam.ac.uk> wrote:
> I posted a full build log earlier today, but I think it is still awaiting
> approval of moderators because of file size limitations.

There are no moderators to approve it. The file size limitation is
just a limitation. If you have web space to place the log, you can
give us a URL.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From vandemeent at damtp.cam.ac.uk  Thu Oct 30 18:39:45 2008
From: vandemeent at damtp.cam.ac.uk (Jan-Willem van de Meent)
Date: Thu, 30 Oct 2008 22:39:45 +0000
Subject: [Numpy-discussion] Numpy matrix multiplication slow even though
	ATLAS linked
In-Reply-To: <3d375d730810301532u2823218ak29f1540e90f3c15@mail.gmail.com>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<200810302219.01782.vandemeent@damtp.cam.ac.uk>
	<3d375d730810301532u2823218ak29f1540e90f3c15@mail.gmail.com>
Message-ID: <200810302239.46008.vandemeent@damtp.cam.ac.uk>

On Thursday 30 October 2008 22:32:22 Robert Kern wrote:
> On Thu, Oct 30, 2008 at 17:19, Jan-Willem van de Meent
> <vandemeent at damtp.cam.ac.uk> wrote:
> > I posted a full build log earlier today, but I think it is still awaiting
> > approval of moderators because of file size limitations.
>
> There are no moderators to approve it. The file size limitation is
> just a limitation. If you have web space to place the log, you can
> give us a URL.

Thanks for the heads up. The compilation log can be found here:

http://www.damtp.cam.ac.uk/user/meent/numpy-build-1.2.1-1jw.log

JW


From rgiuly at gmail.com  Thu Oct 30 21:41:44 2008
From: rgiuly at gmail.com (Rick Giuly)
Date: Thu, 30 Oct 2008 18:41:44 -0700
Subject: [Numpy-discussion] Large array using 4 times as much memory as it
	should
Message-ID: <490A6258.3050904@gmail.com>

Hello All,

I find that python is using about four times as much memory as it should 
need for arrays. This is problematic as I need to use all available 
memory for large 3D imaging datasets. Is there a way to get around this 
problem? Am I making a mistake? Is it a bug?

(I'm running windowsXP 32bit with 760M of memory "Available" according 
to the "Performance" pane of the task manager.)

versions: numpy 1.2.0 with python 2.5.2

Any help is appreciated


-Rick




**************************
Details of my testing:

Each test was run from the command line and for each test python was 
restarted.

Testing a 50M array:
a = numpy.ones((1024,1024,50), dtype=numpy.uint32)
The available memory dropped by 200M


Testing a 100M array:
a = numpy.ones((1024,1024,100), dtype=numpy.uint32)
The available memory dropped by 400M


Testing a 200M array:
a = numpy.ones((1024,1024,200), dtype=numpy.uint32)
The available memory dropped by 750M


Testing a 300M array:
a = numpy.ones((1024,1024,300), dtype=numpy.uint32)
an error occurs:
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File 
"o:\software\pythonxy\python\lib\site-packages\numpy\core\numeric.py", li
ne 1445, in ones
     a = empty(shape, dtype, order)
MemoryError









From robert.kern at gmail.com  Thu Oct 30 21:46:30 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 20:46:30 -0500
Subject: [Numpy-discussion] Large array using 4 times as much memory as
	it should
In-Reply-To: <490A6258.3050904@gmail.com>
References: <490A6258.3050904@gmail.com>
Message-ID: <3d375d730810301846t3d34d1f9lce07c51653151139@mail.gmail.com>

On Thu, Oct 30, 2008 at 20:41, Rick Giuly <rgiuly at gmail.com> wrote:
> Hello All,
>
> I find that python is using about four times as much memory as it should
> need for arrays. This is problematic as I need to use all available
> memory for large 3D imaging datasets. Is there a way to get around this
> problem? Am I making a mistake? Is it a bug?

You are making a mistake. uint32s take up 4 bytes each. So
(1024,1024,50) takes up 4*50*1024*1024 bytes == 200 Mb.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From rgiuly at gmail.com  Thu Oct 30 21:48:01 2008
From: rgiuly at gmail.com (Rick Giuly)
Date: Thu, 30 Oct 2008 18:48:01 -0700
Subject: [Numpy-discussion] [Fwd: Large array using 4 times as much memory
	as it should]
Message-ID: <490A63D1.1060106@gmail.com>

Please disregard the last message.

I just realised 8*4=32, very sorry about this. Apparently I need some sleep.

-Rick

-------- Original Message --------
Subject: Large array using 4 times as much memory as it should
Date: Thu, 30 Oct 2008 18:41:44 -0700
From: Rick Giuly <rgiuly at gmail.com>
To: numpy-discussion at scipy.org

Hello All,

I find that python is using about four times as much memory as it should
need for arrays. This is problematic as I need to use all available
memory for large 3D imaging datasets. Is there a way to get around this
problem? Am I making a mistake? Is it a bug?

(I'm running windowsXP 32bit with 760M of memory "Available" according
to the "Performance" pane of the task manager.)

versions: numpy 1.2.0 with python 2.5.2

Any help is appreciated


-Rick




**************************
Details of my testing:

Each test was run from the command line and for each test python was
restarted.

Testing a 50M array:
a = numpy.ones((1024,1024,50), dtype=numpy.uint32)
The available memory dropped by 200M


Testing a 100M array:
a = numpy.ones((1024,1024,100), dtype=numpy.uint32)
The available memory dropped by 400M


Testing a 200M array:
a = numpy.ones((1024,1024,200), dtype=numpy.uint32)
The available memory dropped by 750M


Testing a 300M array:
a = numpy.ones((1024,1024,300), dtype=numpy.uint32)
an error occurs:
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File
"o:\software\pythonxy\python\lib\site-packages\numpy\core\numeric.py", li
ne 1445, in ones
     a = empty(shape, dtype, order)
MemoryError









From fperez.net at gmail.com  Thu Oct 30 21:57:25 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 30 Oct 2008 18:57:25 -0700
Subject: [Numpy-discussion] [Fwd: Large array using 4 times as much
	memory as it should]
In-Reply-To: <490A63D1.1060106@gmail.com>
References: <490A63D1.1060106@gmail.com>
Message-ID: <db6b5ecc0810301857r38af27c0gaea73bb5515708c4@mail.gmail.com>

On Thu, Oct 30, 2008 at 6:48 PM, Rick Giuly <rgiuly at gmail.com> wrote:
> Please disregard the last message.
>
> I just realised 8*4=32, very sorry about this. Apparently I need some sleep.
>
> -Rick
>
> -------- Original Message --------
> Subject: Large array using 4 times as much memory as it should
> Date: Thu, 30 Oct 2008 18:41:44 -0700
> From: Rick Giuly <rgiuly at gmail.com>
> To: numpy-discussion at scipy.org
>
> Hello All,
>
> I find that python is using about four times as much memory as it should

You may find this handy for looking at your variable's memory use:

In [12]: a = numpy.ones((1024,1024,50), dtype=numpy.uint32)

In [13]: print 'approx size in Mb:',(a.size*a.itemsize)/1e6
approx size in Mb: 209.7152

numpy.who() gives you a nice summary:

In [14]: numpy.who()
Name            Shape                  Bytes            Type
=================================================================

a               1024 x 1024 x 50       209715200        uint32

Upper bound on total bytes  =       209715200


Cheers,

f


From alex.lacoste.ml at gmail.com  Thu Oct 30 23:14:34 2008
From: alex.lacoste.ml at gmail.com (Alexandre Lacoste)
Date: Thu, 30 Oct 2008 23:14:34 -0400
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
Message-ID: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>

Hi I'm trying to install numpy on a x86_64 ubuntu hardy and I'm having a
hard time to get atlas to be linked correctly.

Installing it through apt-get install with the astraw repository worked on
my intel dual core. But now I need to install it in on a x86_64 and the
astraw binaries will link against libatlas3gf-base instead of
libatlas3gf-3dnow because the 3dnow version simply doesn't exist for x86_64
arch. So I still get slow matrix multiplications ...

Then I went through the instructions for building atlas by hand. Then I
build numpy. Everything went ok... Now when comes the time to build scipy, I
get the following error :

from numpy.linalg import lapack_lite
ImportError: /usr/local/atlas/lib/liblapack.so: undefined symbol: ztbsv_

$ uname -a
Linux graal1 2.6.24-21-generic #1 SMP Mon Aug 25 16:57:51 UTC 2008 x86_64
GNU/Linux

$ ldd liblapack.so
        linux-vdso.so.1 =>  (0x00007fffe29fe000)
        libgfortran.so.2 => /usr/lib/libgfortran.so.2 (0x00007febd9ee3000)
        libm.so.6 => /lib/libm.so.6 (0x00007febd9c62000)
        libc.so.6 => /lib/libc.so.6 (0x00007febd98ff000)
        /lib64/ld-linux-x86-64.so.2 (0x00007febda93b000)


I'm getting clueless and googling for the error message return noting useful
any thoughts ?
thanks...

????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081030/7bb35960/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 00:49:00 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 13:49:00 +0900
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
In-Reply-To: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
Message-ID: <490A8E3C.8070005@ar.media.kyoto-u.ac.jp>

Alexandre Lacoste wrote:
> Hi I'm trying to install numpy on a x86_64 ubuntu hardy and I'm having
> a hard time to get atlas to be linked correctly.
>
> Installing it through apt-get install with the astraw repository
> worked on my intel dual core. But now I need to install it in on a
> x86_64 and the astraw binaries will link against libatlas3gf-base
> instead of libatlas3gf-3dnow because the 3dnow version simply doesn't
> exist for x86_64 arch. So I still get slow matrix multiplications ...

You can simply use the sse2 version of atlas: SSE2 is available on any
x86_64 CPU.

Note that on Ubuntu, you can link to libatals3fg-base (the gfortran,
basic version of ATLAS), and after building numpy, installing atlas for
SSE2 and still get the speed up, thanks to the hwcap capability of the
GNU loader. The hwcap means that when the loaded code (numpy) request a
library (atlas), it will first look for a specially optimized one in
some directory (/usr/lib/sse2, for example), and will use the default
one if the optimized one is not there.


>
> Then I went through the instructions for building atlas by hand. Then
> I build numpy. Everything went ok... Now when comes the time to build
> scipy, I get the following error :
>
> from numpy.linalg import lapack_lite
> ImportError: /usr/local/atlas/lib/
> liblapack.so: undefined symbol: ztbsv_

You made a mistake when building lapack and/or atlas. Those are
difficult to build correctly, you should avoid build them by yourself as
much as possible,

cheers,

David


From f.yw at hotmail.com  Fri Oct 31 01:25:49 2008
From: f.yw at hotmail.com (frank wang)
Date: Thu, 30 Oct 2008 23:25:49 -0600
Subject: [Numpy-discussion] (no subject)
In-Reply-To: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
Message-ID: <BLU105-W3478430AF282418CBA6D8489200@phx.gbl>


Hi,
 
In my work, I want to implement a fir filter with an input array. Since performing the filter on each input sample is slow, are there fast way to perform the fir filter operation?  Are there ways to convert input into an array and perform the array multipication?
 
Thanks
 
Frank
_________________________________________________________________
When your life is on the go?take your life with you.
http://clk.atdmt.com/MRT/go/115298558/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081030/1f5cc6bf/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 01:16:33 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 14:16:33 +0900
Subject: [Numpy-discussion] (no subject)
In-Reply-To: <BLU105-W3478430AF282418CBA6D8489200@phx.gbl>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
	<BLU105-W3478430AF282418CBA6D8489200@phx.gbl>
Message-ID: <490A94B1.1010107@ar.media.kyoto-u.ac.jp>

frank wang wrote:
> Hi,
>  
> In my work, I want to implement a fir filter with an input array.
> Since performing the filter on each input sample is slow, are there
> fast way to perform the fir filter operation?  Are there ways to
> convert input into an array and perform the array multipication?

look at scipy.signal.lfilter,

David


From alex.lacoste.ml at gmail.com  Fri Oct 31 01:34:53 2008
From: alex.lacoste.ml at gmail.com (Alexandre Lacoste)
Date: Fri, 31 Oct 2008 01:34:53 -0400
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
In-Reply-To: <c9a625a90810302233k147b7550he24530f56528f821@mail.gmail.com>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
	<490A8E3C.8070005@ar.media.kyoto-u.ac.jp>
	<c9a625a90810302233k147b7550he24530f56528f821@mail.gmail.com>
Message-ID: <c9a625a90810302234t1d3dd805v8a0c3d81e71d5d12@mail.gmail.com>

Thanks for your answer


> You can simply use the sse2 version of atlas: SSE2 is available on any
> x86_64 CPU.
>

Are you sure ? I can't find it...
http://packages.ubuntu.com/search?arch=amd64&searchon=names&keywords=atlas%20sse2


>
> Note that on Ubuntu, you can link to libatals3fg-base (the gfortran,
> basic version of ATLAS), and after building numpy, installing atlas for
> SSE2 and still get the speed up, thanks to the hwcap capability of the
> GNU loader. The hwcap means that when the loaded code (numpy) request a
> library (atlas), it will first look for a specially optimized one in
> some directory (/usr/lib/sse2, for example), and will use the default
> one if the optimized one is not there.
>

That's what I tried in the first place but I must have done something wrong
on that part too ;)



> You made a mistake when building lapack and/or atlas. Those are
> difficult to build correctly, you should avoid build them by yourself as
> much as possible,
>

I did it a couple of time in the past ... even on 64 bits machines... I
guess I was lucky back then.
I really need to get it work. I'll give it a couple of other tries. It is
only annoying to have to wait half an hour each time. If you can give me a
couple of other hints, maybe I'll target the problem faster.

Thanks again


????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/c56f0af1/attachment.html>

From f.yw at hotmail.com  Fri Oct 31 01:44:32 2008
From: f.yw at hotmail.com (frank wang)
Date: Thu, 30 Oct 2008 23:44:32 -0600
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl> 
	<3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
Message-ID: <BLU105-W52318A1E037C38CB20310789200@phx.gbl>


Hi, Bob,
 
The problem is that I want to resample my data with another sampling rate. the two rates is very close. I use the formula:
 
s(t)=sum(a_k*sinc(t-kTs)).
 
the new sampling rate is Ts', so I have 
s(nTs')=sum(a_k*sinc(nTs'-kTs)). The sum index k is over the (-P, P), Centered at n. The n is start from zero. THe code is using two for loops and it is slow. The length of s(nTs) is very long, so it takes quite long time to do it. 
 
Thanks
 
Frank> Date: Sun, 26 Oct 2008 22:45:42 -0500> From: robert.kern at gmail.com> To: numpy-discussion at scipy.org> Subject: Re: [Numpy-discussion] help to speed up the python code> > On Fri, Oct 24, 2008 at 11:30, frank wang <f.yw at hotmail.com> wrote:> > Hi,> >> > I have to send this request second time since my first message contains the> > attached data file which is too big and was blocked by the system. So this> > time I will not attach the data file.> >> > I have converted a matlab function to python using numpy. both matlab and> > python run slow. I know that numpy has a lot of features, so I hope some> > experts can help me to speed up the code.> > Can you describe in higher level terms what you are trying to do? I'm> having trouble following the code.> > -- > Robert Kern> > "I have come to believe that the whole world is an enigma, a harmless> enigma that is made terrible by our own mad attempt to interpret it as> though it had an underlying truth."> -- Umberto Eco> _______________________________________________> Numpy-discussion mailing list> Numpy-discussion at scipy.org> http://projects.scipy.org/mailman/listinfo/numpy-discussion
_________________________________________________________________
Stay organized with simple drag and drop from Windows Live Hotmail.
http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_102008
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081030/74300736/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 01:33:43 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 14:33:43 +0900
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <BLU105-W52318A1E037C38CB20310789200@phx.gbl>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
	<3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
	<BLU105-W52318A1E037C38CB20310789200@phx.gbl>
Message-ID: <490A98B7.4080108@ar.media.kyoto-u.ac.jp>

frank wang wrote:
> Hi, Bob,
>  
> The problem is that I want to resample my data with another sampling
> rate. the two rates is very close. I use the formula:
>  
> s(t)=sum(a_k*sinc(t-kTs)).
>  
> the new sampling rate is Ts', so I have
> s(nTs')=sum(a_k*sinc(nTs'-kTs)). The sum index k is over the (-P, P),
> Centered at n. The n is start from zero. THe code is using two for
> loops and it is slow. The length of s(nTs) is very long, so it takes
> quite long time to do it.

If you want to do high quality resampling, you may want to look at the
samplerate scikits. It uses sinc interpolation for resampling (the
scikits is just a wrapper around SRC: http://www.mega-nerd.com/SRC/, if
you want more details on the implementation)

The documentation is really light unfortunately, but there is an example:

http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/samplerate/index.html

The implementation of SRC is in C, and you can choose between 3 modes
for quality (plus other interpolations like linear, but those are ill
suited for anything serious if you can afford the computational cost),

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 01:40:04 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 14:40:04 +0900
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <BLU105-W52318A1E037C38CB20310789200@phx.gbl>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
	<3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
	<BLU105-W52318A1E037C38CB20310789200@phx.gbl>
Message-ID: <490A9A34.8010308@ar.media.kyoto-u.ac.jp>

frank wang wrote:
> Hi, Bob,
>  
> The problem is that I want to resample my data with another sampling
> rate. the two rates is very close. I use the formula:
>  
> s(t)=sum(a_k*sinc(t-kTs)).
>  
> the new sampling rate is Ts', so I have
> s(nTs')=sum(a_k*sinc(nTs'-kTs)). The sum index k is over the (-P, P),
> Centered at n. The n is start from zero. THe code is using two for
> loops and it is slow. The length of s(nTs) is very long, so it takes
> quite long time to do it.

Additionally, assuming you still want to do it yourself in python, the
above is a convolution, and both numpy and matlab have 'fast' (e.g.
implemented in C) implementation of this; see numpy.convolve. You can
have much faster speed than O(M * P) where M is the length of the
original signal using FFT, but if either the sinc 'filter' or your
signal is large, it is more difficult to do it without using large
amount of memory (you have to split the input signal, and even the
filter impulse response  if it is large, and use overlapp-kind of
techniques, see
http://ccrma.stanford.edu/~jos/sasp/Example_Overlap_Add_Convolution.html
for an example in matlab).

cheers,

David


From shuntim.luk at polyu.edu.hk  Fri Oct 31 02:31:38 2008
From: shuntim.luk at polyu.edu.hk (LUK ShunTim)
Date: Fri, 31 Oct 2008 14:31:38 +0800
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
In-Reply-To: <c9a625a90810302234t1d3dd805v8a0c3d81e71d5d12@mail.gmail.com>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>	<490A8E3C.8070005@ar.media.kyoto-u.ac.jp>	<c9a625a90810302233k147b7550he24530f56528f821@mail.gmail.com>
	<c9a625a90810302234t1d3dd805v8a0c3d81e71d5d12@mail.gmail.com>
Message-ID: <490AA64A.9090703@polyu.edu.hk>

Alexandre Lacoste wrote:
> 
> Thanks for your answer
> 
> 
>     You can simply use the sse2 version of atlas: SSE2 is available on any
>     x86_64 CPU.
> 
> 
> Are you sure ? I can't find it...
> http://packages.ubuntu.com/search?arch=amd64&searchon=names&keywords=atlas%20sse2
> <http://packages.ubuntu.com/search?arch=amd64&searchon=names&keywords=atlas%20sse2>

Hi,

Try "cat /proc/cpuinfo |grep sse2" :-)

In my debian/sid/amd64 box, there's only libatlas3gf now. I guess it's
the same for ubuntu.

Regards,
ST
--



From alex.lacoste.ml at gmail.com  Fri Oct 31 02:59:25 2008
From: alex.lacoste.ml at gmail.com (Alexandre Lacoste)
Date: Fri, 31 Oct 2008 02:59:25 -0400
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
In-Reply-To: <490AA64A.9090703@polyu.edu.hk>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>
	<490A8E3C.8070005@ar.media.kyoto-u.ac.jp>
	<c9a625a90810302233k147b7550he24530f56528f821@mail.gmail.com>
	<c9a625a90810302234t1d3dd805v8a0c3d81e71d5d12@mail.gmail.com>
	<490AA64A.9090703@polyu.edu.hk>
Message-ID: <c9a625a90810302359v1ec58206ra02deabba96488ff@mail.gmail.com>

>
> Hi,
>
> Try "cat /proc/cpuinfo |grep sse2" :-)
>
> In my debian/sid/amd64 box, there's only libatlas3gf now. I guess it's
> the same for ubuntu.
>
>
Yes the cpu supports sse2 instructions. However, I can't find any repository
that have libatlas3gf-sse2 built for x86_64. Should i try to force an
install of a i386 deb. How can I do that? will it work ?

Regards,
????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/54477304/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 02:51:04 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 15:51:04 +0900
Subject: [Numpy-discussion] newaxis and contiguity
Message-ID: <490AAAD8.7030909@ar.media.kyoto-u.ac.jp>

Hi,

    I was wondering about the following behavior:

a = np.random.randn(10)
b = a[:. np.newaxis]
print b.flags # F_CONTIGUOUS and C_CONTIGUOUS are both false

Is it for simplicity purpose, or is there a more fundamental reason why
a rank 2, 'column' vector can't be F_CONTIGUOUS in this case ? Does the
operation involving newaxis modifies the data buffer ?

thanks,

David


From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 02:52:40 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 15:52:40 +0900
Subject: [Numpy-discussion] unable to build atlas and lapack correctly
In-Reply-To: <c9a625a90810302359v1ec58206ra02deabba96488ff@mail.gmail.com>
References: <c9a625a90810302014q386ad34cxdc64c8c0b2cb8e6f@mail.gmail.com>	<490A8E3C.8070005@ar.media.kyoto-u.ac.jp>	<c9a625a90810302233k147b7550he24530f56528f821@mail.gmail.com>	<c9a625a90810302234t1d3dd805v8a0c3d81e71d5d12@mail.gmail.com>	<490AA64A.9090703@polyu.edu.hk>
	<c9a625a90810302359v1ec58206ra02deabba96488ff@mail.gmail.com>
Message-ID: <490AAB38.9060805@ar.media.kyoto-u.ac.jp>

Alexandre Lacoste wrote:
> Yes the cpu supports sse2 instructions. However, I can't find any
> repository that have libatlas3gf-sse2 built for x86_64. Should i try
> to force an install of a i386 deb. How can I do that? will it work ?

No, it won't, and no, don't force it. I doubt dpkg would allow it, but
if it did, it would mess up your system. Installing i386 binary on the
x86_64 to use it at runtime has as much sense as putting PPC or SPARC
binary and hoping it to work (at least on linux),

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Fri Oct 31 06:25:31 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 31 Oct 2008 19:25:31 +0900
Subject: [Numpy-discussion] Simplifying compiler optimization flags logic
	(fortran compilers)
Message-ID: <490ADD1B.9050207@ar.media.kyoto-u.ac.jp>

Hi,

    I was wondering whether it was really worth having a lot of magic
going on in fcompilers for flags like -msse2 and co (everything done in
get_flags_arch, for example). It is quite fragile (we had several
problems wrt buggy compilers, buggy CPU detection), and I am not sure it
buys us much anyway. Did some people notice a difference between
gfortran -O3 -msse2 and gfortran -O3 ?

thanks,

David


From faltet at pytables.org  Fri Oct 31 09:43:19 2008
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 31 Oct 2008 14:43:19 +0100
Subject: [Numpy-discussion] ANN: PyTables 2.1rc1 is ready to be tested
Message-ID: <200810311443.20172.faltet@pytables.org>

============================
 Announcing PyTables 2.1rc1
============================

PyTables is a library for managing hierarchical datasets and designed to
efficiently cope with extremely large amounts of data with support for
full 64-bit file addressing.  PyTables runs on top of the HDF5 library
and NumPy package for achieving maximum throughput and convenient use.

In PyTables 2.1rc1 many new features and a handful of bugs have been
addressed.  This is a release candidate, so, in addition to the tarball,
binaries for Windows are provided too.  Also, the API has been frozen
and you should only expect bug fixes and documentation improvements for
2.1 final (due to release in a couple of weeks now).

This version introduces important improvements, like much faster node
opening, creation or navigation, a file-based way to fine-tune the
different PyTables parameters (fully documented now in a new appendix of
the UG) and support for multidimensional atoms in EArray/CArray objects.

Regarding the Pro edition, 3 different kind of indexes have been added
so that the user can choose the best for her needs.  Also, and due to
the introduction of the concept of chunkmaps in OPSI, the responsiveness
of complex queries with low selectivity has improved quite a lot.  And
last but not least, it is possible now to sort completely tables that
are ordered by a specific field, with no practical limit in size (up to
2**48 rows, that is, around 281 trillion of rows).  More info in:
http://www.pytables.org/moin/PyTablesPro#WhatisnewinforthcomingPyTablesPro2.1

In case you want to know more in detail what has changed in this
version, have a look at ``RELEASE_NOTES.txt`` in the tarball.  Find the
HTML version for this document at:
http://www.pytables.org/moin/ReleaseNotes/Release_2.1rc1

You can download a source package of the version 2.1rc1 with
generated PDF and HTML docs and binaries for Windows from
http://www.pytables.org/download/preliminary

Finally, and for the first time, an evaluation version for PyTables Pro
has been made available in:
http://www.pytables.org/download/evaluation
Please read the evaluation license for terms of use of this version:
http://www.pytables.org/moin/PyTablesProEvaluationLicense

For an on-line version of the manual, visit:
http://www.pytables.org/docs/manual-2.1rc1


Resources
=========

Go to the PyTables web site for more details:

http://www.pytables.org

About the HDF5 library:

http://hdfgroup.org/HDF5/

About NumPy:

http://numpy.scipy.org/

Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Many
thanks also to SourceForge who have helped to make and distribute this
package!  And last, but not least thanks a lot to the HDF5 and NumPy
(and numarray!) makers. Without them PyTables simply would not exist.

Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.

----

  **Enjoy data!**

  -- The PyTables Team


From pav at iki.fi  Fri Oct 31 09:45:56 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 31 Oct 2008 13:45:56 +0000 (UTC)
Subject: [Numpy-discussion] Numpy matrix multiplication slow even
	though	ATLAS linked
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<e06186140810301141q69a0d524v69e9aae8ec1ac71a@mail.gmail.com>
	<200810302219.01782.vandemeent@damtp.cam.ac.uk>
Message-ID: <gef26k$e8t$1@ger.gmane.org>

Thu, 30 Oct 2008 22:19:01 +0000, Jan-Willem van de Meent wrote:

> On Thursday 30 October 2008 18:41:51 Charles R Harris wrote:
>> On Thu, Oct 30, 2008 at 5:19 AM, Jan-Willem van de Meent <
>>
>> vandemeent at damtp.cam.ac.uk> wrote:
>> > Dear all,
>> >
>> > This is my first post to this list. I am having perfomance issues
>> > with with numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes 
>> > about 1m40s, even though numpy is appears to link to my atlas
>> > libraries:

Can you try to benchmark your ATLAS library using a simple C or Fortran 
program to check if the problem is in Numpy, or in Atlas itself.

For comparison,

	gfortran -o test test.f90 -lblas

	time ./test   # ATLAS
	-> 0.55 s

	LD_PRELOAD=/usr/lib/libblas.so.3.0 time ./test  # reference BLAS
	-> 5.6 s


test.f90
--------
program main
    integer, parameter :: n = 1000
    double precision, dimension(n,n) :: a, b, c
    integer :: i, j

    do i = 1, n
        do j = 1,n
            a(i,j) = i+j
            b(i,j) = i-j
        end do
    end do

    call dgemm('N', 'N', n, n, n, 1d0, a, n, b, n, 0d0, c, n)
end program main



From soren.skou.nielsen at gmail.com  Fri Oct 31 11:28:38 2008
From: soren.skou.nielsen at gmail.com (=?ISO-8859-1?Q?S=F8ren_Nielsen?=)
Date: Fri, 31 Oct 2008 16:28:38 +0100
Subject: [Numpy-discussion] indexing numpy arrays in extensions created with
	ext_tools
Message-ID: <de1aa14a0810310828t7597d21etc531d97e87f52b31@mail.gmail.com>

Hi,

I'm trying to make a weave python extension to use in my program. I already
did it in inline, but that doesn't work with py2exe (needs compiler), so I'm
creating extensions instead using ext_tools.

Is there a way I can use blitz with ext_tools? so that I can refer to numpy
arrays like a(x,y) in the c code?? I have alot of 2D arrays that needs
indexing in the C code.. right now I can only index one dimension since the
first is always zero for some reason.. so if I write a[1,1] in the c code, I
still get returned the a[0,1]

Any help is appreciated!

Thanks,
Soren

Some silly test code:
-----------------------------------------

mod = ext_tools.ext_module('test_ext')

    a = zeros((2,2))
    x = 1
    y = 0

    test_code = """
                py::tuple args(1);

                if(a[x,y] == 1)
                {
                    args[0] = 1;
                }
                else
                {
                    args[0] = 0;
                }

                args[0] = a[x,y];

                return_val = args;
                """
    ravg = ext_tools.ext_function('test', test_code, ['a', 'x', 'y'])
    mod.add_function(ravg)
    mod.compile()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/21f43403/attachment.html>

From charlesr.harris at gmail.com  Fri Oct 31 12:16:27 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 31 Oct 2008 10:16:27 -0600
Subject: [Numpy-discussion] newaxis and contiguity
In-Reply-To: <490AAAD8.7030909@ar.media.kyoto-u.ac.jp>
References: <490AAAD8.7030909@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140810310916wcc23259haf1640b58407794b@mail.gmail.com>

On Fri, Oct 31, 2008 at 12:51 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Hi,
>
>    I was wondering about the following behavior:
>
> a = np.random.randn(10)
> b = a[:. np.newaxis]
> print b.flags # F_CONTIGUOUS and C_CONTIGUOUS are both false
>
> Is it for simplicity purpose, or is there a more fundamental reason why
> a rank 2, 'column' vector can't be F_CONTIGUOUS in this case ? Does the
> operation involving newaxis modifies the data buffer ?
>

There are other cases like this. The problem seems to be detecting the
contiguous cases with the result that the flags don't always reflect
reality, i.e., they are on the pessimistic side. This is an area where
improvements could be made.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/c89c8e81/attachment.html>

From w.kejia at gmail.com  Fri Oct 31 13:20:41 2008
From: w.kejia at gmail.com (Wu, Kejia)
Date: Fri, 31 Oct 2008 10:20:41 -0700
Subject: [Numpy-discussion] About Random Number Generation
Message-ID: <1225473641.7737.1.camel@localhost>

Hi all,

I tried the example code here:
http://numpy.scipy.org/numpydoc/numpy-20.html#71863
But failed:
--------------------------------------
rng.py, line 5, in <module>
    import RNG
ImportError: No module named RNG
--------------------------------------

Any suggestion? Thanks at first.

Also, can any body tell me whether the random number algorithm in RNG
package is a pseudorandom one or a real-random one? And is there an
available implementation for Monte Carlo method in NumPy?

Thanks a lot for any reply.

Regards,
Kejia






From vandemeent at damtp.cam.ac.uk  Fri Oct 31 13:40:21 2008
From: vandemeent at damtp.cam.ac.uk (Jan-Willem van de Meent)
Date: Fri, 31 Oct 2008 17:40:21 +0000
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Numpy_matrix_multiplication_slo?=
	=?iso-8859-1?q?w_even_though=09ATLAS_linked?=
In-Reply-To: <gef26k$e8t$1@ger.gmane.org>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<200810302219.01782.vandemeent@damtp.cam.ac.uk>
	<gef26k$e8t$1@ger.gmane.org>
Message-ID: <200810311740.22048.vandemeent@damtp.cam.ac.uk>

On Friday 31 October 2008 13:45:56 Pauli Virtanen wrote:
> Thu, 30 Oct 2008 22:19:01 +0000, Jan-Willem van de Meent wrote:
> > On Thursday 30 October 2008 18:41:51 Charles R Harris wrote:
> >> On Thu, Oct 30, 2008 at 5:19 AM, Jan-Willem van de Meent <
> >>
> >> vandemeent at damtp.cam.ac.uk> wrote:
> >> > Dear all,
> >> >
> >> > This is my first post to this list. I am having perfomance issues
> >> > with with numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes
> >> > about 1m40s, even though numpy is appears to link to my atlas
> >> > libraries:
>
> Can you try to benchmark your ATLAS library using a simple C or Fortran
> program to check if the problem is in Numpy, or in Atlas itself.
>
> For comparison,
>
> 	gfortran -o test test.f90 -lblas
>
> 	time ./test   # ATLAS
> 	-> 0.55 s
>
> 	LD_PRELOAD=/usr/lib/libblas.so.3.0 time ./test  # reference BLAS
> 	-> 5.6 s
>
>
> test.f90
> --------
> program main
>     integer, parameter :: n = 1000
>     double precision, dimension(n,n) :: a, b, c
>     integer :: i, j
>
>     do i = 1, n
>         do j = 1,n
>             a(i,j) = i+j
>             b(i,j) = i-j
>         end do
>     end do
>
>     call dgemm('N', 'N', n, n, n, 1d0, a, n, b, n, 0d0, c, n)
> end program main
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

I must admit have no experience calling Atlas routines from either C or Fortan 
and am a bit clumsy with compilers. However, I got your test case to compile 
by doing: 

   fortran -o test_atlas test_atlas.f90 -lptf77blas -latlas

which gives

    time ./test_atlas
    -> 0.85 s

I don't understand what the LD_PRELOAD directive is supposed to do, but timing 
it gives

  time LD_PRELOAD=/usr/lib/libblas.so.3.0.3 ./test_atlas
  -> 0.86 s

For reference, here are the results of xatlbench and xdmmtst_big (generated at 
compile time by Atlas). As far as I can tell from comparison with on-line 
posted results, these should be pretty normal.

./xatlbench

Clock rate=1667Mhz
               single precision        double precision
            *********************    ********************
               real      complex       real      complex
Benchmark   %   Clock   %   Clock   %   Clock   %   Clock
=========   =========   =========   =========   =========
  kSelMM       264.6      264.6       86.1       84.6
  kGenMM        86.7       89.3       84.7       84.6
  kMM_NT        78.8       77.6       75.5       75.9
  kMM_TN        87.2       84.2       77.7       82.1
  BIG_MM       261.4      261.9       85.2       85.7
   kMV_N        27.2       91.4       50.7       78.3
   kMV_T        75.2       76.9       53.9       63.1
    kGER        49.0       84.7       23.6       46.5

./xdmmtst_big

TEST  TA  TB    M    N    K  alpha   beta    Time  Mflop  SpUp  PASS
====  ==  ==  ===  ===  ===  =====  =====  ======  =====  ====  ====

   1   N   N  100  100  100    1.0    1.0    0.00 600.1  1.00   ---
   1   N   N  100  100  100    1.0    1.0    0.00 600.1  1.00   YES
   2   N   N  200  200  200    1.0    1.0    0.01 1600.0  1.00   ---
   2   N   N  200  200  200    1.0    1.0    0.01 1600.2  1.00   YES
   3   N   N  300  300  300    1.0    1.0    0.04 1350.1  1.00   ---
   3   N   N  300  300  300    1.0    1.0    0.04 1350.1  1.00   YES
   4   N   N  400  400  400    1.0    1.0    0.09 1371.5  1.00   ---
   4   N   N  400  400  400    1.0    1.0    0.09 1422.3  1.04   YES
   5   N   N  500  500  500    1.0    1.0    0.18 1389.0  1.00   ---
   5   N   N  500  500  500    1.0    1.0    0.18 1389.0  1.00   YES
   6   N   N  600  600  600    1.0    1.0    0.31 1408.8  1.00   ---
   6   N   N  600  600  600    1.0    1.0    0.31 1408.8  1.00   YES
   7   N   N  700  700  700    1.0    1.0    0.49 1409.7  1.00   ---
   7   N   N  700  700  700    1.0    1.0    0.49 1409.7  1.00   YES
   8   N   N  800  800  800    1.0    1.0    0.73 1409.3  1.00   ---
   8   N   N  800  800  800    1.0    1.0    0.73 1409.3  1.00   YES
   9   N   N  900  900  900    1.0    1.0    1.03 1411.1  1.00   ---
   9   N   N  900  900  900    1.0    1.0    1.03 1411.1  1.00   YES
  10   N   N 1000 1000 1000    1.0    1.0    1.41 1418.5  1.00   ---
  10   N   N 1000 1000 1000    1.0    1.0    1.42 1408.5  0.99   YES


From robert.kern at gmail.com  Fri Oct 31 13:49:29 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 31 Oct 2008 12:49:29 -0500
Subject: [Numpy-discussion] About Random Number Generation
In-Reply-To: <1225473641.7737.1.camel@localhost>
References: <1225473641.7737.1.camel@localhost>
Message-ID: <3d375d730810311049s350c010fk30572cc93636ca8a@mail.gmail.com>

On Fri, Oct 31, 2008 at 12:20, Wu, Kejia <w.kejia at gmail.com> wrote:
> Hi all,
>
> I tried the example code here:
> http://numpy.scipy.org/numpydoc/numpy-20.html#71863
> But failed:
> --------------------------------------
> rng.py, line 5, in <module>
>    import RNG
> ImportError: No module named RNG
> --------------------------------------
>
> Any suggestion? Thanks at first.

Despite the confusing URL, that is actually documentation for Numeric,
numpy's predecessor. You can see documentation for the current version
of numpy here:

  http://docs.scipy.org/doc/

> Also, can any body tell me whether the random number algorithm in RNG
> package is a pseudorandom one or a real-random one?

Pseudorandom. The Mersenne Twister, to be precise.

> And is there an
> available implementation for Monte Carlo method in NumPy?

"Monte Carlo" is more a general description than a specification of a
particular algorithm. There are many such methods. Which one are you
thinking of?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Fri Oct 31 13:54:54 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 31 Oct 2008 11:54:54 -0600
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <BLU105-W52318A1E037C38CB20310789200@phx.gbl>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
	<3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
	<BLU105-W52318A1E037C38CB20310789200@phx.gbl>
Message-ID: <e06186140810311054u502b957dvc73a95fa13c2e356@mail.gmail.com>

On Thu, Oct 30, 2008 at 11:44 PM, frank wang <f.yw at hotmail.com> wrote:

>  Hi, Bob,
>
> The problem is that I want to resample my data with another sampling rate.
> the two rates is very close. I use the formula:
>
> s(t)=sum(a_k*sinc(t-kTs)).
>
> the new sampling rate is Ts', so I have
> s(nTs')=sum(a_k*sinc(nTs'-kTs)). The sum index k is over the (-P, P),
> Centered at n. The n is start from zero. THe code is using two for loops and
> it is slow. The length of s(nTs) is very long, so it takes quite long time
> to do it.
>
>

I think you can use some form of chirpz/Bluestein for this. Just think of
the data as the Fourier transform of its spectrum.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/6ecdff5a/attachment.html>

From pav at iki.fi  Fri Oct 31 13:58:44 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 31 Oct 2008 17:58:44 +0000 (UTC)
Subject: [Numpy-discussion] About Random Number Generation
References: <1225473641.7737.1.camel@localhost>
Message-ID: <gefh0k$fsu$1@ger.gmane.org>

Fri, 31 Oct 2008 10:20:41 -0700, Wu, Kejia wrote:

> Hi all,
> 
> I tried the example code here:
> http://numpy.scipy.org/numpydoc/numpy-20.html#71863 But failed:

That's the documentation of the old Numeric package, though the name of 
the URL is severly misleading.

> -------------------------------------- rng.py, line 5, in <module>
>     import RNG
> ImportError: No module named RNG
> --------------------------------------
> 
> Any suggestion? Thanks at first.

If you want to stick with the old Numeric (an obsolete software package), 
at least in Debian the random number generator is in a separate package 
(python-numeric-ext).

The random number generator available in current Numpy can be used like 
this

	import numpy.random
	data = numpy.random.rand(100, 100)

> Also, can any body tell me whether the random number algorithm in RNG
> package is a pseudorandom one or a real-random one?

Like most random number generators, it is a pseudorandom one. However, it 
seeds itself from system's random source initially.

> And is there an available implementation for Monte Carlo method in
> NumPy?

"Monte Carlo" is a generic name for a group of methods. For Monte Carlo 
Markov Chain sampling, you can eg. see PyMC [1]. I believe there are also 
other packages implementing these methods and written in Python.

.. [1] http://code.google.com/p/pymc/

-- 
Pauli Virtanen



From vandemeent at damtp.cam.ac.uk  Fri Oct 31 14:06:39 2008
From: vandemeent at damtp.cam.ac.uk (Jan-Willem van de Meent)
Date: Fri, 31 Oct 2008 18:06:39 +0000
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Numpy_matrix_multiplication_slo?=
	=?iso-8859-1?q?w_even_though=09ATLAS_linked?=
In-Reply-To: <200810311740.22048.vandemeent@damtp.cam.ac.uk>
References: <200810301119.00700.vandemeent@damtp.cam.ac.uk>
	<gef26k$e8t$1@ger.gmane.org>
	<200810311740.22048.vandemeent@damtp.cam.ac.uk>
Message-ID: <200810311806.39868.vandemeent@damtp.cam.ac.uk>

I resolved the issue after having another look at the build scripts. Thanks 
everyone for your help!

For reference, the problem arose from a bad site.cfg file. In the past this 
file was necessary on ArchLinux to make numpy find the atlas libs. I tried 
removing it and now numpy to compiles correctly!

(offending) site.cfg
--------------------
[atlas]
library_dirs = /usr/lib
atlas_libs   = atlas, lapack, cblas, f77blas, ptcblas, ptf77blas

For reference #2: During compilation of the improperly linked numpy, the 
script displays the info:

FOUND:
  libraries = ['atlas', 'lapack', 'cblas', 'f77blas', 'ptcblas', 'ptf77blas']
  library_dirs = ['/usr/lib']
  language = c
  define_macros = [('NO_ATLAS_INFO', 2)]
  include_dirs = ['/usr/include']

Whereas a good compile shows:

  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')]
    include_dirs = ['/usr/include']

Cheers! Now I can get back to work :-)

JW



From dfranci at seas.upenn.edu  Fri Oct 31 15:42:21 2008
From: dfranci at seas.upenn.edu (Frank Lagor)
Date: Fri, 31 Oct 2008 15:42:21 -0400
Subject: [Numpy-discussion] Complete LAPACK needed (Frank Lagor)
Message-ID: <9fddf64a0810311242i451df222xf4026aa74f74b7f7@mail.gmail.com>

Hi Everyone,

I am trying to give an update here so that maybe it will be easier for
someone to help me out now.

I have gone through the lapack installation and the atlas installation and
everything seems to have worked fine.  Everything was compiled using the
'fPIC' option and I figured out how to make shared blas and lapack library
files with atlas.  Also, I figured out how to tell Atlas that I want a
COMPLETE lapack.  Now I am having trouble with the numpy install.

In the site.cfg file, I included the location where I have the include and
lib files for lapack.  After install, when I test, I get the following:
[dfranci at alpha ~]$ python -c 'import numpy; numpy.test()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/ronaldo/pkg/dfranci/Python-2.5.2/lib/python2.5/site-packages/numpy/__init__.py",
line 143, in <module>
    import linalg
  File
"/ronaldo/pkg/dfranci/Python-2.5.2/lib/python2.5/site-packages/numpy/linalg/__init__.py",
line 47, in <module>
    from linalg import *
  File
"/ronaldo/pkg/dfranci/Python-2.5.2/lib/python2.5/site-packages/numpy/linalg/linalg.py",
line 29, in <module>
    from numpy.linalg import lapack_lite
ImportError: liblapack.so: cannot open shared object file: No such file or
directory

I notice one thing here that concerns me: numpy is trying to import a module
that it calls "lapack_lite".  Does numpy always build such a module?  If I
am trying to get the complete lapack, did I do something wrong?

Thanks in advance,
Frank
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20081031/4da6e0d8/attachment.html>

From yakov.keselman at gmail.com  Fri Oct 31 20:13:24 2008
From: yakov.keselman at gmail.com (Yakov Keselman)
Date: Fri, 31 Oct 2008 17:13:24 -0700
Subject: [Numpy-discussion] help to speed up the python code
In-Reply-To: <e06186140810311054u502b957dvc73a95fa13c2e356@mail.gmail.com>
References: <BLU105-W5167024FC821A50E46E5DA932B0@phx.gbl>
	<BLU105-W27433EE8AA2ECEDA05E132892B0@phx.gbl>
	<3d375d730810262045r70f61d8er56d5970c29e29fa@mail.gmail.com>
	<BLU105-W52318A1E037C38CB20310789200@phx.gbl>
	<e06186140810311054u502b957dvc73a95fa13c2e356@mail.gmail.com>
Message-ID: <dc3116ef0810311713j6bb8a225p9ad69ac6e6854148@mail.gmail.com>

My understanding of the root of the problem is that you end up with
doing many evaluations of sinc. If this is so, one suggestion is to go
with pre-computed filters. For example, if you are resampling from 9
points to 10, essentially you're trying to go from a function that is
defined on points 0, 1, 2..., 8 to a function defined on points 0,
0.9, 1.8, 2.7, ... 0.8. If you take point 2.7, it falls in between 2
and 3, and the value for your new function can be approximately
computed as the weighted average: f(2.7) = 0.7*f(3) + 0.3*f(2). This
is equivalent to convolving with a filter [0.3, 0.7]. The only problem
is that the filter will vary from point to point. So, perhaps, you can
pre-compute several filters for each type of new point (there will be
10 point types for this specific case). Your filters would also be
larger in length, so that your interpolation is more precise. Hope
this helps.

On 10/31/08, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On Thu, Oct 30, 2008 at 11:44 PM, frank wang <f.yw at hotmail.com> wrote:
>
>>  Hi, Bob,
>>
>> The problem is that I want to resample my data with another sampling rate.
>> the two rates is very close. I use the formula:
>>
>> s(t)=sum(a_k*sinc(t-kTs)).
>>
>> the new sampling rate is Ts', so I have
>> s(nTs')=sum(a_k*sinc(nTs'-kTs)). The sum index k is over the (-P, P),
>> Centered at n. The n is start from zero. THe code is using two for loops
>> and
>> it is slow. The length of s(nTs) is very long, so it takes quite long time
>> to do it.
>>
>>
>
> I think you can use some form of chirpz/Bluestein for this. Just think of
> the data as the Fourier transform of its spectrum.
>
> Chuck
>


-- 
Not to laugh, not to lament, not to curse, but to understand. -- Spinoza