From hjn253 at tom.com Mon Sep 4 16:29:12 2006 From: hjn253 at tom.com (=?GB2312?B?IjnUwjktMTDI1S/Jz7qjIg==?=) Date: Tue, 5 Sep 2006 04:29:12 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU61MvTw0VYQ0VMuMS9+MrQs6HTqs/60+uyxs7xudzA7Q==?= Message-ID: An HTML attachment was scrubbed... URL: From emilee at firstinsurancefunding.com Fri Sep 1 20:18:43 2006 From: emilee at firstinsurancefunding.com (Guendolen Houze) Date: Fri, 1 Sep 2006 17:18:43 -0700 Subject: [Numpy-discussion] iiR Xje Message-ID: <000001c6ce25$597fb010$96f9a8c0@xqyicw> Hi All yo e ur PHA o RRMAC s Y d f irec g tly from the m t anufa a cture j r, Your ch v an b ce to eco i nomiz v e wit o h us http://omgandesunkerion.com r z y floor was smooth and hard, as were the walls when I brushed my fingers What about her? Steengo asked. be a crackup in the old kaserne tonight! -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 2 22:26:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 2 Sep 2006 20:26:42 -0600 Subject: [Numpy-discussion] Keyword added to searchsorted. Message-ID: Hi all, I added the keyword side to the searchsorted method and functions. Documentation: """a.searchsorted(values=v, side='left') -> array of indices. Required Arguments: v -- keys to be searched for in a. Keyword arguments side -- {'left', 'right'}, default('left'). If a is a 1-D array in ascending order, then a.searchsorted(v, side='left') returns an array of indices i such that for each element of values the following holds: a[j] < key <= a[i] for all j < i, If such an index does not exist, a.size() is used. The result is such that if the key were to be inserted in the slot before the index i, then the order of a would be preserved and i would be the smallest index with that property. If a is a 1-D array in ascending order, then a.searchsorted(v, side='right') returns an array of indices i such that for each element of values the following holds: a[j] <= key < a[i] for all j < i, If such an index does not exist, a.size() is used. The result is that if the key were to be inserted in the slot before the index i, then the order of a would be preserved and i would be the largest index with that property. """ I also replaced the old search routine as it was linear time worst case: In [27]: t1 = t.Timer('N.searchsorted(a,1,side="left")','import numpy as N; a = N.ones(1000000)') In [28]: t2 = t.Timer('N.searchsorted(a,1,side="right")','import numpy as N; a = N.ones(1000000)') In [29]: t1.repeat(3,100) Out[29]: [0.5301368236541748, 0.4924161434173584, 0.46317386627197266] In [30]: t2.repeat(3,100) Out[30]: [0.0011379718780517578, 0.00081586837768554688, 0.00083994865417480469] where left was the original routine. It is now noticeably faster in some situations: In [2]: t1 = t.Timer('N.searchsorted(a,1,side="left")','import numpy as N; a = N.ones(1000000)') In [3]: t1.repeat(3,100) Out[3]: [0.00082802772521972656, 0.00077795982360839844, 0.00076913833618164062] I am open to suggestions as to better names for the keyword kind. It also might be worth it to make type-specific routines if anyone commonly uses large lists of keys and large sorted arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Sep 2 22:37:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 02 Sep 2006 21:37:04 -0500 Subject: [Numpy-discussion] Keyword added to searchsorted. In-Reply-To: References: Message-ID: Charles R Harris wrote: > Hi all, > > I added the keyword side to the searchsorted method and functions. Thank you! Just the other day, I was wishing that we had such a thing. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wfspotz at sandia.gov Sun Sep 3 00:47:55 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Sat, 2 Sep 2006 22:47:55 -0600 Subject: [Numpy-discussion] (no subject) Message-ID: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> I think I see a bug in lib.user_array.container class. The __eq__ method is def __eq__(self,other): return self._rc(equal(self.array,other)) the expression equal(self.array,other) will return an ndarray of bools, which is then converted, by way of self._rc(), to whatever the derived class is. In my case, this does not make sense, because an array of bools is not a valid argument to the constructor (actually, the data buffer is accepted, but the results are not reliable). What I want, and it seem most people would want in this situation, is just the array of bools; i.e. don't apply the self._rc() method. Assuming there is agreement that this is the desirable behavior, the same would be true for __lt__, __le__, etc. I will probably override this behavior by defining my own __eq__, etc., in my derived class, just for safety. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From robert.kern at gmail.com Sun Sep 3 01:10:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 03 Sep 2006 00:10:47 -0500 Subject: [Numpy-discussion] user_array.container and __eq__ [was: Re: (no subject)] In-Reply-To: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> References: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> Message-ID: Bill Spotz wrote: > I think I see a bug in lib.user_array.container class. The __eq__ > method is > > def __eq__(self,other): return self._rc(equal(self.array,other)) > > the expression equal(self.array,other) will return an ndarray of > bools, which is then converted, by way of self._rc(), to whatever the > derived class is. In my case, this does not make sense, because an > array of bools is not a valid argument to the constructor (actually, > the data buffer is accepted, but the results are not reliable). What > I want, and it seem most people would want in this situation, is just > the array of bools; i.e. don't apply the self._rc() method. No, it's not a bug. It's just the design of that class: always return the same class of object. Of course, since that class only exists to be subclassed, if you need different behavior from those methods, override them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sun Sep 3 14:20:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 12:20:09 -0600 Subject: [Numpy-discussion] diagflat Message-ID: Travis has introduced the function diagflat for creating diagonal arrays. It flattens whatever array is supplied and returns its values as the diagonal of a 2-D array or matrix. As the function is currently undocumented I thought now might be a good time to discuss other possible choices for the name. Here is a quick list that comes to mind diagflat (current name) asdiagonal todiagonal asdiag todiag ... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fernando.Perez at colorado.edu Sun Sep 3 15:15:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun, 03 Sep 2006 13:15:22 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays Message-ID: <44FB29CA.9070808@colorado.edu> Hi all, I'm wondering if the following difference in behavior of object arrays should be considered a bug. Let a and b be: In [21]: a = [0,1] In [22]: b = [ None, None] If we concatenate a with an empty list, it works: In [23]: numpy.concatenate(([],a)) Out[23]: array([0, 1]) But not so for b: In [24]: numpy.concatenate(([],b)) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/fperez/ ValueError: 0-d arrays can't be concatenated This behavior changed recently (it used to work with r2788), and I realize it's probably part of all the reworkings of the object arrays which have been discussed on the list, and all of whose details I have to admit I haven't followed. But this behavior strikes me as a bit inconsistent, since concatenation with a non-empty object array works fine: In [26]: numpy.concatenate(([None],b)) Out[26]: array([None, None, None], dtype=object) This is biting us in some code which keeps object arrays, because when operations of the kind N.concatenate((some_list_of_objects[:nn],other_object_array)) are taken and nn happens to be 0, the code just explodes. In our case, the variable nn is a runtime computed quantity that comes from a numerical algorithm, for which 0 is a perfectly reasonable value. Are we just misusing things and is there a reasonable alternative, or should this be considered a numpy bug? The r2788 behavior was certainly a lot less surprising as far as our code was concerned. I realize that one alternative is to wrap everything into arrays: N.concatenate((N.asarray(some_list_of_objects[:nn]),other_object_array)) Is this the only solution moving forward, or could the previous behavior be restored without breaking other areas of the new code/design? Thanks for any input, f From nwagner at iam.uni-stuttgart.de Sun Sep 3 15:31:53 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 03 Sep 2006 21:31:53 +0200 Subject: [Numpy-discussion] diagflat In-Reply-To: References: Message-ID: On Sun, 3 Sep 2006 12:20:09 -0600 "Charles R Harris" wrote: > Travis has introduced the function diagflat for creating >diagonal arrays. It > flattens whatever array is supplied and returns its >values as the diagonal > of a 2-D array or matrix. As the function is currently >undocumented I > thought now might be a good time to discuss other >possible choices for the > name. Here is a quick list that comes to mind > > diagflat (current name) > asdiagonal > todiagonal > asdiag > todiag > ... > > Chuck +1 for todiag Nils From charlesr.harris at gmail.com Sun Sep 3 16:43:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 14:43:42 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FB29CA.9070808@colorado.edu> References: <44FB29CA.9070808@colorado.edu> Message-ID: On 9/3/06, Fernando Perez wrote: > > Hi all, > > I'm wondering if the following difference in behavior of object arrays > should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated I think it's propably a bug: >>> concatenate((array([]),b)) array([None, None], dtype=object) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Sep 3 17:54:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 03 Sep 2006 16:54:06 -0500 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> Message-ID: Charles R Harris wrote: > On 9/3/06, *Fernando Perez* > wrote: > > Hi all, > > I'm wondering if the following difference in behavior of object > arrays should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > --------------------------------------------------------------------------- > exceptions.ValueError Traceback > (most recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated > > > I think it's propably a bug: > > >>> concatenate((array([]),b)) > array([None, None], dtype=object) Well, if you can fix it without breaking anything else, then it's a bug. However, I would suggest that a rule of thumb for using object arrays is to always be explicit. Never rely on automatic conversion from Python containers to object arrays. Since Python containers are also objects, it is usually ambiguous what the user meant. I kind of liked numarray's choice to move the object array into a separate constructor. I think that gave them some flexibility to choose different syntax and semantics from the generic array() constructor. Since constructing object arrays is so different from constructing numeric arrays, I think that difference is warranted. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From R.Jager at mapperlithography.com Sun Sep 3 19:00:43 2006 From: R.Jager at mapperlithography.com (R.Jager at mapperlithography.com) Date: Mon, 4 Sep 2006 01:00:43 +0200 Subject: [Numpy-discussion] Remco Jager/mapper is out of the office. Message-ID: I will be out of the office starting 09/04/2006 and will not return until 09/25/2006. I will respond to your message when I return. For urgent matters please contact Ton van de Peut: T.vdpeut at mapperlithography.com From haase at msg.ucsf.edu Sun Sep 3 20:03:39 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 03 Sep 2006 17:03:39 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals Message-ID: <44FB6D5B.5090103@msg.ucsf.edu> Hi, I just learn about the existence of round(). Is the following exposing a bug? >>> N.__version__ '1.0b4' >>> N.array([65.0]) [ 65.] >>> _.round(-1) [ 60.] >>> round(65, -1) 70.0 >>> N.array([65]) [65] >>> _.round(-1) [60] >>> N.array([66]) [66] >>> _.round(-1) [60] >>> N.array([66.2]) [ 66.2] >>> _.round(-1) [ 70.] 65 should round *up* to 70. (when decimals is given as -1) In numpy even 66 rounds down, but 66.2 rounds up ... - Sebastian Haase From charlesr.harris at gmail.com Sun Sep 3 20:18:37 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 18:18:37 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FB6D5B.5090103@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> Message-ID: On 9/3/06, Sebastian Haase wrote: > > Hi, > I just learn about the existence of round(). > Is the following exposing a bug? > >>> N.__version__ > '1.0b4' > >>> N.array([65.0]) > [ 65.] > >>> _.round(-1) > [ 60.] > >>> round(65, -1) > 70.0 > >>> N.array([65]) > [65] > >>> _.round(-1) > [60] > >>> N.array([66]) > [66] > >>> _.round(-1) > [60] > >>> N.array([66.2]) > [ 66.2] > >>> _.round(-1) > [ 70.] > > 65 should round *up* to 70. (when decimals is given as -1) > In numpy even 66 rounds down, but 66.2 rounds up ... Numpy rounds to even. >>> around(arange(10)*.5) array([ 0., 0., 1., 2., 2., 2., 3., 4., 4., 4.]) i.e., when at those x.5 boundaries, round to the nearest even number. Always rounding in the same direction biases the numbers when you have gazillions of them. This is most likely of import when the fpu is rounding intermediate results to register length, but numpy does it too. Floats can be weird anyway, .15, for instance, can't be represented exactly as an ieee float. It does tend to throw folks for a loop when they don't keep this in mind. - Sebastian Haase Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 4 00:55:57 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 22:55:57 -0600 Subject: [Numpy-discussion] Random number generators Message-ID: Hi Robert, I am about to get started on some stuff for the random number generators but thought I would run it by you first. I envisage the following: uniform short_doubles -- doubles generated from a single 32 bit random number (advantage: speed) uniform double, short_doubles on the interval (0,1) -- don't touch singularities in functions like log (this is my preferred default) fast_normal -- ziggurat method using single 32 bit random numbers (advantage: speed) fast_exponential -- ziggurat method using single 32 bit random numbers (advantage: speed) MWC8222 random number generator (advantage: speed on some machines, different from mtrand) Except for the last, none conflict with current routines and can be added without a branch. I expect adding MWC8222 might need more extensive work and I will branch for that. So the questions are of utility and naming. I see some utility for myself, otherwise I wouldn't be considering doing the work. OTOH, I already have (C++) routines that I use for these things, so a larger question might be if anyone else sees a use for these. I like speed, but it is not always that important in everyday apps. I see that Pyrex is used for the interface, so I suppose that is one more tool to become familiar with ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Sep 4 01:34:28 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:34:28 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Charles R Harris wrote: > Except for the last, none conflict with current routines and can be added > without a branch. I expect adding MWC8222 might need more extensive work and > I will branch for that. So the questions are of utility and naming. I see > some utility for myself, otherwise I wouldn't be considering doing the work. > OTOH, I already have (C++) routines that I use for these things, so a larger > question might be if anyone else sees a use for these. I like speed, but it > is not always that important in everyday apps. How painful would it be to allow users to drop in their own sources of raw random numbers? A couple of years ago I was trying to analyze some X-ray timing data and we saw some peaks at the end of a long chain of periodicity-detecting software (Fourier transforms, parameter searching, et cetera). To rule out the possibility that they were coming from the random numbers we were using at one stage, I wanted to supply raw random numbers from /dev/random. Unfortunately, that forced me to write my own program to convert uniform random numbers to exponential. Allowing the function that generates raw random bytes to be overwritten would be extremely handy, even if it were painfully slow. In the same project I also noticed it would be nice to be able to (say) do "exponential(2+sin(arange(10)))" to get an array of exponentials with varying parameters. I realize all this is outside the scope of what you were asking. I would have found another pseudorandom number generator that I could just switch to a benefit; the rest of them are less exciting. When you say "32-bit integers" do you really mean 32-bit, or do you mean "machine width"? 32-bit integers may not be faster on 64-bit platforms... A. M. Archibald From haase at msg.ucsf.edu Mon Sep 4 01:42:17 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 03 Sep 2006 22:42:17 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> Message-ID: <44FBBCB9.6050408@msg.ucsf.edu> Charles R Harris wrote: > > > On 9/3/06, *Sebastian Haase* > wrote: > > Hi, > I just learn about the existence of round(). > Is the following exposing a bug? > >>> N.__version__ > '1.0b4' > >>> N.array([65.0]) > [ 65.] > >>> _.round(-1) > [ 60.] > >>> round(65, -1) > 70.0 > >>> N.array([65]) > [65] > >>> _.round(-1) > [60] > >>> N.array([66]) > [66] > >>> _.round(-1) > [60] > >>> N.array([66.2]) > [ 66.2] > >>> _.round(-1) > [ 70.] > > 65 should round *up* to 70. (when decimals is given as -1) > In numpy even 66 rounds down, but 66.2 rounds up ... > > > Numpy rounds to even. > > >>> around(arange(10)*.5) > array([ 0., 0., 1., 2., 2., 2., 3., 4., 4., 4.]) > > i.e., when at those x.5 boundaries, round to the nearest even number. > Always rounding in the same direction biases the numbers when you have > gazillions of them. This is most likely of import when the fpu is > rounding intermediate results to register length, but numpy does it too. > Floats can be weird anyway, .15, for instance, can't be represented > exactly as an ieee float. It does tend to throw folks for a loop when > they don't keep this in mind. > > - Sebastian Haase > > > Chuck Thanks for the reply - but read what the doc says: >>> N.around.__doc__ 'Round 'a' to the given number of decimal places. Rounding behaviour is equivalent to Python. Return 'a' if the array is not floating point. Round both the real and imaginary parts separately if the array is complex. ' it is *not* done this way in Python: >>> round(.5) 1.0 >>> round(1.5) 2.0 >>> round(2.5) 3.0 ( the round obj. method is missing this doc string ) I really think we should stick to what the doc string say - everybody expects x.5 to round up !! look even at this: >>> repr(2.05) '2.0499999999999998' >>> round(2.05, 1) 2.1 (don't ask me how Python does this, but it's "nice" ) Thanks, Sebastian From peridot.faceted at gmail.com Mon Sep 4 01:43:10 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:43:10 -0400 Subject: [Numpy-discussion] diagflat In-Reply-To: References: Message-ID: On 03/09/06, Charles R Harris wrote: > Travis has introduced the function diagflat for creating diagonal arrays. It > flattens whatever array is supplied and returns its values as the diagonal > of a 2-D array or matrix. As the function is currently undocumented I > thought now might be a good time to discuss other possible choices for the > name. Here is a quick list that comes to mind > > diagflat (current name) > asdiagonal > todiagonal > asdiag > todiag > ... I was wishing for that just the other day... I think my vote is for "diagflat", to emphasize the initial flattening. In particular, there's another function I could wish for: given an array, pick an axis, treat the array like an array of vectors along that axis, and replace each vector with the diagonal matrix. So, supposing that it was called todiagonal: >>> A = todiagonal(ones(2,3,4),axis=1) >>> shape(A) (2,3,3,4) >>> A[1,:,:,2] array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) Specifically I find the forcible flattening a bit monstrous. And you can always just do todiagonal(A.flat). No such function seems to exist at the moment, so I'm attaching one; probably not as efficient as it could be with appropriate fancy indexing wizardry. A. M. Archibald -------------- next part -------------- A non-text attachment was scrubbed... Name: diag.py Type: text/x-python Size: 841 bytes Desc: not available URL: From robert.kern at gmail.com Mon Sep 4 01:47:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 00:47:11 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: Charles R Harris wrote: > Hi Robert, > > I am about to get started on some stuff for the random number generators > but thought I would run it by you first. I envisage the following: > > uniform short_doubles -- doubles generated from a single 32 bit random > number (advantage: speed) > uniform double, short_doubles on the interval (0,1) -- don't touch > singularities in functions like log (this is my preferred default) > fast_normal -- ziggurat method using single 32 bit random numbers > (advantage: speed) > fast_exponential -- ziggurat method using single 32 bit random numbers > (advantage: speed) > MWC8222 random number generator (advantage: speed on some machines, > different from mtrand) > > Except for the last, none conflict with current routines and can be > added without a branch. I expect adding MWC8222 might need more > extensive work and I will branch for that. So the questions are of > utility and naming. I see some utility for myself, otherwise I wouldn't > be considering doing the work. OTOH, I already have (C++) routines that > I use for these things, so a larger question might be if anyone else > sees a use for these. I like speed, but it is not always that important > in everyday apps. I would prefer not to expand the API of numpy.random. If it weren't necessary for numpy to provide all of the capabilities that came with Numeric's RandomArray, I wouldn't want numpy.random in there at all. Now, a very productive course of action would be to refactor numpy.random such that the distributions (the first four items on your list fall into this category) and the underlying PRNG (the fifth) are separated from one another such that they can be mixed and matched at runtime. A byproduct of this would expose the C API of both of these in order to be usable by other C extension modules, something that's been asked for about a dozen times now. The five items on your list could be implemented in an extension module distributed in scipy. > I see that Pyrex is used for the interface, so I suppose that is one > more tool to become familiar with ;) Possibly not. Pyrex was getting in the way of exposing a C API the last time I took a stab at it. A possibility that just occurred to me is to make an extension module that *only* exposes the C API and mtrand could be rewritten to use that API. Hmmm. I like that. I can give some guidance about how to proceed and help you navigate the current code, but I'm afraid I don't have much time to actually code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 01:53:40 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:53:40 -0400 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FBBCB9.6050408@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: On 04/09/06, Sebastian Haase wrote: > Thanks for the reply - but read what the doc says: > >>> N.around.__doc__ > 'Round 'a' to the given number of decimal places. Rounding > behaviour is equivalent to Python. > > Return 'a' if the array is not floating point. Round both the real > and imaginary parts separately if the array is complex. > ' > > it is *not* done this way in Python: > >>> round(.5) > 1.0 > >>> round(1.5) > 2.0 > >>> round(2.5) > 3.0 > > ( the round obj. method is missing this doc string ) Um, well, the doc is wrong. Numpy should *not* always follow python's lead, and in partciular it's explicit that Numeric's floating point is closer to IEEE floating-point behaviour than python's - compare, for example 1./0. and array(1.)/0. > I really think we should stick to what the doc string say - everybody > expects x.5 to round up !! Not everybody. This (admittedly odd) behaviour wasn't decided on because it was more efficient to implement, it was decided on because a group of very smart numerical analysts agreed that it was the best way to avoid surprising naive users with biased results. (Non-naive users can normally pick from any of several other rounding modes if they want.) A better question to ask is, "Can I change numpy's rounding behaviour for my programs?" (And, more generally, "can I set all the various floating-point options that the IEEE standard and my processor both support?") I don't know the answer to that one, but it does seem to be a goal that numpy is trying for (hence, for example, the difference between numpy float scalars and python floats). A. M. Archibald From robert.kern at gmail.com Mon Sep 4 01:51:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 00:51:19 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: A. M. Archibald wrote: > In the same project I also noticed it would be nice to be able to > (say) do "exponential(2+sin(arange(10)))" to get an array of > exponentials with varying parameters. Travis O. recently added this capability. The distribution parameters (loc, scale, alpha, whatever) are broadcast against each other using the same rules as ufunc parameters. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 02:00:56 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 02:00:56 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Robert Kern wrote: > A. M. Archibald wrote: > > > In the same project I also noticed it would be nice to be able to > > (say) do "exponential(2+sin(arange(10)))" to get an array of > > exponentials with varying parameters. > > Travis O. recently added this capability. The distribution parameters (loc, > scale, alpha, whatever) are broadcast against each other using the same rules as > ufunc parameters. Wonderful! Thank you, Travis O. A. M. Archibald From charlesr.harris at gmail.com Mon Sep 4 02:42:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 00:42:23 -0600 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 9/3/06, Robert Kern wrote: > > Charles R Harris wrote: > > Hi Robert, > > > > I am about to get started on some stuff for the random number generators > > but thought I would run it by you first. I envisage the following: > > > > uniform short_doubles -- doubles generated from a single 32 bit random > > number (advantage: speed) > > uniform double, short_doubles on the interval (0,1) -- don't touch > > singularities in functions like log (this is my preferred default) > > fast_normal -- ziggurat method using single 32 bit random numbers > > (advantage: speed) > > fast_exponential -- ziggurat method using single 32 bit random numbers > > (advantage: speed) > > MWC8222 random number generator (advantage: speed on some machines, > > different from mtrand) > > > > Except for the last, none conflict with current routines and can be > > added without a branch. I expect adding MWC8222 might need more > > extensive work and I will branch for that. So the questions are of > > utility and naming. I see some utility for myself, otherwise I wouldn't > > be considering doing the work. OTOH, I already have (C++) routines that > > I use for these things, so a larger question might be if anyone else > > sees a use for these. I like speed, but it is not always that important > > in everyday apps. > > I would prefer not to expand the API of numpy.random. If it weren't > necessary > for numpy to provide all of the capabilities that came with Numeric's > RandomArray, I wouldn't want numpy.random in there at all. Yes, good point. Now, a very productive course of action would be to refactor numpy.randomsuch > that the distributions (the first four items on your list fall into this > category) and the underlying PRNG (the fifth) are separated from one > another > such that they can be mixed and matched at runtime. A byproduct of this > would > expose the C API of both of these in order to be usable by other C > extension > modules, something that's been asked for about a dozen times now. The five > items > on your list could be implemented in an extension module distributed in > scipy. What sort of api should this be? It occurs to me that there are already 4 sources of random bytes: Initialization: /dev/random (pseudo random, I think) /dev/urandom crypto system on windows Pseudo random generators: mtrand I suppose we could add some cryptologically secure source as well. That indicates to me that one set of random number generators would just be streams of random bytes, possibly in 4 byte chunks. If I were doing this for linux these would all look like file systems, FUSE comes to mind. Another set of functions would transform these into the different distributions. So, how much should stay in numpy? What sort of API are folks asking for? > I see that Pyrex is used for the interface, so I suppose that is one > > more tool to become familiar with ;) > > Possibly not. Pyrex was getting in the way of exposing a C API the last > time I > took a stab at it. A possibility that just occurred to me is to make an > extension module that *only* exposes the C API and mtrand could be > rewritten to > use that API. Hmmm. I like that. Good, I can do without pyrex. I can give some guidance about how to proceed and help you navigate the > current > code, but I'm afraid I don't have much time to actually code. Thanks, that is all I ask. -- > Robert Kern Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 4 04:31:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 03:31:02 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: Charles R Harris wrote: > What sort of api should this be? It occurs to me that there are already > 4 sources of random bytes: > > Initialization: > > /dev/random (pseudo random, I think) > /dev/urandom > crypto system on windows > > Pseudo random generators: > > mtrand > > I suppose we could add some cryptologically secure source as well. That > indicates to me that one set of random number generators would just be > streams of random bytes, possibly in 4 byte chunks. If I were doing this > for linux these would all look like file systems, FUSE comes to mind. > Another set of functions would transform these into the different > distributions. So, how much should stay in numpy? What sort of API are > folks asking for? It would be good to also support quasi-random generators like Sobol and Halton sequences. They tend to only generate floating point numbers in [0, 1] (or (0, 1) or [0, 1) or (0, 1]; I think it varies). There are probably other PRNGs that only output floating point numbers, but I doubt we want to implement any of them. You can look at the 1.6 version of RandomKit for an implementation of Sobol sequences and ISAAC, a cryptographic PRNG. http://www.jeannot.org/~js/code/index.en.html I'm thinking that the core struct that we're going to pass around will look something like this: struct random_state { void* state; unsigned int (*gen_uint32)(struct random_state*); double (*gen_double)(struct random_state*); } state -- A pointer to the generator-specific struct. gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. Possibly NULL if the generator can not implement such a generator. gen_double -- A function pointer that yields a double-precision number in [0, 1] (possibly omitting one or both of the endpoints depending on the implementation). Possibly this struct needs to be expanded to include function pointers for destroying the state struct and for a copying the state to a new object. The seeding function(s) probably don't need to travel in the struct. We should determine if C++ will work for us, here. Unfortunately, that will force all extension modules that want to use this API to be C++ modules. One other feature of the current implementation of state struct is that a few of the distributions cache information between calls. Notably, the Box-M?ller Gaussian distribution algorithm generates two normal deviates at a time; one is returned, the other is cached until the next call. The binomial distribution computes some intermediate values that are reused if the next call has the same parameters. We should probably store such things separately from the core state struct this time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 10:34:36 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 10:34:36 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Robert Kern wrote: > Charles R Harris wrote: > > > What sort of api should this be? It occurs to me that there are already > > 4 sources of random bytes: > > > > Initialization: > > > > /dev/random (pseudo random, I think) /dev/random is (supposed to be) true randomness derived from various sources. It's also fed through a bunch of hashing and CRC functions chosen for convenience, but is probably pretty unbiased. > > /dev/urandom > > crypto system on windows > > > > Pseudo random generators: > > > > mtrand > > > > I suppose we could add some cryptologically secure source as well. That > > indicates to me that one set of random number generators would just be > > streams of random bytes, possibly in 4 byte chunks. If I were doing this > > for linux these would all look like file systems, FUSE comes to mind. > > Another set of functions would transform these into the different > > distributions. So, how much should stay in numpy? What sort of API are > > folks asking for? "Cryptographically secure random number generator" means something different to everybody, and implementing one that everyone is happy with is going to be a colossal pain. Look at Yarrow and its design documents for some idea of the complexities involved. However, a random number generator based on a stream cipher is probably going to be pretty good, if slow, so implementingone if those can't hurt. But I think leaving the possibility open that one could use custom sources of random bytes is a very good idea. There's no particular need that custom ones should be fast, as they're for use when you really want *good* random numbers. > It would be good to also support quasi-random generators like Sobol and Halton > sequences. They tend to only generate floating point numbers in [0, 1] (or (0, > 1) or [0, 1) or (0, 1]; I think it varies). There are probably other PRNGs that > only output floating point numbers, but I doubt we want to implement any of > them. You can look at the 1.6 version of RandomKit for an implementation of > Sobol sequences and ISAAC, a cryptographic PRNG. > > http://www.jeannot.org/~js/code/index.en.html Sobol' sequences may require special care in constructing non-uniform distributions in order to preserve their uniformity properties. If we can simply import ISAAC, it won't hurt, but I'd rather trust (say) AES in counter mode than some custom cryptosystem that hasn't been analyzed as thoroughly. > I'm thinking that the core struct that we're going to pass around will look > something like this: > > struct random_state { > void* state; > unsigned int (*gen_uint32)(struct random_state*); > double (*gen_double)(struct random_state*); > } > > state -- A pointer to the generator-specific struct. > > gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. Possibly > NULL if the generator can not implement such a generator. > > gen_double -- A function pointer that yields a double-precision number in [0, 1] > (possibly omitting one or both of the endpoints depending on the implementation). Are 32-bit numbers really the right least common denominator? There are plenty of 64-bit platforms out there... Given this API, implementing a subclassable class that exports it should satisfy most people's needs for interchangeable generators. A. M. Archibald From pjssilva at ime.usp.br Mon Sep 4 11:08:28 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 12:08:28 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FBBCB9.6050408@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: <1157382509.3087.14.camel@localhost.localdomain> Interesting, I was just reading about the round rule in IEEE standard last Friday. What numpy's "around" function does is called "round to even" (round is take to make the next digit even), instead of "round up". According to "What every computer scientist should know about floating-point arithmetic" (do a Google search), the reason to prefer "round to even" is exemplified by the following result from Reiser and Knuth: Theorem Let x and y be floating-point numbers, and define x0 = x, x1 = (x0 - y) + y, ..., xn = (xn-1 - y) + y. If + and - are exactly rounded using round to even, then either xn = x for all n or xn = x1 for all n >= 1. If you use "round up" the sequence xn can start increasing slowly "for ever", and as the paper says: "...This example suggests that when using the round up rule, computations can gradually drift upward, whereas when using round to even the theorem says this cannot happen." Best, Paulo From charlesr.harris at gmail.com Mon Sep 4 11:30:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 09:30:04 -0600 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 9/4/06, A. M. Archibald wrote: > > On 04/09/06, Robert Kern wrote: > > Charles R Harris wrote: > > > > > What sort of api should this be? It occurs to me that there are > already > > > 4 sources of random bytes: > > > > > > Initialization: > > > > > > /dev/random (pseudo random, I think) > > /dev/random is (supposed to be) true randomness derived from various > sources. It's also fed through a bunch of hashing and CRC functions > chosen for convenience, but is probably pretty unbiased. > > > > /dev/urandom > > > crypto system on windows > > > > > > Pseudo random generators: > > > > > > mtrand > > > > > > I suppose we could add some cryptologically secure source as well. > That > > > indicates to me that one set of random number generators would just be > > > streams of random bytes, possibly in 4 byte chunks. If I were doing > this > > > for linux these would all look like file systems, FUSE comes to mind. > > > Another set of functions would transform these into the different > > > distributions. So, how much should stay in numpy? What sort of API are > > > folks asking for? > > "Cryptographically secure random number generator" means something > different to everybody, and implementing one that everyone is happy > with is going to be a colossal pain. Look at Yarrow and its design > documents for some idea of the complexities involved. > > However, a random number generator based on a stream cipher is > probably going to be pretty good, if slow, so implementingone if those > can't hurt. But I think leaving the possibility open that one could > use custom sources of random bytes is a very good idea. There's no > particular need that custom ones should be fast, as they're for use > when you really want *good* random numbers. I was thinking it might be nice to have one, just to generate seeds if nothing else. This could actually be written in python and use something like the python cryptography toolkit, no need to reinvent the wheel. It might be worth looking at that code just to see how someone else thought through the interface. It's under the Python license, so I need to know if that license is compatible with the BSD license used for numpy. > It would be good to also support quasi-random generators like Sobol and > Halton > > sequences. They tend to only generate floating point numbers in [0, 1] > (or (0, > > 1) or [0, 1) or (0, 1]; I think it varies). There are probably other > PRNGs that > > only output floating point numbers, but I doubt we want to implement any > of > > them. You can look at the 1.6 version of RandomKit for an implementation > of > > Sobol sequences and ISAAC, a cryptographic PRNG. > > > > http://www.jeannot.org/~js/code/index.en.html > > Sobol' sequences may require special care in constructing non-uniform > distributions in order to preserve their uniformity properties. If we > can simply import ISAAC, it won't hurt, but I'd rather trust (say) AES > in counter mode than some custom cryptosystem that hasn't been > analyzed as thoroughly. > > > I'm thinking that the core struct that we're going to pass around will > look > > something like this: > > > > struct random_state { > > void* state; > > unsigned int (*gen_uint32)(struct random_state*); > > double (*gen_double)(struct random_state*); > > } > > > > state -- A pointer to the generator-specific struct. > > > > gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. > Possibly > > NULL if the generator can not implement such a generator. > > > > gen_double -- A function pointer that yields a double-precision number > in [0, 1] > > (possibly omitting one or both of the endpoints depending on the > implementation). > > Are 32-bit numbers really the right least common denominator? There > are plenty of 64-bit platforms out there... MWC8222 runs faster (2x) on some 64 bit platforms because it multiplies two 32 bit numbers and uses the 64 bit product. The mtrand generator really uses a polynomial with coefficients in z_2 and has 624 words worth, an even number, so could probably be adapted to run directly with 64 bit registers although the gain might not be much. Most other generators I know of produce 32 bit numbers. Not that I am an expert on the subject. Given this API, implementing a subclassable class that exports it > should satisfy most people's needs for interchangeable generators. A. M. Archibald Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 4 12:32:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 10:32:52 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <1157382509.3087.14.camel@localhost.localdomain> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> Message-ID: On 9/4/06, Paulo J. S. Silva wrote: > > Interesting, I was just reading about the round rule in IEEE standard > last Friday. > > What numpy's "around" function does is called "round to even" (round is > take to make the next digit even), instead of "round up". According to > "What every computer scientist should know about floating-point > arithmetic" (do a Google search), the reason to prefer "round to even" > is exemplified by the following result from Reiser and Knuth: > > Theorem > > Let x and y be floating-point numbers, and define x0 = x, x1 = (x0 - y) > + y, ..., xn = (xn-1 - y) + y. If + and - are exactly rounded using > round to even, then either xn = x for all n or xn = x1 for all n >= 1. > > If you use "round up" the sequence xn can start increasing slowly "for > ever", and as the paper says: > > "...This example suggests that when using the round up rule, > computations can gradually drift upward, whereas when using round to > even the theorem says this cannot happen." > > Best, > > Paulo However, around does set the sign bit when rounding -.5 to zero: >>> around(-.5).tostring() '\x00\x00\x00\x00\x00\x00\x00\x80' so that >>> around(-.5) -0.0 on the other hand >>> around(-.5) == 0.0 True I didn't know -0.0 was part of the IEEE spec. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Mon Sep 4 13:07:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 14:07:05 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> Message-ID: <1157389625.3087.19.camel@localhost.localdomain> Once again, the information that singed zero is part of IEEE standard is in the paper I cited in my last message. It is very important to be able to compute the sign of an overflowed quantity in expressions like 1/x when x goes to zero. Best, Paulo From peridot.faceted at gmail.com Mon Sep 4 14:17:23 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 14:17:23 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Charles R Harris wrote: > > > > On 9/4/06, A. M. Archibald wrote: > > On 04/09/06, Robert Kern wrote: > > > Charles R Harris wrote: > > However, a random number generator based on a stream cipher is > > probably going to be pretty good, if slow, so implementingone if those > > can't hurt. But I think leaving the possibility open that one could > > use custom sources of random bytes is a very good idea. There's no > > particular need that custom ones should be fast, as they're for use > > when you really want *good* random numbers. > > > I was thinking it might be nice to have one, just to generate seeds if > nothing else. This could actually be written in python and use something > like the python cryptography toolkit, no need to reinvent the wheel. It > might be worth looking at that code just to see how someone else thought > through the interface. It's under the Python license, so I need to know if > that license is compatible with the BSD license used for numpy. Of the generators they have, only their random pool is of any use for seeding, and even that needs some clever feeding of random data. However, on practically any platofrm this will run on, random bytes based on real entropy can be extracted from the system (/dev/random, windows crypto API, something on the Mac); a uniform API for those would be a handy thing to have. But I can't recommend spending much effort connecting the python crypto stuff to numpy's random number generators; it's the sort of thing application writers can easily cook up by subclassing RandomGenerator (or whatever). A. M. Archibald From haase at msg.ucsf.edu Mon Sep 4 15:03:51 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 04 Sep 2006 12:03:51 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <1157389625.3087.19.camel@localhost.localdomain> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> Message-ID: <44FC7897.5010005@msg.ucsf.edu> Paulo J. S. Silva wrote: > Once again, the information that singed zero is part of IEEE standard is > in the paper I cited in my last message. > > It is very important to be able to compute the sign of an overflowed > quantity in expressions like 1/x when x goes to zero. > > Best, > > Paulo Hi, This is all very interesting ( and confusing (to me, maybe others) at the same time ...) ... Question 0: Are you sure this is not a bug ? >>> N.array([66]).round(-1) [60] >>> N.array([66.2]).round(-1) [ 70.] Question 1: Was this way of round already in Numeric and /or in numarray ? Question 2: Does this need to be better documented (complete and corrected(!) docstrings - maybe a dedicated wiki page ) !? This is related to "How does Matlab or IDL or others do rounding ?" - This would at least determine how many people would be surprised by this. (I would say that *if* Matlab rounds the same way, we might need less documentation ...) Thanks, Sebastian Haase From fperez.net at gmail.com Mon Sep 4 15:18:45 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 4 Sep 2006 13:18:45 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> Message-ID: On 9/3/06, Robert Kern wrote: > > I think it's propably a bug: > > > > >>> concatenate((array([]),b)) > > array([None, None], dtype=object) > > Well, if you can fix it without breaking anything else, then it's a bug. > > However, I would suggest that a rule of thumb for using object arrays is to > always be explicit. Never rely on automatic conversion from Python containers to > object arrays. Since Python containers are also objects, it is usually ambiguous > what the user meant. > > I kind of liked numarray's choice to move the object array into a separate > constructor. I think that gave them some flexibility to choose different syntax > and semantics from the generic array() constructor. Since constructing object > arrays is so different from constructing numeric arrays, I think that difference > is warranted. This is something that should probably be sorted out before 1.0 is out. IMHO, the current behavior is a bit too full of subtle pitfalls to be a good long-term solution, and I think that predictability trumps convenience for a good API. I know that N.array() is already very complex; perhaps the idea of moving all object array construction into a separate function would be a long-term win. I think that object arrays are actually very important for numpy: they provide the bridge between pure numerical, Fortran-like computing and the richer world of Python datatypes and complex objects. But it's important to acknowledge this bridge character: they connect into a world where the basic assumptions of homogeneity of numpy arrays don't apply anymore. I'd be +1 on forcing this acknowledgement by having a separate N.oarray constructor, accessible via the dtype flag to N.array as well, but /without/ N.array trying to invoke it automatically by guessing the contents of what it was fed. The downside of this approach is that much of the code that 'magically' just works with N.array(foo) today, would now break. I'm becoming almost of the opinion that the code is broken already, it just hasn't failed yet :) Over time, I've become more and more paranoid of what constructors (and factory-type functions that build objects) do with their inputs, how strongly they validate them, and how explicit they require their users to be about their intent. While I'm a big fan of Python's duck-typing, I've also learned (the hard way) that in larger codebases, the place to be very strict about input validation is object constructors. It's easy to let garbage seep into an object by not validating a constructor input, and to later (often MUCH later) have your code bizarrely explode with an unrecognizable exception, because that little piece of garbage was fed to some third-party code which was expecting something else. If I've understood history correctly, some of the motivations behind Enthought's Traits are of a similar nature. For now I've dealt with our private problem that spurred my original posting. But I think this issue is worth clarifying for numpy, before 1.0 paints us into a backwards-compatibility corner with a fairly fundamental datatype and constructor. Regards, f From charlesr.harris at gmail.com Mon Sep 4 15:40:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 13:40:01 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FC7897.5010005@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: On 9/4/06, Sebastian Haase wrote: > > Paulo J. S. Silva wrote: > > Once again, the information that singed zero is part of IEEE standard is > > in the paper I cited in my last message. > > > > It is very important to be able to compute the sign of an overflowed > > quantity in expressions like 1/x when x goes to zero. > > > > Best, > > > > Paulo > Hi, > > This is all very interesting ( and confusing (to me, maybe others) at > the same time ...) ... > > Question 0: Are you sure this is not a bug ? > >>> N.array([66]).round(-1) > [60] That does look like a bug. >>> array([66]).round(-1) array([60]) >>> array([66.]).round(-1) array([ 70.]) I suspect it is related to the integer data type of the first example. The code probably does something like this: round(66/10)*10 and 66/10 == 6 in integer arithmetic. Chuck >>> N.array([66.2]).round(-1) > [ 70.] > > Question 1: Was this way of round already in Numeric and /or in numarray ? Don't recall. Question 2: Does this need to be better documented (complete and > corrected(!) docstrings - maybe a dedicated wiki page ) !? > This is related to "How does Matlab or IDL or others do rounding ?" - > This would at least determine how many people would be surprised by this. > (I would say that *if* Matlab rounds the same way, we might need less > documentation ...) I have improved the document strings and the example at scipy.org. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Mon Sep 4 15:41:16 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 16:41:16 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FC7897.5010005@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: <1157398876.3087.35.camel@localhost.localdomain> Ops, I believe you were caught by int versus float here: In [16]:around(66.0, decimals=-1) Out[16]:70.0 In [17]:around(66, decimals=-1) Out[17]:60 Note that in the int case, it seems like round is simply truncation. Paulo From cookedm at physics.mcmaster.ca Mon Sep 4 16:29:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 4 Sep 2006 16:29:06 -0400 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: <20060904162906.7ac14de9@arbutus.physics.mcmaster.ca> On Mon, 4 Sep 2006 01:53:40 -0400 "A. M. Archibald" wrote: > > A better question to ask is, "Can I change numpy's rounding behaviour > for my programs?" (And, more generally, "can I set all the various > floating-point options that the IEEE standard and my processor both > support?") I don't know the answer to that one, but it does seem to be > a goal that numpy is trying for (hence, for example, the difference > between numpy float scalars and python floats). (looks) I think we've missed rounding. And the inexact flag. You can control how overflow, underflow, divide-by-0, and invalid-operand are handled using geterr and seterr, though. Unfortunately, getting/setting the rounding mode is hard to do in a platform-independent way :( gcc has fegetround() and fesetround() (they're part of the C99 standard, I believe). I don't know what to use on other machines (esp. Windows with MSVC), although the Boost Interval library looks like a good place to start: http://boost.cvs.sourceforge.net/boost/boost/boost/numeric/interval/detail/ -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rw679aq02 at sneakemail.com Mon Sep 4 17:27:51 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 14:27:51 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1156926065.27232.269739888@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> Message-ID: <1157405271.21974.270111020@webmail.messagingengine.com> Forgive me if I missed any replies. Since I have seen none, I will rephrase the request. Please demonstrate an irregular array in numpy with time complexity measurement. The shape does not matter so long as it is non-rectangular and includes a complexity measurement. A sparse matrix is conceptually rectangular, so it does not fit the request at all. The question is whether numpy has such support; if not, is it planned. The question is not about the general-purpose Python language. (Although Python demonstrations are of interest if they include the time complexity measure from first principles or experiment.) From charlesr.harris at gmail.com Mon Sep 4 17:53:25 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 15:53:25 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: On 9/4/06, Charles R Harris wrote: > > > > On 9/4/06, Sebastian Haase wrote: > > > > Paulo J. S. Silva wrote: > > > Once again, the information that singed zero is part of IEEE standard > > is > > > in the paper I cited in my last message. > > > > > > It is very important to be able to compute the sign of an overflowed > > > quantity in expressions like 1/x when x goes to zero. > > > > > > Best, > > > > > > Paulo > > Hi, > > > > This is all very interesting ( and confusing (to me, maybe others) at > > the same time ...) ... > > > > Question 0: Are you sure this is not a bug ? > > >>> N.array([66]).round(-1) > > [60] > > > That does look like a bug. > > >>> array([66]).round(-1) > array([60]) > >>> array([66.]).round(-1) > array([ 70.]) > > I suspect it is related to the integer data type of the first example. The > code probably does something like this: > > round(66/10)*10 > > and 66/10 == 6 in integer arithmetic. > The problem is somewhere in this bit of code: // f will be a double f = PyFloat_FromDouble(power_of_ten(decimals)); if (f==NULL) return NULL; // op1 will be division ret = PyObject_CallFunction(op1, "OOO", a, f, out); if (ret==NULL) goto finish; // round in place. tmp = PyObject_CallFunction(n_ops.rint, "OO", ret, ret); if (tmp == NULL) {Py_DECREF(ret); ret=NULL; goto finish;} I suspect the problem is the division, which has integer 'a', double 'f', and integer 'out'. Integer out is probably the killer. Let's see: # use doubles for output >>> out = zeros(1) >>> array([66]).round(decimals=-1, out=out) array([ 70.]) yep, that's the problem Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 4 17:58:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 16:58:29 -0500 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157405271.21974.270111020@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: rw679aq02 at sneakemail.com wrote: > Forgive me if I missed any replies. Since I have seen none, I will > rephrase the request. > > Please demonstrate an irregular array in numpy with time complexity > measurement. The shape does not matter so long as it is non-rectangular > and includes a complexity measurement. A sparse matrix is conceptually > rectangular, so it does not fit the request at all. > > The question is whether numpy has such support; if not, is it planned. No, and no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Mon Sep 4 18:34:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 04 Sep 2006 16:34:08 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FB29CA.9070808@colorado.edu> References: <44FB29CA.9070808@colorado.edu> Message-ID: <44FCA9E0.4040908@ieee.org> Fernando Perez wrote: > Hi all, > > I'm wondering if the following difference in behavior of object arrays should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated > This is a result of PyArray_FromAny changing when object arrays are explicitly requested (which they are in this case --- although behind the scenes). I decided to revert to the previous behavior and only use the Object_FromNestedLists code when an error occurs and the user explicitly requested an object array. The downside is that you can not place empty lists (or tuples) as objects in an object-array construct. as you could before. Given the trouble people had with the "feature," it seems wise to use it only when previous code would have raised an error. -Travis From rw679aq02 at sneakemail.com Mon Sep 4 18:46:57 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 15:46:57 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: <1157410017.28372.270115201@webmail.messagingengine.com> >> The question is whether numpy has such support; if not, is it planned. > No, and no. Thank you for answering, and I am sorry to hear that. I will be dropping my membership on the scipy-numpy email list shortly. Many systems handle rectangular arrays quite well already, and are more fully developed. It is a common fallacy rectangles are the only shapes one ever needs. Physical geometry is only rarely rectangular, and solution of actual physical problems, and even number-theoretical problem, is a far larger problem domain. From charlesr.harris at gmail.com Mon Sep 4 19:35:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 17:35:04 -0600 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157410017.28372.270115201@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157410017.28372.270115201@webmail.messagingengine.com> Message-ID: On 9/4/06, rw679aq02 at sneakemail.com wrote: > > >> The question is whether numpy has such support; if not, is it planned. > > > No, and no. > > Thank you for answering, and I am sorry to hear that. > > I will be dropping my membership on the scipy-numpy email list shortly. > Many systems handle rectangular arrays quite well already, and are more > fully developed. > > It is a common fallacy rectangles are the only shapes one ever needs. > Physical geometry is only rarely rectangular, and solution of actual > physical problems, and even number-theoretical problem, is a far larger > problem domain. Thanks for blessing us with your exalted presence and superior intellect. You will be missed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Sep 4 19:55:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 04 Sep 2006 17:55:01 -0600 Subject: [Numpy-discussion] Going to cut NumPy 1.0b5 tonight Message-ID: <44FCBCD5.3000407@ieee.org> This is a last reminder that I'm going to cut a release of NumPy 1.0b5 tonight. Please have any fixes and/or problems worked out before then. -Travis From cookedm at physics.mcmaster.ca Mon Sep 4 20:30:45 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 4 Sep 2006 20:30:45 -0400 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157410017.28372.270115201@webmail.messagingengine.com> Message-ID: <20060904203045.37f31874@arbutus.physics.mcmaster.ca> On Mon, 4 Sep 2006 17:35:04 -0600 "Charles R Harris" wrote: > On 9/4/06, rw679aq02 at sneakemail.com wrote: > > > > >> The question is whether numpy has such support; if not, is it planned. > > > > > No, and no. > > > > Thank you for answering, and I am sorry to hear that. > > > > I will be dropping my membership on the scipy-numpy email list shortly. > > Many systems handle rectangular arrays quite well already, and are more > > fully developed. > > > > It is a common fallacy rectangles are the only shapes one ever needs. > > Physical geometry is only rarely rectangular, and solution of actual > > physical problems, and even number-theoretical problem, is a far larger > > problem domain. > > > Thanks for blessing us with your exalted presence and superior intellect. > You will be missed. Well now, that's just snarky. rw679aq02 (if that is indeed your real name!), the reason that numpy will not support irregular "arrays" anytime soon comes down to multiple reasons: 1) It would require a complete rework; better to make a new package. Irregular arrays would require an entirely different approach than regular arrays. 2) While most things are not rectangular, the equations that describe them are, for the most part. Finite-element methods, for instance, use a triangulation of the physical object, and the equations can then be cast as very large set of array equations. 3) I would guess that problems that could be described by irregular arrays could be better recast with a different data structure. There's a saying that you can write Fortran in any language; however, that doesn't mean you should! 4) No one else has asked for them, and the developers don't need them (this is how open source works: scratching one's own itch) 5) If, instead, you're looking for efficient memory representations of of sparse matrices (most elements being 0), then there are various ways to do this. I'm not familiar with them, but scipy has packages to handle sparse matrices. A lot of work happens in this field (those finite-element methods tend to make sparse matrices). Note that you could simulate an irregular array (with a maximum size in the dimensions) using the masked arrays provided by NumPy. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rw679aq02 at sneakemail.com Tue Sep 5 00:27:17 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 21:27:17 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157405271.21974.270111020@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: <1157430437.25529.270131342@webmail.messagingengine.com> Exalted presences and superior intellects aside, the point is not hard to get: Motivational examples are everywhere. Think about gridding physical problems expressed in cylindrical or spherical coordinates. The natural slices are not rectangles. You can use rectangular storage but only with O(n^3) waste. More abstract solution spaces of math and physics do not usually lend themselves to rectangular treatments. (I understand finite element techniques and am not referring to those.) Again, rectangular storage is possible only with O(n^d) waste, where commonly d>3. Granted one may overcome these issues with software development effort; that insight begs the question. I am looking for teaching software that already does so. I agree that rectangular storage is easiest for software programmers and hence common. It is not easiest for solving all problems. Students should explore solutiuon spaces in a proper setting. So I just asked what numpy could do in this regard. Now I have the plain answer, and am grateful for it. From cwmoad at gmail.com Tue Sep 5 08:00:26 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Tue, 5 Sep 2006 08:00:26 -0400 Subject: [Numpy-discussion] [matplotlib-devel] Fwd: Going to cut NumPy 1.0b5 tonight In-Reply-To: References: <44FCBCD5.3000407@ieee.org> <6382066a0609041800l504bfbb9t637f6700de9d65a@mail.gmail.com> Message-ID: <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> I also get a compile error when trying to build against the win32-py2.4 release. src\_na_nxutils.c(213) : error C2275: 'PyObject' : illegal use of this type as an expression c:\Python24\include\object.h(104) : see declaration of 'PyObject' src\_na_nxutils.c(213) : error C2065: 'ret' : undeclared identifier src\_na_nxutils.c(215) : warning C4047: 'return' : 'PyObject *' differs in levels of indirection from 'int' Do we need to modify our use of the c-api at all? - Charlie On 9/5/06, Boyd Waters wrote: > Very sorry: here is the initial error: > numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use > in this function) > > > On Sep 5, 2006, at 12:01 AM, Boyd Waters wrote: > > > 1.0b5 > > Fails to compile? > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Matplotlib-devel mailing list > Matplotlib-devel at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > > > > From jdhunter at ace.bsd.uchicago.edu Tue Sep 5 08:04:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue, 05 Sep 2006 07:04:04 -0500 Subject: [Numpy-discussion] [matplotlib-devel] Fwd: Going to cut NumPy 1.0b5 tonight In-Reply-To: <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> ("Charlie Moad"'s message of "Tue, 5 Sep 2006 08:00:26 -0400") References: <44FCBCD5.3000407@ieee.org> <6382066a0609041800l504bfbb9t637f6700de9d65a@mail.gmail.com> <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> Message-ID: <877j0i7c5n.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Charlie" == Charlie Moad writes: Charlie> I also get a compile error when trying to build against Charlie> the win32-py2.4 release. src\_na_nxutils.c(213) : error Charlie> C2275: 'PyObject' : illegal use of this type as an Charlie> expression c:\Python24\include\object.h(104) : see It looks like I made the typical C++ programmer writing C mistake: putting a type declaration in code where it is first used rather than at the beginning of the function. I just committed a change that should fix this. JDH From matthew.brett at gmail.com Tue Sep 5 09:34:58 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 5 Sep 2006 14:34:58 +0100 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FCA9E0.4040908@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> Message-ID: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Hi, > This is a result of PyArray_FromAny changing when object arrays are > explicitly requested (which they are in this case --- although behind > the scenes). Hmm - I think I am hitting a related bug/feature/surprising change in behavior, which is showing up rather obscurely in a failure of the scipy.io matlab loading tests: http://projects.scipy.org/scipy/scipy/ticket/258 Here's the change I wasn't expecting, present with current SVN: a = arange(2) b = arange(1) c = array([a, b], dtype=object) c -> array([[0, 1], [0, 0]], dtype=object) On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: array([[0], [0 1]], dtype=object) Best, Matthew From charlesr.harris at gmail.com Tue Sep 5 12:40:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 10:40:53 -0600 Subject: [Numpy-discussion] order keyword Message-ID: I am trying to write more document strings for numpy and have a question about the order keyword in tostring. The usual allowed values of this keyword are "C", "F", or None, but in the tostring method there is also the value "Any" which has the same effect as None. I wonder if the "Any" shouldn't be removed as None seems to be the preferred form in other methods. Also, the default value of order in the tostring method is "C" and it seems to me that the principal of least surprise requires None as the default so that the order of the array being converted is the default. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 5 12:44:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 10:44:41 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Message-ID: <44FDA979.6080502@ieee.org> Matthew Brett wrote: > Hi, > > >> This is a result of PyArray_FromAny changing when object arrays are >> explicitly requested (which they are in this case --- although behind >> the scenes). >> > > Hmm - I think I am hitting a related bug/feature/surprising change in > behavior, which is showing up rather obscurely in a failure of the > scipy.io matlab loading tests: > > http://projects.scipy.org/scipy/scipy/ticket/258 > > Here's the change I wasn't expecting, present with current SVN: > > a = arange(2) > b = arange(1) > c = array([a, b], dtype=object) > c > -> > array([[0, 1], > [0, 0]], dtype=object) > > On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: > > array([[0], [0 1]], dtype=object) > Grrr.. Object arrays are very hard to get right. I have no idea why this is happening, but I'll look into it. I think it's the bug that led me to put in the special-case object-array handling in the first place. Now, that special-case object-array handling is only done on an error condition, I need to fix this right and raise an inconsistent shape error. It will probably help with the TypeError messages that are currently raised in this situation with other types as well. -Travis From charlesr.harris at gmail.com Tue Sep 5 13:17:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 11:17:09 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: Message-ID: On 9/5/06, Charles R Harris wrote: > > I am trying to write more document strings for numpy and have a question > about the order keyword in tostring. The usual allowed values of this > keyword are "C", "F", or None, but in the tostring method there is also the > value "Any" which has the same effect as None. I wonder if the "Any" > shouldn't be removed as None seems to be the preferred form in other > methods. Also, the default value of order in the tostring method is "C" and > it seems to me that the principal of least surprise requires None as the > default so that the order of the array being converted is the default. > > Chuck > > What is the actual order of data in memory? Is it always row major? In [38]: array([[1,2],[3,4]], dtype=int8, order="F").tofile('test.txt', sep=" ") In [39]: cat test.txt 1 2 3 4 In [40]: array([[1,2],[3,4]], dtype=int8, order="F").tostring(order="A") Out[40]: '\x01\x03\x02\x04' -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 5 13:02:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 11:02:52 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: Message-ID: <44FDADBC.5030906@ieee.org> Charles R Harris wrote: > I am trying to write more document strings for numpy and have a > question about the order keyword in tostring. The usual allowed values > of this keyword are "C", "F", or None, but in the tostring method > there is also the value "Any" which has the same effect as None. I > wonder if the "Any" shouldn't be removed as None seems to be the > preferred form in other methods. I don't think keeping 'Any' as a keyword here is a problem. > Also, the default value of order in the tostring method is "C" and it > seems to me that the principal of least surprise requires None as the > default so that the order of the array being converted is the default. I've thought this through several times. There may be a few cases where 'Any' is appropriate but the user will be expecting 'C' as the default because that was the only possibility for a long time. It's actually very problematic to switch to 'Any' as the default. You end up with lots of surprises as things start behaving very differently then they used to under Numeric once you transpose the array. -Travis From kortmann at ideaworks.com Tue Sep 5 14:04:11 2006 From: kortmann at ideaworks.com (kortmann at ideaworks.com) Date: Tue, 5 Sep 2006 11:04:11 -0700 (PDT) Subject: [Numpy-discussion] (no subject) Message-ID: <2794.12.216.231.149.1157479451.squirrel@webmail.ideaworks.com> Hey guys I posted this on matplotlibs mailing list > hey guys i got the subversion from the site and I am trying to install it > on windows. > > I changed dir into the matplotlib dir that includes the setup.py file. > > run python setup.py install, and im getting a wierd error. i left the > topmost lines along with the error. has anyone seen anything like this > before? > > building 'matplotlib.enthought.traits.ctraits' extension > creating build\temp.win32-2.4\Release\lib > creating build\temp.win32-2.4\Release\lib\matplotlib > creating build\temp.win32-2.4\Release\lib\matplotlib\enthought > creating build\temp.win32-2.4\Release\lib\matplotlib\enthought\traits > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c > /nologo /Ox > /MD /W3 /GX /DNDEBUG -Ic:\Python24\include -Ic:\Python24\PC > /Tclib/matplotlib/e > nthought/traits/ctraits.c > /Fobuild\temp.win32-2.4\Release\lib/matplotlib/enthoug > ht/traits/ctraits.obj > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /DLL > /nologo > /INCREMENTAL:NO /LIBPATH:c:\Python24\libs /LIBPATH:c:\Python24\PCBuild > /EXPORT: > initctraits > build\temp.win32-2.4\Release\lib/matplotlib/enthought/traits/ctraits > .obj /OUT:build\lib.win32-2.4\matplotlib\enthought\traits\ctraits.pyd > /IMPLIB:bu > ild\temp.win32-2.4\Release\lib/matplotlib/enthought/traits\ctraits.lib > building 'matplotlib.backends._tkagg' extension > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c > /nologo /Ox > /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc -Iswig > -Iagg23/incl > ude -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetype2 > -Isrc\freetype2 > -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\freetype2 > -Ic:\Pyt > hon24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp > /Fobuild\temp.win32-2.4\Release > \src/_tkagg.obj > _tkagg.cpp > src\_tkagg.cpp(28) : fatal error C1083: Cannot open include file: 'tk.h': > No suc > h file or directory > error: Command ""C:\Program Files\Microsoft Visual Studio .NET > 2003\Vc7\bin\cl.e > xe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. > -Isrc - > Iswig -Iagg23/include -I. -I. -Iwin32_static/include/tcl84\freetype2 > -I.\freetyp > e2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 > -I.\freetype2 -I.\ > freetype2 -Ic:\Python24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp > /Fobuild\temp > .win32-2.4\Release\src/_tkagg.obj" failed with exit status 2 and recieved this response > > You need to install the tcl/tk headers as Darren mentioned. I just > install ActiveTcl and the build should pick up on it no problem. > And then after that I ran into another error that looks like it might be numpy related. So i am posting this here and on Matplotlibs list. The second error is why i am posting here but since some of you use the latest SVN on windows maybe you could offer more general help of how to compile matplotlib on windows I went ahead and installed Active Tcl I removed microsft visual studio 2003 .net from my computer because I never used it, and I did not want it on here in the first place but i had it on here from being an intern over the summer. C:\matplotlib\trunk\matplotlib>c:\Python24\python.exe setup.py install GTK requires pygtk building tkagg 2 4 Building for python24 GTKAgg requires pygtk running install running build running build_py running build_ext No module named msvccompiler in numpy.distutils, trying from distutils.. building 'matplotlib.backends._tkagg' extension C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc -Iswig -Iagg23/incl ude -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetype2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\freetype2 -Ic:\Pyt hon24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp /Fobuild\temp.win32-2.4\Release \src/_tkagg.obj _tkagg.cpp src\_tkagg.cpp(28) : fatal error C1083: Cannot open include file: 'tk.h': No suc h file or directory error: Command ""C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc - Iswig -Iagg23/include -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetyp e2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\ freetype2 -Ic:\Python24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp /Fobuild\temp .win32-2.4\Release\src/_tkagg.obj" failed with exit status 2 This is the error i got after my first try at python setup.py install after it did not work is when i uninstalled .net 2003. and i recieve this error currently. C:\matplotlib\trunk\matplotlib>c:\Python24\python.exe setup.py install GTK requires pygtk building tkagg 2 4 Building for python24 GTKAgg requires pygtk running install running build running build_py running build_ext No module named msvccompiler in numpy.distutils, trying from distutils.. error: The .NET Framework SDK needs to be installed before building extensions f or Python. Has anyone seen this one? I would not be so persistent as to trying to install the current SVN except that I need one of the algorithms in numpy version 1.0b2 and above. I am also going to post this on the numpy mailing list because it says the error is in numpy.distutils. From oliphant.travis at ieee.org Tue Sep 5 13:05:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 11:05:00 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Message-ID: <44FDAE3C.7000901@ieee.org> Matthew Brett wrote: > Hi, > > >> This is a result of PyArray_FromAny changing when object arrays are >> explicitly requested (which they are in this case --- although behind >> the scenes). >> > > Hmm - I think I am hitting a related bug/feature/surprising change in > behavior, which is showing up rather obscurely in a failure of the > scipy.io matlab loading tests: > > http://projects.scipy.org/scipy/scipy/ticket/258 > > Here's the change I wasn't expecting, present with current SVN: > > a = arange(2) > b = arange(1) > c = array([a, b], dtype=object) > c > -> > array([[0, 1], > [0, 0]], dtype=object) > > On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: > > array([[0], [0 1]], dtype=object) > This should now be fixed. The code was inappropriately not checking for dimensions when object arrays were being constructed. Now, it raises the appropriate error and then interprets it correctly using the extra object creation code. Users of scipy 0.5.1 will only have to upgrade NumPy to get the fix (the SciPy install won't have to be re-built). -Travis From dalcinl at gmail.com Tue Sep 5 14:33:03 2006 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Sep 2006 15:33:03 -0300 Subject: [Numpy-discussion] order keyword In-Reply-To: <44FDADBC.5030906@ieee.org> References: <44FDADBC.5030906@ieee.org> Message-ID: BTW, in numpy-1.0b1 numpy.zeros((3,3), order='QQQ') pass without generating any error... Is this intended behaviour? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From charlesr.harris at gmail.com Tue Sep 5 14:37:19 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 12:37:19 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: <44FDADBC.5030906@ieee.org> References: <44FDADBC.5030906@ieee.org> Message-ID: On 9/5/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > I am trying to write more document strings for numpy and have a > > question about the order keyword in tostring. The usual allowed values > > of this keyword are "C", "F", or None, but in the tostring method > > there is also the value "Any" which has the same effect as None. I > > wonder if the "Any" shouldn't be removed as None seems to be the > > preferred form in other methods. > I don't think keeping 'Any' as a keyword here is a problem. Yeah, I noticed that the PyArray_OrderConverter just replaces None by "A" and is common to all the methods but the default is set in the calling code. > Also, the default value of order in the tostring method is "C" and it > > seems to me that the principal of least surprise requires None as the > > default so that the order of the array being converted is the default. > I've thought this through several times. There may be a few cases where > 'Any' is appropriate but the user will be expecting 'C' as the default > because that was the only possibility for a long time. It's actually > very problematic to switch to 'Any' as the default. You end up with > lots of surprises as things start behaving very differently then they > used to under Numeric once you transpose the array. > > -Travis Ok, I will add a comment to tofile that the data is written out in "C" order. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 5 14:40:50 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 12:40:50 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: <44FDADBC.5030906@ieee.org> Message-ID: On 9/5/06, Lisandro Dalcin wrote: > > BTW, in numpy-1.0b1 > > numpy.zeros((3,3), order='QQQ') > > pass without generating any error... Is this intended behaviour? This has been fixed: In [3]: zeros((3,3), order='QQQ') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/charris/ TypeError: order not understood Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From torgil.svensson at gmail.com Tue Sep 5 15:49:34 2006 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Tue, 5 Sep 2006 21:49:34 +0200 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157430437.25529.270131342@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157430437.25529.270131342@webmail.messagingengine.com> Message-ID: > Think about gridding physical problems expressed in cylindrical or > spherical coordinates. The natural slices are not rectangles. You can > use rectangular storage but only with O(n^3) waste. I don't get this argument. Are you slicing your spherical coordinates with a cartesian coordinate system? That's surely a waste. Many times bases can be chosen so that you even in an array doesn't waste space. I can't see the point attacking numpy with this "irregular" generalisation. Numpy is a tool very good for it's purpose, we don't really want to trash it with features for solving unspecified general problems. Instead, specify a real world problem, and perhaps someone might have a good tip, otherwise look for other tools. Pytables for example can efficiently store varying length arrays. Maybe start on developing an variable length array yourself that integrates perfectly with Numpy where applicable? //Torgil On 9/5/06, rw679aq02 at sneakemail.com wrote: > Exalted presences and superior intellects aside, the point is not hard > to get: Motivational examples are everywhere. > > Think about gridding physical problems expressed in cylindrical or > spherical coordinates. The natural slices are not rectangles. You can > use rectangular storage but only with O(n^3) waste. > > More abstract solution spaces of math and physics do not usually lend > themselves to rectangular treatments. (I understand finite element > techniques and am not referring to those.) Again, rectangular storage > is possible only with O(n^d) waste, where commonly d>3. > > Granted one may overcome these issues with software development effort; > that insight begs the question. I am looking for teaching software that > already does so. > > I agree that rectangular storage is easiest for software programmers and > hence common. It is not easiest for solving all problems. Students > should explore solutiuon spaces in a proper setting. So I just asked > what numpy could do in this regard. Now I have the plain answer, and am > grateful for it. > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From scipy at mspacek.mm.st Tue Sep 5 19:46:20 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Tue, 05 Sep 2006 16:46:20 -0700 Subject: [Numpy-discussion] Keyword added to searchsorted. In-Reply-To: References: Message-ID: <44FE0C4C.80402@mspacek.mm.st> I agree. This'll allow me to delete some messy code I have to get the same behaviour. I'm amazed by how often I use searchsorted. 'side' sounds like a good keyword name to me. Martin Robert Kern wrote: > Charles R Harris wrote: >> Hi all, >> >> I added the keyword side to the searchsorted method and functions. > > Thank you! Just the other day, I was wishing that we had such a thing. > From pgmdevlist at gmail.com Wed Sep 6 03:57:38 2006 From: pgmdevlist at gmail.com (PGM) Date: Wed, 6 Sep 2006 03:57:38 -0400 Subject: [Numpy-discussion] convention in numpy.correlate? Message-ID: <200609060357.38906.pgmdevlist@gmail.com> Folks, I need to compute some cross-correlations between time series, and I naturally started to use numpy.correlate. Howver, I'm not sure about the convention being used: The crosscorrelation is usually defined as $\gamma_{xy}[k] = \sum_{i}{x[i] y[i+k]}$ So, when I compute >>> numpy.dot(x[:-1],y[1:]) on two 1D series of same size $n$, I should have $\gamma[1]$. With numpy.correlate, I have to use >>>numpy.correlate(x,y,'full')[(n-1)-1] or reverse x and y to get $\gamma[1]$ Is that correct ? P. From seguridad at santander.com.mx Wed Sep 6 05:07:50 2006 From: seguridad at santander.com.mx (Santander Serfin) Date: Wed, 6 Sep 2006 04:07:50 -0500 Subject: [Numpy-discussion] Seguridad Santander (Actualizacion) Message-ID: An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Sep 6 05:52:11 2006 From: pgmdevlist at gmail.com (PGM) Date: Wed, 6 Sep 2006 05:52:11 -0400 Subject: [Numpy-discussion] numpy.fftpack: axis=None ? Message-ID: <200609060552.11394.pgmdevlist@gmail.com> Folks, I was playing around the numpy.fftpack when I ran into the problem below: it seems that axis=None is not valid with fft. Is there a reason for that ? I was assuming a behavior similar to other functions, where axis=None translates to "use a flat array". ------------------------------------------------------ >>> N.fft.fft(N.arange(100).reshape(10,10),128,None) /usr/lib64/python2.4/site-packages/numpy/fft/fftpack.py in fft(a, n, axis) 85 different n's.""" 86 ---> 87 return _raw_fft(a, n, axis, fftpack.cffti, fftpack.cfftf, _fft_cache) 88 89 /usr/lib64/python2.4/site-packages/numpy/fft/fftpack.py in _raw_fft(a, n, axis, init_function, work_function, fft_cache) 44 fft_cache[n] = wsave 45 ---> 46 if a.shape[axis] != n: 47 s = list(a.shape) 48 if s[axis] > n: TypeError: tuple indices must be integers ------------------------------------------------- From ryanlists at gmail.com Wed Sep 6 13:18:03 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 12:18:03 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version Message-ID: I am a Linux user trying to install Numpy/Scipy on a Windows machine in my office. I went to the website and grabbed the two latest versions: scipy = scipy-0.5.0.win32-py2.4.exe numpy = numpy-1.0b5.win32-py2.4.exe Trying to import scipy I get: Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy Overwriting info= from scipy.misc.helpmod (was from numpy.lib.utils) Overwriting who= from scipy.misc.common (was from numpy.lib.utils) Overwriting source= from scipy.misc.helpmod (was from numpy.lib.utils) RuntimeError: module compiled against version 1000000 of C-API but this version of numpy is 1000002 Fatal Python error: numpy.core.multiarray failed to import... exiting. I read about this problem on the list and there was mention of needing a different Numpy version. With a slightly older Numpy I got (numpy-0.9.8.win32-py2.4.exe): Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\scipy\__init__.py", line 32, in ? from numpy import oldnumeric ImportError: cannot import name oldnumeric Are there windows installers available for compatible versions of Scipy and Numpy? Thanks, Ryan From scipy at mspacek.mm.st Wed Sep 6 13:23:01 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Wed, 06 Sep 2006 10:23:01 -0700 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: Message-ID: <44FF03F5.60208@mspacek.mm.st> Ryan, Try installing the latest scipy version 0.51. There's a windows binary for it. Worked fine for me. Martin Ryan Krauss wrote: > I am a Linux user trying to install Numpy/Scipy on a Windows machine > in my office. > > I went to the website and grabbed the two latest versions: > scipy = scipy-0.5.0.win32-py2.4.exe > numpy = numpy-1.0b5.win32-py2.4.exe > > From ryanlists at gmail.com Wed Sep 6 13:30:17 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 12:30:17 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF03F5.60208@mspacek.mm.st> References: <44FF03F5.60208@mspacek.mm.st> Message-ID: I would be glad to try that. Where do I get the installer for scipy 0.51? Thanks, Ryan On 9/6/06, Martin Spacek wrote: > Ryan, > > Try installing the latest scipy version 0.51. There's a windows binary > for it. Worked fine for me. > > Martin > > Ryan Krauss wrote: > > I am a Linux user trying to install Numpy/Scipy on a Windows machine > > in my office. > > > > I went to the website and grabbed the two latest versions: > > scipy = scipy-0.5.0.win32-py2.4.exe > > numpy = numpy-1.0b5.win32-py2.4.exe > > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jg307 at cam.ac.uk Wed Sep 6 13:37:17 2006 From: jg307 at cam.ac.uk (James Graham) Date: Wed, 06 Sep 2006 18:37:17 +0100 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> Message-ID: <44FF074D.5070204@cam.ac.uk> Ryan Krauss wrote: > I would be glad to try that. Where do I get the installer for scipy 0.51? http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download The SciPy website needs to be updated. -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead From ryanlists at gmail.com Wed Sep 6 14:17:57 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 13:17:57 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF074D.5070204@cam.ac.uk> References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: O.K. I think I am up and running. I actually installed Entought and Numpy and Scipy worked together there. I then upgraded the scipy and numpy packages. Is there a compatible matplotlib as well? I was o.k. with mpl from enthought until I switched numerix to numpy. That made mpl unhappy. I downloaded 0.87.5 but I broke something in the process because now even switching back to Numeric doesn't make mpl happy. I may switch to the mpl list if this keeps giving me problems. Ryan On 9/6/06, James Graham wrote: > Ryan Krauss wrote: > > I would be glad to try that. Where do I get the installer for scipy 0.51? > > http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download > > The SciPy website needs to be updated. > > -- > "Eternity's a terrible thought. I mean, where's it all going to end?" > -- Tom Stoppard, Rosencrantz and Guildenstern are Dead > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From bhendrix at enthought.com Wed Sep 6 14:33:54 2006 From: bhendrix at enthought.com (bryce hendrix) Date: Wed, 06 Sep 2006 13:33:54 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: <44FF1492.9060604@enthought.com> Ryan, The Enthought 1.0.0 release shipped a pre-release of matplotlib 0.87.3, svn revision 2478. If you update numpy, you have to update to matching versions of scipy and matplotlib. If you are using binary releases, guessing which version of each is needed is difficult unless builders of the binaries specify exactly which version of numpy was used. If you're comfortable building these packages yourself, the simple solution is to download the latest numpy, install it over the existing numpy, get the latest scipy & matplot lib sources, build them and install them over the existing packages. Bryce Ryan Krauss wrote: > O.K. I think I am up and running. I actually installed Entought and > Numpy and Scipy worked together there. I then upgraded the scipy and > numpy packages. > > Is there a compatible matplotlib as well? I was o.k. with mpl from > enthought until I switched numerix to numpy. That made mpl unhappy. > I downloaded 0.87.5 but I broke something in the process because now > even switching back to Numeric doesn't make mpl happy. I may switch > to the mpl list if this keeps giving me problems. > > Ryan > > On 9/6/06, James Graham wrote: > >> Ryan Krauss wrote: >> >>> I would be glad to try that. Where do I get the installer for scipy 0.51? >>> >> http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download >> >> The SciPy website needs to be updated. >> >> -- >> "Eternity's a terrible thought. I mean, where's it all going to end?" >> -- Tom Stoppard, Rosencrantz and Guildenstern are Dead >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Sep 6 16:50:12 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 06 Sep 2006 13:50:12 -0700 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: <44FF3484.2040700@noaa.gov> Ryan Krauss wrote: > Is there a compatible matplotlib as well? I was o.k. with mpl from > enthought until I switched numerix to numpy. That made mpl unhappy. > I downloaded 0.87.5 but I broke something in the process because now > even switching back to Numeric doesn't make mpl happy. There was an error with the first MPL 0.87.5 builds uploaded. See the MPL list for more info, but these builds should work: http://euclid.uits.iupui.edu/mplfiles/ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Sep 6 17:08:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 15:08:08 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FDAE3C.7000901@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> Message-ID: On 9/5/06, Travis Oliphant wrote: > > Matthew Brett wrote: > > Hi, > > > > > >> This is a result of PyArray_FromAny changing when object arrays are > >> explicitly requested (which they are in this case --- although behind > >> the scenes). > >> > > > > Hmm - I think I am hitting a related bug/feature/surprising change in > > behavior, which is showing up rather obscurely in a failure of the > > scipy.io matlab loading tests: > > > > http://projects.scipy.org/scipy/scipy/ticket/258 > > > > Here's the change I wasn't expecting, present with current SVN: > > > > a = arange(2) > > b = arange(1) > > c = array([a, b], dtype=object) > > c > > -> > > array([[0, 1], > > [0, 0]], dtype=object) > > > > On a previous version of numpy (1.02b.dev2975) I get the answer I was > expecting: > > > > array([[0], [0 1]], dtype=object) > > > > This should now be fixed. The code was inappropriately not checking for > dimensions when object arrays were being constructed. Now, it raises > the appropriate error and then interprets it correctly using the extra > object creation code. > > Users of scipy 0.5.1 will only have to upgrade NumPy to get the fix (the > SciPy install won't have to be re-built). > > -Travis Where is array at this point? I would like to review the documented behaviour and make modifications to the document string if required. What about Robert's idea of a separate constructor for object arrays? Is it something we could introduce on top of the current array constructor? I realize that if we restrict the current array constructor there might be compatibility problems with Numeric code, but introducing something like oarray as a shorthand for object arrays might incourage it's use. Robert also said that Numarray dealt with object arrays as a separate issue and I wonder what they did that we should think about. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Sep 6 19:33:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 06 Sep 2006 17:33:42 -0600 Subject: [Numpy-discussion] numpy.fftpack: axis=None ? In-Reply-To: <200609060552.11394.pgmdevlist@gmail.com> References: <200609060552.11394.pgmdevlist@gmail.com> Message-ID: <44FF5AD6.6050800@ieee.org> PGM wrote: > Folks, > I was playing around the numpy.fftpack when I ran into the problem below: > it seems that axis=None is not valid with fft. Is there a reason for that ? Not really, other than history and the fact that sub-packages of NumPy will often define their own behavior. Perhaps we could implement it, but would this create more confusion as usually when axis=None is accepted then it is the default. -Travis From oliphant.travis at ieee.org Wed Sep 6 19:39:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 06 Sep 2006 17:39:35 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> Message-ID: <44FF5C37.4090909@ieee.org> Charles R Harris wrote: > > Where is array at this point? Basically it supports the old Numeric behavior wherein object array's are treated as before *except* for when an error would have occurred previously when the "new behavior" kicks in. Anything that violates that is a bug needing to be fixed. This leaves the new object-array constructor used less often. It could be exported explicitly into an oarray constructor, but I'm not sure about the advantages of that approach. There are benefits to having object arrays constructed in the same way as other arrays. It turns out many people actually like that feature of Numeric, which is the reason I didn't go the route of numarray which pulled object arrays out. At this point, however, object arrays can even be part of records and so need to be an integral part of the data-type description. Pulling that out is not going to happen. A more intelligent object-array constructor, however, may be a useful tool. -Travis From ryanlists at gmail.com Wed Sep 6 20:44:38 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 19:44:38 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF3484.2040700@noaa.gov> References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> <44FF3484.2040700@noaa.gov> Message-ID: Thanks Chris, that worked great. Ryan On 9/6/06, Christopher Barker wrote: > Ryan Krauss wrote: > > Is there a compatible matplotlib as well? I was o.k. with mpl from > > enthought until I switched numerix to numpy. That made mpl unhappy. > > I downloaded 0.87.5 but I broke something in the process because now > > even switching back to Numeric doesn't make mpl happy. > > There was an error with the first MPL 0.87.5 builds uploaded. See the > MPL list for more info, but these builds should work: > > http://euclid.uits.iupui.edu/mplfiles/ > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From fccoelho at gmail.com Wed Sep 6 21:05:51 2006 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed, 6 Sep 2006 22:05:51 -0300 Subject: [Numpy-discussion] f2py with xmingw Message-ID: Hi, I have a module that uses a Fortran extension which I would like to compile for windows with f2py. I wonder If I could do this from Linux using xmingw. Has anyone tried this? thanks, -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 6 21:18:33 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 19:18:33 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FF5C37.4090909@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 9/6/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > Where is array at this point? > Basically it supports the old Numeric behavior wherein object array's > are treated as before *except* for when an error would have occurred > previously when the "new behavior" kicks in. Anything that violates > that is a bug needing to be fixed. > > This leaves the new object-array constructor used less often. It could > be exported explicitly into an oarray constructor, but I'm not sure > about the advantages of that approach. There are benefits to having > object arrays constructed in the same way as other arrays. It turns out > many people actually like that feature of Numeric, which is the reason I > didn't go the route of numarray which pulled object arrays out. > > At this point, however, object arrays can even be part of records and so > need to be an integral part of the data-type description. Pulling that > out is not going to happen. A more intelligent object-array > constructor, however, may be a useful tool. OK. I do have a couple of questions. Let me insert the docs for array and asarray : """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) Return an array from object with the specified date-type. Inputs: object - an array, any object exposing the array interface, any object whose __array__ method returns an array, or any (nested) sequence. dtype - The desired data-type for the array. If not given, then the type will be determined as the minimum type required to hold the objects in the sequence. This argument can only be used to 'upcast' the array. For downcasting, use the .astype(t) method. copy - If true, then force a copy. Otherwise a copy will only occur if __array__ returns a copy, obj is a nested sequence, or a copy is needed to satisfy any of the other requirements order - Specify the order of the array. If order is 'C', then the array will be in C-contiguous order (last-index varies the fastest). If order is 'FORTRAN', then the returned array will be in Fortran-contiguous order (first-index varies the fastest). If order is None, then the returned array may be in either C-, or Fortran-contiguous order or even discontiguous. subok - If True, then sub-classes will be passed-through, otherwise the returned array will be forced to be a base-class array ndmin - Specifies the minimum number of dimensions that the resulting array should have. 1's will be pre-pended to the shape as needed to meet this requirement. """) asarray(a, dtype=None, order=None) Returns a as an array. Unlike array(), no copy is performed if a is already an array. Subclasses are converted to base class ndarray. 1) Is it true that array doesn't always return a copy except by default? asarray says it contrasts with array in this regard. Maybe copy=0 should be deprecated. 2) Is asarray is basically array with copy=0? 3) Is asanyarray basically array with copy=0 and subok=1? 4) Is there some sort of precedence table for conversions? To me it looks like the most deeply nested lists are converted to arrays first, numeric if they contain all numeric types, object otherwise. I assume the algorithm then ascends up through the hierarchy like traversing a binary tree in postorder? 5) All nesting must be to the same depth and the deepest nested items must have the same length. 6) How is the difference between lists and "lists" determined, i.e., In [3]: array([list([1,2,3]),list([1,2])], dtype = object) Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) In [8]: array([array([1,2,3]),array([1,2])], dtype = object) Out[8]: array([[1 2 3], [1 2]], dtype=object) In [9]: array([1,2,3],[1,2]], dtype = object) ------------------------------------------------------------ File "", line 1 array([1,2,3],[1,2]], dtype = object) ^ SyntaxError: invalid syntax Is the difference that list(...) and array(...) are passed as functions (lazy evaluation), but a list is just a list? Sorry to be asking all these questions, but I would like to try making the documentation be a bit of a reference. I am sure I will have more questions ;) -Travis Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 6 21:52:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 19:52:42 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 9/6/06, Charles R Harris wrote: > > > > On 9/6/06, Travis Oliphant wrote: > > > > Charles R Harris wrote: > > > > > > Where is array at this point? > > Basically it supports the old Numeric behavior wherein object array's > > are treated as before *except* for when an error would have occurred > > previously when the "new behavior" kicks in. Anything that violates > > that is a bug needing to be fixed. > > > > This leaves the new object-array constructor used less often. It could > > be exported explicitly into an oarray constructor, but I'm not sure > > about the advantages of that approach. There are benefits to having > > object arrays constructed in the same way as other arrays. It turns out > > many people actually like that feature of Numeric, which is the reason I > > didn't go the route of numarray which pulled object arrays out. > > > > At this point, however, object arrays can even be part of records and so > > need to be an integral part of the data-type description. Pulling that > > out is not going to happen. A more intelligent object-array > > constructor, however, may be a useful tool. > > > OK. I do have a couple of questions. Let me insert the docs for array and > asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > Inputs: > object - an array, any object exposing the array interface, any > object whose __array__ method returns an array, or any > (nested) sequence. > dtype - The desired data-type for the array. If not given, then > the type will be determined as the minimum type required > to hold the objects in the sequence. This argument can > only > be used to 'upcast' the array. For downcasting, use the > .astype(t) method. > copy - If true, then force a copy. Otherwise a copy will only > occur > if __array__ returns a copy, obj is a nested sequence, or > a copy is needed to satisfy any of the other requirements > order - Specify the order of the array. If order is 'C', then the > array will be in C-contiguous order (last-index varies the > fastest). If order is 'FORTRAN', then the returned array > will be in Fortran-contiguous order (first-index varies > the > fastest). If order is None, then the returned array may > be in either C-, or Fortran-contiguous order or even > discontiguous. > subok - If True, then sub-classes will be passed-through, otherwise > the returned array will be forced to be a base-class array > ndmin - Specifies the minimum number of dimensions that the > resulting > array should have. 1's will be pre-pended to the shape as > needed to meet this requirement. > > """) > > asarray(a, dtype=None, order=None) > Returns a as an array. > > Unlike array(), no copy is performed if a is already an array. > Subclasses > are converted to base class ndarray. > > 1) Is it true that array doesn't always return a copy except by default? > asarray says it contrasts with array in this regard. Maybe copy=0 should be > deprecated. > > 2) Is asarray is basically array with copy=0? > > 3) Is asanyarray basically array with copy=0 and subok=1? > > 4) Is there some sort of precedence table for conversions? To me it looks > like the most deeply nested lists are converted to arrays first, numeric if > they contain all numeric types, object otherwise. I assume the algorithm > then ascends up through the hierarchy like traversing a binary tree in > postorder? > > 5) All nesting must be to the same depth and the deepest nested items must > have the same length. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax > > Is the difference that list(...) and array(...) are passed as functions > (lazy evaluation), but a list is just a list? > > Sorry to be asking all these questions, but I would like to try making the > documentation be a bit of a reference. I am sure I will have more questions > ;) > > -Travis > > And, voila, ragged arrays: In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) In [10]: a*2 Out[10]: array([[2 4 6], [2 4]], dtype=object) In [11]: a + a Out[11]: array([[2 4 6], [2 4]], dtype=object) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Sep 6 22:18:39 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Wed, 6 Sep 2006 22:18:39 -0400 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 06/09/06, Charles R Harris wrote: > On 9/6/06, Charles R Harris wrote: > > > order - Specify the order of the array. If order is 'C', then the > > array will be in C-contiguous order (last-index varies the > > fastest). If order is 'FORTRAN', then the returned array > > will be in Fortran-contiguous order (first-index varies > the > > fastest). If order is None, then the returned array may > > be in either C-, or Fortran-contiguous order or even > > discontiguous. This one's a bit complicated. If array() is passed a list of lists, there are two different orders that are relevant - the output order of the array, and the order used to interpret the input. I suppose that if L is a lost of lists, array(L)[2,3]==L[2][3], that is, in some sense the arrays are always logically C-ordered even if the underlying representation is different. Does it make sense to specify this somewhere in the docstring? At least it would be good to make it clear that the order parameter affects only the underlying storage format, and not the indexing of the array. A. M. Archibald From oliphant.travis at ieee.org Thu Sep 7 02:54:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 07 Sep 2006 00:54:41 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: <44FFC231.4060507@ieee.org> Charles R Harris wrote: > On 9/6/06, *Charles R Harris* > wrote: > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > wrote: > > Charles R Harris wrote: > > > > Where is array at this point? > Basically it supports the old Numeric behavior wherein object > array's > are treated as before *except* for when an error would have > occurred > previously when the "new behavior" kicks in. Anything that > violates > that is a bug needing to be fixed. > > This leaves the new object-array constructor used less > often. It could > be exported explicitly into an oarray constructor, but I'm not > sure > about the advantages of that approach. There are benefits to > having > object arrays constructed in the same way as other arrays. It > turns out > many people actually like that feature of Numeric, which is > the reason I > didn't go the route of numarray which pulled object arrays out. > > At this point, however, object arrays can even be part of > records and so > need to be an integral part of the data-type description. > Pulling that > out is not going to happen. A more intelligent object-array > constructor, however, may be a useful tool. > > > OK. I do have a couple of questions. Let me insert the docs for > array and asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > Inputs: > object - an array, any object exposing the array interface, any > object whose __array__ method returns an array, or any > (nested) sequence. > dtype - The desired data-type for the array. If not given, > then > the type will be determined as the minimum type > required > to hold the objects in the sequence. This > argument can only > be used to 'upcast' the array. For downcasting, > use the > .astype(t) method. > copy - If true, then force a copy. Otherwise a copy will > only occur > if __array__ returns a copy, obj is a nested > sequence, or > a copy is needed to satisfy any of the other > requirements > order - Specify the order of the array. If order is 'C', > then the > array will be in C-contiguous order (last-index > varies the > fastest). If order is 'FORTRAN', then the > returned array > will be in Fortran-contiguous order (first-index > varies the > fastest). If order is None, then the returned > array may > be in either C-, or Fortran-contiguous order or even > discontiguous. > subok - If True, then sub-classes will be passed-through, > otherwise > the returned array will be forced to be a > base-class array > ndmin - Specifies the minimum number of dimensions that the > resulting > array should have. 1's will be pre-pended to the > shape as > needed to meet this requirement. > > """) > > asarray(a, dtype=None, order=None) > Returns a as an array. > > Unlike array(), no copy is performed if a is already an array. > Subclasses > are converted to base class ndarray. > > 1) Is it true that array doesn't always return a copy except by > default? asarray says it contrasts with array in this regard. > Maybe copy=0 should be deprecated. > > 2) Is asarray is basically array with copy=0? > > 3) Is asanyarray basically array with copy=0 and subok=1? > > 4) Is there some sort of precedence table for conversions? To me > it looks like the most deeply nested lists are converted to arrays > first, numeric if they contain all numeric types, object > otherwise. I assume the algorithm then ascends up through the > hierarchy like traversing a binary tree in postorder? > > 5) All nesting must be to the same depth and the deepest nested > items must have the same length. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax > > Is the difference that list(...) and array(...) are passed as > functions (lazy evaluation), but a list is just a list? > > Sorry to be asking all these questions, but I would like to try > making the documentation be a bit of a reference. I am sure I will > have more questions ;) > > -Travis > > > And, voila, ragged arrays: > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > In [10]: a*2 > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > In [11]: a + a > Out[11]: array([[2 4 6], [2 4]], dtype=object) Now I remember that this was my original motivation for futzing with the object-array constructor in the first place. So, now you get there only after an attempt to make a "rectangular" array first. -Travis From oliphant.travis at ieee.org Thu Sep 7 03:03:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 07 Sep 2006 01:03:03 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: <44FFC427.2030101@ieee.org> Charles R Harris wrote: > OK. I do have a couple of questions. Let me insert the docs for array > and asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > 1) Is it true that array doesn't always return a copy except by > default? asarray says it contrasts with array in this regard. Maybe > copy=0 should be deprecated. array is the main creation function. It is a loose wrapper around PyArray_fromAny. copy=0 means don't copy unless you have to. > 2) Is asarray is basically array with copy=0? Yes. > > 3) Is asanyarray basically array with copy=0 and subok=1? Yes. > > 4) Is there some sort of precedence table for conversions? To me it > looks like the most deeply nested lists are converted to arrays first, > numeric if they contain all numeric types, object otherwise. I assume > the algorithm then ascends up through the hierarchy like traversing a > binary tree in postorder? I'm not sure I understand what you mean. The discover-depth and discover-dimensions algorithm figures out what the shape should be and then recursive PySequence_GetItem and PySequence_SetItem is used to copy the information over to the ndarray from the nested sequence. > > 5) All nesting must be to the same depth and the deepest nested items > must have the same length. Yes, there are routines discover_depth and discover_dimensions that are the actual algorithm used. These are adapted from Numeric. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax I think this is just due to a missing [ in In [9]. There is no semantic difference between list([1,2,3]) and [1,2,3] (NumPy will see those things as exactly the same). > > Is the difference that list(...) and array(...) are passed as > functions (lazy evaluation), but a list is just a list? There is nothing like "lazy evaluation" going on. array([1,2,3]) is evaluated returning an object and array([1,2]) is evaluated returning an object and then the two are put into another object array. Equivalent code a = array([1,2,3]) b = array([1,2]) c = array([a,b],dtype=object) Thanks for all your help with documentation. It is very-much appreciated. -Travis From zdm105 at tom.com Sun Sep 10 03:09:07 2006 From: zdm105 at tom.com (=?GB2312?B?IjnUwjIxLTIyyNUvyc+6oyI=?=) Date: Sun, 10 Sep 2006 15:09:07 +0800 Subject: [Numpy-discussion] =?GB2312?B?RE9FzO+/2sq9xrfWyrmks8w=?= Message-ID: An HTML attachment was scrubbed... URL: From ivilata at carabos.com Thu Sep 7 08:12:10 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Thu, 07 Sep 2006 14:12:10 +0200 Subject: [Numpy-discussion] Fix to Int64/string numexpr support Message-ID: <45000C9A.5050902@carabos.com> Hi all, I've detected some small errors in the patches I sent some time ago for adding Int64 and string support to numexpr (see http://www.mail-archive.com/numpy-discussion%40lists.sourceforge.net/msg01551.html). Basically: * ``numpy.string`` was accessed instead of ``numpy.string_`` (looks like one of those not-so-just-aesthetical last-time changes). * The ``copy_args`` argument to ``evaluate()`` and others was no longer needed since discontiguous/unaligned arrays are no longer a special case. I have attached the necessary patch. Bye! :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr_no_copy_args.diff Type: text/x-patch Size: 2712 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: OpenPGP digital signature URL: From svetosch at gmx.net Thu Sep 7 09:29:24 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 07 Sep 2006 15:29:24 +0200 Subject: [Numpy-discussion] reshape: missing arg check? Message-ID: <45001EB4.7080001@gmx.net> Hi, never mind that the following syntax is wrong, but is it supposed to yield that SystemError instead of something more informative? (This is with b5 on win32 and python 2.4.3) >>> b.reshape(3,3,axis = 1) Traceback (most recent call last): File "", line 1, in ? SystemError: NULL result without error in PyObject_Call -sven From charlesr.harris at gmail.com Thu Sep 7 15:22:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 13:22:01 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FFC231.4060507@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: On 9/7/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > On 9/6/06, *Charles R Harris* > > wrote: > > > > > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > > wrote: > > > > Charles R Harris wrote: > > > > > > Where is array at this point? > > Basically it supports the old Numeric behavior wherein object > > array's > > are treated as before *except* for when an error would have > > occurred > > previously when the "new behavior" kicks in. Anything that > > violates > > that is a bug needing to be fixed. > > > > This leaves the new object-array constructor used less > > often. It could > > be exported explicitly into an oarray constructor, but I'm not > > sure > > about the advantages of that approach. There are benefits to > > having > > object arrays constructed in the same way as other arrays. It > > turns out > > many people actually like that feature of Numeric, which is > > the reason I > > didn't go the route of numarray which pulled object arrays out. > > > > At this point, however, object arrays can even be part of > > records and so > > need to be an integral part of the data-type description. > > Pulling that > > out is not going to happen. A more intelligent object-array > > constructor, however, may be a useful tool. > > > > > > OK. I do have a couple of questions. Let me insert the docs for > > array and asarray : > > > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > > > Return an array from object with the specified date-type. > > > > Inputs: > > object - an array, any object exposing the array interface, > any > > object whose __array__ method returns an array, or > any > > (nested) sequence. > > dtype - The desired data-type for the array. If not given, > > then > > the type will be determined as the minimum type > > required > > to hold the objects in the sequence. This > > argument can only > > be used to 'upcast' the array. For downcasting, > > use the > > .astype(t) method. > > copy - If true, then force a copy. Otherwise a copy will > > only occur > > if __array__ returns a copy, obj is a nested > > sequence, or > > a copy is needed to satisfy any of the other > > requirements > > order - Specify the order of the array. If order is 'C', > > then the > > array will be in C-contiguous order (last-index > > varies the > > fastest). If order is 'FORTRAN', then the > > returned array > > will be in Fortran-contiguous order (first-index > > varies the > > fastest). If order is None, then the returned > > array may > > be in either C-, or Fortran-contiguous order or even > > discontiguous. > > subok - If True, then sub-classes will be passed-through, > > otherwise > > the returned array will be forced to be a > > base-class array > > ndmin - Specifies the minimum number of dimensions that the > > resulting > > array should have. 1's will be pre-pended to the > > shape as > > needed to meet this requirement. > > > > """) > > > > asarray(a, dtype=None, order=None) > > Returns a as an array. > > > > Unlike array(), no copy is performed if a is already an array. > > Subclasses > > are converted to base class ndarray. > > > > 1) Is it true that array doesn't always return a copy except by > > default? asarray says it contrasts with array in this regard. > > Maybe copy=0 should be deprecated. > > > > 2) Is asarray is basically array with copy=0? > > > > 3) Is asanyarray basically array with copy=0 and subok=1? > > > > 4) Is there some sort of precedence table for conversions? To me > > it looks like the most deeply nested lists are converted to arrays > > first, numeric if they contain all numeric types, object > > otherwise. I assume the algorithm then ascends up through the > > hierarchy like traversing a binary tree in postorder? > > > > 5) All nesting must be to the same depth and the deepest nested > > items must have the same length. > > > > 6) How is the difference between lists and "lists" determined, i.e., > > > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > > > > In [9]: array([1,2,3],[1,2]], dtype = object) > > ------------------------------------------------------------ > > File "", line 1 > > array([1,2,3],[1,2]], dtype = object) > > ^ > > SyntaxError: invalid syntax > > > > Is the difference that list(...) and array(...) are passed as > > functions (lazy evaluation), but a list is just a list? > > > > Sorry to be asking all these questions, but I would like to try > > making the documentation be a bit of a reference. I am sure I will > > have more questions ;) > > > > -Travis > > > > > > And, voila, ragged arrays: > > > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > > > In [10]: a*2 > > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > > > In [11]: a + a > > Out[11]: array([[2 4 6], [2 4]], dtype=object) > > Now I remember that this was my original motivation for futzing with the > object-array constructor in the first place. So, now you get there only > after an attempt to make a "rectangular" array first. > > -Travis So is this intentional? In [24]: a = array([[],[],[]], dtype=object) In [25]: a.shape Out[25]: (3, 0) In [26]: a = array([], dtype=object) In [27]: a.shape Out[27]: (0,) One could argue that the first array should have shape (3,) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 7 15:29:45 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 13:29:45 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: On 9/7/06, Charles R Harris wrote: > > > > On 9/7/06, Travis Oliphant wrote: > > > > Charles R Harris wrote: > > > On 9/6/06, *Charles R Harris* > > > wrote: > > > > > > > > > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > > > wrote: > > > > > > Charles R Harris wrote: > > > > > > > > Where is array at this point? > > > Basically it supports the old Numeric behavior wherein object > > > array's > > > are treated as before *except* for when an error would have > > > occurred > > > previously when the "new behavior" kicks in. Anything that > > > violates > > > that is a bug needing to be fixed. > > > > > > This leaves the new object-array constructor used less > > > often. It could > > > be exported explicitly into an oarray constructor, but I'm not > > > > > sure > > > about the advantages of that approach. There are benefits to > > > having > > > object arrays constructed in the same way as other arrays. It > > > turns out > > > many people actually like that feature of Numeric, which is > > > the reason I > > > didn't go the route of numarray which pulled object arrays > > out. > > > > > > At this point, however, object arrays can even be part of > > > records and so > > > need to be an integral part of the data-type description. > > > Pulling that > > > out is not going to happen. A more intelligent object-array > > > constructor, however, may be a useful tool. > > > > > > > > > OK. I do have a couple of questions. Let me insert the docs for > > > array and asarray : > > > > > > """array(object, dtype=None, copy=1,order=None, > > subok=0,ndmin=0) > > > > > > Return an array from object with the specified date-type. > > > > > > Inputs: > > > object - an array, any object exposing the array interface, > > any > > > object whose __array__ method returns an array, or > > any > > > (nested) sequence. > > > dtype - The desired data-type for the array. If not given, > > > then > > > the type will be determined as the minimum type > > > required > > > to hold the objects in the sequence. This > > > argument can only > > > be used to 'upcast' the array. For downcasting, > > > use the > > > .astype(t) method. > > > copy - If true, then force a copy. Otherwise a copy will > > > only occur > > > if __array__ returns a copy, obj is a nested > > > sequence, or > > > a copy is needed to satisfy any of the other > > > requirements > > > order - Specify the order of the array. If order is 'C', > > > then the > > > array will be in C-contiguous order (last-index > > > varies the > > > fastest). If order is 'FORTRAN', then the > > > returned array > > > will be in Fortran-contiguous order (first-index > > > varies the > > > fastest). If order is None, then the returned > > > array may > > > be in either C-, or Fortran-contiguous order or > > even > > > discontiguous. > > > subok - If True, then sub-classes will be passed-through, > > > otherwise > > > the returned array will be forced to be a > > > base-class array > > > ndmin - Specifies the minimum number of dimensions that the > > > resulting > > > array should have. 1's will be pre-pended to the > > > shape as > > > needed to meet this requirement. > > > > > > """) > > > > > > asarray(a, dtype=None, order=None) > > > Returns a as an array. > > > > > > Unlike array(), no copy is performed if a is already an array. > > > Subclasses > > > are converted to base class ndarray. > > > > > > 1) Is it true that array doesn't always return a copy except by > > > default? asarray says it contrasts with array in this regard. > > > Maybe copy=0 should be deprecated. > > > > > > 2) Is asarray is basically array with copy=0? > > > > > > 3) Is asanyarray basically array with copy=0 and subok=1? > > > > > > 4) Is there some sort of precedence table for conversions? To me > > > it looks like the most deeply nested lists are converted to arrays > > > first, numeric if they contain all numeric types, object > > > otherwise. I assume the algorithm then ascends up through the > > > hierarchy like traversing a binary tree in postorder? > > > > > > 5) All nesting must be to the same depth and the deepest nested > > > items must have the same length. > > > > > > 6) How is the difference between lists and "lists" determined, i.e > > ., > > > > > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > > > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > > > > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > > > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > > > > > > > In [9]: array([1,2,3],[1,2]], dtype = object) > > > ------------------------------------------------------------ > > > File "", line 1 > > > array([1,2,3],[1,2]], dtype = object) > > > ^ > > > SyntaxError: invalid syntax > > > > > > Is the difference that list(...) and array(...) are passed as > > > functions (lazy evaluation), but a list is just a list? > > > > > > Sorry to be asking all these questions, but I would like to try > > > making the documentation be a bit of a reference. I am sure I will > > > have more questions ;) > > > > > > -Travis > > > > > > > > > And, voila, ragged arrays: > > > > > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > > > > > In [10]: a*2 > > > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > > > > > In [11]: a + a > > > Out[11]: array([[2 4 6], [2 4]], dtype=object) > > > > Now I remember that this was my original motivation for futzing with the > > > > object-array constructor in the first place. So, now you get there only > > after an attempt to make a "rectangular" array first. > > > > -Travis > > > So is this intentional? > > In [24]: a = array([[],[],[]], dtype=object) > > In [25]: a.shape > Out[25]: (3, 0) > > In [26]: a = array([], dtype=object) > > In [27]: a.shape > Out[27]: (0,) > > One could argue that the first array should have shape (3,) > And this doesn't look quite right: In [38]: a = array([[1],[2],[3]], dtype=object) In [39]: a.shape Out[39]: (3, 1) In [40]: a = array([[1],[2,3],[4,5]], dtype=object) In [41]: a.shape Out[41]: (3,) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Glen.Mabey at swri.org Thu Sep 7 15:34:14 2006 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Thu, 7 Sep 2006 14:34:14 -0500 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files Message-ID: <20060907193414.GA7225@swri16wm.electro.swri.edu> A long time ago, Travis wrote: > On a related, but orthogonal note: > > My understanding is that using memory-mapped files for *very* large > files will require modification to the mmap module in Python --- > something I think we should push. One part of that process would be > to add the C-struct array interface to the mmap module and the buffer > object -- perhaps this is how we get the array interface into Python > quickly. Then, if we could make a base-type mmap that did not use > the buffer interface or the sequence interface (similar to the > bigndarray in scipy_core) and therefore by-passed the problems with > Python in those areas, then the current mmap object could inherit from > the base class and provide current functionality while still exposing > the array interface for access to >2GB files on 64-bit systems. > > Who would like to take up the ball for modifying mmap in Python in > this fashion? > > -Travis Did anyone ever "pick up the ball" on this issue? Glen From scipy at mspacek.mm.st Thu Sep 7 16:17:58 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Thu, 07 Sep 2006 13:17:58 -0700 Subject: [Numpy-discussion] ndarray.count() ? Message-ID: <45007E76.8030102@mspacek.mm.st> What's the most straightforward way to count, say, the number of 1s or Trues in the array? Or the number of any integer? I was surprised to discover recently that there isn't a count() method as there is for Python lists. Sorry if this has been discussed already, but I'm wondering if there's a reason for its absence. I came across a thread in March: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3066460 that talked a bit about this in terms of speed, but what about just the convenience of having a count() method? Looks like masked arrays have a count method, don't know much about them though. Also, I understand the inaccuracies when converting between binary and decimal floating point representations, and therefore making counting of a specific float value in an array somewhat undefined, yet it seems to work in Python lists: >>> 1.1 1.1000000000000001 >>> a=[1.1, 1.1, 1.2] >>> a [1.1000000000000001, 1.1000000000000001, 1.2] >>> a.count(1.1) 2 >>> a.count(1.1000000000000001) 2 >>> a.count(1.2) 1 Comments? Martin From charlesr.harris at gmail.com Thu Sep 7 16:37:07 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 14:37:07 -0600 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. I don't know about count, but you can gin up something like this In [78]: a = ran.randint(0,2, size=(10,)) In [79]: a Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) In [80]: b = sort(a) In [81]: b.searchsorted(1, side='right') - b.searchsorted(1, side='left') Out[81]: 6 Which counts the number of ones in a. I came across a thread in March: > > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3066460 > > that talked a bit about this in terms of speed, but what about just the > convenience of having a count() method? > > Looks like masked arrays have a count method, don't know much about them > though. > > Also, I understand the inaccuracies when converting between binary and > decimal floating point representations, and therefore making counting of > a specific float value in an array somewhat undefined, yet it seems to > work in Python lists: > > >>> 1.1 > 1.1000000000000001 > >>> a=[1.1, 1.1, 1.2] > >>> a > [1.1000000000000001, 1.1000000000000001, 1.2] > >>> a.count(1.1) > 2 > >>> a.count(1.1000000000000001) > 2 > >>> a.count(1.2) > 1 Well, 1.1 == 1.1000000000000001 and that doesn't change. You probably need to use different precisions to run into problems. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 16:40:24 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 15:40:24 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. Mostly, it's simply easy enough to implement yourself. Not all one-liners should be methods on the array object. (a == value).sum() Of course, there are several different things you might do. You might want to have multiple values broadcast across the array. You might want to reduce the count along a given axis. You might want to use floating point comparison with a tolerance. Putting all of those options into one method reduces the convenience of that method. Putting a crippled implementation (i.e. just that one-liner) expands the already-enormous API of the ndarray object without much benefit. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Thu Sep 7 16:25:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 7 Sep 2006 13:25:04 -0700 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files In-Reply-To: <20060907193414.GA7225@swri16wm.electro.swri.edu> References: <20060907193414.GA7225@swri16wm.electro.swri.edu> Message-ID: <200609071325.04777.haase@msg.ucsf.edu> Hi Glen ! How is that quote really !? The new Python2.5 *is* implementing the needed changes - so go ahead install Python2.5 (rc1 is the latest I think) and report how it works. I would also be very intersted to hear ;-) -Sebastian Haase On Thursday 07 September 2006 12:34, Glen W. Mabey wrote: > A long time ago, Travis wrote: > > On a related, but orthogonal note: > > > > My understanding is that using memory-mapped files for *very* large > > files will require modification to the mmap module in Python --- > > something I think we should push. One part of that process would be > > to add the C-struct array interface to the mmap module and the buffer > > object -- perhaps this is how we get the array interface into Python > > quickly. Then, if we could make a base-type mmap that did not use > > the buffer interface or the sequence interface (similar to the > > bigndarray in scipy_core) and therefore by-passed the problems with > > Python in those areas, then the current mmap object could inherit from > > the base class and provide current functionality while still exposing > > the array interface for access to >2GB files on 64-bit systems. > > > > Who would like to take up the ball for modifying mmap in Python in > > this fashion? > > > > -Travis > > Did anyone ever "pick up the ball" on this issue? > > Glen > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From alexander.belopolsky at gmail.com Thu Sep 7 16:47:07 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 7 Sep 2006 16:47:07 -0400 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. > You don't really need count with ndarrays: >>> from numpy import * >>> a = array([1,2,3,1,2,3,1,2]) >>> (a==3).sum() 2 From ndarray at mac.com Thu Sep 7 16:48:07 2006 From: ndarray at mac.com (Sasha) Date: Thu, 7 Sep 2006 16:48:07 -0400 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. > You don't really need count with ndarrays: >>> from numpy import * >>> a = array([1,2,3,1,2,3,1,2]) >>> (a==3).sum() 2 From mspacek at mm.st Thu Sep 7 17:21:10 2006 From: mspacek at mm.st (Martin Spacek) Date: Thu, 07 Sep 2006 14:21:10 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: <45008D46.5050008@mm.st> Great! That's exactly what I wanted. Works with floats too. Thanks, Martin Robert Kern wrote: > Mostly, it's simply easy enough to implement yourself. Not all one-liners should > be methods on the array object. > > (a == value).sum() > From mike.ressler at gmail.com Thu Sep 7 16:18:19 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Thu, 7 Sep 2006 13:18:19 -0700 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files In-Reply-To: <20060907193414.GA7225@swri16wm.electro.swri.edu> References: <20060907193414.GA7225@swri16wm.electro.swri.edu> Message-ID: <268febdf0609071318la00d3evaac8d6640b967acb@mail.gmail.com> On 9/7/06, Glen W. Mabey wrote: > > A long time ago, Travis wrote: > > > > My understanding is that using memory-mapped files for *very* large > > files will require modification to the mmap module in Python --- > > Did anyone ever "pick up the ball" on this issue? This works with python-2.5 betas and above, and numpy 1.0 betas and above. I frequently do 10+ GB mmaps. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Sep 7 18:15:33 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 07 Sep 2006 16:15:33 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: <45009A05.5020007@ee.byu.edu> Charles R Harris wrote: > > So is this intentional? > > In [24]: a = array([[],[],[]], dtype=object) > > In [25]: a.shape > Out[25]: (3, 0) > > In [26]: a = array([], dtype=object) > > In [27]: a.shape > Out[27]: (0,) > > One could argue that the first array should have shape (3,) > Yes, it's intentional because it's the old behavior of Numeric. And it follows the rule that object arrays don't do anything special unless the old technique of using [] as 'dimension delimiters' breaks down. > > And this doesn't look quite right: > > In [38]: a = array([[1],[2],[3]], dtype=object) > > In [39]: a.shape > Out[39]: (3, 1) > > In [40]: a = array([[1],[2,3],[4,5]], dtype=object) > > In [41]: a.shape > Out[41]: (3,) > Again, same reason as before. The first example works fine to construct a rectangular array of object arrays of dimension 2. The second only does if we limit the number of dimensions to 1. The rule is that array needs nested lists with the same number of dimensions unless you have object arrays. Then, the dimensionality will be determined by finding the largest number of dimensions possible for consistency of shape. -Travis From rex at nosyntax.com Thu Sep 7 18:33:36 2006 From: rex at nosyntax.com (rex) Date: Thu, 7 Sep 2006 15:33:36 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: <20060907223336.GH5025@x2.nosyntax.com> Charles R Harris [2006-09-07 15:04]: > I don't know about count, but you can gin up something like this > > In [78]: a = ran.randint(0,2, size=(10,)) > > In [79]: a > Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) This exposed inconsistent randint() behavior between SciPy and the Python random module. The Python randint includes the upper endpoint. The SciPy version excludes it. >>> from random import randint >>> for i in range(50): ... print randint(0,2), ... 0 1 1 1 1 1 0 0 2 1 1 0 2 2 1 2 0 2 0 0 0 2 2 2 2 2 2 2 1 2 2 0 0 1 2 2 0 1 1 0 2 0 1 2 1 2 2 2 1 1 >>> from scipy import * >>> print random.randint(0,2, size=(100,)) [0 1 1 1 1 0 1 1 0 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 1 1 0 1 1 1 0 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 1 0 1 1 1 1 1 1 1 0 1 1 0 0 1 0 1 1 0 0 0 0 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 0 0] -rex From a.h.jaffe at gmail.com Thu Sep 7 18:46:15 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 07 Sep 2006 23:46:15 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy Message-ID: Hi all, It seems that scipy and numpy define rfft differently. numpy returns n/2+1 complex numbers (so the first and last numbers are actually real) with the frequencies equivalent to the positive part of the fftfreq, whereas scipy returns n real numbers with the frequencies as in rfftfreq (i.e., two real numbers at the same frequency, except for the highest and lowest) [All of the above for even n; but the difference between numpy and scipy remains for odd n.] I think the numpy behavior makes more sense, as it doesn't require any unpacking after the fact, at the expense of a tiny amount of wasted space. But would this in fact require scipy doing extra work from whatever the 'native' real_fft (fftw, I assume) produces? Anyone else have an opinion? Andrew From charlesr.harris at gmail.com Thu Sep 7 18:48:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 16:48:51 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <45009A05.5020007@ee.byu.edu> References: <44FB29CA.9070808@colorado.edu> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: On 9/7/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > So is this intentional? > > > > In [24]: a = array([[],[],[]], dtype=object) > > > > In [25]: a.shape > > Out[25]: (3, 0) > > > > In [26]: a = array([], dtype=object) > > > > In [27]: a.shape > > Out[27]: (0,) > > > > One could argue that the first array should have shape (3,) > > > Yes, it's intentional because it's the old behavior of Numeric. And it > follows the rule that object arrays don't do anything special unless the > old technique of using [] as 'dimension delimiters' breaks down. > > > > > And this doesn't look quite right: > > > > In [38]: a = array([[1],[2],[3]], dtype=object) > > > > In [39]: a.shape > > Out[39]: (3, 1) > > > > In [40]: a = array([[1],[2,3],[4,5]], dtype=object) > > > > In [41]: a.shape > > Out[41]: (3,) > > > > Again, same reason as before. The first example works fine to construct > a rectangular array of object arrays of dimension 2. The second only > does if we limit the number of dimensions to 1. > > The rule is that array needs nested lists with the same number of > dimensions unless you have object arrays. Then, the dimensionality will > be determined by finding the largest number of dimensions possible for > consistency of shape. So there is a 'None' trick: In [93]: a = array([[[2]], None], dtype=object) In [94]: a[0] Out[94]: [[2]] I wonder if it wouldn't be useful to have a 'depth' keyword. Thus depth=None is current behavior, but array([], depth=0) would produce a zero dimensional array containing an empty list. Although I notice from playing with dictionaries that a zero dimensional array containing a dictionary isn't very useful. array([[],[]], depth=1) would produce a one dimensional array containing two empty lists, etc. I can see it is difficult to get something truely general with the current syntax without a little bit of extra information. Another question, what property must an object possess to be a container type argument in array? There are sequence type objects, and array type objects. Are there more or is everything else treated as an object? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 18:57:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 17:57:00 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <20060907223336.GH5025@x2.nosyntax.com> References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> Message-ID: rex wrote: > Charles R Harris [2006-09-07 15:04]: >> I don't know about count, but you can gin up something like this >> >> In [78]: a = ran.randint(0,2, size=(10,)) >> >> In [79]: a >> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) > > This exposed inconsistent randint() behavior between SciPy and the Python > random module. The Python randint includes the upper endpoint. The SciPy > version excludes it. numpy.random.random_integers() includes the upper bound, if you like. numpy.random does not try to emulate the standard library's random module. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Sep 7 19:03:59 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 17:03:59 -0600 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: On 9/7/06, Andrew Jaffe wrote: > > Hi all, > > It seems that scipy and numpy define rfft differently. > > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? > > Anyone else have an opinion? Yes, I prefer the scipy version because the result is actually a complex array and can immediately be use as the coefficients of the fft for frequencies <= Nyquist. I suspect, without checking, that what you get in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as the coefficient of the second frequency, etc. This makes it difficult to multiply by a complex transfer function or phase shift the result to rotate the original points by some fractional amount. As to unpacking, for some algorithms the two real coefficients are packed into the real and complex parts of the zero frequency so all that is needed is an extra complex slot at the end. Other algorithms produce what you describe. I just think it is more convenient to think of the real fft as an efficient complex fft that only computes the coefficients <= Nyquist because Hermitean symmetry determines the rest. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstrunk at enthought.com Thu Sep 7 19:19:01 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Thu, 7 Sep 2006 18:19:01 -0500 Subject: [Numpy-discussion] ISP changeover Tuesday 9/12 7:00 PM Central Message-ID: <200609071819.01770.jstrunk@enthought.com> Good afternoon, Unfortunately, our recent change in internet service providers is not working out. We will be switching to a more reliable provider on Tuesday 9/12 at 7:00 PM Central. Please allow for up to two hours of downtime. I will send an email announcing the start and completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. The problem was that our in-building internet service provider neglected to mention that our internet connection was over a wireless link to a different building. This link had very high latency. They have fixed this problem, but we feel that wireless is not stable enough for our needs. In order to provide higher quality service, we will be using 3 bonded T1s from AT&T after this switchover. Please pass this message along to people that I have missed. If you have any questions, please direct them to me. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From peridot.faceted at gmail.com Thu Sep 7 19:49:08 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Thu, 7 Sep 2006 19:49:08 -0400 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: Maybe I should stay out of this, but it seems like constructing object arrays is complicated and involves a certain amount of guesswork on the part of Numeric. For example, if you do array([a,b,c]).shape(), the answer is normally (3,) unless a b and c happen to all be lists of the same length, at which point your array could have a much more complicated shape... but as the person who wrote "array([a,b,c])" it's tempting to assume that the result has shape (3,), only to discover subtle bugs much later. If we were writing an array-creation function from scratch, would there be any reason to include object-array creation in the same function as uniform array creation? It seems like a bad idea to me. If not, the problem is just compatibility with Numeric. Why not simply write a wrapper function in python that does Numeric-style guesswork, and put it in the compatibility modules? How much code will actually break? A. M. Archibald From a.h.jaffe at gmail.com Thu Sep 7 20:04:52 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 08 Sep 2006 01:04:52 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: <4500B3A4.7040704@gmail.com> Hi Charles, Charles R Harris wrote: > On 9/7/06, *Andrew Jaffe* > wrote: > > Hi all, > > It seems that scipy and numpy define rfft differently. > > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the > difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? > > Anyone else have an opinion? > > Yes, I prefer the scipy version because the result is actually a complex > array and can immediately be use as the coefficients of the fft for > frequencies <= Nyquist. I suspect, without checking, that what you get > in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as > the coefficient of the second frequency, etc. This makes it difficult to > multiply by a complex transfer function or phase shift the result to > rotate the original points by some fractional amount. > > As to unpacking, for some algorithms the two real coefficients are > packed into the real and complex parts of the zero frequency so all that > is needed is an extra complex slot at the end. Other algorithms produce > what you describe. I just think it is more convenient to think of the > real fft as an efficient complex fft that only computes the coefficients > <= Nyquist because Hermitean symmetry determines the rest. Unless I misunderstand, I think you've got it backwards: - numpy.fft.rfft produces the correct complex array (with no strange packing), with frequencies (0, 1, 2, ... n/2) - scipy.fftpack.rfft produces a single real array, in the correct order, but with frequencies (0, 1, 1, 2, 2, ..., n/2) -- as given by scipy.fftpack's rfftfreq function. So I think you prefer numpy, not scipy. This is complicated by the fact that (I think) numpy.fft shows up as scipy.fft, so functions with the same name in scipy.fft and scipy.fftpack aren't actually the same! Andrew From rex at nosyntax.com Thu Sep 7 20:14:22 2006 From: rex at nosyntax.com (rex) Date: Thu, 7 Sep 2006 17:14:22 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> Message-ID: <20060908001422.GJ5025@x2.nosyntax.com> Robert Kern [2006-09-07 16:35]: > rex wrote: > > Charles R Harris [2006-09-07 15:04]: > >> I don't know about count, but you can gin up something like this > >> > >> In [78]: a = ran.randint(0,2, size=(10,)) > >> > >> In [79]: a > >> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) > > > > This exposed inconsistent randint() behavior between SciPy and the Python > > random module. The Python randint includes the upper endpoint. The SciPy > > version excludes it. > > numpy.random.random_integers() includes the upper bound, if you like. > numpy.random does not try to emulate the standard library's random module. I'm not in a position to argue the merits, but IMHO, when code that previously worked silently starts returning subtly bad results after importing numpy, there is a problem. What possible upside is there in having randint() behave one way in the random module and silently behave differently in numpy? More generally, since numpy.random does not try to emulate the random module, how does one convert from code that uses the random module to numpy? Is randint() the only silent problem, or are there others? If so, how does one discover them? Are they documented anywhere? I deeply appreciate the countless hours the core developers have contributed to numpy/scipy, but sometimes I think you are too close to the problems to fully appreciate the barriers to widespread adoption such silent "gotchas" present. If the code breaks, fine, you know there's a problem. When it runs, but returns wrong -- but not obviously wrong -- results, there's a serious problem that will deter a significant number of people from ever trying the product again. Again, what is the upside of changing the behavior of the standard library's randint() without also changing the name? -rex From charlesr.harris at gmail.com Thu Sep 7 20:42:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 18:42:08 -0600 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: <4500B3A4.7040704@gmail.com> References: <4500B3A4.7040704@gmail.com> Message-ID: On 9/7/06, Andrew Jaffe wrote: > > Hi Charles, > > Charles R Harris wrote: > > On 9/7/06, *Andrew Jaffe* > > wrote: > > > > Hi all, > > > > It seems that scipy and numpy define rfft differently. > > > > numpy returns n/2+1 complex numbers (so the first and last numbers > are > > actually real) with the frequencies equivalent to the positive part > of > > the fftfreq, whereas scipy returns n real numbers with the > frequencies > > as in rfftfreq (i.e., two real numbers at the same frequency, except > for > > the highest and lowest) [All of the above for even n; but the > > difference > > between numpy and scipy remains for odd n.] > > > > I think the numpy behavior makes more sense, as it doesn't require > any > > unpacking after the fact, at the expense of a tiny amount of wasted > > space. But would this in fact require scipy doing extra work from > > whatever the 'native' real_fft (fftw, I assume) produces? > > > > Anyone else have an opinion? > > > > Yes, I prefer the scipy version because the result is actually a complex > > array and can immediately be use as the coefficients of the fft for > > frequencies <= Nyquist. I suspect, without checking, that what you get > > in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as > > the coefficient of the second frequency, etc. This makes it difficult to > > multiply by a complex transfer function or phase shift the result to > > rotate the original points by some fractional amount. > > > > As to unpacking, for some algorithms the two real coefficients are > > packed into the real and complex parts of the zero frequency so all that > > is needed is an extra complex slot at the end. Other algorithms produce > > what you describe. I just think it is more convenient to think of the > > real fft as an efficient complex fft that only computes the coefficients > > <= Nyquist because Hermitean symmetry determines the rest. > > Unless I misunderstand, I think you've got it backwards: > - numpy.fft.rfft produces the correct complex array (with no strange > packing), with frequencies (0, 1, 2, ... n/2) > > - scipy.fftpack.rfft produces a single real array, in the correct > order, but with frequencies (0, 1, 1, 2, 2, ..., n/2) -- as given by > scipy.fftpack's rfftfreq function. > > So I think you prefer numpy, not scipy. Ah, well then, yes. I prefer Numpy. IIRC, one way to get the Scipy ordering is to use the Hartley transform as the front to the real transform. And, now that you mention it, there was a big discussion about it in the scipy list way back when with yours truly pushing the complex form. I don't recall that anything was settled, rather the natural output of the algorithm they were using prevailed by default. Maybe you could write a front end that did the right thing? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fccoelho at gmail.com Thu Sep 7 21:18:46 2006 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu, 7 Sep 2006 22:18:46 -0300 Subject: [Numpy-discussion] Fwd: f2py with xmingw In-Reply-To: References: Message-ID: Hi, I have a module that uses a Fortran extension which I would like to compile for windows with f2py. I wonder If I could do this from Linux using xmingw. Has anyone tried this? thanks, -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 23:35:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 22:35:06 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <20060908001422.GJ5025@x2.nosyntax.com> References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> <20060908001422.GJ5025@x2.nosyntax.com> Message-ID: rex wrote: > Robert Kern [2006-09-07 16:35]: >> rex wrote: >>> Charles R Harris [2006-09-07 15:04]: >>>> I don't know about count, but you can gin up something like this >>>> >>>> In [78]: a = ran.randint(0,2, size=(10,)) >>>> >>>> In [79]: a >>>> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) >>> This exposed inconsistent randint() behavior between SciPy and the Python >>> random module. The Python randint includes the upper endpoint. The SciPy >>> version excludes it. >> numpy.random.random_integers() includes the upper bound, if you like. >> numpy.random does not try to emulate the standard library's random module. > > I'm not in a position to argue the merits, but IMHO, when code that > previously worked silently starts returning subtly bad results after > importing numpy, there is a problem. What possible upside is there in > having randint() behave one way in the random module and silently behave > differently in numpy? I don't understand you. Importing numpy does not change the standard library's random module in any way. There is no silent difference in behavior. If you use numpy.random you get one set of behavior. If you use random, you get another. Pick the one you want. They're not interchangeable, and nothing suggests that they ought to be. > More generally, since numpy.random does not try to emulate the random > module, how does one convert from code that uses the random module to > numpy? Is randint() the only silent problem, or are there others? If so, > how does one discover them? Are they documented anywhere? The docstrings in that module are complete. > I deeply appreciate the countless hours the core developers have > contributed to numpy/scipy, but sometimes I think you are too close to > the problems to fully appreciate the barriers to widespread adoption such > silent "gotchas" present. If the code breaks, fine, you know there's a > problem. When it runs, but returns wrong -- but not obviously wrong -- > results, there's a serious problem that will deter a significant number > of people from ever trying the product again. > > Again, what is the upside of changing the behavior of the standard > library's randint() without also changing the name? Again, numpy.random has nothing to do with the standard library module random. The names of the functions match those in the PRNG facilities that used to be in Numeric and scipy which numpy.random is replacing. Specifically, numpy.random.randint() derives its behavior from Numeric's RandomArray.randint(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rahn at avg-joe.com Fri Sep 8 06:35:35 2006 From: rahn at avg-joe.com (Mabelle Reach) Date: Fri, 8 Sep 2006 03:35:35 -0700 Subject: [Numpy-discussion] PHAruRMACY Message-ID: <000001c6d332$84d20620$cc58a8c0@nyaz> Hi All yo k ur P n HAR c MAC x Y di f rect i ly from the man a ufa d cture o r, Your c w hanc f e to eco q no p mize w f ith us http://www.pumationdesun.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From listservs at mac.com Fri Sep 8 09:20:27 2006 From: listservs at mac.com (listservs at mac.com) Date: Fri, 8 Sep 2006 09:20:27 -0400 Subject: [Numpy-discussion] numpy test failure: "ctypes is not available" Message-ID: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> I have built and installed numpy from svn on an Intel mac, and am having test failures that do not occur on PPC: In [8]: numpy.test() Found 13 tests for numpy.core.umath Found 8 tests for numpy.lib.arraysetops Found 3 tests for numpy.fft.helper Found 1 tests for numpy.lib.ufunclike Found 4 tests for numpy.ctypeslib Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 3 tests for numpy.lib.getlimits Found 31 tests for numpy.core.numerictypes Found 4 tests for numpy.core.scalarmath Found 10 tests for numpy.lib.twodim_base Found 46 tests for numpy.lib.shape_base Found 4 tests for numpy.lib.index_tricks Found 32 tests for numpy.linalg.linalg Found 5 tests for numpy.distutils.misc_util Found 42 tests for numpy.lib.type_check Found 163 tests for numpy.core.multiarray Found 36 tests for numpy.core.ma Found 10 tests for numpy.core.defmatrix Found 39 tests for numpy.lib.function_base Found 0 tests for __main__ .........................EEEE........................................... ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ......................................................... ====================================================================== ERROR: check_dtype (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 10, in check_dtype p = ndpointer(dtype=dt) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_flags (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 54, in check_flags p = ndpointer(flags='FORTRAN') File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_ndim (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 36, in check_ndim p = ndpointer(ndim=0) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_shape (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 46, in check_shape p = ndpointer(shape=(1,2)) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ---------------------------------------------------------------------- Ran 489 tests in 0.528s FAILED (errors=4) Out[8]: Any clues? -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From faltet at carabos.com Fri Sep 8 10:41:21 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 8 Sep 2006 16:41:21 +0200 Subject: [Numpy-discussion] numpy test failure: "ctypes is not available" In-Reply-To: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> References: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> Message-ID: <200609081641.22048.faltet@carabos.com> A Divendres 08 Setembre 2006 15:20, listservs at mac.com va escriure: > I have built and installed numpy from svn on an Intel mac, and am > having test failures that do not occur on PPC: > > In [8]: numpy.test() > Found 13 tests for numpy.core.umath > Found 8 tests for numpy.lib.arraysetops > Found 3 tests for numpy.fft.helper > Found 1 tests for numpy.lib.ufunclike > Found 4 tests for numpy.ctypeslib > Found 1 tests for numpy.lib.polynomial > Found 8 tests for numpy.core.records > Found 26 tests for numpy.core.numeric > Found 3 tests for numpy.lib.getlimits > Found 31 tests for numpy.core.numerictypes > Found 4 tests for numpy.core.scalarmath > Found 10 tests for numpy.lib.twodim_base > Found 46 tests for numpy.lib.shape_base > Found 4 tests for numpy.lib.index_tricks > Found 32 tests for numpy.linalg.linalg > Found 5 tests for numpy.distutils.misc_util > Found 42 tests for numpy.lib.type_check > Found 163 tests for numpy.core.multiarray > Found 36 tests for numpy.core.ma > Found 10 tests for numpy.core.defmatrix > Found 39 tests for numpy.lib.function_base > Found 0 tests for __main__ > .........................EEEE........................................... > ........................................................................ > ........................................................................ > ........................................................................ > ........................................................................ > ........................................................................ > ......................................................... > ====================================================================== > ERROR: check_dtype (numpy.tests.test_ctypeslib.test_ndpointer) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 10, in > check_dtype > p = ndpointer(dtype=dt) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy > raise ImportError, "ctypes is not available." > ImportError: ctypes is not available. This may be due to the fact that you are using Python 2.4 here and ctypes comes with Python2.5. Switch to 2.5, install ctypes separately or feel free to ignore this. I suppose that a check has to be set up in the tests to avoid ctypes ones to be checked in case ctypes is not available. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From dd55 at cornell.edu Fri Sep 8 11:03:02 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri, 8 Sep 2006 11:03:02 -0400 Subject: [Numpy-discussion] missing import in numpy.oldnumeric.mlab Message-ID: <200609081103.02628.dd55@cornell.edu> This was just reported on mpl-dev: In [1]: from numpy.oldnumeric import mlab In [2]: mlab.eye(3) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /home/darren/ /usr/lib64/python2.4/site-packages/numpy/oldnumeric/mlab.py in eye(N, M, k, typecode, dtype) 22 dtype = convtypecode(typecode, dtype) 23 if M is None: M = N ---> 24 m = nn.equal(nn.subtract.outer(nn.arange(N), nn.arange(M)),-k) 25 if m.dtype != dtype: 26 return m.astype(dtype) NameError: global name 'nn' is not defined From listservs at mac.com Fri Sep 8 11:31:01 2006 From: listservs at mac.com (listservs at mac.com) Date: Fri, 8 Sep 2006 11:31:01 -0400 Subject: [Numpy-discussion] problems with numpy/f2py module on Intel Mac Message-ID: <9A3362CE-CA49-4241-A5C6-9FA478BAD0F9@mac.com> After building and installing numpy from svn on an Intel Mac, I can successfully build f2py modules, but I get the following linking error: ImportError: Failure linking new module: /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/ flib.so: Symbol not found: ___dso_handle Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/PyMC/flib.so Expected in: dynamic lookup I'm using gfortran. Has anyone seen these types of errors before? I do not get them on PPC. Thanks, -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From zdm105 at tom.com Mon Sep 11 14:30:49 2006 From: zdm105 at tom.com (=?GB2312?B?IjnUwjIxLTIyyNUvyc+6oyI=?=) Date: Tue, 12 Sep 2006 02:30:49 +0800 Subject: [Numpy-discussion] =?GB2312?B?RE9FzO+/2sq9xrfWyrmks8w=?= Message-ID: An HTML attachment was scrubbed... URL: From jsz at kalliness.fsnet.co.uk Fri Sep 8 16:16:53 2006 From: jsz at kalliness.fsnet.co.uk (Sadie Craft) Date: Fri, 8 Sep 2006 22:16:53 +0200 Subject: [Numpy-discussion] regurgitation Message-ID: <001a01c6d384$3131d142$d8d71853@ybxm> Is it really true that no influence is ever brought to bear onanybody about anything? If Diana did want anything youd be on the spot. Andchanging her tone, she said coldly: After all, Captain Cherrellis a D. A good many, said Dinny, seem to drop first, and then do therunning. She did not talk, however,leaving him to begin if he wished. Personally, I dontbelieve Aunt May can put us up. Especially when theyve been to the same school. Not more than I am, dont worry about that. Its as good a place, I suppose, asyou could find. Arrived at Shropshire House Sir Lawrence said:Can we see the Marquess, Pommett? Her knowledge of London was small, and she hailed the first cab. She passed Dinny andAdrian as they were coming in. The man came in again and drewthe curtains. Itrepresented with a moderate degree of certainty a young womanwithout clothes. The man came in again and drewthe curtains. Is your extradition really likely,Hubert? The hat ropes did it, Dinny, and let me tell you that the grapeshave been sour ever since. Professor Hallorsen was expected in at five and should at once begiven the message. Lord Saxenden knows me, and raisedher eyes. I shall sleep with my emergency suit-case; one can always get a taxi here at a moments notice. Shake hands, Millie, and remember what I said. Then, swiftly, looking neither to left nor right, she returned toAdrian. Go off your chump, Cherrell,then youll know what it means to be lonely for the rest of yourdays. But privilege is only justified nowadays byrunning till you drop. Your Auntnever ceases to throw them in my teeth. Puzzlin, theyheard her say as they entered the library. Either Im over-sensitive, Dinny, or this particular afflictiondoes seem to me too dreadful. She bent her head, and still didnt speak. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sister.gif Type: image/gif Size: 15866 bytes Desc: not available URL: From dpinte at itae.be Fri Sep 8 20:32:20 2006 From: dpinte at itae.be (Didrik Pinte) Date: Sat, 09 Sep 2006 02:32:20 +0200 Subject: [Numpy-discussion] gdal numpy/numeric/matplotlib problem Message-ID: <1157761940.23138.27.camel@geru-itae> Hi, I am running a Debian/Sid system and face a problem when using the gdal python bindinds. The gdal python bindings are linked with Numeric. In one of my application i'm using some numpy methods. This is working fine BUT when I add the use of the matploblib library to my application all the calls to gdal specific methods are broken. I have attached a basic example. The first test fails if I import the pylab module. The second one that can be run with any shapefile shows that when pylab is loaded, some GDAL methods raises GEOS exceptions. Commenting the "import pylab" line shows that without pylab there is no exceptions, nor problems. They are two workarounds : [1] a real one : using matplotlib with the Numeric lib [2] a fake one : renaming /usr/lib/python2.4/site-packages/numpy/core/multiarray.so to another name, the tests does not fail. Does anybody have a suggestion to correct this problem ? I can provide more details if needed. Best regards, -- Didrik -------------- next part -------------- A non-text attachment was scrubbed... Name: test_gdalnumpy.py Type: text/x-python Size: 1469 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From mmb at pwdfoundation.org Sat Sep 9 09:20:20 2006 From: mmb at pwdfoundation.org (Roger Mclaughlin) Date: Sat, 9 Sep 2006 15:20:20 +0200 Subject: [Numpy-discussion] cobblestone Message-ID: <001a01c6d413$1cecdd3c$7c824b52@shc.rp> We are taught that in the Bible, youremember. Possibly he meant to melt away in the darkness, leaving us in thewilderness to our fate. He did not seem in the least surprised, indeed he said he knew thatthis was so. Why must he be our companion, Macumazahn? In the end I did so, very unwillingly,to please my relations. Well, hereplied, with a twinkle in his eye, if I were you I should go andlook for the lady there. I realizedthis when I made up my mind to continue the journey to Lake Monealone. Also she was rude enoughto believe and to tell me she believed that I was mad. Odd to have met you in this fashion,but so is everything in this place. Do you mean to say that you tried to do that, Mr. Oh, only that I seemed different in every way. No, always when I was awake and looking at the stars, andgenerally when I was in the open air. I willfollow with the wanderer as soon as he can walk. Then he lay down in the tentand fell at once into a profound sleep. My father was cracked on astrology andcast mine when I was born. I suppose I inherited it frommy father, at any rate there it is. Then he lay down in the tentand fell at once into a profound sleep. This pistol is much lighter triggered than I thought or perhaps theheat has affected the spring. From that moment, for example, I hated thedissipations which had attracted me. Go with them to the borders of the lake, where perchanceone will meet you. From that moment, for example, I hated thedissipations which had attracted me. Indeed, you would be wise to turnback, for I think that death is very fond of Lake Mone. There you will find those who will guideyou. He did not seem in the least surprised, indeed he said he knew thatthis was so. Lastly it was necessary to provide him withone of the spare Winchester rifles and some cartridges. Quatermain, I am adreamer and what is called a mystic. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: obediently.gif Type: image/gif Size: 12923 bytes Desc: not available URL: From ndbecker2 at gmail.com Sat Sep 9 10:18:24 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 9 Sep 2006 10:18:24 -0400 Subject: [Numpy-discussion] 1.0b5 build failure Message-ID: <200609091018.24835.ndbecker2@gmail.com> Fedora FC5 linux, with BLAS, LAPACK, ATLAS, FFTW compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c numpy/core/src/multiarraymodule.c: In function ?initmultiarray?: numpy/core/src/multiarraymodule.c:6860: error: ?NPY_ALLOW_THREADS? undeclared (first use in this function) numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier is reported only once numpy/core/src/multiarraymodule.c:6860: error: for each function it appears in.) numpy/core/src/multiarraymodule.c: In function ?initmultiarray?: numpy/core/src/multiarraymodule.c:6860: error: ?NPY_ALLOW_THREADS? undeclared (first use in this function) numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier is reported only once numpy/core/src/multiarraymodule.c:6860: error: for each function it appears in.) error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC -Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-x86_64-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 From ndbecker2 at gmail.com Sat Sep 9 10:50:24 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 09 Sep 2006 10:50:24 -0400 Subject: [Numpy-discussion] 1.0b5 build failure References: <200609091018.24835.ndbecker2@gmail.com> Message-ID: Sorry, actually that's today's numpy svn - not 1.0b5. From fperez.net at gmail.com Sat Sep 9 11:10:37 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 9 Sep 2006 09:10:37 -0600 Subject: [Numpy-discussion] 1.0b5 build failure In-Reply-To: References: <200609091018.24835.ndbecker2@gmail.com> Message-ID: On 9/9/06, Neal Becker wrote: > Sorry, actually that's today's numpy svn - not 1.0b5. Make sure you have a clean build environment, the current SVN builds just fine on my Dapper 6.06 box; I just rebuilt it a second ago. Cheers, f ps. Here's my little build script, which I use to rebuild numpy/scipy/matplotlib in one shot: longs[scipy]> cat makeall #!/bin/sh ./makepkg numpy ./makepkg scipy ./makepkg matplotlib # EOF which uses the simple makepkg (just edit the PYTHON and PYPREFIX vars to suit your config): longs[scipy]> cat makepkg #!/bin/sh PACKAGE=$1 PYTHON=python2.4 PYPREFIX=$HOME/tmp/local #REVISION="HEAD" #REVISION=2772 # for numpy #REVISION=2054 # for scipy #REVISION="{2006-07-06}" #svn update -r $REVISION ${PACKAGE} svn update ${PACKAGE} export PYTHONPATH=${PYPREFIX}/lib/${PYTHON}/site-packages:${PYTHONPATH}: # remove existing ${PACKAGE} to make sure the build doesn't pick up spurious things rm -rf $PYPREFIX/lib/${PYTHON}/site-packages/${PACKAGE} # make/install cd ${PACKAGE} rm -rf build $PYTHON setup.py install --prefix=$PYPREFIX From stevenj at alum.mit.edu Sat Sep 9 12:50:58 2006 From: stevenj at alum.mit.edu (Steven G. Johnson) Date: Sat, 09 Sep 2006 09:50:58 -0700 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> Andrew Jaffe wrote: > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? As an author of FFTW, let me interject a couple of points into this discussion. First, if you are using FFTW, then its real-input r2c routines "natively" produce output in the "unpacked" numpy format as described above: an array of n/2+1 complex numbers. Any "packed" format would require some data permutations. Other FFT implementations use a variety of formats. Second, the *reason* why FFTW's r2c routines produce unpacked output is largely because "packed" formats do not generalize well to multi-dimensional FFTs, while the "unpacked" format does. (Packed formats are *possible* for multidimensional transforms, but become increasingly intricate as you add more dimensions.) Additionally, I personally find the unpacked format more convenient in most applications. (FFTW does actually support a different, "packed" format in its r2r interface, but only for 1d FFTs. The reason for this has do to with advantages of that format for odd sizes. We recommend that most users employ the r2c interface, however; our r2c interface is generally faster for even sizes.) I hope this is helpful. Cordially, Steven G. Johnson From tzdorsgmsb at esmas.com Sat Sep 9 18:11:21 2006 From: tzdorsgmsb at esmas.com (whieofvv llolkgcls) Date: Sat, 09 Sep 2006 23:11:21 +0100 Subject: [Numpy-discussion] Your PR man AETR Message-ID: <0429640572.fNGGRG-46475387-3458@esmas.com> W A T C H O U T! Here comes the big one! All signs show that AETR is going to Explode! ALLIANCE ENTERPRISE (AETR) Current Price 0.50 Add this gem to your watch list, and watch it trad closely! NEWS RELEASE TAECORP ANNOUNCES BREAKTHROUGH IN REMOVING DEADLY LANDMINES. TaeCorp Announces That NASA Technology Will Be a Key Contributor to Resolving World's Landmine Dilemma Sept. 6, 2006--The Alliance Enterprise Corporation ("TaeCorp") is pleased to announce that NASA technology will be the key component in keeping our landmine locator (LML) system operational year round, especially in inclement weather. TaeCorp and Ice Management have agreed that Ice Management will customize their de-icer system exclusively for TaeCorp's LML air vehicles that are to be deployed for landmine removal and home security MILL VALLEY, CALIFORNIA August 25, 2006 - The Alliance Enterprise Corporation announced today a BREAKTHROUGH in developing an Aerial Landmine System aimed at locating, detecting and mapping deadly landmines. More than 100 million landmines in 83 countries are holding international communities and industries hostage, preventing the investment in and development of productive lands and the re-building of infrastructure. A broad variety of landmines have been scattered over productive areas effectively crippling the economy and disabling thousands of children and adults. There are no reliable records that accurately show where these devastating landmines lie in wait for their victims. With the present day costs to clear a single land mine ranging between ,000 to ,500, solving the problem of de-mining lands will reach billions of dollars. TaeCorp has developed a technology based, cost effective solution to this problem using its three tiered approach to scanning, mapping and removing landmines. TaeCorp's System will provide many social and economic benefits to countries and their industries including oil and gas, mining, agriculture, roads and infrastructure development. About TaeCorp. TaeCorp's vision is to be the recognized leader in providing Aerial Detection Systems including global de-mining, clearing a path to a safer planet for all humankind. TaeCorp's mission is to reclaim lands around the globe embedded with landmines that victimize countries and their stakeholders. Watch AETR like a HAWK come Monday You will be convinced if you see ----------------------- What goes up must come down. Stop and smell the roses. Your barking up the wrong tree. Season of mists and mellow fruitfulness. You can't squeeze blood out of a turnip. A rolling stone gathers no moss. Through the grapevine. Say it with flowers. What goes up must come down. Plant kindness and gather love. You never miss the water till the well runs dry. When the cows come home. Water under the bridge. Tools of the trade. Two peas in a pod. Sitting on the fence. Put to bed with a shovel. We hung them out to dry. When you get lemons, make lemonade.(When life gives you scraps make quilts.) Rare as walking on water. A tree does not move unless there is wind. Slow as molasses in January. We'll cross that bridge when we come to it. Sweet as apple pie. There is always next year. Stone cold sober. Raking in the dough. Waking up with the chickens. A weed is no more than a flower in disguise. That's water under the bridge. Stop, look and listen. Spring to mind. Tastes like chicken. Tools of the trade. A snail's pace. Plant kindness and gather love. Still water runs dirty and deep. What goes down usually comes up. You have to separate the chaff from the wheat. Thick as a brick. When it rains it pours. Put off the scent. Walking on water. Take time to smell the roses. Schools out for summer. When pigs fly. From a.h.jaffe at bakerjaffe.plus.com Mon Sep 11 11:02:08 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 11 Sep 2006 16:02:08 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> References: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> Message-ID: <45057A70.2040709@bakerjaffe.plus.com> Steven G. Johnson wrote: > Andrew Jaffe wrote: >> numpy returns n/2+1 complex numbers (so the first and last numbers are >> actually real) with the frequencies equivalent to the positive part of >> the fftfreq, whereas scipy returns n real numbers with the frequencies >> as in rfftfreq (i.e., two real numbers at the same frequency, except for >> the highest and lowest) [All of the above for even n; but the difference >> between numpy and scipy remains for odd n.] >> >> I think the numpy behavior makes more sense, as it doesn't require any >> unpacking after the fact, at the expense of a tiny amount of wasted >> space. But would this in fact require scipy doing extra work from >> whatever the 'native' real_fft (fftw, I assume) produces? > > As an author of FFTW, let me interject a couple of points into this > discussion. > > First, if you are using FFTW, then its real-input r2c routines > "natively" produce output in the "unpacked" numpy format as described > above: an array of n/2+1 complex numbers. Any "packed" format would > require some data permutations. Other FFT implementations use a > variety of formats. > > Second, the *reason* why FFTW's r2c routines produce unpacked output is > largely because "packed" formats do not generalize well to > multi-dimensional FFTs, while the "unpacked" format does. (Packed > formats are *possible* for multidimensional transforms, but become > increasingly intricate as you add more dimensions.) Additionally, I > personally find the unpacked format more convenient in most > applications. > > I hope this is helpful. OK -- so it appears that all (three) of the votes so far are in support of the numpy convention -- a complex result. If this is something that can be done in pure python, I'm willing to give it a stab, but I'm probably not capable of handling any python/C issues. Does anyone out there understand the interaction between fftpack (C & python?), fftw (C), scipy and numpy in this context well enough to give some advice? Yours, Andrew From listservs at mac.com Mon Sep 11 11:45:08 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 11 Sep 2006 11:45:08 -0400 Subject: [Numpy-discussion] sum returns grand total? Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I must have missed something -- since when does summing a multidimensional array return the grand total? I thought that it should sum over axis zero by default (at least, thats what the docs say). Here's what I am getting: (Pdb) pre Out[6]: array([[[ 13, 3, 1, 33, 52], [ 1, 0, 0, 0, 2], [ 4, 0, 0, 2, 6], [ 15, 1, 0, 20, 54]], [[380, 101, 9, 267, 321], [ 20, 4, 0, 7, 16], [ 31, 3, 1, 31, 22], [314, 27, 4, 276, 391]]], dtype=int64) (Pdb) sum(pre) Out[6]: 2432L Is this correct behaviour? - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFBYSJGKaDpEGshJ0RAiPhAJ0d/MCGMciLgB2ZOwXf1FVj26NpTgCdG7h8 M83+mZsYGP2svVBGn8RtbNw= =mby0 -----END PGP SIGNATURE----- From oliphant at ee.byu.edu Mon Sep 11 11:47:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 11 Sep 2006 09:47:18 -0600 Subject: [Numpy-discussion] sum returns grand total? In-Reply-To: References: Message-ID: <45058506.3060403@ee.byu.edu> listservs at mac.com wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >I must have missed something -- since when does summing a >multidimensional array return the grand total? I thought that it >should sum over axis zero by default (at least, thats what the docs >say). Here's what I am getting: > >(Pdb) pre >Out[6]: >array([[[ 13, 3, 1, 33, 52], > [ 1, 0, 0, 0, 2], > [ 4, 0, 0, 2, 6], > [ 15, 1, 0, 20, 54]], > > [[380, 101, 9, 267, 321], > [ 20, 4, 0, 7, 16], > [ 31, 3, 1, 31, 22], > [314, 27, 4, 276, 391]]], dtype=int64) >(Pdb) sum(pre) >Out[6]: 2432L > >Is this correct behaviour? > > Yes, you need to specify which axis you want to sum over if you have a multi-dimensional array. This is the "basic" behavior of most functions in numpy unless you import them from oldnumeric. See the fix_default_axis.py file for help on converting code that worked for numpy 0.9.8 to 1.0 where this switch was made. -Travis From aisaac at american.edu Mon Sep 11 12:21:29 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 11 Sep 2006 12:21:29 -0400 Subject: [Numpy-discussion] sum returns grand total? In-Reply-To: <45058506.3060403@ee.byu.edu> References: <45058506.3060403@ee.byu.edu> Message-ID: On Mon, 11 Sep 2006, Travis Oliphant apparently wrote: > Yes, you need to specify which axis you want to sum over > if you have a multi-dimensional array. This is the > "basic" behavior of most functions And a good behavior indeed! Cheers, Alan Isaac From rex at nosyntax.com Mon Sep 11 16:38:33 2006 From: rex at nosyntax.com (rex) Date: Mon, 11 Sep 2006 13:38:33 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> <20060908001422.GJ5025@x2.nosyntax.com> Message-ID: <20060911203833.GE5064@x2.nosyntax.com> Robert Kern [2006-09-08 06:51]: > rex wrote: > > Robert Kern [2006-09-07 16:35]: > >> rex wrote: > >>> This exposed inconsistent randint() behavior between SciPy and the Python > >>> random module. The Python randint includes the upper endpoint. The SciPy > >>> version excludes it. > > > > I'm not in a position to argue the merits, but IMHO, when code that > > previously worked silently starts returning subtly bad results after > > importing numpy, there is a problem. What possible upside is there in > > having randint() behave one way in the random module and silently behave > > differently in numpy? > > I don't understand you. That's because I wasn't making any sense. :( > Importing numpy does not change the standard library's > random module in any way. There is no silent difference in behavior. If you use > numpy.random you get one set of behavior. If you use random, you get another. > Pick the one you want. They're not interchangeable, and nothing suggests that > they ought to be. Of course you're right. I thought the name would be overwritten, and it isn't. Sorry for wasting your time. :( Thanks, -rex From dgssgsd at tom.com Mon Sep 11 16:58:31 2006 From: dgssgsd at tom.com (=?GB2312?B?1cXPyMn6?=) Date: Tue, 12 Sep 2006 04:58:31 +0800 Subject: [Numpy-discussion] =?GB2312?B?tPq/qreixrE=?= Message-ID: ????????(??/??)??? ????????????????????????????????.???? ?????????????????6%?????????1-1.5%????????? ???????????????? ???????????????????????????????????? ??????????????????? ? ?????13590116835 ??????? E- MAIL ?szhw1359011 at 126.com ? ??????????????? ????????????????????????? ? ?? From michael.sorich at gmail.com Tue Sep 12 03:38:55 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue, 12 Sep 2006 17:08:55 +0930 Subject: [Numpy-discussion] creating a masked array with a sequence containing a masked scalar array Message-ID: <16761e100609120038m2aa21600qbea2c71d571cc9e9@mail.gmail.com> It seems that ma.array does not accept a sequence which contains a masked scalar array (not sure if this is the correct term). Is this deliberate? #numpy 1.0b5, python 2.3.x, winXP import numpy as N print N.ma.array([1,2,3, N.ma.array(1, mask=True)]) Traceback (most recent call last): File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_comm.py", line 529, in doIt result = pydevd_vars.evaluateExpression( self.thread_id, self.frame_id, self.expression, self.doExec ) File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_vars.py", line 258, in evaluateExpression exec expression in frame.f_globals, frame.f_locals File "", line 1, in ? File "C:\Python23\Lib\site-packages\numpy\core\ma.py", line 562, in __init__ c = numeric.array(data, dtype=tc, copy=True, order=order) TypeError: an integer is required #works fine if scalar masked array is not masked print N.ma.array([1,2,3, N.ma.array(1)]) [1 2 3 1] From thibault at physics.cornell.edu Tue Sep 12 09:52:03 2006 From: thibault at physics.cornell.edu (Pierre Thibault) Date: Tue, 12 Sep 2006 09:52:03 -0400 Subject: [Numpy-discussion] In-place operations Message-ID: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Hi, I would like to have information on the best techniques to do in-place calculations and to minimize temporary array creations. To me this seems to be very important whenever the arrays become very large. I already know about the ufunc in-place functionality (which is great). More specifically, here are examples that occured in my code 1) FFTs: Let A and B be two large arrays, already allocated. I want the fft of A to be stored in B. If I just type B = fft(A), there is a temprary array creation, right? Is it possible to avoid that? 2) Function output: In general, I think the same thing happens with functions like def f1(array_in): array_out = # something using array_in return array_out Then, if B is already allocated, writing B = f1(A) involves again a temporary array creation I thought instead of doing something like def f2(array_in, array_out): array_out[:] = # something # Is this good practice? and call f2(A,B). If I understand well, this still requires a temporary array creation. Is there another way of doing that (appart from actually looping through the indices of A and B)? I know that the use of f2 has one clear advantage: it makes sure that whatever was in B is discarded. With f1, the following could happen: A # contains something B # contains something C = B B = f1(A) C still contains whatever was in B. This could be what you wanted, but if you consider C just as a reference to B, this is not good. I guess these considerations are not standard python problems because you expect python to take care of memory issues. With big arrays in scientific computations, I feel the question is more relevant. I might be wrong... Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From jstrunk at enthought.com Tue Sep 12 10:26:51 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 09:26:51 -0500 Subject: [Numpy-discussion] REMINDER: ISP changeover today 9/12 7:00 PM Central Message-ID: <200609120926.51762.jstrunk@enthought.com> Good morning, This is a reminder for this evening's network maintenance. Unfortunately, our recent change in internet service providers is not working out. We will be switching to a more reliable provider tonight at 7:00 PM Central. Please allow for up to two hours of downtime. I will send an email announcing the start and completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. Please pass this message along to people that I have missed. If you have any questions, please direct them to me. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From faltet at carabos.com Tue Sep 12 10:51:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 12 Sep 2006 16:51:03 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: <1158072663.3973.21.camel@localhost.localdomain> Hello Pierre, El dt 12 de 09 del 2006 a les 09:52 -0400, en/na Pierre Thibault va escriure: > Hi, > > I would like to have information on the best techniques to do in-place > calculations and to minimize temporary array creations. To me this > seems to be very important whenever the arrays become very large. > > I already know about the ufunc in-place functionality (which is great). > > More specifically, here are examples that occured in my code > > 1) FFTs: Let A and B be two large arrays, already allocated. I want > the fft of A to be stored in B. If I just type B = fft(A), there is a > temprary array creation, right? Is it possible to avoid that? Well, in some way, there is a temporary array creation that is immediately bound to B, so in the end, the temporary is not so temporary, but a new (bounded) object. Obviously, the object that was referencing B is freed (unless there is another reference to it). > > 2) Function output: In general, I think the same thing happens with > functions like > > def f1(array_in): > array_out = # something using array_in > return array_out > > Then, if B is already allocated, writing B = f1(A) involves again a > temporary array creation Again, it depends what do you understand by 'temporary'. > > I thought instead of doing something like > > def f2(array_in, array_out): > array_out[:] = # something > # Is this good practice? > > and call f2(A,B). > > If I understand well, this still requires a temporary array creation. > Is there another way of doing that (appart from actually looping > through the indices of A and B)? Here I'd say that yes, you are creating a truly temporary object and then assign element by element to B. > > I know that the use of f2 has one clear advantage: it makes sure that > whatever was in B is discarded. With f1, the following could happen: > > A # contains something > B # contains something > C = B > B = f1(A) > > C still contains whatever was in B. This could be what you wanted, but > if you consider C just as a reference to B, this is not good. This is not good if you want to get rid of the object pointed by B, but in general, this is considered a nice feature of python. > > I guess these considerations are not standard python problems because > you expect python to take care of memory issues. With big arrays in > scientific computations, I feel the question is more relevant. I might > be wrong... If you are worried about wasting memory, just get familiar with this rule: an object only exists in memory when it is referenced (bounded) by a variable. When the object is no longer referenced, its memory becomes freed and is available to the system for later reuse. With this, B = fft(A) will create a new object (the FFT of A), the object pointed by B will be freed (if there is not any other reference to it) and the new object will be bound to B. If what you want is to avoid having in memory the three objects (namely A, old B and new B) at the same time, you can do something like: del B # deletes reference to object pointed by B B = fft(A) # B gets bounded to new FFT object HTH, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From dd55 at cornell.edu Tue Sep 12 12:26:10 2006 From: dd55 at cornell.edu (Darren Dale) Date: Tue, 12 Sep 2006 12:26:10 -0400 Subject: [Numpy-discussion] numpy.distutils compatibility with subversion-1.4 Message-ID: <200609121226.10259.dd55@cornell.edu> This morning I updated to subversion-1.4. There have been several significant enhancements in this release (see http://subversion.tigris.org/svn_1.4_releasenotes.html), and it appears that numpy.distutils is not compatible with working copies created with the new svn. I submitted a ticket (#276), but I figured it was worth a post here, just a "heads up" for anyone not watching the trac website. $ python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' No module named __svn_version__ Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/home/darren/src/numpy/numpy/distutils/core.py", line 144, in setup config = configuration() File "setup.py", line 48, in configuration config.add_subpackage('numpy') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in add_subpackage caller_level = 2) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in get_subpackage caller_level = caller_level + 1) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "./numpy/setup.py", line 8, in configuration config.add_subpackage('f2py') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in add_subpackage caller_level = 2) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in get_subpackage caller_level = caller_level + 1) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/f2py/setup.py", line 39, in configuration config.make_svn_version_py() File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1298, in make_svn_version_py self.add_data_files(('', generate_svn_version_py())) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1281, in generate_svn_version_py assert revision is not None,'hmm, why I am not inside SVN tree???' AssertionError: hmm, why I am not inside SVN tree??? Darren From thibault at physics.cornell.edu Tue Sep 12 13:17:22 2006 From: thibault at physics.cornell.edu (Pierre Thibault) Date: Tue, 12 Sep 2006 13:17:22 -0400 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158072663.3973.21.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> Message-ID: <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> Hello again, On 9/12/06, Francesc Altet wrote: > Hello Pierre, > [...] > > Well, in some way, there is a temporary array creation that is > immediately bound to B, so in the end, the temporary is not so > temporary, but a new (bounded) object. Obviously, the object that was > referencing B is freed (unless there is another reference to it). ok, I guess I was aware of all that. My worries are related to two cases: 1) When the mere creation of a new array is prohibitive. 2) When what I really want to change is the _content_ of an array. Using assignment (=) disconnects B from the array it refers to, so that what used to be true (C=B) is not anymore. I understant that B[:] = ... solves the problem 2), though I don't know if this notation is recommended; but I would like to know if there is anyway to solve 1), in the way ufuncs can do. I had fun using kde's "performance monitor" (ksysguard) to see the difference between a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) a = numpy.exp(a) and a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) numpy.exp(a,out=a) Not only is the latter faster, but I could see a large glitch in the memory usage for the former. Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From peridot.faceted at gmail.com Tue Sep 12 13:19:05 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 12 Sep 2006 13:19:05 -0400 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: On 12/09/06, Pierre Thibault wrote: > I would like to have information on the best techniques to do in-place > calculations and to minimize temporary array creations. To me this > seems to be very important whenever the arrays become very large. The first rule of optimization: don't do it yet. You can usually go through and banish temporary arrays (using ufuncs and so on) at the cost of readability, code encapsulation, and thread-safe-ness. But it may not do what you want. I had an image-processing code that was taking longer than I thought it should and using two hundred megabytes or so of RAM. So I rewrote it, with a certain amount of pain, in a way that it used the fewest possible temporary arrays. It didn't run any faster, and it then took five hundred megabytes. Because all the arrays ended up being in memory at once, the memory footprint increased drastically. malloc() is fast, typically just a handful of instructions; if you're allocating a giant array, it's almost certainly being allocated using mmap(), and it can be released back to the OS on deallocation. But you probably still want to avoid temporary arrays. So: > More specifically, here are examples that occured in my code > > 1) FFTs: Let A and B be two large arrays, already allocated. I want > the fft of A to be stored in B. If I just type B = fft(A), there is a > temprary array creation, right? Is it possible to avoid that? Doing an FFT in-place is a major challenge, and involves its own slowdowns, so generally high-level toolkits don't bother. But fft seems to be like many functions (those generated by interp1d, for example) that insist on malloc()ing their own arrays to return. Short of rooting around in the numpy/scipy code, there's no real way around this for such functions. The best you can do is make actual use of the allocated array (rather than copy its contents to *another* array and discard it). > 2) Function output: In general, I think the same thing happens with > functions like > > def f1(array_in): > array_out = # something using array_in > return array_out > > Then, if B is already allocated, writing B = f1(A) involves again a > temporary array creation Uh, no, not really. The way you have written f1, it probably malloc()s space for array_out. the address of that space (roughly) is saved in the array_out variable. If you write B=f1(A), you are just storing the address in B. The memory is not copied. Even if you do B=f1(A)[::10] you don't copy the memory. > I thought instead of doing something like > > def f2(array_in, array_out): > array_out[:] = # something > # Is this good practice? > > and call f2(A,B). This is a ufunc-like solution; you could even make array_out an optional argument, and return it. > If I understand well, this still requires a temporary array creation. > Is there another way of doing that (appart from actually looping > through the indices of A and B)? It depends what #something is. If, say, it is 2*array_in, you can simply do multiply(array_in,2,array_out) to avoid any dynamic allocation. > I guess these considerations are not standard python problems because > you expect python to take care of memory issues. With big arrays in > scientific computations, I feel the question is more relevant. I might > be wrong... Some of these issues come up when dealing with mutable objects (lists, dictionaries, and so on). Some of them (the fact that python variables contain simply references) are discussed in various python FAQs. A. M. Archibald From faltet at carabos.com Tue Sep 12 13:49:28 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 12 Sep 2006 19:49:28 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> Message-ID: <1158083368.3973.34.camel@localhost.localdomain> El dt 12 de 09 del 2006 a les 13:17 -0400, en/na Pierre Thibault va escriure: > Hello again, > > On 9/12/06, Francesc Altet wrote: > > Hello Pierre, > > [...] > > > > Well, in some way, there is a temporary array creation that is > > immediately bound to B, so in the end, the temporary is not so > > temporary, but a new (bounded) object. Obviously, the object that was > > referencing B is freed (unless there is another reference to it). > > ok, I guess I was aware of all that. My worries are related to two cases: > 1) When the mere creation of a new array is prohibitive. As Archibald said in other message, creation of a big array is not an issue (malloc is very fast) and indepent of the size: >>> Timer("a=numpy.array(100,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.19819307327270508, 0.14915895462036133, 0.14999985694885254] >>> Timer("a=numpy.array(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.15171599388122559, 0.14998698234558105, 0.14901280403137207] that is 15 us (in my old machine) irregardingly of the size. [BTW, numpy.empty seems twice as slower in my machine. Why? >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.37033700942993164, 0.31780219078063965, 0.31607294082641602] ] > 2) When what I really want to change is the _content_ of an array. > Using assignment (=) disconnects B from the array it refers to, so > that what used to be true (C=B) is not anymore. > > I understant that B[:] = ... solves the problem 2), though I don't > know if this notation is recommended; but I would like to know if > there is anyway to solve 1), in the way ufuncs can do. > > I had fun using kde's "performance monitor" (ksysguard) to see the > difference between > > a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) > a = numpy.exp(a) > > and > > a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) > numpy.exp(a,out=a) > > Not only is the latter faster, but I could see a large glitch in the > memory usage for the former. Yes, it seems that some ufuncs have an additional "out" parameter that I was not aware of. Well, it that case, this parameter in the fft function would solve your needs, although I don't know how complicated would be this. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Sep 12 15:28:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 12 Sep 2006 13:28:01 -0600 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158083368.3973.34.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> Message-ID: <45070A41.90001@ee.byu.edu> Francesc Altet wrote: >>>>Timer("a=numpy.array(100,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.19819307327270508, 0.14915895462036133, 0.14999985694885254] > > >>>>Timer("a=numpy.array(10000,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.15171599388122559, 0.14998698234558105, 0.14901280403137207] > >that is 15 us (in my old machine) irregardingly of the size. > > Ummm.. You are not creating empty arrays here. You are creating a 0-d array with a single entry. >[BTW, numpy.empty seems twice as slower in my machine. Why? > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] >] > > Now, you are creating an empty array with 10000 elements in it. Best, -Travis From haase at msg.ucsf.edu Tue Sep 12 16:33:21 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 12 Sep 2006 13:33:21 -0700 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? Message-ID: <200609121333.21254.haase@msg.ucsf.edu> Hi ! I have a nice ndarray image viewer built on OpenGL - one annoyance though is that is crashes if the array contains any NaN, or inf, ... So, I found N.nan_to_num - but OpenGL (read: video cards) supports only single precision float (N.float32) So I get this: >>> N.nan_to_num([N.inf]) [ 1.79769313e+308] >>> N.nan_to_num([N.inf]).astype(N.float32) [ inf] Could nan_to_num() get an optional dtype argument ? Thanks, Sebastian Haase From haase at msg.ucsf.edu Tue Sep 12 16:39:54 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 12 Sep 2006 13:39:54 -0700 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? In-Reply-To: <200609121333.21254.haase@msg.ucsf.edu> References: <200609121333.21254.haase@msg.ucsf.edu> Message-ID: <200609121339.54126.haase@msg.ucsf.edu> On Tuesday 12 September 2006 13:33, Sebastian Haase wrote: > Hi ! > I have a nice ndarray image viewer built on OpenGL - one annoyance though > is that is crashes if the array contains any NaN, or inf, ... > > So, I found N.nan_to_num - but OpenGL (read: video cards) supports only > single precision float (N.float32) > > So I get this: > >>> N.nan_to_num([N.inf]) > > [ 1.79769313e+308] > > >>> N.nan_to_num([N.inf]).astype(N.float32) > > [ inf] > > Could nan_to_num() get an optional dtype argument ? > > Thanks, > Sebastian Haase OK - sorry for the noise... this is how to do it: >>> N.nan_to_num( N.array([N.nan,N.inf,-N.inf],N.float32) ) [ 0.00000000e+00 3.40282347e+38 -3.40282347e+38] In the meantime I noticed that there is a N.asarray_chkfinite() function but no equivalent for N.asanyarray() ! Should N.asanyarray_chkfinite() be added !? -Sebastian From jstrunk at enthought.com Tue Sep 12 19:30:03 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 18:30:03 -0500 Subject: [Numpy-discussion] ISP changeover beginning in 30 minutes (7:00 PM Central) Message-ID: <200609121830.03566.jstrunk@enthought.com> Good evening, We will begin switching our internet service over in about 30 minutes. Please allow for up to two hours of downtime. I will send an email announcing the completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From aisaac at american.edu Tue Sep 12 20:51:19 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 12 Sep 2006 20:51:19 -0400 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? In-Reply-To: <200609121333.21254.haase@msg.ucsf.edu> References: <200609121333.21254.haase@msg.ucsf.edu> Message-ID: On Tue, 12 Sep 2006, Sebastian Haase apparently wrote: > I have a nice ndarray image viewer built on OpenGL Please post (with license). Thanks! Alan Isaac From jstrunk at enthought.com Tue Sep 12 21:11:58 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 20:11:58 -0500 Subject: [Numpy-discussion] ISP Changeover complete Message-ID: <200609122011.58960.jstrunk@enthought.com> Good evening, We have completed our internet service switchover. Thank you for your patience. If you have any trouble, please let me know. Thank you, Jeff Strunk IT Administrator Enthought, Inc. From faltet at carabos.com Wed Sep 13 03:15:49 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 13 Sep 2006 09:15:49 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <45070A41.90001@ee.byu.edu> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> Message-ID: <1158131750.3968.36.camel@localhost.localdomain> El dt 12 de 09 del 2006 a les 13:28 -0600, en/na Travis Oliphant va escriure: > >[BTW, numpy.empty seems twice as slower in my machine. Why? > > > > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > >>>> > >>>> > >numpy").repeat(3,10000) > >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] > >] > > > > > Now, you are creating an empty array with 10000 elements in it. Ups, my bad. So, here are the correct times for array creation: >>> Timer("a=numpy.empty(10,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.083303928375244141, 0.080381870269775391, 0.077172040939331055] >>> Timer("a=numpy.empty(100,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.086454868316650391, 0.084085941314697266, 0.083555936813354492] >>> Timer("a=numpy.empty(1000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.084996223449707031, 0.082299947738647461, 0.081347942352294922] >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.31068897247314453, 0.30376386642456055, 0.30176281929016113] >>> Timer("a=numpy.empty(100000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.42552995681762695, 0.36864185333251953, 0.36870002746582031] >>> Timer("a=numpy.empty(1000000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.48045611381530762, 0.41251182556152344, 0.40645909309387207] So, it seems that there are a certain time dependency with size array of 10 elements --> 7.7 us array of 100 elements --> 8.4 us array of 1000 elements --> 8.1 us array of 10000 elements --> 30.2 us array of 100000 elements --> 36.9 us array of 1000000 elements --> 40.6 us Well, it seems that malloc actually takes more time when asking for more space. However, this can't be the reason why Pierre is seeing that: a = numpy.exp(a) [1] is slower than numpy.exp(a,out=a) [2] as I'd say that this increment in time is negligible compared with processing times of those big arrays. In fact, here are my times: >>> Timer("a = numpy.exp(a)", "import numpy;a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)").repeat(3,1) [2.5527338981628418, 2.5427830219268799, 2.5074479579925537] >>> Timer("numpy.exp(a,out=a)", "import numpy;a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)").repeat(3,1) [2.5298278331756592, 2.5082788467407227, 2.5222280025482178] So, both times are comparable. Perhaps what Pierre is seeing is that he is approaching the limits of memory in his system and because [1] takes more memory than [2] (two objects in memory instead of one) perhaps the former is causing the OS to start swapping. However a quick look with top at the processes, says that both [1] and [2] takes similar amounts of memory (~ 170 MB peak) and, as arrays take 64 MB each, in both cases the used memory seems higher than the required at first sight. Mmmm, the only explanation is that the exp() ufunc does require temporaries, although this is a bit strange as exp() works element wise. I recognize that I'm a bit lost here... -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From a.u.r.e.l.i.a.n at gmx.net Wed Sep 13 04:23:50 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 13 Sep 2006 10:23:50 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: <200609131023.50534.a.u.r.e.l.i.a.n@gmx.net> Hi, one word in advance, instead of optimizing it is advisable to seek for a way to refactorize the algorithm using smaller arrays, since this kind of optimization almost certainly reduces readability. If you do it, comment well. ;-) If you have very large arrays and want to do some arithmetics on it, say B = 2*B + C you can use inplace operators to avoid memory overhead: B *= 2 B += C Another trick which works in most situations is to do the outermost loop in python: for i in xrange(len(B)): B[i] = 2*B[i] + C[i] This reduces the temporary array size to 1/len(B) while still being fast (if the other dimensions are large enough). For very large 1d arrays, you could split them into chunks of a certain size. However, you have to be careful that your calculation does not access already-calculated elements of B. Consider the following example: In [2]: B=arange(10) In [3]: B+B[::-1] Out[3]: array([9, 9, 9, 9, 9, 9, 9, 9, 9, 9]) In [4]: B += B[::-1] In [5]: B Out[5]: array([ 9, 9, 9, 9, 9, 14, 15, 16, 17, 18]) Johannes From zeldey at i-enternet.com Wed Sep 13 07:08:55 2006 From: zeldey at i-enternet.com (Paxton Pereda) Date: Wed, 13 Sep 2006 04:08:55 -0700 Subject: [Numpy-discussion] PHAlxcRMA Message-ID: <000001c6d725$00f3d760$790ea8c0@fnue> Hi QUIT OVE t RPA e YIN a G FOR YOU e R P o HAR z MAC j Y S u AV k E u p p to 50 w % wi v th http://www.prlawoec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From klemm at phys.ethz.ch Wed Sep 13 11:09:40 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Wed, 13 Sep 2006 17:09:40 +0200 Subject: [Numpy-discussion] numpy-1.0b5 can't find ATLAS Message-ID: Hello, when I try to build numpy-1.0b5 it doesn't find my ATLAS libraries. I had to build hem from scratch (version 3.7.17) and the compile went well. I have installed them under /scratch/python2.4/atlas furthermore, I have blas and lapack installed under /scatch/python2.4/lib I adjusted the environment variables as: BLAS=/scratch/python2.4/lib/libfblas.a LAPACK=/scratch/python2.4/lib/libflapack.a ATLAS=/scratch/python2.4/atlas and my site.cfg looks like [atlas] library_dirs = /scratch/python2.4/atlas/lib atlas_libs = lapack, blas, cblas, atlas include_dirs = /scratch/python2.4/atlas/include python setup.py config then finds the blas and lapack libraries under /scratch/python2.4/lib but does not find the atlas libraries. What am I doing wrong here? Hanno P.S.: The output of python setup.py config reads: atlas_threads_info: Setting PTATLAS=ATLAS libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include libraries lapack_atlas not found in /scratch/python2.4/atlas/include libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/lib libraries lapack_atlas not found in /scratch/python2.4/atlas/lib libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/lib libraries lapack_atlas not found in /scratch/python2.4/lib libraries lapack,blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack,blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include libraries lapack_atlas not found in /scratch/python2.4/atlas/include libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/lib libraries lapack_atlas not found in /scratch/python2.4/atlas/lib libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/lib libraries lapack_atlas not found in /scratch/python2.4/lib libraries lapack,blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack,blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /scratch/src/numpy-1.0b5/numpy/distutils/system_info.py:1205: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/scratch/python2.4/lib'] language = f77 FOUND: libraries = ['lapack', 'fblas'] library_dirs = ['/scratch/python2.4/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running config -- Hanno Klemm klemm at phys.ethz.ch From rng7 at cornell.edu Wed Sep 13 14:32:29 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Wed, 13 Sep 2006 14:32:29 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? Message-ID: <45084EBD.5050205@cornell.edu> Hi all, I'm migrating an application from Numeric to numpy, and I've run into a significant application slowdown related to arithmetic on array-scalars. The inner loop of the application is integrating a nonlinear set of differential equations using odeint, with the rhs a dynamically-generated (only once) python function. In that function I copy the entries of the current x array to a bunch of local variables, do a bunch of arithmetic, and assign the results to a dx_dt array. The arithmetic is approximately 3x slower using numpy than Numeric, because numpy returns array-scalars while Numeric returns normal scalars. (Simple example below.) I can wrap all my arrays accesses with float() casts, but that introduces a noticable overhead (~50% for problems of interest). I'm guessing speeding up the scalar-array math would be difficult, if not impossible. (Maybe I'm wrong?) I notice that numpy_array.item() will give me the first element as a normal scalar. Would it be possible for numpy_array.item(N) to return the Nth element of the array as a normal scalar? Thanks a bunch, Ryan The effect can be isolated as (running in python 2.4 on a 32-bit Athlon): In [1]: import Numeric, numpy In [2]: a_old, a_new = Numeric.array([1.0, 2.0]), numpy.array([1.0, 2.0]) In [3]: b_old, b_new = a_old[0], a_new[0] In [4]: %time for ii in xrange(1000000):c = b_old + 1.0 CPU times: user 0.40 s, sys: 0.00 s, total: 0.40 s Wall time: 0.40 In [5]: %time for ii in xrange(1000000):c = b_new + 1.0 CPU times: user 1.20 s, sys: 0.00 s, total: 1.20 s Wall time: 1.22 In [6]: Numeric.__version__, numpy.__version__ Out[6]: ('24.2', '1.0b5') -- Ryan Gutenkunst | Cornell LASSP | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)227-7914 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From oliphant.travis at ieee.org Wed Sep 13 15:07:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 13:07:03 -0600 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <45084EBD.5050205@cornell.edu> References: <45084EBD.5050205@cornell.edu> Message-ID: <450856D7.6040601@ieee.org> Ryan Gutenkunst wrote: > Hi all, > > I'm migrating an application from Numeric to numpy, and I've run into a > significant application slowdown related to arithmetic on array-scalars. > > Yeah, that is a common thing. There are two factors: 1) array indexing and 2) array scalar math. I don't think array scalar math has to be slower in principle then Python code. I think it's ability to handle interaction with multiple scalars that is operating more slowly right now. It could be sped up. The array indexing code is slower (in fact Numeric's indexing code is slower then just using lists also). > I'm guessing speeding up the scalar-array math would be difficult, if > not impossible. (Maybe I'm wrong?) > I think scalar-array math could be sped up quite a bit. I haven't done much in that area at all. Right now a lot of setup code is handled generically instead of type-specifically like it could be. > I notice that numpy_array.item() will give me the first element as a > normal scalar. Would it be possible for numpy_array.item(N) to return > the Nth element of the array as a normal scalar? > Now this is an interesting idea. It would allow you to by-pass the slow-indexing as well as the array scalar computation should you desire it. I like it and am going to add it unless there are convincing objections. -Travis From oliphant.travis at ieee.org Wed Sep 13 16:18:28 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 14:18:28 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend Message-ID: <45086794.8020704@ieee.org> I'd like to make the first release-candidate of NumPy 1.0 this weekend. Any additions wanting to make the first official release candidate should be checked in by then. -Travis From fullung at gmail.com Wed Sep 13 18:41:34 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 14 Sep 2006 00:41:34 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> Message-ID: Hello all I just ran NumPy and SciPy through Valgrind, and everything looks clean on that the NumPy side. Some other things that could be fixed for RC1: - GCC 4.1.1 warning in ufuncobject.c: numpy/core/src/ufuncobject.c: In function ?PyUFunc_RegisterLoopForType?: numpy/core/src/ufuncobject.c:3215: warning: "cmp" may be used uninitialized in this function - Time to kill the dft package? /usr/lib/python2.4/site-packages/numpy/dft/__init__.py:2: UserWarning: The dft subpackage will be removed by 1.0 final -- it is now called fft warnings.warn("The dft subpackage will be removed by 1.0 final -- it is now called fft") - Test failure ====================================================================== ERROR: check_instance_methods (numpy.core.tests.test_defmatrix.test_matrix_return) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/numpy/core/tests/test_defmatrix.py", line 166, in check_instance_methods b = f(*args) ValueError: setitem must have at least one argument Although not strictly NumPy issues, the following crops up when you run the SciPy test suite through Valgrind: Valgrind warnings when running scipy.integrate.tests.test_integrate.test_odeint http://projects.scipy.org/scipy/scipy/ticket/246 Valgrind warnings when running Cephes tests http://projects.scipy.org/scipy/scipy/ticket/247 Memory leak in fitpack http://projects.scipy.org/scipy/scipy/ticket/248 I think I've figured out #248. #246 should be relatively easy to fix. #247 is... interesting. Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Travis Oliphant > Sent: 13 September 2006 22:18 > To: numpy-discussion > Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend > > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis From haase at msg.ucsf.edu Wed Sep 13 18:39:38 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 13 Sep 2006 15:39:38 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: <200609131539.39147.haase@msg.ucsf.edu> Hi! I would like to hear about three tickets I submitted some time ago: Ticket #230 a**2 not executed as a*a if a.dtype = int32 is this easy to fix ? Ticket #229 numpy.random.poisson(0) should return 0 I hope there is agreement that the edge-case of 0 should/could be handled without raising an exception. I submitted a patch (please test first!) any comments on this one. Ticket #188 dtype should have nicer str representation Is this one now officially dead ? " I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Wed Sep 13 20:08:43 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 18:08:43 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <200609131539.39147.haase@msg.ucsf.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> Message-ID: <45089D8B.3070602@ieee.org> Sebastian Haase wrote: > Hi! > I would like to hear about three tickets I submitted some time ago: > > Ticket #230 a**2 not executed as a*a if a.dtype = int32 > is this easy to fix ? > > Fixed. Now, all arrays with a**2 are executed as a*a (float arrays are still executed as square(a) (is this needed)? > Ticket #229 numpy.random.poisson(0) should return 0 > I hope there is agreement that the edge-case of 0 should/could be handled > without raising an exception. I submitted a patch (please test first!) > any comments on this one. > Fixed. This seems reasonable to me. > Ticket #188 dtype should have nicer str representation > Is this one now officially dead ? > " but str() should rather return 'int32 (little endian)' > It's not necessarily dead, the problem is complexity of implementation and more clarity about how all dtypes are supposed to be printed, not just this particular example. A patch would be very helpful here. If desired it could be implemented in _internal.py and called from there in arrayobject.c But, to get you thinking... How should the following be printed dtype('c4') dtype('a4,i8,3f4') dtype('3f4') -Travis From matthew.brett at gmail.com Wed Sep 13 20:09:50 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 14 Sep 2006 01:09:50 +0100 Subject: [Numpy-discussion] iscomplex on strings Message-ID: <1e2af89e0609131709v3b57a005j9056112f2021858b@mail.gmail.com> Hi, I was surprised by this - but maybe I shouldn't have been: In [7]:iscomplex('a') Out[7]:True In [8]:iscomplex(u'a') Out[8]:True Best, Matthew From matthew.brett at gmail.com Wed Sep 13 20:13:52 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 14 Sep 2006 01:13:52 +0100 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Hi, > For example, if you do array([a,b,c]).shape(), the answer is normally > (3,) unless a b and c happen to all be lists of the same length, at > which point your array could have a much more complicated shape... but > as the person who wrote "array([a,b,c])" it's tempting to assume that > the result has shape (3,), only to discover subtle bugs much later. Very much agree with this. > If we were writing an array-creation function from scratch, would > there be any reason to include object-array creation in the same > function as uniform array creation? It seems like a bad idea to me. > > If not, the problem is just compatibility with Numeric. Why not simply > write a wrapper function in python that does Numeric-style guesswork, > and put it in the compatibility modules? How much code will actually > break? Can I encourage any more comments? This suggestion seems very sensible to me, and I guess this is our very last chance to change this. The current behavior does seem to violate least surprise - at least to my eye. Best, Matthew From charlesr.harris at gmail.com Wed Sep 13 21:07:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:07:39 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Message-ID: On 9/13/06, Matthew Brett wrote: > > Hi, > > > For example, if you do array([a,b,c]).shape(), the answer is normally > > (3,) unless a b and c happen to all be lists of the same length, at > > which point your array could have a much more complicated shape... but > > as the person who wrote "array([a,b,c])" it's tempting to assume that > > the result has shape (3,), only to discover subtle bugs much later. > > Very much agree with this. > > > If we were writing an array-creation function from scratch, would > > there be any reason to include object-array creation in the same > > function as uniform array creation? It seems like a bad idea to me. > > > > If not, the problem is just compatibility with Numeric. Why not simply > > write a wrapper function in python that does Numeric-style guesswork, > > and put it in the compatibility modules? How much code will actually > > break? > > Can I encourage any more comments? This suggestion seems very > sensible to me, and I guess this is our very last chance to change > this. The current behavior does seem to violate least surprise - at > least to my eye. I've been thinking about how to write a new constructor for objects. Because array has been at the base of numpy for many years I think it is too late to change it now, but perhaps a new and more predictable constructor for objects may eventually displace it. The main problem in constructing arrays of objects is more information needs to be supplied because the user's intention can't be reliably deduced from the current syntax. That said, I have no idea how widespread the use of object arrays is and so don't know how much it really matters. I don't use them much myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 13 21:18:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:18:39 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: On 9/13/06, Travis Oliphant wrote: > > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. There are a few cleanups and added functionality I have in mind but nothing that would affect the release. Do you plan to keep the 1.0 release as is with only added fixes and then make a 1.1 release not too long after that contains additions, or are you thinking that modifications that don't affect the API should all go into 1.0.x or some such? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 13 21:24:28 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:24:28 -0600 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158131750.3968.36.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> <1158131750.3968.36.camel@localhost.localdomain> Message-ID: On 9/13/06, Francesc Altet wrote: > > El dt 12 de 09 del 2006 a les 13:28 -0600, en/na Travis Oliphant va > escriure: > > >[BTW, numpy.empty seems twice as slower in my machine. Why? > > > > > > > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > > >>>> > > >>>> > > >numpy").repeat(3,10000) > > >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] > > >] > > > > > > > > Now, you are creating an empty array with 10000 elements in it. > > Ups, my bad. So, here are the correct times for array creation: > > >>> Timer("a=numpy.empty(10,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.083303928375244141, 0.080381870269775391, 0.077172040939331055] > >>> Timer("a=numpy.empty(100,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.086454868316650391, 0.084085941314697266, 0.083555936813354492] > >>> Timer("a=numpy.empty(1000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.084996223449707031, 0.082299947738647461, 0.081347942352294922] > >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.31068897247314453, 0.30376386642456055, 0.30176281929016113] > >>> Timer("a=numpy.empty(100000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.42552995681762695, 0.36864185333251953, 0.36870002746582031] > >>> Timer("a=numpy.empty(1000000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.48045611381530762, 0.41251182556152344, 0.40645909309387207] > > So, it seems that there are a certain time dependency with size > > array of 10 elements --> 7.7 us > array of 100 elements --> 8.4 us > array of 1000 elements --> 8.1 us > array of 10000 elements --> 30.2 us > array of 100000 elements --> 36.9 us > array of 1000000 elements --> 40.6 us The transition looks a bit like a cache effect, although I don't see why the cache should enter in. But all the allocations look pretty fast to me. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Sep 13 23:25:39 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 13 Sep 2006 20:25:39 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45089D8B.3070602@ieee.org> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> Message-ID: <4508CBB3.1070104@msg.ucsf.edu> Travis Oliphant wrote: >> Ticket #188 dtype should have nicer str representation >> Is this one now officially dead ? >> "> but str() should rather return 'int32 (little endian)' >> > It's not necessarily dead, the problem is complexity of implementation > and more clarity about how all dtypes are supposed to be printed, not > just this particular example. A patch would be very helpful here. If > desired it could be implemented in _internal.py and called from there in > arrayobject.c > > But, to get you thinking... How should the following be printed > > dtype('c4') > > dtype('a4,i8,3f4') > > dtype('3f4') > > > -Travis I would argue that if the simple cases were addressed first those would cover 90% (if not 99% for most people) of the cases occurring in people's daily use. For complex types (like 'a4,i8,3f4') I actually think the current text is compact and good. (Lateron one could think about 'c4' --> '4 chars' '3f4' --> '3 float32s' but already I don't know: is there any difference between 'c4' and '4c1'? What is the difference between 'c4' and 'a4' !? ) My main focus is on the fact that you might read ' References: <072.99fbb56cad8c4060e4cd98531f18384d@scipy.net> <081.27063a6fc6f370624d8d3a3c75880a1e@scipy.net> Message-ID: <4508CDF9.4010706@msg.ucsf.edu> Travis, what is the "new string directives as the first element of the item tuple" !? I always liked the idea of having a "shortest possible" way for creating (or concatenating) rows with "r_" *and* columns with "c_" ! Why did the "c_" have to be removed !? Thanks, Sebastan NumPy wrote: > #235: r_, c_, hstack, vstack, column_stack should be made more consistent > -------------------------+-------------------------------------------------- > Reporter: baxissimo | Owner: somebody > Type: enhancement | Status: closed > Priority: normal | Milestone: > Component: numpy.lib | Version: devel > Severity: normal | Resolution: fixed > Keywords: | > -------------------------+-------------------------------------------------- > Changes (by oliphant): > > * status: new => closed > * resolution: => fixed > > Comment: > > r_ is the only current quick-creator. You can now get the functionality > of all others using string directives as the first element of the item > tuple. > > Column stack was fixed. > From oliphant.travis at ieee.org Wed Sep 13 23:40:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:40:59 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: References: <45086794.8020704@ieee.org> Message-ID: <4508CF4B.3010904@ieee.org> Charles R Harris wrote: > > > On 9/13/06, *Travis Oliphant* > wrote: > > I'd like to make the first release-candidate of NumPy 1.0 this > weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > > There are a few cleanups and added functionality I have in mind but > nothing that would affect the release. Do you plan to keep the 1.0 > release as is with only added fixes and then make a 1.1 release not > too long after that contains additions, or are you thinking that > modifications that don't affect the API should all go into 1.0.x or > some such? The plan is for 1.0.x to contain modifications that don't affect the API (good additions should be O.K.). We want extensions compiled against 1.0.x to work for a long time. The 1.1 release won't be for at least a year and probably longer. 1.0.1 would be a maintenance release of the 1.0 release. -Travis From oliphant.travis at ieee.org Wed Sep 13 23:44:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:44:08 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <4508CBB3.1070104@msg.ucsf.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> Message-ID: <4508D008.8010305@ieee.org> Sebastian Haase wrote: > Travis Oliphant wrote: > > >> It's not necessarily dead, the problem is complexity of implementation >> and more clarity about how all dtypes are supposed to be printed, not >> just this particular example. A patch would be very helpful here. If >> desired it could be implemented in _internal.py and called from there in >> arrayobject.c >> >> But, to get you thinking... How should the following be printed >> >> dtype('c4') >> >> dtype('a4,i8,3f4') >> >> dtype('3f4') >> >> >> -Travis >> > > > I would argue that if the simple cases were addressed first those would > cover 90% (if not 99% for most people) of the cases occurring in > people's daily use. > For complex types (like 'a4,i8,3f4') I actually think the current text > is compact and good. > (Lateron one could think about > 'c4' --> '4 chars' > '3f4' --> '3 float32s' > > but already I don't know: is there any difference between 'c4' and > '4c1'? What is the difference between 'c4' and 'a4' !? > ) > > > My main focus is on the fact that you might read ' "less" than 4-bytes int, which is very confusing ! > I can agree it's confusing at first, but it's the same syntax the struct module uses which is the Python precedent for this. > As far as a patch is concerned: is _internal.py already being called now > from arrayobject.c for the str() and repr() methods ? And is there so > Yes, you can easily make a call to _internal.py from arrayobject.c (it's how some things are actually implemented). If you just provide a Python function to call for dtype.__str__ then that would suffice. > far any difference in str() and repr() ? > I assume that repr() has to stay exactly the way it is right now - right !? > > Yeah, the repr() probably needs to stay the same -Travis From oliphant.travis at ieee.org Wed Sep 13 23:56:22 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:56:22 -0600 Subject: [Numpy-discussion] [Numpy-tickets] [NumPy] #235: r_, c_, hstack, vstack, column_stack should be made more consistent In-Reply-To: <4508CDF9.4010706@msg.ucsf.edu> References: <072.99fbb56cad8c4060e4cd98531f18384d@scipy.net> <081.27063a6fc6f370624d8d3a3c75880a1e@scipy.net> <4508CDF9.4010706@msg.ucsf.edu> Message-ID: <4508D2E6.1020109@ieee.org> Sebastian Haase wrote: > Travis, > what is the "new string directives as the first element of the item > tuple" !? > These have been there for a while, but I recently added a couple of capabilities. > I always liked the idea of having a "shortest possible" way for creating > (or concatenating) > rows with "r_" > *and* > columns with "c_" > ! > > Why did the "c_" have to be removed !? > It wasn't removed, I thought to deprecate it. Owing to your response and the fact that others seem to use c_ quite a bit, I've kept it as a short hand for r_['1,2,0', ...] This means that arrays will be concatenated along the 1st axis after being up-graded to (at-least) 2-dimensional arrays with 1's placed at the end of the new shape. Thus, c_[[1,2,3],[4,5,6] produces array([[1, 4], [2, 5], [3, 6]]) This is a bit different if you were using c_ when you should have been using r_. -Travis From hfgi at grover5352.freeserve.co.uk Thu Sep 14 01:41:07 2006 From: hfgi at grover5352.freeserve.co.uk (Herbert Mcdermott) Date: Thu, 14 Sep 2006 08:41:07 +0300 Subject: [Numpy-discussion] learning Message-ID: <001b01c6d7c1$2292490a$a64d6055@anorwq.vfvdf> A slow strangling gurgle escaped from Eugenes screwed lips. Then, after a moments silence, he went on:Would you boys care to see the body? And Old Bens got the nicest place of all. A slow strangling gurgle escaped from Eugenes screwed lips. The air was a chill dusky pearl: above him a few palestars were out. He watcheduntil the first shovel of dirt had been thrown into the grave. Where is the can in which they throw the parts? Luke, said Harry Tugman, looking up from his paper, I wascertainly sorry to hear about Ben. He stood awkwardly, chilled, not knowinghow to continue. Bens grave had been that day freshlysodded: there was a sharp cold smell of earth there. He stood awkwardly, chilled, not knowinghow to continue. When he came to the gate of the cemetery he found it open. He was a man past fifty, with apleasant red face, brown mustaches, and a gentle placid manner. Well, then, said Horse Hines decisively, I was going to suggestto you boys that you take this one. And it will not lookso bad, if only I can save the front ones. Boys, he said, coming up to them sorrowfully,I was mighty sorry to hear of your trouble. Now, said Horse Hines quietly, I know the family doesnt wantanything cheap. The laurel, the lizard, and the stone will come no more. He put his hand affectionatelyupon a handsome casket at his side. The long barbarism of burial was at an end. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ice skate.gif Type: image/gif Size: 13290 bytes Desc: not available URL: From nwagner at iam.uni-stuttgart.de Thu Sep 14 02:47:32 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 14 Sep 2006 08:47:32 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: <4508FB04.1060709@iam.uni-stuttgart.de> Travis Oliphant wrote: > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Is it possible to circumvent the error messages if one uses Python2.4 ? ImportError: ctypes is not available. Nils From strawman at astraw.com Thu Sep 14 05:11:05 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 14 Sep 2006 02:11:05 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <4508D008.8010305@ieee.org> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> Message-ID: <45091CA9.60702@astraw.com> Travis Oliphant wrote: > Sebastian Haase wrote: > >> Travis Oliphant wrote: >> >> >> >>> It's not necessarily dead, the problem is complexity of implementation >>> and more clarity about how all dtypes are supposed to be printed, not >>> just this particular example. A patch would be very helpful here. If >>> desired it could be implemented in _internal.py and called from there in >>> arrayobject.c >>> >>> But, to get you thinking... How should the following be printed >>> >>> dtype('c4') >>> >>> dtype('a4,i8,3f4') >>> >>> dtype('3f4') >>> >>> >>> -Travis >>> >>> >> I would argue that if the simple cases were addressed first those would >> cover 90% (if not 99% for most people) of the cases occurring in >> people's daily use. >> For complex types (like 'a4,i8,3f4') I actually think the current text >> is compact and good. >> (Lateron one could think about >> 'c4' --> '4 chars' >> '3f4' --> '3 float32s' >> >> but already I don't know: is there any difference between 'c4' and >> '4c1'? What is the difference between 'c4' and 'a4' !? >> ) >> >> >> My main focus is on the fact that you might read '> "less" than 4-bytes int, which is very confusing ! >> >> > I can agree it's confusing at first, but it's the same syntax the struct > module uses which is the Python precedent for this. > I'm happy with seeing the repr() value since I know what it means, but I can see Sebastian's point. Perhaps there's a middle ground -- the str() representation for simple dtypes could contain both the repr() value and an English description. For example, something along the lines of "dtype(' References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> <1158131750.3968.36.camel@localhost.localdomain> Message-ID: <1b1c766f0609140601h294cb1beq3dfd4e83d2ff3cba@mail.gmail.com> On 9/13/06, Francesc Altet wrote: > Well, it seems that malloc actually takes more time when asking for more > space. However, this can't be the reason why Pierre is seeing that: > > a = numpy.exp(a) [1] > > is slower than > > numpy.exp(a,out=a) [2] > > as I'd say that this increment in time is negligible compared with > processing times of those big arrays. In fact, here are my times: > > >>> Timer("a = numpy.exp(a)", "import numpy;a = > numpy.random.rand(2048,2048) + 1j * > numpy.random.rand(2048,2048)").repeat(3,1) > [2.5527338981628418, 2.5427830219268799, 2.5074479579925537] > >>> Timer("numpy.exp(a,out=a)", "import numpy;a = > numpy.random.rand(2048,2048) + 1j * > numpy.random.rand(2048,2048)").repeat(3,1) > [2.5298278331756592, 2.5082788467407227, 2.5222280025482178] > > So, both times are comparable. Yeah, sorry about that: I had not checked carefully the timing. It seemed slower to me, but you're right, this is not a problem as long as there is enough free RAM. Ok, I'll go back to my coding and do like I should always do: care about optimization later. Thanks for all the comments and explanations! Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From martin.wiechert at gmx.de Thu Sep 14 10:16:26 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Thu, 14 Sep 2006 16:16:26 +0200 Subject: [Numpy-discussion] `_import_array' defined but not used Message-ID: <200609141616.26963.martin.wiechert@gmx.de> Hi gurus, I'm debugging a C-extension module that uses numpy. My version is 1.0b1. Can I safely ignore the following compiler warning? .../lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:903: warning: `_import_array' defined but not used Any help would be appreciated. Regards, Martin Wiechert From faltet at carabos.com Thu Sep 14 11:10:19 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 14 Sep 2006 17:10:19 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45091CA9.60702@astraw.com> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> Message-ID: <1158246619.3963.5.camel@localhost.localdomain> El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va escriure: > >> My main focus is on the fact that you might read ' >> "less" than 4-bytes int, which is very confusing ! > >> > >> > > I can agree it's confusing at first, but it's the same syntax the struct > > module uses which is the Python precedent for this. > > > I'm happy with seeing the repr() value since I know what it means, but I > can see Sebastian's point. Perhaps there's a middle ground -- the str() > representation for simple dtypes could contain both the repr() value and > an English description. For example, something along the lines of > "dtype(' the repr() string could be given without any kind of English translation. +1 I was very used (and happy) to the numarray string representation for types ('Int32', 'Complex64'...) and looking at how NumPy represents it now, I'd say that this is a backwards step in readability. Something like '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From laidler at stsci.edu Thu Sep 14 11:17:53 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 14 Sep 2006 11:17:53 -0400 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <1158246619.3963.5.camel@localhost.localdomain> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> <1158246619.3963.5.camel@localhost.localdomain> Message-ID: <450972A1.5080802@stsci.edu> Francesc Altet wrote: >El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va >escriure: > > >>>>My main focus is on the fact that you might read '>>>"less" than 4-bytes int, which is very confusing ! >>>> >>>> >>>> >>>> >>>I can agree it's confusing at first, but it's the same syntax the struct >>>module uses which is the Python precedent for this. >>> >>> >>> >>I'm happy with seeing the repr() value since I know what it means, but I >>can see Sebastian's point. Perhaps there's a middle ground -- the str() >>representation for simple dtypes could contain both the repr() value and >>an English description. For example, something along the lines of >>"dtype('>the repr() string could be given without any kind of English translation. >> >> > >+1 > >I was very used (and happy) to the numarray string representation for >types ('Int32', 'Complex64'...) and looking at how NumPy represents it >now, I'd say that this is a backwards step in readability. Something >like 'high-level one like NumPy, IMO. > > I agree entirely. The first type I got ' She explained the route and the exact situation of the house. Theres plenty of time, answered the dutiful nephew coolly. Somebody turned the handle of the door and drew back the bolts. Getting down, he tried the gates, and found one was fastened by a slip catch. No blood on the floor, mused Sneed, and stared up at the open skylight. She tried to pull the bed from the wall, but it was a heavy oaken affair and beyond her strength. In a few words he gave the gist of the terrible message which had reached him. She listened at the door, her senses tense, and heard a faint, deep sobbing, then heard no more. I do not expect you to do, that, said Mr Cody hastily. My idea, the man went on, calmness itself, was to meet her halfway. Your daughters friend is a fairly rich young lady? You dare accuse your aunt of being - -Ive got a great respect for my aunt. Ive got a car down here - - Mine is faster. The two men looked at one another without a word. She has a girl friend and often dines with her - probably she will go on to a theatre afterwards. She explained the route and the exact situation of the house. You dare accuse your aunt of being - -Ive got a great respect for my aunt. The hand gripped the bottom of the door and strove to lift it. Ill be under the railway arch in Brixton Road. There came no sound but the slow, solemn ticking of a dock on a landing above. Presently Mrs Cody came out alone, and, going downstairs, unlocked the library and went in. In the hall the chauffeur was lighting a cigarette. I shall not sign any document that I havent read, she replied, and laid down the pen. You ask too many questions; hes been complainin about you. The excuse was so flimsy, she told herself. We do not support such a luxury, Mrs Cody, she said. She knew it was covered with oilcloth, and it was on this that the feet were moving. Whilst the stout man watched admiringly, he removed the whole pane and drew it out. Bare feet must have wandered aimlessly here - the footmarks are on every rug. Ive sent for the Scotland Yard photographer and the local police, he said. He was on a level ledge of roof about three feet wide. As the machine sped southward he told of Sybils disappearance. A spring lock, explained Sneed; fastens automatically when its closed. She listened, and for a long time there was nothing to break the silence. Until then he had not made any reference to the story the librarian had told him. I dont know what your dirty business is - -Mr Cody exploded in anger. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: foreseeable.gif Type: image/gif Size: 7943 bytes Desc: not available URL: From charlesr.harris at gmail.com Thu Sep 14 12:24:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 14 Sep 2006 10:24:51 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <450972A1.5080802@stsci.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> <1158246619.3963.5.camel@localhost.localdomain> <450972A1.5080802@stsci.edu> Message-ID: On 9/14/06, Victoria G. Laidler wrote: > > Francesc Altet wrote: > > >El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va > >escriure: > > > > > >>>>My main focus is on the fact that you might read ' >>>>"less" than 4-bytes int, which is very confusing ! > >>>> > >>>> > >>>> > >>>> > >>>I can agree it's confusing at first, but it's the same syntax the > struct > >>>module uses which is the Python precedent for this. > >>> > >>> > >>> > >>I'm happy with seeing the repr() value since I know what it means, but I > >>can see Sebastian's point. Perhaps there's a middle ground -- the str() > >>representation for simple dtypes could contain both the repr() value and > >>an English description. For example, something along the lines of > >>"dtype(' >>the repr() string could be given without any kind of English > translation. > >> > >> > > > >+1 > > > >I was very used (and happy) to the numarray string representation for > >types ('Int32', 'Complex64'...) and looking at how NumPy represents it > >now, I'd say that this is a backwards step in readability. Something > >like ' >high-level one like NumPy, IMO. > > > > > I agree entirely. > The first type I got ' hell is that?" > It looked disturbingly like line-noise corrupted text to me! (Blast from > the past...) > > +1 from me as well. Just to balance the voting, I think things are fine as they are. As Travis says, the '<' is already used in the Python structure module and the rest doesn't take much time getting used to. However, the docstring for the dtype class is a bit lacking. It shouldn't be too much work to fix that up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Sep 14 12:29:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 14 Sep 2006 09:29:02 -0700 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Message-ID: <4509834E.4000007@noaa.gov> Charles R Harris wrote: >> > Why not simply >> > write a wrapper function in python that does Numeric-style guesswork, >> > and put it in the compatibility modules? >> Can I encourage any more comments? +1 > The main problem in constructing arrays > of objects is more information needs to be supplied because the user's > intention can't be reliably deduced from the current syntax. I wrote about this a bit early in this conversation, and as I thought about it. I'm not sure it's possible _- you could specify a rank, or a shape, but in general, there wouldn't be a unique way to translate an given hierarchy of sequences into a particular shape: imagine four levels of nested lists, asked to turn into a rank-3 array. This is why it may be best to simply recommend that people create an empty array of the shape they need, then put the objects into it - it's the only way to construct what you need reliably. However, an object array constructor that take a rank as an argument might well work for most cases, as long as there is a clearly documented and consistent way to handle extra levels of sequences: perhaps specify that any extra levels of nesting always go to the last dimension (or the first). That being said, it's still dangerous -- what levels of nesting are allowed would depend on which sequences *happen* to be the same size. Also the code would be a pain to write! I wonder how often people need to use objects arrays when they don't know when writing the code what shape they need? this is making me think that maybe all we really need is a little syntactic sugar for creating empty object arrays: numpy.ObjectArray(shape) Not much different than: numpy.empty(shape, dtype=numpy.object) but a little cleaner an more obvious to new users that are primarily interested in object arrays -- analogous to ones() and zeros() > That said, I > have no idea how widespread the use of object arrays is and so don't know > how much it really matters. If we ever get nd-arrays into the standard lib (or want to see wider use of them in any case), I think that object arrays are critical. Right now, people think they don't have a use for numpy if they aren't doing serious number crunching -- it's seen mostly as a way to speed up computations on lots of numbers. However, I think nd-arrays have LOTS of other applications, for anything where the data fits well in to a "rectangular" data structure. n-d slicing is a wonderful thing! As numpy gets wider use -- object arrays will be a very big draw. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rng7 at cornell.edu Thu Sep 14 18:15:15 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Thu, 14 Sep 2006 18:15:15 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <450856D7.6040601@ieee.org> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> Message-ID: <4509D473.7020905@cornell.edu> Hi Travis, Travis Oliphant wrote: > Ryan Gutenkunst wrote: >> I notice that numpy_array.item() will give me the first element as a >> normal scalar. Would it be possible for numpy_array.item(N) to return >> the Nth element of the array as a normal scalar? >> > Now this is an interesting idea. It would allow you to by-pass the > slow-indexing as well as the array scalar computation should you desire > it. I like it and am going to add it unless there are convincing > objections. > > -Travis Thanks for the quick response, but either there's a bug, or I'm using things wrong. It appears to work to work in the N-D case, but not 1-D. I looked at the change you made, but my grasp of the C-API is too weak to isolate the problem. >>> import numpy >>> numpy.__version__ '1.0rc1.dev3154' >>> a = numpy.array([[1.0, 2.0], [3.0, 4.0]]) >>> a.item(1,1) 4.0 >>> a = numpy.array([1.0, 2.0]) >>> a.item(0) 1.0 >>> a.item(1) 1.7765824089018436e-307 >>> a.item((1,)) 1.7765824089018436e-307 Thanks for your help, Ryan -- Ryan Gutenkunst | Cornell LASSP | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)227-7914 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From oliphant at ee.byu.edu Thu Sep 14 19:40:56 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 17:40:56 -0600 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <4509D473.7020905@cornell.edu> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> <4509D473.7020905@cornell.edu> Message-ID: <4509E888.2050800@ee.byu.edu> Ryan Gutenkunst wrote: >Hi Travis, > >Travis Oliphant wrote: > > >>Ryan Gutenkunst wrote: >> >> >>>I notice that numpy_array.item() will give me the first element as a >>>normal scalar. Would it be possible for numpy_array.item(N) to return >>>the Nth element of the array as a normal scalar? >>> >>> >>> >>Now this is an interesting idea. It would allow you to by-pass the >>slow-indexing as well as the array scalar computation should you desire >>it. I like it and am going to add it unless there are convincing >>objections. >> >>-Travis >> >> > >Thanks for the quick response, but either there's a bug, or I'm using >things wrong. It appears to work to work in the N-D case, but not 1-D. I >looked at the change you made, but my grasp of the C-API is too weak to >isolate the problem. > > It's a silly bug. Please check out the latest SVN. -Travis From oliphant at ee.byu.edu Thu Sep 14 19:58:17 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 17:58:17 -0600 Subject: [Numpy-discussion] Experience with Visit? Message-ID: <4509EC99.2060002@ee.byu.edu> Has anybody had any experience with the 3-D visualization software VISIT? It has Python bindings and seems to be pretty sophisticated. I'm wondering why I haven't heard more about it. http://www.llnl.gov/visit/ -Travis From brendansimons at yahoo.ca Thu Sep 14 20:20:28 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Thu, 14 Sep 2006 20:20:28 -0400 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: Message-ID: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Hi all, Just wondering if there was an arbitrary axis iterator in numpy, or if not, if there's demand for one. What I'm looking for is something which would allow me to do something like (vectorFunc(column) for column in array.axisIter(1) ) without a bunch of for loops and slicing. Thoughts? Brendan __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From oliphant at ee.byu.edu Thu Sep 14 20:31:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 18:31:43 -0600 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: <4509F46F.6090000@ee.byu.edu> Brendan Simons wrote: >Hi all, > >Just wondering if there was an arbitrary axis iterator in numpy, or >if not, if there's demand for one. What I'm looking for is something >which would allow me to do something like (vectorFunc(column) for >column in array.axisIter(1) ) without a bunch of for loops and slicing. > > Hmm... I can't think of something directly in Python, but it would be pretty easy to add. You could probably also do something clever by creating your own ufunc with frompyfunc and object arrays. In C, this would be easy. In the C-API there is a function PyArray_IterAllButAxis which provides an iterator that iterates over all axes but one. Then, you would call the vectorFunc for each element in the loop. The ufuncs use this functionality to call the underlying 1-d loops. -Travis From wbaxter at gmail.com Thu Sep 14 20:53:00 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 15 Sep 2006 09:53:00 +0900 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: Iteration over axis 0 is built-in, so you can already do (vectorFunc(row) for row in array) And you can use transpose() to make it so the axis you want to iterate over is axis 0. (vectorFunc(col) for col in array.transpose(1,0)) Or just use the .T attribute (vectorFunc(col) for col in array.T) So it seems kind of a toss-up whether it's worth adding a specific API to do that. The implementation would probably just return the transpose with the given axis in the zero slot. Something like: def axisiter(arr, i): ax = [i] + range(arr.ndim) del ax[i+1] return arr.transpose(ax) --bb On 9/15/06, Brendan Simons wrote: > Hi all, > > Just wondering if there was an arbitrary axis iterator in numpy, or > if not, if there's demand for one. What I'm looking for is something > which would allow me to do something like (vectorFunc(column) for > column in array.axisIter(1) ) without a bunch of for loops and slicing. > > Thoughts? > Brendan From rng7 at cornell.edu Thu Sep 14 20:59:59 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Thu, 14 Sep 2006 20:59:59 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <4509E888.2050800@ee.byu.edu> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> <4509D473.7020905@cornell.edu> <4509E888.2050800@ee.byu.edu> Message-ID: Travis, Thanks for the quick response. My application is back up to its old speed. Thanks also for spearheading the numpy/scipy projects. It's certainly made my work much, much more productive. Cheers, Ryan On Sep 14, 2006, at 7:40 PM, Travis Oliphant wrote: > Ryan Gutenkunst wrote: >> Thanks for the quick response, but either there's a bug, or I'm using >> things wrong. It appears to work to work in the N-D case, but not >> 1-D. I >> looked at the change you made, but my grasp of the C-API is too weak >> to >> isolate the problem. >> >> > It's a silly bug. Please check out the latest SVN. > > -Travis -- Ryan Gutenkunst | Cornell Dept. of Physics | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)255-6068 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From tim.hochberg at ieee.org Thu Sep 14 21:20:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 14 Sep 2006 18:20:08 -0700 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: <4509FFC8.5000309@ieee.org> Bill Baxter wrote: > Iteration over axis 0 is built-in, so you can already do > (vectorFunc(row) for row in array) > And you can use transpose() to make it so the axis you want to iterate > over is axis 0. > (vectorFunc(col) for col in array.transpose(1,0)) > Or just use the .T attribute > (vectorFunc(col) for col in array.T) > > So it seems kind of a toss-up whether it's worth adding a specific API > to do that. The implementation would probably just return the > transpose with the given axis in the zero slot. Something like: > > def axisiter(arr, i): > ax = [i] + range(arr.ndim) > del ax[i+1] > return arr.transpose(ax) > > --bb > Isn't swapaxis appropriate for this? In other words: for x in arr.swapaxis(0, i): #... should be more or less the same as: for x in axister(arr, i): #.... and it already exists. I'm in a hurry, so I haven't checked the details here, but the basic idea seems sound. -tim > On 9/15/06, Brendan Simons wrote: > >> Hi all, >> >> Just wondering if there was an arbitrary axis iterator in numpy, or >> if not, if there's demand for one. What I'm looking for is something >> which would allow me to do something like (vectorFunc(column) for >> column in array.axisIter(1) ) without a bunch of for loops and slicing. >> >> Thoughts? >> Brendan >> > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From haase at msg.ucsf.edu Thu Sep 14 21:20:53 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 14 Sep 2006 18:20:53 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? Message-ID: <200609141820.53325.haase@msg.ucsf.edu> Hi, what I'm asking is if numpy has an equivalent to numarray's info() function: >>> na.arange(10).info() class: shape: (10,) strides: (4,) byteoffset: 0 bytestride: 4 itemsize: 4 aligned: 1 contiguous: 1 buffer: data pointer: 0x085b7ec8 (DEBUG ONLY) byteorder: 'little' byteswap: 0 type: Int32 This was always helpful to me when debugging C binding code. Especially I'm asking if there is any way to get the memory address of an array - for debugging purposes only - of course ;-) Thanks, Sebastian Haase From wbaxter at gmail.com Thu Sep 14 22:05:13 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 15 Sep 2006 11:05:13 +0900 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <4509FFC8.5000309@ieee.org> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> <4509FFC8.5000309@ieee.org> Message-ID: On 9/15/06, Tim Hochberg wrote: > Isn't swapaxis appropriate for this? In other words: > You're right. Just didn't think of that. Never used swapaxes before. def axisiter(arr, i): return arr.swapaxes(0,i) --bb From brendansimons at yahoo.ca Thu Sep 14 23:27:59 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Thu, 14 Sep 2006 23:27:59 -0400 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: Message-ID: <9E3412CA-464A-409D-8AAE-A3CE8D28AA2F@yahoo.ca> Oh that's cool. For some reason I thought that the built in iterator (for i in array) iterated over cells, not the first axis. I also didn't think about swapaxes. Is there any desire to add a convenience function or method as follows? def axisIter(selfOrArr, i): return iter(selfOrArr.swapAxes(0,i)) Thanks for everyone who helped out. I've got something that works now. Cheers. Brendan On 14-Sep-06, at 10:05 PM, numpy-discussion- request at lists.sourceforge.net wrote: > Date: Fri, 15 Sep 2006 11:05:13 +0900 > From: "Bill Baxter" > Subject: Re: [Numpy-discussion] Axis Iterator? > To: "Discussion of Numerical Python" > > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 9/15/06, Tim Hochberg wrote: >> Isn't swapaxis appropriate for this? In other words: >> > > You're right. Just didn't think of that. Never used swapaxes before. > > def axisiter(arr, i): > return arr.swapaxes(0,i) > > --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Sep 15 02:10:29 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 08:10:29 +0200 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: <1158300630.3964.9.camel@localhost.localdomain> El dj 14 de 09 del 2006 a les 18:20 -0700, en/na Sebastian Haase va escriure: > Especially I'm asking if there is any way to get the memory address of an > array - for debugging purposes only - of course ;-) For this, you can print the data buffer: In [1]:import numpy In [2]:a=numpy.array([1]) In [3]:a.data Out[3]: although I'm not sure which number is the memory address I'd say it's the last one. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From fullung at gmail.com Fri Sep 15 05:51:31 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 11:51:31 +0200 Subject: [Numpy-discussion] how to get info about internals of an arrayobject ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase > Sent: 15 September 2006 03:21 > To: numpy-discussion > Subject: [Numpy-discussion] how to get info about internals of an > arrayobject ? > > Hi, > what I'm asking is if numpy has an equivalent to numarray's info() > function: > > This was always helpful to me when debugging C binding code. > > Especially I'm asking if there is any way to get the memory address of an > array - for debugging purposes only - of course ;-) numpy.array([]).__array_interface__['data'][0] Cheers, Albert From lroubeyrie at limair.asso.fr Fri Sep 15 08:49:51 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Fri, 15 Sep 2006 14:49:51 +0200 Subject: [Numpy-discussion] recarray Message-ID: <200609151449.51529.lroubeyrie@limair.asso.fr> Hi all, I try to use recarray with rec.fromrecords on time-series, datas come from a file where they are stored in csv format, with after each data colum there is one column meanning the state of the data, and the first column is for dates. Then, is it possible to directly transform column of strings to a integer one (or datetime one), and to remove a not used column? thanks -- Lionel From satyaupadhya at yahoo.co.in Fri Sep 15 09:09:25 2006 From: satyaupadhya at yahoo.co.in (Satya Upadhya) Date: Fri, 15 Sep 2006 14:09:25 +0100 (BST) Subject: [Numpy-discussion] degree matrix construction Message-ID: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Dear Friends, my question is the following: Suppose i have the following code: >>> from LinearAlgebra import * >>> from Numeric import * >>> A = [1,2,1,3,1,3,4,1,2] >>> B = reshape(A,(3,3)) >>> C = sum(B,1) >>> C array([4, 7, 7]) >>> Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix with diagonal elements 4,7,7 (obtained from the elements of C) and all off-diagonal elements equal to 0. Could some kind soul kindly tell me how to do this. I've looked at the help for the diagonal function and i am unable to do what i wish to. Furthermore i dont understand the meaning of axis1 and axis2: >>> help (diagonal) Help on function diagonal in module Numeric: diagonal(a, offset=0, axis1=0, axis2=1) diagonal(a, offset=0, axis1=0, axis2=1) returns all offset diagonals defined by the given dimensions of the array. >>> Thanking you, Satya --------------------------------- Find out what India is talking about on - Yahoo! Answers India Send FREE SMS to your friend's mobile from Yahoo! Messenger Version 8. Get it NOW -------------- next part -------------- An HTML attachment was scrubbed... URL: From joris at ster.kuleuven.be Fri Sep 15 09:18:55 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Fri, 15 Sep 2006 15:18:55 +0200 Subject: [Numpy-discussion] degree matrix construction In-Reply-To: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> References: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Message-ID: <200609151518.55585.joris@ster.kuleuven.be> Hi, [SU]: Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix with diagonal elements 4,7,7 You might have a look at the Numpy Example List, at the function diag(). Ciao, Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From joris at ster.kuleuven.be Fri Sep 15 09:22:11 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Fri, 15 Sep 2006 15:22:11 +0200 Subject: [Numpy-discussion] degree matrix construction Message-ID: <200609151522.11577.joris@ster.kuleuven.be> Forgot the link to the NEL: http://www.scipy.org/Numpy_Example_List J. Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From charlesr.harris at gmail.com Fri Sep 15 09:40:45 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Sep 2006 07:40:45 -0600 Subject: [Numpy-discussion] degree matrix construction In-Reply-To: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> References: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Message-ID: On 9/15/06, Satya Upadhya wrote: > > Dear Friends, > my question is the following: > > Suppose i have the following code: > > >>> from LinearAlgebra import * > > >>> from Numeric import * > >>> A = [1,2,1,3,1,3,4,1,2] > >>> B = reshape(A,(3,3)) > >>> C = sum(B,1) > >>> C > array([4, 7, 7]) > >>> > > Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix > with diagonal elements 4,7,7 (obtained from the elements of C) and all > off-diagonal elements equal to 0. > Is this what you want to do? In [2]: a = array([4, 7, 7]) In [3]: diagflat(a) Out[3]: array([[4, 0, 0], [0, 7, 0], [0, 0, 7]]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 15 09:57:28 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Sep 2006 08:57:28 -0500 Subject: [Numpy-discussion] recarray In-Reply-To: <200609151449.51529.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> Message-ID: Lionel Roubeyrie wrote: > Hi all, > I try to use recarray with rec.fromrecords on time-series, datas come from a > file where they are stored in csv format, with after each data colum there is > one column meanning the state of the data, and the first column is for dates. > Then, is it possible to directly transform column of strings to a integer one > (or datetime one), and to remove a not used column? When I import CSV files into record arrays, I usually read in all of the data and transpose the list of rows to get a list of columns. Then I can remove columns and transform them _en masse_, usually with map(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Fri Sep 15 10:05:55 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 16:05:55 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: References: <200609151449.51529.lroubeyrie@limair.asso.fr> Message-ID: <200609151605.56207.faltet@carabos.com> A Divendres 15 Setembre 2006 15:57, Robert Kern va escriure: > Lionel Roubeyrie wrote: > > Hi all, > > I try to use recarray with rec.fromrecords on time-series, datas come > > from a file where they are stored in csv format, with after each data > > colum there is one column meanning the state of the data, and the first > > column is for dates. Then, is it possible to directly transform column of > > strings to a integer one (or datetime one), and to remove a not used > > column? > > When I import CSV files into record arrays, I usually read in all of the > data and transpose the list of rows to get a list of columns. Then I can > remove columns and transform them _en masse_, usually with map(). Another possibility is to play with columns directly from the initial recarray. The next is an example: In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) In [102]: ra Out[102]: recarray([('1111', 825307441, 2.5784852031307537e-09), ('1111', 825307441, 2.5784852031307537e-09), ('1111', 825307441, 2.5784852031307537e-09)], dtype=[('f0', '|S4'), ('f1', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Fri Sep 15 10:13:41 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:13:41 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes Message-ID: <200609151613.42137.martin.wiechert@gmx.de> Hi list, I'm using PyArray_DescrConverter with a dict object to create a "struct-like" dtype from C. As the struct contains different data types I run into "unaligned access" problems. Is there a way to force alignment or to get trailing unused bytes in the dtpye? Thanks, Martin From martin.wiechert at gmx.de Fri Sep 15 10:15:19 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:15:19 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <200609151615.19950.martin.wiechert@gmx.de> On Friday 15 September 2006 16:13, Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a > "struct-like" dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. on IA64 I should have mentioned > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > > Thanks, Martin > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cimrman3 at ntc.zcu.cz Fri Sep 15 10:28:30 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 15 Sep 2006 16:28:30 +0200 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: <4509EC99.2060002@ee.byu.edu> References: <4509EC99.2060002@ee.byu.edu> Message-ID: <450AB88E.7040802@ntc.zcu.cz> Travis Oliphant wrote: > Has anybody had any experience with the 3-D visualization software > VISIT? It has Python bindings and seems to be pretty sophisticated. > I'm wondering why I haven't heard more about it. > > http://www.llnl.gov/visit/ No reaction up to now, so... I have just tried the 'getting started' part and was quite impressed, thanks for posting the link! Up to now I have used ParaView and was very satisfied, but the Python bindings of VisIt are a great lure. r. From faltet at carabos.com Fri Sep 15 10:40:39 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 16:40:39 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <200609151640.40220.faltet@carabos.com> A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a > "struct-like" dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? One possible solution is to declare void ('V' charcode) types for filling the gaps. For example: In [118]: ra=numpy.rec.array("1"*300, dtype=[('sval','0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Fri Sep 15 10:56:31 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:56:31 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151640.40220.faltet@carabos.com> References: <200609151613.42137.martin.wiechert@gmx.de> <200609151640.40220.faltet@carabos.com> Message-ID: <200609151656.31753.martin.wiechert@gmx.de> On Friday 15 September 2006 16:40, Francesc Altet wrote: > A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure: > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a > > "struct-like" dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > One possible solution is to declare void ('V' charcode) types for filling > the gaps. For example: > > > In [118]: ra=numpy.rec.array("1"*300, > dtype=[('sval',' > In [119]: ra['sval'] > Out[119]: > recarray([1111, 1111], > dtype='|S4') > > In [120]: ra['dval'] > Out[120]: array([ 9.73041595e-72, 9.73041595e-72]) > Hm, not exactly elegant but should work. Thanks Francesc! Note that all but trailing gaps can be controlled using a dictionary descriptor with an offsets key: >>> ra = zeros ((10,), dtype = {'names': ['sval', 'dval'], 'formats': ['>> ra.dtype dtype([('sval', '|S4'), ('', '|V4'), ('dval', ' You can still access the empty spaces if you want (although it is > nonsense): > > In [121]: ra['unused'] > Out[121]: > recarray([1111, 1111], > dtype='|V4') > > Cheers, From fullung at gmail.com Fri Sep 15 11:53:11 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 17:53:11 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: Hello all In [1]: import numpy as N In [3]: N.dtype({'names' : ['x', 'y'], 'formats' : [N.intc, N.float64]}, align=True) Out[3]: dtype([('x', ' String Form: Namespace: Interactive File: c:\python24\lib\site-packages\numpy\__init__.py Docstring: Cheers, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > Sent: 15 September 2006 16:14 > To: numpy-discussion > Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / > trailingunused bytes > > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a "struct- > like" > dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > > Thanks, Martin From martin.wiechert at gmx.de Fri Sep 15 11:56:37 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 17:56:37 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: References: Message-ID: <200609151756.37869.martin.wiechert@gmx.de> On Friday 15 September 2006 17:53, Albert Strasheim wrote: > Hello all > > In [1]: import numpy as N > > In [3]: N.dtype({'names' : ['x', 'y'], > 'formats' : [N.intc, N.float64]}, > align=True) > Out[3]: dtype([('x', ' > The reason you might not have discovered this: > > In [2]: N.dtype? > Type: type > Base Class: > String Form: > Namespace: Interactive > File: c:\python24\lib\site-packages\numpy\__init__.py > Docstring: > > Thanks Albert! Do you also know the corresponding C-API function? It cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no "align", right? > Cheers, > > Albert > > > -----Original Message----- > > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > > Sent: 15 September 2006 16:14 > > To: numpy-discussion > > Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / > > trailingunused bytes > > > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a "struct- > > like" > > dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > > > Thanks, Martin > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Michael_OKeefe at nrel.gov Fri Sep 15 12:23:24 2006 From: Michael_OKeefe at nrel.gov (O'Keefe, Michael) Date: Fri, 15 Sep 2006 10:23:24 -0600 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: <450AB88E.7040802@ntc.zcu.cz> Message-ID: I haven't tried VisIT before but thanks for the link. I also downloaded and am checking it out. Along this same line of discussion, has anyone tried OOF2 which is an FEA package that also has some strong python connections? http://www.ctcms.nist.gov/oof/oof2.html I'm working on a Windows machine and it the current code-base doesn't seem to support Windows out of the box (if at all). Looks like you can put it together for *nix or Mac OS X, though... --Michael > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net > [mailto:numpy-discussion-bounces at lists.sourceforge.net] On > Behalf Of Robert Cimrman > Sent: Friday, September 15, 2006 8:29 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Experience with Visit? > > Travis Oliphant wrote: > > Has anybody had any experience with the 3-D visualization software > > VISIT? It has Python bindings and seems to be pretty > sophisticated. > > I'm wondering why I haven't heard more about it. > > > > http://www.llnl.gov/visit/ > > No reaction up to now, so... > > I have just tried the 'getting started' part and was quite impressed, > thanks for posting the link! Up to now I have used ParaView > and was very > satisfied, but the Python bindings of VisIt are a great lure. > > r. > > -------------------------------------------------------------- > ----------- > Using Tomcat but need to do more? Need to support web > services, security? > Get stuff done quickly with pre-integrated technology to make > your job easier > Download IBM WebSphere Application Server v.1.0.1 based on > Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057& > dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Fri Sep 15 12:44:23 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 10:44:23 -0600 Subject: [Numpy-discussion] how to get info about internals of an arrayobject ? In-Reply-To: References: Message-ID: <450AD867.1020301@ieee.org> Albert Strasheim wrote: >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase >> Sent: 15 September 2006 03:21 >> To: numpy-discussion >> Subject: [Numpy-discussion] how to get info about internals of an >> arrayobject ? >> >> Hi, >> what I'm asking is if numpy has an equivalent to numarray's info() >> function: >> > > >> >> > > >> This was always helpful to me when debugging C binding code. >> >> Especially I'm asking if there is any way to get the memory address of an >> array - for debugging purposes only - of course ;-) >> > > numpy.array([]).__array_interface__['data'][0] > Or numpy.ctypes._as_parameter_ (should work even if you don't have ctypes installed) -Travis From emsellem at obs.univ-lyon1.fr Fri Sep 15 12:49:09 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Fri, 15 Sep 2006 18:49:09 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ? Message-ID: <450AD985.3050005@obs.univ-lyon1.fr> Hi, I am facing a rather frustrating problem with numpy/scipy: after upgrading to svn numpy and scipy, and trying to remove most of the now unnecessary casting (floats) the program I wrote does not give me the right answer. It seems that the answer is somewhat scaled down (but not in a simple way). I had this kind of behaviour some time ago when scipy and numpy were not fully compatible and then I had to cast the floats explicitely in my program to make it work. But now it SHOULD be fully compatible as far as I understand the issue. So my question is: is there any remaining flaws in terms of casts/format between scipy and numpy? I am specifically using the scipy functions: - special.erf, special.erfc, orthogonal.ps_roots, integrate.quad... and the numpy functions : sum, sqrt, exp, sin, cos, arctan. I am doing all the calculations using numpy.float32. Did anybody witness weird behaviour recently in this context? (I don't really know what to do now since I already spent hours trying to find the problem: the point is that the final answer is wrong for sure, but the intermediate calculations are not easily checked since I do not know what to really expect there. The program was working before I touched the casting + upgraded numpy/scipy) thanks for any help here, and cheers. Eric From oliphant.travis at ieee.org Fri Sep 15 13:00:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 11:00:14 -0600 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: <450ADC1E.2060004@ieee.org> Sebastian Haase wrote: > Hi, > what I'm asking is if numpy has an equivalent to numarray's info() function: > >>>> na.arange(10).info() >>>> numpy.numarray.info(numpy.arange(10)) (get recent SVN as there were some bugs just fixed. -Travis From oliphant.travis at ieee.org Fri Sep 15 13:14:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 11:14:15 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151756.37869.martin.wiechert@gmx.de> References: <200609151756.37869.martin.wiechert@gmx.de> Message-ID: <450ADF67.5000709@ieee.org> Martin Wiechert wrote: > Thanks Albert! Do you also know the corresponding C-API function? It cannot be > PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no > "align", right? > The DescrConverter function is meant for "O&"-style conversions. It can't accept an align function. We could possibly add something to the converter to allow specification of alignment through the object to be converted. Or, you can just call the __new__ method of the PyArrayDescr_Type object res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", "Oi", dict_object, 1)) or call the tp->new method directly: args = Py_BuildValue("Oi", dict_object, 1); PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); Py_DECREF(args); (I think passing in NULL for the keywords is O.K., but I haven't checked it). -Travis From martin.wiechert at gmx.de Fri Sep 15 13:53:14 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 19:53:14 +0200 Subject: [Numpy-discussion] =?iso-8859-1?q?PyArray=5FDescrConverter_-_alig?= =?iso-8859-1?q?nment_/=09trailingunused_bytes?= In-Reply-To: <450ADF67.5000709@ieee.org> References: <200609151756.37869.martin.wiechert@gmx.de> <450ADF67.5000709@ieee.org> Message-ID: <200609151953.14405.martin.wiechert@gmx.de> On Friday 15 September 2006 19:14, Travis Oliphant wrote: > Martin Wiechert wrote: > > Thanks Albert! Do you also know the corresponding C-API function? It > > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > > signature has no "align", right? > > The DescrConverter function is meant for "O&"-style conversions. It > can't accept an align function. We could possibly add something to the > converter to allow specification of alignment through the object to be > converted. > I begin to see the light.... For dictionaries one could maybe just add an optional key "align". Also an optional key "elsize" or "itemsize" to force the total size of the record may sometimes be useful. E.g. one may want to faithfully map a given C struct. (That's why I was asking for trailing unused bytes.) But I understand that other things have higher priority. > Or, you can just call the __new__ method of the PyArrayDescr_Type object > > res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", > "Oi", dict_object, 1)) > > or call the tp->new method directly: > > args = Py_BuildValue("Oi", dict_object, 1); > PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); > Py_DECREF(args); > Thank you! I'll try this. > (I think passing in NULL for the keywords is O.K., but I haven't checked > it). > > -Travis > One final question. To me the repr of a dtype with gaps is a little bit puzzling: >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Fri Sep 15 14:05:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 12:05:50 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <450AEB7E.3050009@ieee.org> Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a "struct-like" > dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > Besides calling the tp_new function (or the method __new__), I added PyArray_DescrAlignConverter to the C-API. It is at the end and so does not require re-compilation of extension modules (unless you want to use it...) -Travis From martin.wiechert at gmx.de Fri Sep 15 14:03:54 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 20:03:54 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <450AEB7E.3050009@ieee.org> References: <200609151613.42137.martin.wiechert@gmx.de> <450AEB7E.3050009@ieee.org> Message-ID: <200609152003.54715.martin.wiechert@gmx.de> On Friday 15 September 2006 20:05, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a > > "struct-like" dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > Besides calling the tp_new function (or the method __new__), I added > PyArray_DescrAlignConverter to the C-API. > > It is at the end and so does not require re-compilation of extension > modules (unless you want to use it...) > Thanks Travis. I appreciate your work. > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From fullung at gmail.com Fri Sep 15 14:13:29 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 20:13:29 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151953.14405.martin.wiechert@gmx.de> Message-ID: Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > Sent: 15 September 2006 19:53 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion]PyArray_DescrConverter - alignment / > trailingunused bytes > > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > > Martin Wiechert wrote: > > > Thanks Albert! Do you also know the corresponding C-API function? It > > > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > > > signature has no "align", right? > > > > The DescrConverter function is meant for "O&"-style conversions. It > > can't accept an align function. We could possibly add something to the > > converter to allow specification of alignment through the object to be > > converted. > > > > One final question. To me the repr of a dtype with gaps is a little bit > puzzling: > > >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' 'offsets': [0, 16, 24]}) > dtype([('a', '|S4'), ('', '|V12'), ('b', ' ' > There should be no gap between "b" and "c" but still the repr has ('', > '|V12') > between them. Am I missing something? For performance reasons, compilers will typically align integers (and probably floats) on 4-byte boundaries and apparently, doubles on 16-byte boundaries. Because compilers align like this, so does NumPy. This allows you to: 1. Take any kind of C struct definition 2. Convert it to a dtype 3. Create a NumPy array with this dtype 4. Pass the array's data pointer to C code 5. Cast the data pointer to a pointer to your C struct 6. Operate on the pointer to struct as if it were allocated in C Cheers, Albert From fullung at gmail.com Fri Sep 15 14:16:59 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 20:16:59 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment/ trailingunused bytes In-Reply-To: Message-ID: Argh > > > > > One final question. To me the repr of a dtype with gaps is a little bit > > puzzling: > > > > >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > 'offsets': [0, 16, 24]}) > > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > ' > > > There should be no gap between "b" and "c" but still the repr has ('', > > '|V12') > > between them. Am I missing something? I see you're not specifying align=True here. Ignore my last message. Cheers, Albert From oliphant.travis at ieee.org Fri Sep 15 14:27:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 12:27:00 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151953.14405.martin.wiechert@gmx.de> References: <200609151756.37869.martin.wiechert@gmx.de> <450ADF67.5000709@ieee.org> <200609151953.14405.martin.wiechert@gmx.de> Message-ID: <450AF074.1090806@ieee.org> Martin Wiechert wrote: > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > >> Martin Wiechert wrote: >> >>> Thanks Albert! Do you also know the corresponding C-API function? It >>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose >>> signature has no "align", right? >>> >> The DescrConverter function is meant for "O&"-style conversions. It >> can't accept an align function. We could possibly add something to the >> converter to allow specification of alignment through the object to be >> converted. >> >> > > I begin to see the light.... > > For dictionaries one could maybe just add an optional key "align". > Also an optional key "elsize" or "itemsize" to force the total size of the > record may sometimes be useful. E.g. one may want to faithfully map a given C > struct. (That's why I was asking for trailing unused bytes.) > > But I understand that other things have higher priority. > > >> Or, you can just call the __new__ method of the PyArrayDescr_Type object >> >> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", >> "Oi", dict_object, 1)) >> >> or call the tp->new method directly: >> >> args = Py_BuildValue("Oi", dict_object, 1); >> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); >> Py_DECREF(args); >> >> > > Thank you! I'll try this. > > >> (I think passing in NULL for the keywords is O.K., but I haven't checked >> it). >> >> -Travis >> >> > > One final question. To me the repr of a dtype with gaps is a little bit > puzzling: > > >>>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['>>> > 'offsets': [0, 16, 24]}) > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > There should be no gap between "b" and "c" but still the repr has ('', '|V12') > between them. Am I missing something? > There was a bug I just fixed in the representation of these structures with gaps. It should be fixed in SVN, now. -Travis From martin.wiechert at gmx.de Fri Sep 15 14:22:44 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 20:22:44 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <450AF074.1090806@ieee.org> References: <200609151953.14405.martin.wiechert@gmx.de> <450AF074.1090806@ieee.org> Message-ID: <200609152022.45061.martin.wiechert@gmx.de> On Friday 15 September 2006 20:27, Travis Oliphant wrote: > Martin Wiechert wrote: > > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > >> Martin Wiechert wrote: > >>> Thanks Albert! Do you also know the corresponding C-API function? It > >>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > >>> signature has no "align", right? > >> > >> The DescrConverter function is meant for "O&"-style conversions. It > >> can't accept an align function. We could possibly add something to the > >> converter to allow specification of alignment through the object to be > >> converted. > > > > I begin to see the light.... > > > > For dictionaries one could maybe just add an optional key "align". > > Also an optional key "elsize" or "itemsize" to force the total size of > > the record may sometimes be useful. E.g. one may want to faithfully map a > > given C struct. (That's why I was asking for trailing unused bytes.) > > > > But I understand that other things have higher priority. > > > >> Or, you can just call the __new__ method of the PyArrayDescr_Type object > >> > >> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", > >> "Oi", dict_object, 1)) > >> > >> or call the tp->new method directly: > >> > >> args = Py_BuildValue("Oi", dict_object, 1); > >> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); > >> Py_DECREF(args); > > > > Thank you! I'll try this. > > > >> (I think passing in NULL for the keywords is O.K., but I haven't checked > >> it). > >> > >> -Travis > > > > One final question. To me the repr of a dtype with gaps is a little bit > > > > puzzling: > >>>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > > > 'offsets': [0, 16, 24]}) > > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > ' > > > There should be no gap between "b" and "c" but still the repr has ('', > > '|V12') between them. Am I missing something? > > There was a bug I just fixed in the representation of these structures > with gaps. It should be fixed in SVN, now. > > > -Travis > > Thanks again. > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Fri Sep 15 15:07:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Sep 2006 14:07:25 -0500 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ? In-Reply-To: <450AD985.3050005@obs.univ-lyon1.fr> References: <450AD985.3050005@obs.univ-lyon1.fr> Message-ID: Eric Emsellem wrote: > Hi, > > I am facing a rather frustrating problem with numpy/scipy: after > upgrading to svn numpy and scipy, and trying to remove most of the now > unnecessary casting (floats) the program I wrote does not give me the > right answer. It seems that the answer is somewhat scaled down (but not > in a simple way). I had this kind of behaviour some time ago when scipy > and numpy were not fully compatible and then I had to cast the floats > explicitely in my program to make it work. But now it SHOULD be fully > compatible as far as I understand the issue. So my question is: > is there any remaining flaws in terms of casts/format between scipy and > numpy? I am specifically using the scipy functions: > - special.erf, special.erfc, orthogonal.ps_roots, integrate.quad... > and the numpy functions : sum, sqrt, exp, sin, cos, arctan. > I am doing all the calculations using numpy.float32. Could you give us a short piece of code that demonstrates the problem? You don't state what flaws you were working around before. Note that some of those functions in scipy are unavoidably implemented in double precision at the C/FORTRAN level. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hetland at tamu.edu Fri Sep 15 15:15:55 2006 From: hetland at tamu.edu (Rob Hetland) Date: Fri, 15 Sep 2006 14:15:55 -0500 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: References: Message-ID: I have looked at it, and I even know somebody who is using python to put in a data reader for FVCOM (a finite volume ocean model). He has a prototype working. I have no experience with the python bindings personally. I looked into compiling it from source (for an intel mac), but it was daunting so I gave up. I also contacted them to try to put in lower level python bindings, so that we could create 3D graphics from the command line, but they didn't seem too interested in that. Otherwise, it is a very nice workable app. -Rob On Sep 15, 2006, at 11:23 AM, O'Keefe, Michael wrote: > I haven't tried VisIT before but thanks for the link. I also > downloaded and am checking it out. > > Along this same line of discussion, has anyone tried OOF2 which is > an FEA package that also has some strong python connections? > > http://www.ctcms.nist.gov/oof/oof2.html > > I'm working on a Windows machine and it the current code-base > doesn't seem to support Windows out of the box (if at all). Looks > like you can put it together for *nix or Mac OS X, though... > > --Michael > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net >> [mailto:numpy-discussion-bounces at lists.sourceforge.net] On >> Behalf Of Robert Cimrman >> Sent: Friday, September 15, 2006 8:29 >> To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] Experience with Visit? >> >> Travis Oliphant wrote: >>> Has anybody had any experience with the 3-D visualization software >>> VISIT? It has Python bindings and seems to be pretty >> sophisticated. >>> I'm wondering why I haven't heard more about it. >>> >>> http://www.llnl.gov/visit/ >> >> No reaction up to now, so... >> >> I have just tried the 'getting started' part and was quite impressed, >> thanks for posting the link! Up to now I have used ParaView >> and was very >> satisfied, but the Python bindings of VisIt are a great lure. >> >> r. >> >> -------------------------------------------------------------- >> ----------- >> Using Tomcat but need to do more? Need to support web >> services, security? >> Get stuff done quickly with pre-integrated technology to make >> your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on >> Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057& >> dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > > ---------------------------------------------------------------------- > --- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion ---- Rob Hetland, Associate Professor Dept. of Oceanography, Texas A&M University http://pong.tamu.edu/~rob phone: 979-458-0096, fax: 979-845-6331 From haase at msg.ucsf.edu Fri Sep 15 15:13:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 15 Sep 2006 12:13:43 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <450ADC1E.2060004@ieee.org> References: <200609141820.53325.haase@msg.ucsf.edu> <450ADC1E.2060004@ieee.org> Message-ID: <200609151213.43655.haase@msg.ucsf.edu> On Friday 15 September 2006 10:00, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > > > what I'm asking is if numpy has an equivalent to numarray's info() function: > >>>> na.arange(10).info() > > numpy.numarray.info(numpy.arange(10)) > > (get recent SVN as there were some bugs just fixed. > > -Travis Thanks, should this maybe also be added somewhere in the numpy module itself !? I guess the question is, what the original intent was for numarray to put it in (only for debugging ?) -- then we could decide if this would also make sense for numpy. I have never used numpy.info(obj) - I don't know what is does (compared to __doc__) but maybe it could be extended like if isinstance( obj , N.ndarray): print N.numarray.info( obj ) right now it prints nothings (and returns None) Thanks, Sebastian Haase From oliphant at ee.byu.edu Fri Sep 15 15:49:56 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 15 Sep 2006 13:49:56 -0600 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609151213.43655.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> <450ADC1E.2060004@ieee.org> <200609151213.43655.haase@msg.ucsf.edu> Message-ID: <450B03E4.6080101@ee.byu.edu> Sebastian Haase wrote: >On Friday 15 September 2006 10:00, Travis Oliphant wrote: > > >>Sebastian Haase wrote: >> >> >>>Hi, >>> >>>what I'm asking is if numpy has an equivalent to numarray's info() >>> >>> >function: > > >>>>>>na.arange(10).info() >>>>>> >>>>>> >>numpy.numarray.info(numpy.arange(10)) >> >>(get recent SVN as there were some bugs just fixed. >> >>-Travis >> >> > >Thanks, > >should this maybe also be added somewhere in the numpy module itself !? >I guess the question is, what the original intent was for numarray to put it >in (only for debugging ?) -- then we could decide if this would also make >sense for numpy. > >I have never used numpy.info(obj) - I don't know what is does (compared to >__doc__) > It prints the doc string of an object searching for it better than help seems to do and without a "pager". Compare numpy.info(numpy.sin) versus help(numpy.sin) I don't know what the info method was for other than debugging. What about having the __doc__ attribute of an ndarray return the info? -Travis From haase at msg.ucsf.edu Fri Sep 15 16:49:41 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 15 Sep 2006 13:49:41 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <450B03E4.6080101@ee.byu.edu> References: <200609141820.53325.haase@msg.ucsf.edu> <200609151213.43655.haase@msg.ucsf.edu> <450B03E4.6080101@ee.byu.edu> Message-ID: <200609151349.41809.haase@msg.ucsf.edu> On Friday 15 September 2006 12:49, Travis Oliphant wrote: > Sebastian Haase wrote: > >On Friday 15 September 2006 10:00, Travis Oliphant wrote: > >>Sebastian Haase wrote: > >>>Hi, > >>> > >>>what I'm asking is if numpy has an equivalent to numarray's info() > > > >function: > >>>>>>na.arange(10).info() > >> > >>numpy.numarray.info(numpy.arange(10)) > >> > >>(get recent SVN as there were some bugs just fixed. > >> > >>-Travis > > > >Thanks, > > > >should this maybe also be added somewhere in the numpy module itself !? > >I guess the question is, what the original intent was for numarray to put > > it in (only for debugging ?) -- then we could decide if this would also > > make sense for numpy. > > > >I have never used numpy.info(obj) - I don't know what is does (compared to > >__doc__) > > It prints the doc string of an object searching for it better than help > seems to do and without a "pager". > > Compare numpy.info(numpy.sin) versus help(numpy.sin) In PyShell I would miss the output of numpy.info(numpy.sin) since it goes to the (C) stdout and not to the (sometimes redirected ) python sys.stdout. So it seems it prints N.sin.__doc__ The output from help(...) goes to the (python) sys.stdout (or stderr !?) and is quite long (and mostly just generec info about ufuncs ...) > > I don't know what the info method was for other than debugging. > > What about having the __doc__ attribute of an ndarray return the info? > Maybe, but honestly I would *not* expect any kind of "live inspection" to be done by __doc__ I would expect more like "ndarray is an efficient way to handle large data sets in python - it is the bases for Numerical Python, see www.scipy.org " Maybe info() should just be an array attribute - just like it was it numarray. -Sebastian From yukihana at yahoo.co.jp Sat Sep 16 08:15:41 2006 From: yukihana at yahoo.co.jp (=?iso-2022-jp?B?eXVraWhhbmE=?=) Date: Sat, 16 Sep 2006 12:15:41 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?02 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?02 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ?????????????????????????????????? ???????????? ??????????3-6-4-533 ?????? 139-3668-7892 From philluewan at alliedvaughn.com Sat Sep 16 09:18:38 2006 From: philluewan at alliedvaughn.com (Sib Ridenhour) Date: Sat, 16 Sep 2006 06:18:38 -0700 Subject: [Numpy-discussion] PHxoeARMA Message-ID: <01c6d992$e87da9c0$0700000a@vieja> Hi QUIT OVE k RPA y YIN r G FOR Y c OUR PH l AR g MAC n Y S w AV i E up t h o 50 z % wit p h http://www.criaboedri.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From irbdavid at gmail.com Sat Sep 16 11:46:28 2006 From: irbdavid at gmail.com (David Andrews) Date: Sat, 16 Sep 2006 17:46:28 +0200 Subject: [Numpy-discussion] 1.0b5 Installation on OS X Message-ID: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> Hi, I'm unable to install the 1.0b5 release from the .mpkg on OS X - when it comes to the two checkboxes to select the platlib and the scripts, both are greyed out. The installer then claims 'Cannot continue: Nothing to install'. Any suggestions? Cheers, Dave From Carmen at tal.is Sat Sep 16 13:39:48 2006 From: Carmen at tal.is (Casandra) Date: Sat, 16 Sep 2006 23:39:48 +0600 Subject: [Numpy-discussion] A Job offer Message-ID: Dear Friend, Taken into account the expansion of a field of activity of our firm, which is located in Europe and engaged in modeling and fashion industry in the world markets. We require Project Manager. The job is of a partial employment, which means that you will not spend a lot of energy and time on it. The salary depends on the price of projects executed by you (about 30000-50000 dollars a year). If our offer has interested you and you are a serious and hardworking person, who is not afraid to open a new field of activity for yourself, you are probably the person that we are looking for. Also, we will be glad to see you in our company. To suit this job you have to be at least 21 years of age or older and be a US or Australian citizen. Tahnk you. galina_frolova at gawab.com From martin.wiechert at gmx.de Sat Sep 16 14:49:32 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sat, 16 Sep 2006 20:49:32 +0200 Subject: [Numpy-discussion] concatenation changes dtype Message-ID: <200609162049.33244.martin.wiechert@gmx.de> Hi list, Apparently r_[x,x] does not necessarily have the same dtype as x: >>> from numpy import * >>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>> x = zeros ((1,), dt) >>> x.dtype dtype([('a', '|i1'), ('f4', '>> r_[x,x].dtype dtype('object') >>> >>> import numpy >>> numpy.version.version '1.0rc1.dev3171' Does anybody know how to avoid this change of dtype? Thanks, Martin. From oliphant.travis at ieee.org Sat Sep 16 18:29:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 16 Sep 2006 16:29:01 -0600 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <200609162049.33244.martin.wiechert@gmx.de> References: <200609162049.33244.martin.wiechert@gmx.de> Message-ID: <450C7AAD.6030705@ieee.org> Martin Wiechert wrote: > Hi list, > > Apparently r_[x,x] does not necessarily have the same dtype as x: > > >>>> from numpy import * >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>>> x = zeros ((1,), dt) >>>> x.dtype >>>> > dtype([('a', '|i1'), ('f4', ' >>>> r_[x,x].dtype >>>> > dtype('object') > >>>> import numpy >>>> numpy.version.version >>>> > '1.0rc1.dev3171' > > Does anybody know how to avoid this change of dtype? > Thanks for the check. It should be fixed now in SVN. There was an inappropriate force to object arrays which neglected this case. -Travis From oliphant.travis at ieee.org Sat Sep 16 18:36:39 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 16 Sep 2006 16:36:39 -0600 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <200609162049.33244.martin.wiechert@gmx.de> References: <200609162049.33244.martin.wiechert@gmx.de> Message-ID: <450C7C77.4050006@ieee.org> Martin Wiechert wrote: > Hi list, > > Apparently r_[x,x] does not necessarily have the same dtype as x: > > >>>> from numpy import * >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>>> x = zeros ((1,), dt) >>>> x.dtype >>>> > dtype([('a', '|i1'), ('f4', ' Did you mean to make a data-type that was a 1-byte integer followed by a 4-byte integer with field names of 'a' and 'f4'? Perhaps you meant: dtype([('a','f4'),('b','i4')]) which is a 4-byte float followed by a 4-byte integer. -Travis From martin.wiechert at gmx.de Sat Sep 16 20:15:23 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 17 Sep 2006 02:15:23 +0200 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <450C7C77.4050006@ieee.org> References: <200609162049.33244.martin.wiechert@gmx.de> <450C7C77.4050006@ieee.org> Message-ID: <200609170215.23435.martin.wiechert@gmx.de> On Sunday 17 September 2006 00:36, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > Apparently r_[x,x] does not necessarily have the same dtype as x: > >>>> from numpy import * > >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) > >>>> x = zeros ((1,), dt) > >>>> x.dtype > > > > dtype([('a', '|i1'), ('f4', ' > Did you mean to make a data-type that was a 1-byte integer followed by a > 4-byte integer with field names of 'a' and 'f4'? > > Perhaps you meant: > > dtype([('a','f4'),('b','i4')]) > > which is a 4-byte float followed by a 4-byte integer. > Thanks for the hint. Actually, the code I posted was just for the sake of example. Still you correctly read my thoughts. Martin. > > -Travis > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From kaapoudugg at centerpartners.com Sun Sep 17 09:19:38 2006 From: kaapoudugg at centerpartners.com (Aksel Booth) Date: Sun, 17 Sep 2006 06:19:38 -0700 Subject: [Numpy-discussion] PHAxqiRMA Message-ID: <01c6da5b$f1c90500$0a01a8c0@renia> Hi d V x I d A d G o R m A s V w A f L p I p U l M l C q I n A t L e I z S e A x M g B u I b E o N k X d A w N w A q X S y AV o E up to 6 t 0 % w e it c h http://www.ylutriamoa.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Sun Sep 17 14:17:09 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 17 Sep 2006 20:17:09 +0200 Subject: [Numpy-discussion] record descriptors: trailing unused bytes Message-ID: <200609172017.09548.martin.wiechert@gmx.de> Hi list, does anybody know, if in C in the PyArray_Descr struct it is safe to manually change descr->elsize and descr->alignment as long as these are compatible and descr->elsize is large enough to hold all fields? Of course I mean before any array is constructed using descr. Thanks, Martin. From Achim.Gaedke at physik.tu-darmstadt.de Sun Sep 17 16:00:07 2006 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Sun, 17 Sep 2006 22:00:07 +0200 Subject: [Numpy-discussion] compile error with python2.5 Message-ID: <450DA947.4020809@physik.tu-darmstadt.de> Hi! I compiled numpy-1.0b5 with python2.5 on debian testing and raIn file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsIntp': numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use in this function) numpy/core/src/arrayobject.c:564: error: (Each undeclared identifier is reported only once numpy/core/src/arrayobject.c:564: error: for each function it appears in.) numpy/core/src/arrayobject.c:565: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsInt': numpy/core/src/arrayobject.c:654: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'array_subscript': numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsIntp': numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use in this function) numpy/core/src/arrayobject.c:564: error: (Each undeclared identifier is reported only once numpy/core/src/arrayobject.c:564: error: for each function it appears in.) numpy/core/src/arrayobject.c:565: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsInt': numpy/core/src/arrayobject.c:654: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'array_subscript': numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -Ibuild/src.linux-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/home/achim/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-i686-2.5/numpy/core/src/multiarraymodule.o" failed with exit status 1 n into errors I fixed it by guessing the correct arguments. The patch is attached. I hope someone can revise this fix. Achim -------------- next part -------------- diff -ru numpy-1.0b5/numpy/core/src/arrayobject.c numpy-1.0b5_fixed/numpy/core/src/arrayobject.c --- numpy-1.0b5/numpy/core/src/arrayobject.c 2006-09-05 00:41:47.000000000 +0200 +++ numpy-1.0b5_fixed/numpy/core/src/arrayobject.c 2006-09-17 21:41:49.000000000 +0200 @@ -561,8 +561,8 @@ return ret; } #if (PY_VERSION_HEX >= 0x02050000) - if (PyIndex_Check(op)) { - long_value = (longlong) PyNumber_Index(op); + if (PyIndex_Check(o)) { + long_value = (longlong) PyNumber_Index(o); goto finish; } #endif @@ -2691,7 +2691,7 @@ } if (PyInt_Check(op) || PyArray_IsScalar(op, Integer) || - PyLong_Check(op) || PyIndex_Check(index)) { + PyLong_Check(op) || PyIndex_Check(op)) { intp value; value = PyArray_PyIntAsIntp(op); if (!PyErr_Occurred()) { From fullung at gmail.com Sun Sep 17 16:29:25 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sun, 17 Sep 2006 22:29:25 +0200 Subject: [Numpy-discussion] compile error with python2.5 In-Reply-To: <450DA947.4020809@physik.tu-darmstadt.de> Message-ID: Hello all These problems were fixed in SVN just after 1.0rc1 was released. http://projects.scipy.org/scipy/numpy/changeset/3119 Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Achim Gaedke > Sent: 17 September 2006 22:00 > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] compile error with python2.5 > > Hi! > > I compiled numpy-1.0b5 with python2.5 on debian testing and raIn file > included from numpy/core/src/multiarraymodule.c:65: From syzb at garonpr.demon.co.uk Sun Sep 17 18:20:24 2006 From: syzb at garonpr.demon.co.uk (Louisa Mata) Date: Mon, 18 Sep 2006 06:20:24 +0800 Subject: [Numpy-discussion] mournful Message-ID: <000801c6daa8$c6738cb1$3160a2da@qxfi.ncpdr> This may have been regularan then again it may have been queer. Bilyen made a careful inspection of the spot, and then faced Brazos witha curious fire in his eyes. I raisedthe ante to two thousand dollars. Give me the lowdown on rustlin in eastern Colorado. Suddenly she sprang out from the wall, formidable as a tigress. Without any greeting, Brazos flung a query at the banker. Yu-all shorehave heahed of the Sewall McCoy combine with Russ Slaughter. Shes the most fascinating girl I ever met. But he could be made a target for speech thatwould sweep over town like fire in prairie grass. At the foot of the Odd Fellows stairway Brazos halted to load his gun. Father hasa deal on with Surface and Miller. I mixed with cowboys,cattlemen, gamblers, an town folks. Cross the street heah an walk up thetside an down on this side. Yu men held upNeece thet night an robbed him. Wal, this gang of three was after himfor reasons that bear strong in this deal. An lettin that gambler Howard runaround with her. Range talk blames me for alot thet Im innocent of. She met his piercing gaze with understanding, and avisible shudder. If they would be, yud never get a vote,onless from some of yore hired hands. No noticeable change showed in the ranchers pale face. Dealing Surface a powerful left-handed blow,Brazos knocked him flat. Lura designated her bag, which Brazostook up. Whats to keep mefrom shootin Raine Surfaces laig off? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ambiguity.gif Type: image/gif Size: 6995 bytes Desc: not available URL: From lroubeyrie at limair.asso.fr Mon Sep 18 03:38:34 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 09:38:34 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609151605.56207.faltet@carabos.com> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609151605.56207.faltet@carabos.com> Message-ID: <200609180938.34721.lroubeyrie@limair.asso.fr> Le vendredi 15 septembre 2006 16:05, Francesc Altet a ?crit?: > Another possibility is to play with columns directly from the initial > recarray. The next is an example: > > In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) > In [102]: ra > Out[102]: > recarray([('1111', 825307441, 2.5784852031307537e-09), > ('1111', 825307441, 2.5784852031307537e-09), > ('1111', 825307441, 2.5784852031307537e-09)], > dtype=[('f0', '|S4'), ('f1', ' In [103]: rb=numpy.rec.fromarrays([numpy.array(ra['f0'], 'i4'),ra['f2']], > names='f0,f1') > In [104]: rb > Out[104]: > recarray([(1111, 2.5784852031307537e-09), (1111, 2.5784852031307537e-09), > (1111, 2.5784852031307537e-09)], > dtype=[('f0', ' > where ra is the original recarray and rb is a derived one where its first > column is the original from ra, but converted to integers ('i4'), and the > second it's the third column from ra (so the second column from ra has been > stripped out from rb). I have a problem with that : lionel[ETD-2006-01__PM2.5_DALTON]334>datas[0:5] Sortie[334]: [['Dates ', 'PM10 ', 'c10', 'PM2.5 ', 'c2.5'], ['05/01/2006', '33', 'A', '', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A'], ['08/01/2006', '16', 'A', '13', 'A']] lionel[ETD-2006-01__PM2.5_DALTON] 335>ra=rec.array(datas[1:],formats='a10,i2,a1,i2,a1') lionel[ETD-2006-01__PM2.5_DALTON]336>ra[0:5] Sortie[336]: recarray([[('05/01/2006', 0, '', 0, ''), ('33', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('30', 0, '', 0, ''), ('N\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('06/01/2006', 0, '', 0, ''), ('41', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('30', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('07/01/2006', 0, '', 0, ''), ('20', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('16', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('08/01/2006', 0, '', 0, ''), ('16', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('13', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('09/01/2006', 0, '', 0, ''), ('18', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('15', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')]], dtype=[('f1', '|S10'), ('f2', ' BUG in numpy.sum? Message-ID: <450E6D2E.9010706@obs.univ-lyon1.fr> Hi again after some hours of debugging I finally (I think) found the problem: numpy.sum([[0,1,2],[2,3,4]]) 24 numpy.sum([[0,1,2],[2,3,4]],axis=0) array([2, 4, 6]) numpy.sum([[0,1,2],[2,3,4]],axis=1) array([3, 9]) Isn't the first line supposed to act as with "axis=0" by default (see help numpy.sum!)...??? Not setting axis=0 it sums everything! Is that the expected behaviour or is the doc not updated? (I realise it may be better to force axis=.. to indicate what you need but still.. the example in numpy.sum help gives: sum([[0, 1], [0, 5]]) array([0, 6]) which is not what it gives when you do it...) thanks Eric From faltet at carabos.com Mon Sep 18 06:17:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 18 Sep 2006 12:17:03 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609180938.34721.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609151605.56207.faltet@carabos.com> <200609180938.34721.lroubeyrie@limair.asso.fr> Message-ID: <1158574623.4218.6.camel@localhost.localdomain> El dl 18 de 09 del 2006 a les 09:38 +0200, en/na Lionel Roubeyrie va escriure: > Le vendredi 15 septembre 2006 16:05, Francesc Altet a ?crit : > > Another possibility is to play with columns directly from the initial > > recarray. The next is an example: > > > > In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) > > In [102]: ra > > Out[102]: > > recarray([('1111', 825307441, 2.5784852031307537e-09), > > ('1111', 825307441, 2.5784852031307537e-09), > > ('1111', 825307441, 2.5784852031307537e-09)], > > dtype=[('f0', '|S4'), ('f1', ' > In [103]: rb=numpy.rec.fromarrays([numpy.array(ra['f0'], 'i4'),ra['f2']], > > names='f0,f1') > > In [104]: rb > > Out[104]: > > recarray([(1111, 2.5784852031307537e-09), (1111, 2.5784852031307537e-09), > > (1111, 2.5784852031307537e-09)], > > dtype=[('f0', ' > > > where ra is the original recarray and rb is a derived one where its first > > column is the original from ra, but converted to integers ('i4'), and the > > second it's the third column from ra (so the second column from ra has been > > stripped out from rb). > > I have a problem with that : > lionel[ETD-2006-01__PM2.5_DALTON]334>datas[0:5] > Sortie[334]: > [['Dates ', 'PM10 ', 'c10', 'PM2.5 ', 'c2.5'], > ['05/01/2006', '33', 'A', '', 'N'], > ['06/01/2006', '41', 'A', '30', 'A'], > ['07/01/2006', '20', 'A', '16', 'A'], > ['08/01/2006', '16', 'A', '13', 'A']] > > lionel[ETD-2006-01__PM2.5_DALTON] > 335>ra=rec.array(datas[1:],formats='a10,i2,a1,i2,a1') > > lionel[ETD-2006-01__PM2.5_DALTON]336>ra[0:5] > Sortie[336]: > recarray([[('05/01/2006', 0, '', 0, ''), ('33', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('30', 0, '', 0, ''), > ('N\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('06/01/2006', 0, '', 0, ''), ('41', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('30', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('07/01/2006', 0, '', 0, ''), ('20', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('16', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('08/01/2006', 0, '', 0, ''), ('16', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('13', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('09/01/2006', 0, '', 0, ''), ('18', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('15', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')]], > dtype=[('f1', '|S10'), ('f2', ' ('f5', '|S1')]) > > I have some missing entries, is it for that or do I have to make some changes > on the date column? You have two problems here. The first is that you shouldn't have missign entries, or conversion from empty strings to ints (or whatever) will fail: >>> int('') Traceback (most recent call last): File "", line 1, in ? ValueError: invalid literal for int(): Second, you can't feed a string of literals directly into the rec.array constructor (it is not as intelligent to digest this yet). You can achieve what you want by first massaging the data a bit: >>> ra=numpy.rec.array(datas[1:]) >>> numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, 'A'), ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, 'A')], dtype=[('f1', '|S10'), ('f2', '>> ca=numpy.array(datas[1:]) >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, 'A'), ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, 'A')], dtype=[('f1', '|S10'), ('f2', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From fullung at gmail.com Mon Sep 18 06:17:26 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 18 Sep 2006 12:17:26 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: Hey Eric > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! The problem is that the sum axis default is None, not 0. > Is that the expected behaviour or is the doc not updated? (I realise it > may be better to force axis=.. to indicate what you need but still.. the > example in numpy.sum help gives: > sum([[0, 1], [0, 5]]) > array([0, 6]) which is not what it gives when you do it...) The documentation seems to be outdated. I'm not sure, but I think a few of the axis default might have changed in the not too distant past. Not sure if sum was affected. Either way, could you please file a ticket? http://projects.scipy.org/scipy/numpy/register http://projects.scipy.org/scipy/numpy/newticket Regards, Albert From svetosch at gmx.net Mon Sep 18 06:20:49 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 18 Sep 2006 12:20:49 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> References: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: <450E7301.3020106@gmx.net> Eric Emsellem schrieb: > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! The default axis was changed recently to None (btw, which version are you using?), see the release notes on the web. So yes, it's expected. > > Is that the expected behaviour or is the doc not updated? (I realise it > may be better to force axis=.. to indicate what you need but still.. the > example in numpy.sum help gives: > sum([[0, 1], [0, 5]]) > array([0, 6]) which is not what it gives when you do it...) on my 1.0b5 I also see this docstring which indeed seems obsolete. Has it been changed since then? (And similarly for the other functions/methods where axis arguments have changed.) I could check the docstrings systematically and update them, but unfortunately not before the end of this month, sorry. cheers, sven From qwe239 at tom.com Sat Sep 23 08:32:14 2006 From: qwe239 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Sat, 23 Sep 2006 20:32:14 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO3Js8XMxKPE4r/Os8w=?= Message-ID: An HTML attachment was scrubbed... URL: From lroubeyrie at limair.asso.fr Mon Sep 18 11:10:10 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 17:10:10 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <1158574623.4218.6.camel@localhost.localdomain> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609180938.34721.lroubeyrie@limair.asso.fr> <1158574623.4218.6.camel@localhost.localdomain> Message-ID: <200609181710.10496.lroubeyrie@limair.asso.fr> Le lundi 18 septembre 2006 12:17, Francesc Altet a ?crit?: > You have two problems here. The first is that you shouldn't have missign > entries, or conversion from empty strings to ints (or whatever) will > > fail: > >>> int('') > > Traceback (most recent call last): > File "", line 1, in ? > ValueError: invalid literal for int(): > > Second, you can't feed a string of literals directly into the rec.array > constructor (it is not as intelligent to digest this yet). You can > > achieve what you want by first massaging the data a bit: > >>> ra=numpy.rec.array(datas[1:]) > > numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats >='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), > ('06/01/2006', 41, 'A', 30, 'A'), > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > 'A')], > dtype=[('f1', '|S10'), ('f2', ' ' > or, a bit more easier, > > >>> ca=numpy.array(datas[1:]) > >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') > > recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, > 'A'), > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > 'A')], > dtype=[('f1', '|S10'), ('f2', ' ' > > Cheers, Hi, thanks for your help, but I don't understand why is not working here: lionel[ETD-2006-01__PM2.5_DALTON]624>datas Sortie[624]: [['05/01/2006', '33', 'A', '10', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A']] lionel[ETD-2006-01__PM2.5_DALTON]625>ra=rec.array(datas) lionel[ETD-2006-01__PM2.5_DALTON]626>ra Sortie[626]: recarray([('05/01/2006', '33', 'A', '10', 'N'), ('06/01/2006', '41', 'A', '30', 'A'), ('07/01/2006', '20', 'A', '16', 'A')], dtype=[('f1', '|S10'), ('f2', '|S2'), ('f3', '|S1'), ('f4', '|S2'), ('f5', '|S1')]) lionel[ETD-2006-01__PM2.5_DALTON]627>rec.fromarrays( [ra['f1'], ra['f2'], ra['f4']], formats='a10,i2,i2') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/lionel/Etudes_Techniques/ETD-2006-01__PM2.5_DALTON/ /usr/lib/python2.4/site-packages/numpy/core/records.py in fromarrays(arrayList, formats, names, titles, shape, aligned) 235 # populate the record array (makes a copy) 236 for i in range(len(arrayList)): --> 237 _array[_names[i]] = arrayList[i] 238 239 return _array TypeError: array cannot be safely cast to required type -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From faltet at carabos.com Mon Sep 18 11:40:40 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 18 Sep 2006 17:40:40 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609181710.10496.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609180938.34721.lroubeyrie@limair.asso.fr> <1158574623.4218.6.camel@localhost.localdomain> <200609181710.10496.lroubeyrie@limair.asso.fr> Message-ID: <1158594041.4218.11.camel@localhost.localdomain> El dl 18 de 09 del 2006 a les 17:10 +0200, en/na Lionel Roubeyrie va escriure: > Le lundi 18 septembre 2006 12:17, Francesc Altet a ?crit : > > You have two problems here. The first is that you shouldn't have missign > > entries, or conversion from empty strings to ints (or whatever) will > > > > fail: > > >>> int('') > > > > Traceback (most recent call last): > > File "", line 1, in ? > > ValueError: invalid literal for int(): > > > > Second, you can't feed a string of literals directly into the rec.array > > constructor (it is not as intelligent to digest this yet). You can > > > > achieve what you want by first massaging the data a bit: > > >>> ra=numpy.rec.array(datas[1:]) > > > > numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats > >='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), > > ('06/01/2006', 41, 'A', 30, 'A'), > > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > > 'A')], > > dtype=[('f1', '|S10'), ('f2', ' > ' > > > or, a bit more easier, > > > > >>> ca=numpy.array(datas[1:]) > > >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') > > > > recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, > > 'A'), > > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > > 'A')], > > dtype=[('f1', '|S10'), ('f2', ' > ' > > > > > Cheers, > > Hi, > thanks for your help, but I don't understand why is not working here: > lionel[ETD-2006-01__PM2.5_DALTON]624>datas > Sortie[624]: > [['05/01/2006', '33', 'A', '10', 'N'], > ['06/01/2006', '41', 'A', '30', 'A'], > ['07/01/2006', '20', 'A', '16', 'A']] > > lionel[ETD-2006-01__PM2.5_DALTON]625>ra=rec.array(datas) > > lionel[ETD-2006-01__PM2.5_DALTON]626>ra > Sortie[626]: > recarray([('05/01/2006', '33', 'A', '10', 'N'), > ('06/01/2006', '41', 'A', '30', 'A'), > ('07/01/2006', '20', 'A', '16', 'A')], > dtype=[('f1', '|S10'), ('f2', '|S2'), ('f3', '|S1'), ('f4', '|S2'), > ('f5', '|S1')]) > > lionel[ETD-2006-01__PM2.5_DALTON]627>rec.fromarrays( [ra['f1'], ra['f2'], > ra['f4']], formats='a10,i2,i2') > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) Mmm, this works for me: >>> datas [['05/01/2006', '33', 'A', '0', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A'], ['08/01/2006', '16', 'A', '13', 'A']] >>> ra=rec.array(datas) >>> rec.fromarrays([ra['f1'],ra['f2'],ra['f4']], formats='a10,i2,i2') recarray([('05/01/2006', 33, 0), ('06/01/2006', 41, 30), ('07/01/2006', 20, 16), ('08/01/2006', 16, 13)], dtype=[('f1', '|S10'), ('f2', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From lroubeyrie at limair.asso.fr Mon Sep 18 11:47:07 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 17:47:07 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <1158594041.4218.11.camel@localhost.localdomain> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609181710.10496.lroubeyrie@limair.asso.fr> <1158594041.4218.11.camel@localhost.localdomain> Message-ID: <200609181747.07852.lroubeyrie@limair.asso.fr> Le lundi 18 septembre 2006 17:40, Francesc Altet a ?crit?: > I'm running NumPy 1.0b5. Please, check that you are using a recent > version of it. > > Cheers, Arg, sorry, version here was 0.9, an upgrade and it works fine. thanks again -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From myeates at jpl.nasa.gov Mon Sep 18 12:29:24 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 18 Sep 2006 09:29:24 -0700 Subject: [Numpy-discussion] visual programming Message-ID: <450EC964.6080304@jpl.nasa.gov> semi off topic. Does anyone know of any good visual programming tools? Forever ago, I used to use something called Khoros for image processing and I found it very useful. You connect boxes which represent different processing steps. At about the same time, there was something similar to Khoros from SGI. Explorer was its name, I think. Anybody out there use this sort of thing? Matlab doesn't offer it, right? Nor matplotlib? Mathew From mg.mailing-list at laposte.net Mon Sep 18 12:55:31 2006 From: mg.mailing-list at laposte.net (mg) Date: Mon, 18 Sep 2006 18:55:31 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy Message-ID: <450ECF83.9020103@laposte.net> Hi all, I am doing a feseability study to migrate our Python based FEM applications from Numarray to Numpy. First, I tried to install Numpy from Python-2.4 on linux-x86, linux-86-64bit. So, all work fine. Great! Moreover, I change easily the BLAS linked libraries. I tried with ATLAS and GOTO. Great again! Second, I try to do the same think on windows-x86 without success. So my first question is: is Numpy-1.0b5 has been tested and is supported on Windows? Third, I tried to install Numpy from Python-2.5, which is our standard Python, on linux-x86... and the compilation stopped during the compilation of core/src/multiarraymodule.c. So my second question is: is there a workaround or is the porting to Python2.5 is yet schedule? My third question is: is the tool to migrate the numarray based Python scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) We have created a lot of bindings in order to pilote our generic-C++ framework with Python scripts. So, about the Numpy API, is it widely different than the Numarray API? (We will order the Numpy Guide too.) To not duplicate large numerical memory arrays, Numarray allows to aliasing the memory of some bindings with arrays from Numarray, and we have used this feature intensively. So, I wonder if it is currently supported (or even scheduled)? Thanks for your answer, Mathieu. From schaffer at optonline.net Mon Sep 18 12:59:52 2006 From: schaffer at optonline.net (Les Schaffer) Date: Mon, 18 Sep 2006 12:59:52 -0400 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: <450ED088.80309@optonline.net> Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. > if i recall correctly, there is still a version of Khoros floating around that you can compile under Linux. i've done it several time, but it WAS a while ago. Les Schaffer From Chris.Barker at noaa.gov Mon Sep 18 13:08:48 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 18 Sep 2006 10:08:48 -0700 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E7301.3020106@gmx.net> References: <450E6D2E.9010706@obs.univ-lyon1.fr> <450E7301.3020106@gmx.net> Message-ID: <450ED2A0.50208@noaa.gov> Sven Schreiber wrote: > on my 1.0b5 I also see this docstring which indeed seems obsolete. I get this docs string from : >>> import numpy as N >>> N.__version__ '1.0b5' >>> a = N.arange(10) >>> help( a.sum) """ sum(...) a.sum(axis=None, dtype=None) -> Sum of array over given axis. Sum the array over the given axis. If the axis is None, sum over all dimensions of the array. """ that looks right to me. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From peridot.faceted at gmail.com Mon Sep 18 13:17:45 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 18 Sep 2006 13:17:45 -0400 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: On 18/09/06, Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew I doubt it's what you're looking for, but PD and GEM are a visual programming environment, oriented towards music, audio signal processing, 2D and 3D video. (They're related to Max and jMax, which do some of the same things.) A. M. Archibald From charlesr.harris at gmail.com Mon Sep 18 13:35:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 11:35:23 -0600 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: On 9/18/06, Mathew Yeates wrote: > > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew There have been several presentations at the scipy conferences about a visual programming environment for molecular visualization. You might check and see if the presentations are available on the scipy site. Lets see, name is ViPEr, and some papers can be found under publications at this site: http://www.scripps.edu/%7Esanner/python/index.html I have no idea how generally useful the framework is. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Sep 18 14:46:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 12:46:17 -0600 Subject: [Numpy-discussion] record descriptors: trailing unused bytes In-Reply-To: <200609172017.09548.martin.wiechert@gmx.de> References: <200609172017.09548.martin.wiechert@gmx.de> Message-ID: <450EE979.706@ieee.org> Martin Wiechert wrote: > Hi list, > > does anybody know, if in C in the PyArray_Descr struct it is safe to manually > change descr->elsize and descr->alignment as long as these are compatible and > descr->elsize is large enough to hold all fields? Of course I mean before any > array is constructed using descr. > Well, you should really make a copy of the PyArray_Descr structure and then fill it in as desired unless you are sure that no other arrays have are using that particular data-type object to describe their own data: PyArray_DescrNew gives you a new copy based on an old one (you just get a reference to the funciton pointers in the 'f' member so don't go changing those). I guess your statement about "before any array is constructed using descr" means you are sure that there are no other arrays using the old elsize and alignment. Then it should be fine. There are not supposed to be any assumptions about the data-types except for what is explicitly provided in the data-type object (PyArray_Descr *). So, changing them will change the data-type and things should ideally work. -Travis From matthew.brett at gmail.com Mon Sep 18 14:49:54 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Sep 2006 19:49:54 +0100 Subject: [Numpy-discussion] Changing byte ordering with astype fails with 0d arrays Message-ID: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> Hi, As expected: In [67]:a = array([1], dtype='i4').dtype Out[68]:dtype('>i4') I was also expecting this to work for 0d arrays, but it doesn't: In [69]:a = array(1, dtype='i4').dtype Out[70]:dtype(' BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> References: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: <450EEAE8.6060706@ieee.org> Eric Emsellem wrote: > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! > See the Release Notes page on www.scipy.org. It documents everything that has changed. Several things will break old code as indicated. There are several options for keeping old code working: 1) Use the numpy.oldnumeric compatibility layer which keeps the same definitions and defaults as Numeric 2) Use conversion tools (like the recently added fix_default_axis) tool to automatically insert axis=0 arguments in all code where it is not present (or to automatically change the import to oldnumeric). For the future, you must specify which axis you mean for a Nd array or the code will assume you meant to work over the entire N-d array. We all recognize this is a pain to change. That's why the backward compatibilty options are avaiable and the tools have been written. Believe me, I know what a pain it is. I have had to keep SciPy and Matplotlib working with all the changes to NumPy. -Travis From oliphant.travis at ieee.org Mon Sep 18 14:58:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 12:58:26 -0600 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450ECF83.9020103@laposte.net> References: <450ECF83.9020103@laposte.net> Message-ID: <450EEC52.30906@ieee.org> mg wrote: > Hi all, > > I am doing a feseability study to migrate our Python based FEM > applications from Numarray to Numpy. > > First, I tried to install Numpy from Python-2.4 on linux-x86, > linux-86-64bit. So, all work fine. Great! Moreover, I change easily the > BLAS linked libraries. I tried with ATLAS and GOTO. Great again! > > Second, I try to do the same think on windows-x86 without success. So my > first question is: is Numpy-1.0b5 has been tested and is supported on > Windows? > Yes, it should work. Builds for windows were provided. But, perhaps there are configuration issues for your system that we are not handling correctly. > Third, I tried to install Numpy from Python-2.5, which is our standard > Python, on linux-x86... and the compilation stopped during the > compilation of core/src/multiarraymodule.c. So my second question is: is > there a workaround or is the porting to Python2.5 is yet schedule? > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. Look for NumPy 1.0rc1 to come out soon. > My third question is: is the tool to migrate the numarray based Python > scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) > It needs more testing. It would be great if you could help us find and fix bugs in it. I don't have a lot of numarray code to test. > We have created a lot of bindings in order to pilote our generic-C++ > framework with Python scripts. So, about the Numpy API, is it widely > different than the Numarray API? (We will order the Numpy Guide too.) > It is more similar to the Numeric C-API. However, the numarray C-API is completely supported by including numpy/libnumarray.h so you should be able to convert your C code very easily. Any problems encountered should be noted and we'll get them fixed. > To not duplicate large numerical memory arrays, Numarray allows to > aliasing the memory of some bindings with arrays from Numarray, and we > have used this feature intensively. So, I wonder if it is currently > supported (or even scheduled)? I'm pretty sure the answer is yes (because the Numarray C-API is supported), though I'm not exactly sure what you mean. Do you mean that you have memory created in the C/C++ framework and then you have an array use that memory for it's data area? If that is what you mean, then the answer is definitely yes. -Travis From oliphant.travis at ieee.org Mon Sep 18 15:01:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 13:01:19 -0600 Subject: [Numpy-discussion] Changing byte ordering with astype fails with 0d arrays In-Reply-To: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> References: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> Message-ID: <450EECFF.9050809@ieee.org> Matthew Brett wrote: > Hi, > > As expected: > > In [67]:a = array([1], dtype=' > In [68]:a.astype('>i4').dtype > Out[68]:dtype('>i4') > > I was also expecting this to work for 0d arrays, but it doesn't: > > In [69]:a = array(1, dtype=' > In [70]:a.astype('>i4').dtype > Out[70]:dtype(' > The problem is that the astype method is returning an array scalar (it used to be that 0-d arrays were "avoided" at all costs). We've since relaxes this requirement and I think here's another place where it needs to be relaxed. -Travis From Chris.Barker at noaa.gov Mon Sep 18 15:37:36 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 18 Sep 2006 12:37:36 -0700 Subject: [Numpy-discussion] 1.0b5 Installation on OS X In-Reply-To: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> References: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> Message-ID: <450EF580.90103@noaa.gov> David Andrews wrote: > I'm unable to install the 1.0b5 release from the .mpkg on OS X - when > it comes to the two checkboxes to select the platlib and the scripts, > both are greyed out. The installer then claims 'Cannot continue: > Nothing to install'. hmmm. It works just fine for me -- just clicking OK as I go merrily along. OS version? python version? processor? Also, try deleting or renaming any existing install you have from site-packages, then try again. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthew.brett at gmail.com Mon Sep 18 15:57:16 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Sep 2006 20:57:16 +0100 Subject: [Numpy-discussion] Segfault on byteswap() on recarrays Message-ID: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> Hi, I noticed this works: In [5]:a = array((1,), dtype=[('one', ' I need build core numpy (plus possible dft) static into python.a Can someone point me to the right direction. Currently I built some libs but blocked on _internal. From my Setup.local: NPSRC=/home/mrovner/src/numpy-0.9.8 NPBLD=$(NPSRC)/build/src.linux-i686-2.4 NPINC=-I$(NPSRC)/numpy/core/include -I$(NPBLD)/numpy/core -I$(NPSRC)/numpy/core/src multiarray $(NPSRC)/numpy/core/src/multiarraymodule.c $(NPINC) umath $(NPBLD)/numpy/core/src/umathmodule.c $(NPINC) scalarmath $(NPBLD)/numpy/core/src/scalarmathmodule.c $(NPINC) _sort $(NPBLD)/numpy/core/src/_sortmodule.c $(NPINC) _compiled_base $(NPSRC)/numpy/lib/src/_compiled_base.c $(NPINC) Thanks, Mike From svetosch at gmx.net Mon Sep 18 16:30:22 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 18 Sep 2006 22:30:22 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450ED2A0.50208@noaa.gov> References: <450E6D2E.9010706@obs.univ-lyon1.fr> <450E7301.3020106@gmx.net> <450ED2A0.50208@noaa.gov> Message-ID: <450F01DE.9070607@gmx.net> Christopher Barker schrieb: > Sven Schreiber wrote: >> on my 1.0b5 I also see this docstring which indeed seems obsolete. > > I get this docs string from : > > >>> import numpy as N > >>> N.__version__ > '1.0b5' > >>> a = N.arange(10) > >>> help( a.sum) > > """ > sum(...) > a.sum(axis=None, dtype=None) -> Sum of array over given axis. > > Sum the array over the given axis. If the axis is None, sum over > all dimensions of the array. > > """ > > that looks right to me. > > -Chris > Well here's what we had in mind: sum(x, axis=None, dtype=None, out=None) (...) Examples: >>> sum([0.5, 1.5]) 2.0 >>> sum([0.5, 1.5], dtype=Int32) 1 >>> sum([[0, 1], [0, 5]]) array([0, 6]) >>> sum([[0, 1], [0, 5]], axis=1) array([1, 5]) And you can see the third example would now be wrong, giving a scalar of 6 now. -sven From oliphant.travis at ieee.org Mon Sep 18 16:53:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 14:53:44 -0600 Subject: [Numpy-discussion] Segfault on byteswap() on recarrays In-Reply-To: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> References: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> Message-ID: <450F0758.8080607@ieee.org> Matthew Brett wrote: > Hi, > > I noticed this works: > > In [5]:a = array((1,), dtype=[('one', ' > In [6]:a.byteswap() > Out[6]: > array((16777216,), > dtype=[('one', ' > But, extending the recarray leads to a segfault on byteswapping: > > In [8]:a = array((1, 2), dtype=[('one', ' > In [9]:a.byteswap() > Segmentation fault > Great catch. Fixed in SVN. -Travis From bryan at cole.uklinux.net Mon Sep 18 17:53:33 2006 From: bryan at cole.uklinux.net (Bryan Cole) Date: Mon, 18 Sep 2006 22:53:33 +0100 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: <1158616413.8450.49.camel@pc1.cole.uklinux.net> On Mon, 2006-09-18 at 09:29 -0700, Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. Looking at the Khoros commercial site (now AccuSoft), it looks like a more polished version of OpenDX (which also offers a visual programming system). See http://www.opendx.org/index2.php The problem with OpenDX is the cr*ppy Motif-based GUI (particularly so if you use Windows). I prefer VTK (www.vtk.org) which offers similar concepts (pipeline execution of data filtering/visualisation blocks) but in a programmatic form based on python (which we like...). BC > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 From Martin.Wiechert at mpimf-heidelberg.mpg.de Mon Sep 18 18:12:31 2006 From: Martin.Wiechert at mpimf-heidelberg.mpg.de (Martin Wiechert) Date: Tue, 19 Sep 2006 00:12:31 +0200 Subject: [Numpy-discussion] record descriptors: trailing unused bytes In-Reply-To: <450EE979.706@ieee.org> References: <200609172017.09548.martin.wiechert@gmx.de> <450EE979.706@ieee.org> Message-ID: <200609190012.31853.wiechert@mpimf-heidelberg.mpg.de> On Monday 18 September 2006 20:46, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > does anybody know, if in C in the PyArray_Descr struct it is safe to > > manually change descr->elsize and descr->alignment as long as these are > > compatible and descr->elsize is large enough to hold all fields? Of > > course I mean before any array is constructed using descr. > > Well, you should really make a copy of the PyArray_Descr structure and > then fill it in as desired unless you are sure that no other arrays have > are using that particular data-type object to describe their own data: > PyArray_DescrNew gives you a new copy based on an old one (you just get > a reference to the funciton pointers in the 'f' member so don't go > changing those). > > I guess your statement about "before any array is constructed using > descr" means you are sure that there are no other arrays using the old > elsize and alignment. > > Then it should be fine. There are not supposed to be any assumptions > about the data-types except for what is explicitly provided in the > data-type object (PyArray_Descr *). So, changing them will change the > data-type and things should ideally work. > > -Travis > Thanks > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- AV scanned by FortiGate From stefan at sun.ac.za Mon Sep 18 20:07:27 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 19 Sep 2006 02:07:27 +0200 Subject: [Numpy-discussion] unique1d: renaming retindx to return_index Message-ID: <20060919000727.GA12696@mentat.za.net> Hi, Would anyone object if I changed the signature of unique1d(ar1, retindx=False) to unique1d(ar1, return_index=False)? I find retindx both harder to read and to type than return_index. Thanks. St?fan From kslmns at clubtheatre.org.uk Mon Sep 18 20:03:59 2006 From: kslmns at clubtheatre.org.uk (Sam Estes) Date: Mon, 18 Sep 2006 19:03:59 -0500 Subject: [Numpy-discussion] persecute ambidextrous Message-ID: <001c01c6db80$620f899a$7b40d041@xlwmk.qdkvjf> I will shed no blood, said Wi, not even that of those who hate me,for misery makes them mad. That stone was ill aimed, said Pag who stood by. I tried to stay them but theyfelled me with clubs, for they are fierce as wolves and more savage. Aye, let us go to the cave, for if we stay here upon the ice we shallperish, said Pag. He looked down upon the people and, by the shimmering moonlight,watched their faces. The people who called upon the Ice-dwellers, where arethey? If he is afraid todie, then let him give another to the gods. I am sure that it will be so, she replied simply. I tell you nothing, Wife, he answered sternly. Now let Wi the mightyman, our chief who rejects you, make answer to it if he is able. Wi, you have your answer, piped Ngae, as the shouting died. Was it not well that one should die for the sake ofmany? Shall we gather men and fall on them and kill them? I brought the trouble, Wi; surely I should have paid its price. I brought the trouble, Wi; surely I should have paid its price. They cowered before him and muttered together. All the white world was a desolation and a waste. Then he had come thither because theplace was holy to him. For he had sent command to Aaka that she must nomore sleep alone in her hut. They say that the Ice-gods demand ahuman sacrifice and that this sacrifice must be given to them. He, Wi himself,whom they dared not touch because he was chief and too strong forthem. They saw it creep into the sea and there break off in sharp-toppedhills of ice. Yes, a Voice has spoken tothem from the roof of their hut in the dead of night. Now let Wi the mightyman, our chief who rejects you, make answer to it if he is able. They are starvingbrutes, and such will fill themselves. Or, if you dare not, then send us oneof your household. Their prayers finished, they spoke together. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: warm-up.gif Type: image/gif Size: 8483 bytes Desc: not available URL: From seth.olsen at gmail.com Mon Sep 18 20:51:55 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 10:51:55 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi Numpyers, I recently sent the message below to the MMTK mailing list, but it's really a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors() never stops, even when I try to diagonalize a very simple 2x2 matrix. I've tried this on two machines with completely standard FC4 and FC5 installations using the default libraries and atlas libraries downloaded and installed via yum. Does anyone on this list have any experience with this problem? Cheers, Seth ---------- Forwarded message ---------- From: Dr. Seth Olsen Date: Sep 19, 2006 10:38 AM Subject: Re: Collection.findTransformation() never stops To: mmtk at python.net, mmtk at starship.python.net Hi MMTKers, The problem with Collection.findTransformation() that I reported earlier is due to a problem with LinearAlgebra.eigenvectors(). This algortithm does not seem to stop - it can't even manage to find the eigenvectors of the 2x2 square matrix [[2,1],[1,2]]. Asking for this causes a long wait followed by CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with newly installed atlas libraries, but it still won't work. I have had this problem now on 2 machines (one running FC4 with , the other FC5, both pentium 4 machines), with atlas lapack/blas libraries installed freshly via yum and linked as per the recommendations found in setup.py. THE FEDORA INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been downloaded and installed via the fedora yum repositories. As the eigenvectors() routine in LinearAlgebra.py is obviously going to be a heavily used algorithm (not just in MMTK), is it really possible that no one has had this problem before? Cheers, Seth On 9/18/06, Dr. Seth Olsen wrote: > > > Hi MMTKers, > > I'm having a bit of trouble with MMTK.Collection.findTransformation. When > I use this member function, the computer never stops thinking until I kill > the process. I've waited a couple of hours and still nothing. I get the > same thing if I try to take a step back and use > Collection.findTransformationAsQuaternion. Has anyone had this problem > before? > > Cheers, > > Seth > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 21:34:50 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 10:34:50 +0900 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hey there, I don't see anything called "LinearAlgegra.eigenvectors()". Do you maybe mean numpy.linalg.eig? Which version of numpy are you using? The latest release is 1.0b5. >>> import numpy >>> numpy.__version__ '1.0b5' >>> numpy.linalg.eig([[2,1],[1,2]]) (array([ 3., 1.]), array([[ 0.70710678, -0.70710678], [ 0.70710678, 0.70710678]])) I'm on WindowsXP, though. Regards, Bill On 9/19/06, Dr. Seth Olsen wrote: > > Hi Numpyers, > > I recently sent the message below to the MMTK mailing list, but it's really > a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors() > never stops, even when I try to diagonalize a very simple 2x2 matrix. I've > tried this on two machines with completely standard FC4 and FC5 > installations using the default libraries and atlas libraries downloaded and > installed via yum. Does anyone on this list have any experience with this > problem? > > Cheers, > > Seth > > ---------- Forwarded message ---------- > From: Dr. Seth Olsen > Date: Sep 19, 2006 10:38 AM > Subject: Re: Collection.findTransformation() never stops > To: mmtk at python.net, mmtk at starship.python.net > > > Hi MMTKers, > > The problem with Collection.findTransformation() that I reported earlier is > due to a problem with LinearAlgebra.eigenvectors(). This algortithm does > not seem to stop - it can't even manage to find the eigenvectors of the 2x2 > square matrix [[2,1],[1,2]]. Asking for this causes a long wait followed by > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with newly > installed atlas libraries, but it still won't work. I have had this problem > now on 2 machines (one running FC4 with , the other FC5, both pentium 4 > machines), with atlas lapack/blas libraries installed freshly via yum and > linked as per the recommendations found in setup.py. THE FEDORA > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been downloaded > and installed via the fedora yum repositories. > > As the eigenvectors() routine in LinearAlgebra.py is obviously going to be a > heavily used algorithm (not just in MMTK), is it really possible that no one > has had this problem before? > > Cheers, > > Seth > > > On 9/18/06, Dr. Seth Olsen wrote: > > > > > > Hi MMTKers, > > > > I'm having a bit of trouble with > MMTK.Collection.findTransformation. When I use this member > function, the computer never stops thinking until I kill the process. I've > waited a couple of hours and still nothing. I get the same thing if I try > to take a step back and use > Collection.findTransformationAsQuaternion. Has anyone had > this problem before? > > > > Cheers, > > > > Seth > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > > > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From seth.olsen at gmail.com Mon Sep 18 22:06:20 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 12:06:20 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi Bill, MMTK has not made the conversion over to the new numpy module. It is built against the old Numeric code, and the word from its developers is that changing to numpy cannot be a priority now. Cheers, Seth On 9/19/06, Bill Baxter wrote: > > Hey there, > I don't see anything called "LinearAlgegra.eigenvectors()". Do you > maybe mean numpy.linalg.eig? > Which version of numpy are you using? > The latest release is 1.0b5. > > >>> import numpy > >>> numpy.__version__ > '1.0b5' > > >>> numpy.linalg.eig([[2,1],[1,2]]) > (array([ 3., 1.]), > array([[ 0.70710678, -0.70710678], > [ 0.70710678, 0.70710678]])) > > I'm on WindowsXP, though. > Regards, > Bill > > On 9/19/06, Dr. Seth Olsen wrote: > > > > Hi Numpyers, > > > > I recently sent the message below to the MMTK mailing list, but it's > really > > a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors > () > > never stops, even when I try to diagonalize a very simple 2x2 > matrix. I've > > tried this on two machines with completely standard FC4 and FC5 > > installations using the default libraries and atlas libraries downloaded > and > > installed via yum. Does anyone on this list have any experience with > this > > problem? > > > > Cheers, > > > > Seth > > > > ---------- Forwarded message ---------- > > From: Dr. Seth Olsen > > Date: Sep 19, 2006 10:38 AM > > Subject: Re: Collection.findTransformation() never stops > > To: mmtk at python.net, mmtk at starship.python.net > > > > > > Hi MMTKers, > > > > The problem with Collection.findTransformation() that I reported earlier > is > > due to a problem with LinearAlgebra.eigenvectors(). This algortithm > does > > not seem to stop - it can't even manage to find the eigenvectors of the > 2x2 > > square matrix [[2,1],[1,2]]. Asking for this causes a long wait > followed by > > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with > newly > > installed atlas libraries, but it still won't work. I have had this > problem > > now on 2 machines (one running FC4 with , the other FC5, both pentium 4 > > machines), with atlas lapack/blas libraries installed freshly via yum > and > > linked as per the recommendations found in setup.py. THE FEDORA > > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been > downloaded > > and installed via the fedora yum repositories. > > > > As the eigenvectors() routine in LinearAlgebra.py is obviously going to > be a > > heavily used algorithm (not just in MMTK), is it really possible that no > one > > has had this problem before? > > > > Cheers, > > > > Seth > > > > > > On 9/18/06, Dr. Seth Olsen wrote: > > > > > > > > > Hi MMTKers, > > > > > > I'm having a bit of trouble with > > MMTK.Collection.findTransformation. When I use this member > > function, the computer never stops thinking until I kill the > process. I've > > waited a couple of hours and still nothing. I get the same thing if I > try > > to take a step back and use > > Collection.findTransformationAsQuaternion. Has anyone had > > this problem before? > > > > > > Cheers, > > > > > > Seth > > > > > > -- > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > positions of > > the University of Queensland. > > > > > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions > of > > the University of Queensland. > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions > of > > the University of Queensland. > > > ------------------------------------------------------------------------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > > opinions on IT & business topics through brief surveys -- and earn cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 22:07:05 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 11:07:05 +0900 Subject: [Numpy-discussion] unique1d: renaming retindx to return_index In-Reply-To: <20060919000727.GA12696@mentat.za.net> References: <20060919000727.GA12696@mentat.za.net> Message-ID: Grepping through numpy/**/*.py, the only three functions I could find with an argument to specify extra return values are: linspace(start, stop, num=50, endpoint=True, retstep=False) average(a, axis=None weights=None, returned=False) unique1d(ar1, retindx=False) If unique1d is going to change to return_index, then linspace should probably also change to return_step to maintain consistency. And maybe average should change to 'return_count' or [ick] 'return_denominator'. --bb On 9/19/06, Stefan van der Walt wrote: > Hi, > > Would anyone object if I changed the signature of > > unique1d(ar1, retindx=False) > > to > > unique1d(ar1, return_index=False)? > > I find retindx both harder to read and to type than return_index. > From wbaxter at gmail.com Mon Sep 18 22:21:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 11:21:19 +0900 Subject: [Numpy-discussion] max argmax combo Message-ID: I find myself often wanting both the max and the argmax of an array. (And same for the other arg* functions) Of course I can do argmax first then use fancy indexing to get the max as well. But the result of argmax isn't really in a format that's readily usable as an index. You have to do something like a = rand(10,5) imax = a.argmax(axis=0) vmax = a[(imax, range(5))] Which isn't terrible, just always takes me a moment to remember the proper indexing expression. Would a way to get the argmax and the max at the same time be of interest to anyone else? Maybe an extra 'ret' arg on argmax? a = rand(10,5) imax,vmax = a.argmax(axis=0,retmax=True) --Bill From charlesr.harris at gmail.com Mon Sep 18 22:56:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 20:56:39 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Bill Baxter wrote: > > I find myself often wanting both the max and the argmax of an array. > (And same for the other arg* functions) > Of course I can do argmax first then use fancy indexing to get the max as > well. > But the result of argmax isn't really in a format that's readily > usable as an index. > You have to do something like > a = rand(10,5) > imax = a.argmax(axis=0) > vmax = a[(imax, range(5))] > > Which isn't terrible, just always takes me a moment to remember the > proper indexing expression. > Would a way to get the argmax and the max at the same time be of > interest to anyone else? Maybe an extra 'ret' arg on argmax? > > a = rand(10,5) > imax,vmax = a.argmax(axis=0,retmax=True) I don't generally like overloading return values, the function starts to lose its definition and becomes a bit baroque where simply changing a keyword value can destroy the viability of the following code. But I can see the utility of what you want. Hmm, this problem is not unique to argmax. Maybe what we need is a general way to extract values, something like extract(a, imax, axis=0) to go along with all the single axis functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From seth.olsen at gmail.com Mon Sep 18 23:11:35 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 13:11:35 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi MMTKers and NUMPYers, Bill's answer to my inquiry about the problem I'm having with Collection.findTransformation() (and also, incidentally, with the dgesvd call in Subspace.getBasis(), has convinced me that I can no long use MMTK without changing some of the code over to numpy. I have already been able to determine that invoking numpy.oldnumeric.alter_code1.convertall() over the MMTK directory tree is not the answer. Has anyone on either of these lists ever tried this before and, if so, can it be done (without destroying my sanity)? Cheers, Seth On 9/19/06, Dr. Seth Olsen wrote: > > > Hi Bill, > > MMTK has not made the conversion over to the new numpy module. It is > built against the old Numeric code, and the word from its developers is that > changing to numpy cannot be a priority now. > > Cheers, > > Seth > > > On 9/19/06, Bill Baxter wrote: > > > > Hey there, > > I don't see anything called "LinearAlgegra.eigenvectors()". Do you > > maybe mean numpy.linalg.eig? > > Which version of numpy are you using? > > The latest release is 1.0b5. > > > > >>> import numpy > > >>> numpy.__version__ > > '1.0b5' > > > > >>> numpy.linalg.eig([[2,1],[1,2]]) > > (array([ 3., 1.]), > > array([[ 0.70710678, -0.70710678], > > [ 0.70710678, 0.70710678]])) > > > > I'm on WindowsXP, though. > > Regards, > > Bill > > > > On 9/19/06, Dr. Seth Olsen wrote: > > > > > > Hi Numpyers, > > > > > > I recently sent the message below to the MMTK mailing list, but it's > > really > > > a problem with LinearAlgebra.py. The routine > > LinearAlgebra.eigenvectors() > > > never stops, even when I try to diagonalize a very simple 2x2 > > matrix. I've > > > tried this on two machines with completely standard FC4 and FC5 > > > installations using the default libraries and atlas libraries > > downloaded and > > > installed via yum. Does anyone on this list have any experience with > > this > > > problem? > > > > > > Cheers, > > > > > > Seth > > > > > > ---------- Forwarded message ---------- > > > From: Dr. Seth Olsen > > > Date: Sep 19, 2006 10:38 AM > > > Subject: Re: Collection.findTransformation() never stops > > > To: mmtk at python.net, mmtk at starship.python.net > > > > > > > > > Hi MMTKers, > > > > > > The problem with Collection.findTransformation() that I reported > > earlier is > > > due to a problem with LinearAlgebra.eigenvectors(). This algortithm > > does > > > not seem to stop - it can't even manage to find the eigenvectors of > > the 2x2 > > > square matrix [[2,1],[1,2]]. Asking for this causes a long wait > > followed by > > > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with > > newly > > > installed atlas libraries, but it still won't work. I have had this > > problem > > > now on 2 machines (one running FC4 with , the other FC5, both pentium > > 4 > > > machines), with atlas lapack/blas libraries installed freshly via yum > > and > > > linked as per the recommendations found in setup.py . THE FEDORA > > > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been > > downloaded > > > and installed via the fedora yum repositories. > > > > > > As the eigenvectors() routine in LinearAlgebra.py is obviously going > > to be a > > > heavily used algorithm (not just in MMTK), is it really possible that > > no one > > > has had this problem before? > > > > > > Cheers, > > > > > > Seth > > > > > > > > > On 9/18/06, Dr. Seth Olsen < seth.olsen at gmail.com> wrote: > > > > > > > > > > > > Hi MMTKers, > > > > > > > > I'm having a bit of trouble with > > > MMTK.Collection.findTransformation . When I use this member > > > function, the computer never stops thinking until I kill the > > process. I've > > > waited a couple of hours and still nothing. I get the same thing if I > > try > > > to take a step back and use > > > Collection.findTransformationAsQuaternion. Has anyone had > > > this problem before? > > > > > > > > Cheers, > > > > > > > > Seth > > > > > > > > -- > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > > > Dr Seth Olsen, PhD > > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > > Centre for Computational Molecular Science > > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. > > 75) > > > > The University of Queensland > > > > Qld 4072, Brisbane, Australia > > > > > > > > tel (617) 3346 3976 > > > > fax (617) 33654623 > > > > email: s.olsen1 at uq.edu.au > > > > Web: www.ccms.uq.edu.au > > > > > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > > > > > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > ------------------------------------------------------------------------- > > > Take Surveys. Earn Cash. Influence the Future of IT > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > share your > > > opinions on IT & business topics through brief surveys -- and earn > > cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > your > > opinions on IT & business topics through brief surveys -- and earn cash > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 23:19:32 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 12:19:32 +0900 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/19/06, Charles R Harris wrote: > On 9/18/06, Bill Baxter wrote: > > I find myself often wanting both the max and the argmax of an array. > > (And same for the other arg* functions) > > You have to do something like > > a = rand(10,5) > > imax = a.argmax(axis=0) > > vmax = a[(imax, range(5))] > > > I don't generally like overloading return values, the function starts to > lose its definition and becomes a bit baroque where simply changing a > keyword value can destroy the viability of the following code. Agreed. Seems like the only justification is if you get multiple results from one calculation but only rarely want the extra values. It doesn't make sense to always return them, but it's also not worth making a totally different function. > But I can see the utility of what you want. Hmm, this problem is not unique to argmax. > Maybe what we need is a general way to extract values, something like > > extract(a, imax, axis=0) > > to go along with all the single axis functions. Yes, I think that would be easier to remember. It should also work for the axis=None case. imax = a.argmax(axis=None) v = extract(a, imax, axis=None) --Bill From charlesr.harris at gmail.com Tue Sep 19 00:05:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 22:05:42 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Bill Baxter wrote: > > On 9/19/06, Charles R Harris wrote: > > On 9/18/06, Bill Baxter wrote: > > > I find myself often wanting both the max and the argmax of an array. > > > (And same for the other arg* functions) > > > > You have to do something like > > > a = rand(10,5) > > > imax = a.argmax(axis=0) > > > vmax = a[(imax, range(5))] > > > > > I don't generally like overloading return values, the function starts to > > lose its definition and becomes a bit baroque where simply changing a > > keyword value can destroy the viability of the following code. > > Agreed. Seems like the only justification is if you get multiple > results from one calculation but only rarely want the extra values. > It doesn't make sense to always return them, but it's also not worth > making a totally different function. > > > > But I can see the utility of what you want. Hmm, this problem is not > unique to argmax. > > Maybe what we need is a general way to extract values, something like > > > > extract(a, imax, axis=0) > > > > to go along with all the single axis functions. > > Yes, I think that would be easier to remember. > > It should also work for the axis=None case. > imax = a.argmax(axis=None) > v = extract(a, imax, axis=None) It shouldn't be too difficult to jig something up given all the example code. I can do that, but I would like more input first. The questions I have are these. 1) Should it be done? 2) Should it be a method? (functions being somewhat deprecated) 3) What name should it have? I think Travis will have to weigh in on this. IIRC, he felt that the number of methods was getting out of hand. --Bill Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 01:12:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 23:12:11 -0600 Subject: [Numpy-discussion] ***[Possible UCE]*** Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <450F7C2B.5030409@ieee.org> Dr. Seth Olsen wrote: > > Hi MMTKers and NUMPYers, > > Bill's answer to my inquiry about the problem I'm having with > Collection.findTransformation() (and also, incidentally, with the > dgesvd call in Subspace.getBasis(), has convinced me that I can no > long use MMTK without changing some of the code over to numpy. I have > already been able to determine that invoking > numpy.oldnumeric.alter_code1.convertall() over the MMTK directory tree > is not the answer. Why not? It should be. That is the recommended way to begin porting code. If we need to improve alter_code, we cannot do it unless we receive bug reports. Please tell us what difficulty you had. -Travis From charlesr.harris at gmail.com Tue Sep 19 01:18:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 23:18:52 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Charles R Harris wrote: > > > > On 9/18/06, Bill Baxter wrote: > > > > On 9/19/06, Charles R Harris wrote: > > > On 9/18/06, Bill Baxter wrote: > > > > I find myself often wanting both the max and the argmax of an array. > > > > (And same for the other arg* functions) > > > > > > You have to do something like > > > > a = rand(10,5) > > > > imax = a.argmax(axis=0) > > > > vmax = a[(imax, range(5))] > > > > > > > I don't generally like overloading return values, the function starts > > to > > > lose its definition and becomes a bit baroque where simply changing a > > > keyword value can destroy the viability of the following code. > > > > Agreed. Seems like the only justification is if you get multiple > > results from one calculation but only rarely want the extra values. > > It doesn't make sense to always return them, but it's also not worth > > making a totally different function. > > > > > > > But I can see the utility of what you want. Hmm, this problem is not > > unique to argmax. > > > Maybe what we need is a general way to extract values, something like > > > > > > extract(a, imax, axis=0) > > > > > > to go along with all the single axis functions. > > > > Yes, I think that would be easier to remember. > > > > It should also work for the axis=None case. > > imax = a.argmax(axis=None) > > v = extract(a, imax, axis=None) > > > It shouldn't be too difficult to jig something up given all the example > code. I can do that, but I would like more input first. The questions I have > are these. > > 1) Should it be done? > 2) Should it be a method? (functions being somewhat deprecated) > 3) What name should it have? > > I think Travis will have to weigh in on this. IIRC, he felt that the > number of methods was getting out of hand. > > --Bill > > > Chuck > I note that argsort also produces indexes that are hard to use in the ndim>1 case. The two problems aren't quite equivalent because argsort maintains ndim while argmax reduces ndim by one, but it would be nice if there was something that would work for both. Using imax[...,newaxis] would make the shapes compatible for input but then the output would need to lose the newaxis on return. Hmmm. These are both examples of an indirect indexing problem where the indices on the specified axis are a function of the indices on the other axis. Using the other indices in the argmax case yields a scalar index while in the argsort case it yields a 1d array that can be used for fancy indexing. I'm just floating around here trying to think of a consistent way to regard these things. Ummm, and I think this could work. In fact, the arrays could be even deeper and fancy indexing on the specified axis would produce what was essentially an array of arrays. This is sort of the photo-negative version of take. Apropos the overloaded return types, I think that that is precisely what makes it tricky to use functions that may return either copies or views. The results should really be marked read only because otherwise unexpected results can arise. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 02:43:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 00:43:15 -0600 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <450F9183.6040405@ieee.org> Dr. Seth Olsen wrote: > > Hi Bill, > > MMTK has not made the conversion over to the new numpy module. It is > built against the old Numeric code, and the word from its developers > is that changing to numpy cannot be a priority now. > My suggestion is to *kindly* put pressure on them. I've spent at least a hundred hours making it easy for people to port Numeric and Numarray-built code to NumPy. Because of this, I'm a little bit frustrated by this kind of response. I understand it will take time for people to migrate, but it really does not take that long to port code to use NumPy. I've offered to do it for any open source code. In fact, I just spent 30 minutes and ported both Scientific Python and MMTK to use numpy. I'll send you a patch if you want. It is true, that the result needs to be better tested, but I'm confident that any errors which might remain in the compatibility layer will be easily fixable (and we need people who are willing to do the tests to fix them). I'd rather not do this, but if necessary we can easily create an SVN tree of third-party packages ported to use NumPy if the package-owners are not willing to do it. Keeping Numeric packages around except for legacy systems will only make things harder. I'll repeat the same offer I've made before: I will gladly give my book and my help to any open source library author who will make porting to NumPy a priority for their package. Note, however, my (free) ports to use NumPy do not use any "numerix-style" layer. The library is converted to work with NumPy alone. In other words, I won't spend any more "spare" time supporting 3 array packages. Best regards, -Travis From seth.olsen at gmail.com Tue Sep 19 03:26:08 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 17:26:08 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: <450F9183.6040405@ieee.org> References: <450F9183.6040405@ieee.org> Message-ID: Hi Travis, I would very happily accept the Scientific and MMTK patches. Thank you very much for the offer. I hadn't thought much about it until very recently, when the advent of a new IT structure in our laboratory forced me to upgrade my existing software. It was then that it became obvious that the LAPACK routines called by Numeric and MMTK were refusing to work. I've spent the day trying to wrestle with the problem (I am by no means an expert). I finally decided to get around the problems by altering the problem-solving routines in MMTK by using routines from numpy and then immediately applying a=Numeric.array( a.tolist()), which has stopped my script from gagging (although I have still to demonstrate to myself that everything is working). The uses to which I apply MMTK usually mean that the matrices in question are pretty small, so I don't have to worry too much about speed. I was suprised to note, however, that a straightforward application of F2PY to some fresh downloaded LAPACK F77 code did not work, and sent my machine into a similar endless whirr as I had seen at the beginning of the day. Apparently there's something sinister going on... Cheers, Seth On 9/19/06, Travis Oliphant wrote: > > Dr. Seth Olsen wrote: > > > > Hi Bill, > > > > MMTK has not made the conversion over to the new numpy module. It is > > built against the old Numeric code, and the word from its developers > > is that changing to numpy cannot be a priority now. > > > My suggestion is to *kindly* put pressure on them. > > I've spent at least a hundred hours making it easy for people to port > Numeric and Numarray-built code to NumPy. Because of this, I'm a > little bit frustrated by this kind of response. I understand it will > take time for people to migrate, but it really does not take that long > to port code to use NumPy. > > I've offered to do it for any open source code. In fact, I just spent > 30 minutes and ported both Scientific Python and MMTK to use numpy. > I'll send you a patch if you want. It is true, that the result needs > to be better tested, but I'm confident that any errors which might > remain in the compatibility layer will be easily fixable (and we need > people who are willing to do the tests to fix them). > > I'd rather not do this, but if necessary we can easily create an SVN > tree of third-party packages ported to use NumPy if the package-owners > are not willing to do it. Keeping Numeric packages around except for > legacy systems will only make things harder. > > I'll repeat the same offer I've made before: I will gladly give my book > and my help to any open source library author who will make porting to > NumPy a priority for their package. Note, however, my (free) ports to > use NumPy do not use any "numerix-style" layer. The library is > converted to work with NumPy alone. In other words, I won't spend any > more "spare" time supporting 3 array packages. > > Best regards, > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 04:44:39 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 02:44:39 -0600 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: <450F9183.6040405@ieee.org> Message-ID: <450FADF7.5020708@ieee.org> Dr. Seth Olsen wrote: > > Hi Travis, > > I would very happily accept the Scientific and MMTK patches. Thank > you very much for the offer. Look at http://www.scipy.org/Porting_to_NumPy for information (and patches) on how to convert ScientificPython and MMTK to use NumPy. I managed to get the codes to build and install but they are not tested. Any problems you encounter would be useful to know about. You can patch the code by changing to the top-level directory and entering patch -p1 < patch_file If it works for you, please email the developers and let them know how easy it can be (hopefully). Best, -Travis From mg.mailing-list at laposte.net Tue Sep 19 04:48:37 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 10:48:37 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450EEC52.30906@ieee.org> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> Message-ID: <450FAEE5.7040305@laposte.net> Hi Travis, First, thanks for your answer. I just download the source of Numpy from the SVN and recompile the package from Python-2.5 and 2.6a (from svn)... and it still crashes. So I do not investigate the compilation on Windows. So, may I have to wait the release 1.0rc0 of Numpy? About the migration of our python scripts, we can keep contact to help you to test the transcription tool. It is a good idea. About the Numpy API, we prefer to use the native C-Numpy-API in our code. I think it will offer more flexibilities in our generic C++ framework, particularly in our main object (named Field) which use the aliasing memory feature. Nevertheless, the first step of our migration could be using the numarray interface before using the native interface. It could be another test for you. Finally, your description of what i call "aliasing memory" is good. So it is a very good point for us that it is supported. Best regards, - Mathieu Travis Oliphant wrote: > mg wrote: > >> Hi all, >> >> I am doing a feseability study to migrate our Python based FEM >> applications from Numarray to Numpy. >> >> First, I tried to install Numpy from Python-2.4 on linux-x86, >> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >> >> Second, I try to do the same think on windows-x86 without success. So my >> first question is: is Numpy-1.0b5 has been tested and is supported on >> Windows? >> >> > Yes, it should work. Builds for windows were provided. But, perhaps > there are configuration issues for your system that we are not handling > correctly. > > >> Third, I tried to install Numpy from Python-2.5, which is our standard >> Python, on linux-x86... and the compilation stopped during the >> compilation of core/src/multiarraymodule.c. So my second question is: is >> there a workaround or is the porting to Python2.5 is yet schedule? >> >> > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. > Look for NumPy 1.0rc1 to come out soon. > >> My third question is: is the tool to migrate the numarray based Python >> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >> >> > It needs more testing. It would be great if you could help us find and > fix bugs in it. I don't have a lot of numarray code to test. > >> We have created a lot of bindings in order to pilote our generic-C++ >> framework with Python scripts. So, about the Numpy API, is it widely >> different than the Numarray API? (We will order the Numpy Guide too.) >> >> > It is more similar to the Numeric C-API. However, the numarray C-API is > completely supported by including numpy/libnumarray.h so you should be > able to convert your C code very easily. Any problems encountered > should be noted and we'll get them fixed. > >> To not duplicate large numerical memory arrays, Numarray allows to >> aliasing the memory of some bindings with arrays from Numarray, and we >> have used this feature intensively. So, I wonder if it is currently >> supported (or even scheduled)? >> > I'm pretty sure the answer is yes (because the Numarray C-API is > supported), though I'm not exactly sure what you mean. Do you mean that > you have memory created in the C/C++ framework and then you have an > array use that memory for it's data area? If that is what you mean, > then the answer is definitely yes. > > > -Travis > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From mg.mailing-list at laposte.net Tue Sep 19 04:58:57 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 10:58:57 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450EEC52.30906@ieee.org> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> Message-ID: <450FB151.30800@laposte.net> I forget a last question, We currently advise a High Performance Platform for our customers (Opteron cluster) where our applications are linked with GOTO blas library. I found in the Numpy distribution the file site.cfg, used to specify the BLAS to used at the link time. (It could be an interesting feature for us.) I noted some configuration variables have names depending on the name of the chosen BLAS library. So, my question is: is GOTO is supported or is it easy for us to add it? I wonder too if the string "[atlas]" written in site.cfg is used or not ? thanks, Mathieu. Travis Oliphant wrote: > mg wrote: > >> Hi all, >> >> I am doing a feseability study to migrate our Python based FEM >> applications from Numarray to Numpy. >> >> First, I tried to install Numpy from Python-2.4 on linux-x86, >> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >> >> Second, I try to do the same think on windows-x86 without success. So my >> first question is: is Numpy-1.0b5 has been tested and is supported on >> Windows? >> >> > Yes, it should work. Builds for windows were provided. But, perhaps > there are configuration issues for your system that we are not handling > correctly. > > >> Third, I tried to install Numpy from Python-2.5, which is our standard >> Python, on linux-x86... and the compilation stopped during the >> compilation of core/src/multiarraymodule.c. So my second question is: is >> there a workaround or is the porting to Python2.5 is yet schedule? >> >> > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. > Look for NumPy 1.0rc1 to come out soon. > >> My third question is: is the tool to migrate the numarray based Python >> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >> >> > It needs more testing. It would be great if you could help us find and > fix bugs in it. I don't have a lot of numarray code to test. > >> We have created a lot of bindings in order to pilote our generic-C++ >> framework with Python scripts. So, about the Numpy API, is it widely >> different than the Numarray API? (We will order the Numpy Guide too.) >> >> > It is more similar to the Numeric C-API. However, the numarray C-API is > completely supported by including numpy/libnumarray.h so you should be > able to convert your C code very easily. Any problems encountered > should be noted and we'll get them fixed. > >> To not duplicate large numerical memory arrays, Numarray allows to >> aliasing the memory of some bindings with arrays from Numarray, and we >> have used this feature intensively. So, I wonder if it is currently >> supported (or even scheduled)? >> > I'm pretty sure the answer is yes (because the Numarray C-API is > supported), though I'm not exactly sure what you mean. Do you mean that > you have memory created in the C/C++ framework and then you have an > array use that memory for it's data area? If that is what you mean, > then the answer is definitely yes. > > > -Travis > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From nwagner at iam.uni-stuttgart.de Tue Sep 19 05:06:10 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 19 Sep 2006 11:06:10 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450FB151.30800@laposte.net> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> <450FB151.30800@laposte.net> Message-ID: <450FB302.60508@iam.uni-stuttgart.de> mg wrote: > I forget a last question, > > We currently advise a High Performance Platform for our customers > (Opteron cluster) where our applications are linked with GOTO blas library. > > I found in the Numpy distribution the file site.cfg, used to specify the > BLAS to used at the link time. (It could be an interesting feature for > us.) I noted some configuration variables have names depending on the > name of the chosen BLAS library. So, my question is: is GOTO is > supported or is it easy for us to add it? I wonder too if the string > "[atlas]" written in site.cfg is used or not ? > > thanks, > Mathieu. > There is a ticket concerning your inquiry. http://projects.scipy.org/scipy/numpy/ticket/73 Nils > Travis Oliphant wrote: > >> mg wrote: >> >> >>> Hi all, >>> >>> I am doing a feseability study to migrate our Python based FEM >>> applications from Numarray to Numpy. >>> >>> First, I tried to install Numpy from Python-2.4 on linux-x86, >>> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >>> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >>> >>> Second, I try to do the same think on windows-x86 without success. So my >>> first question is: is Numpy-1.0b5 has been tested and is supported on >>> Windows? >>> >>> >>> >> Yes, it should work. Builds for windows were provided. But, perhaps >> there are configuration issues for your system that we are not handling >> correctly. >> >> >> >>> Third, I tried to install Numpy from Python-2.5, which is our standard >>> Python, on linux-x86... and the compilation stopped during the >>> compilation of core/src/multiarraymodule.c. So my second question is: is >>> there a workaround or is the porting to Python2.5 is yet schedule? >>> >>> >>> >> There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. >> Look for NumPy 1.0rc1 to come out soon. >> >> >>> My third question is: is the tool to migrate the numarray based Python >>> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >>> >>> >>> >> It needs more testing. It would be great if you could help us find and >> fix bugs in it. I don't have a lot of numarray code to test. >> >> >>> We have created a lot of bindings in order to pilote our generic-C++ >>> framework with Python scripts. So, about the Numpy API, is it widely >>> different than the Numarray API? (We will order the Numpy Guide too.) >>> >>> >>> >> It is more similar to the Numeric C-API. However, the numarray C-API is >> completely supported by including numpy/libnumarray.h so you should be >> able to convert your C code very easily. Any problems encountered >> should be noted and we'll get them fixed. >> >> >>> To not duplicate large numerical memory arrays, Numarray allows to >>> aliasing the memory of some bindings with arrays from Numarray, and we >>> have used this feature intensively. So, I wonder if it is currently >>> supported (or even scheduled)? >>> >>> >> I'm pretty sure the answer is yes (because the Numarray C-API is >> supported), though I'm not exactly sure what you mean. Do you mean that >> you have memory created in the C/C++ framework and then you have an >> array use that memory for it's data area? If that is what you mean, >> then the answer is definitely yes. >> >> >> -Travis >> >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From william.grant at ubuntu-au.org Tue Sep 19 05:23:17 2006 From: william.grant at ubuntu-au.org (William Grant) Date: Tue, 19 Sep 2006 19:23:17 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 Message-ID: <450FB705.7010002@ubuntu-au.org> Hi, I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it currently cannot happen as numpy doesn't build with Python 2.5. I note that changeset 3109 (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to give 2.5 compatibility, but it is those bits which seem to break it. The end of the build log with the errors can be found at http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. Has anybody got any ideas on how to fix this? A number of Ubuntu users want scipy 0.5.1, but that can't happen while it won't build with Python 2.5. Thanks, William Grant. From mg.mailing-list at laposte.net Tue Sep 19 07:26:00 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 13:26:00 +0200 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FB705.7010002@ubuntu-au.org> References: <450FB705.7010002@ubuntu-au.org> Message-ID: <450FD3C8.90308@laposte.net> Hi, I have the same problem with Python-2.5 and Python-2.6a with which i can not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of Scipy is based on one of these Numpy.) I posted my problem yesterday. The answer from Travis was it will be corrected in Numpy-1.0rc1 which will come soon. regards, Mathieu. William Grant wrote: > Hi, > > I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it > currently cannot happen as numpy doesn't build with Python 2.5. I note > that changeset 3109 > (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to > give 2.5 compatibility, but it is those bits which seem to break it. The > end of the build log with the errors can be found at > http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. > > Has anybody got any ideas on how to fix this? A number of Ubuntu users > want scipy 0.5.1, but that can't happen while it won't build with Python > 2.5. > > Thanks, > > William Grant. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From william.grant at ubuntu.com.au Tue Sep 19 07:34:20 2006 From: william.grant at ubuntu.com.au (William Grant) Date: Tue, 19 Sep 2006 21:34:20 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD3C8.90308@laposte.net> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> Message-ID: <450FD5BC.9040208@ubuntu.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 mg wrote: > Hi, > > I have the same problem with Python-2.5 and Python-2.6a with which i can > not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of > Scipy is based on one of these Numpy.) > I posted my problem yesterday. The answer from Travis was it will be > corrected in Numpy-1.0rc1 which will come soon. > > regards, > Mathieu. Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or after the 28th? Thanks, William Grant. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFFD9W8rbS+qZ0dQCURAk+7AKDkQSnrzs6SIYBmJc3QW9j48Zpu7QCePsqU AnLTk9YQzYAD/Ps/3JIBEU8= =QU5m -----END PGP SIGNATURE----- From pearu.peterson at gmail.com Tue Sep 19 09:19:40 2006 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 19 Sep 2006 16:19:40 +0300 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD5BC.9040208@ubuntu.com.au> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> <450FD5BC.9040208@ubuntu.com.au> Message-ID: <2c406a760609190619x10376f8fv7d983f3b6255cb74@mail.gmail.com> On 9/19/06, William Grant wrote: > mg wrote: > > Hi, > > > > I have the same problem with Python-2.5 and Python-2.6a with which i can > > not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of > > Scipy is based on one of these Numpy.) > > I posted my problem yesterday. The answer from Travis was it will be > > corrected in Numpy-1.0rc1 which will come soon. > > > > regards, > > Mathieu. > > Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or > after the 28th? I have the corresponding fixed compiler errors in numpy svn. Regards, Pearu From luszczek at cs.utk.edu Tue Sep 19 10:21:35 2006 From: luszczek at cs.utk.edu (Piotr Luszczek S) Date: Tue, 19 Sep 2006 10:21:35 -0400 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: <450F9183.6040405@ieee.org> Message-ID: <200609191021.36093.luszczek@cs.utk.edu> Seth, this problem is most likely related to LAPACK's SLAMCH and DLAMCH routines. If these routines get translated with f2c the compiler can optimize things away and create infinite loops. This functions get called when you start using singular and eigenvalue routines. It's because Fortran 77 didn't have a way to specify "volatile" storage. In any case, you can avoid the problem by linking different SLAMCH and DLAMCH routines. I'm attaching two possible impementations that use C99 and Fortran 95. Probably you'll have to ensure that name mangling occurs correctly (rename slamch to slamch_ and dlamch to dlamch_). Piotr On Tuesday 19 September 2006 03:26, Dr. Seth Olsen wrote: > Hi Travis, > > I would very happily accept the Scientific and MMTK patches. Thank > you very much for the offer. > > I hadn't thought much about it until very recently, when the advent > of a new IT structure in our laboratory forced me to upgrade my > existing software. It was then that it became obvious that the LAPACK > routines called by Numeric and MMTK were refusing to work. I've > spent the day trying to wrestle with the problem (I am by no means an > expert). I finally decided to get around the problems by altering > the problem-solving routines in MMTK by using routines from numpy and > then immediately applying a=Numeric.array( a.tolist()), which has > stopped my script from gagging (although I have still to demonstrate > to myself that everything is working). The uses to which I apply > MMTK usually mean that the matrices in question are pretty small, so > I don't have to worry too much about speed. > > I was suprised to note, however, that a straightforward application > of F2PY to some fresh downloaded LAPACK F77 code did not work, and > sent my machine into a similar endless whirr as I had seen at the > beginning of the day. Apparently there's something sinister going > on... > > Cheers, > > Seth > > On 9/19/06, Travis Oliphant wrote: > > Dr. Seth Olsen wrote: > > > Hi Bill, > > > > > > MMTK has not made the conversion over to the new numpy module. > > > It is built against the old Numeric code, and the word from its > > > developers is that changing to numpy cannot be a priority now. > > > > My suggestion is to *kindly* put pressure on them. > > > > I've spent at least a hundred hours making it easy for people to > > port Numeric and Numarray-built code to NumPy. Because of this, > > I'm a little bit frustrated by this kind of response. I understand > > it will take time for people to migrate, but it really does not > > take that long to port code to use NumPy. > > > > I've offered to do it for any open source code. In fact, I just > > spent 30 minutes and ported both Scientific Python and MMTK to use > > numpy. I'll send you a patch if you want. It is true, that the > > result needs to be better tested, but I'm confident that any errors > > which might remain in the compatibility layer will be easily > > fixable (and we need people who are willing to do the tests to fix > > them). > > > > I'd rather not do this, but if necessary we can easily create an > > SVN tree of third-party packages ported to use NumPy if the > > package-owners are not willing to do it. Keeping Numeric packages > > around except for legacy systems will only make things harder. > > > > I'll repeat the same offer I've made before: I will gladly give my > > book and my help to any open source library author who will make > > porting to NumPy a priority for their package. Note, however, my > > (free) ports to use NumPy do not use any "numerix-style" layer. > > The library is converted to work with NumPy alone. In other words, > > I won't spend any more "spare" time supporting 3 array packages. > > > > Best regards, > > > > -Travis > > > > > > ------------------------------------------------------------------- > >------ Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to > > share your > > opinions on IT & business topics through brief surveys -- and earn > > cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID= > >DEVDEV _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: lamch.c Type: text/x-csrc Size: 2371 bytes Desc: not available URL: -------------- next part -------------- program lamch ! implicit none integer i external tlamch do i = 1, 2 call tlamch('B', i) call tlamch('E', i) call tlamch('L', i) call tlamch('M', i) call tlamch('N', i) call tlamch('P', i) call tlamch('R', i) call tlamch('O', i) call tlamch('S', i) call tlamch('U', i) end do end program lamch subroutine tlamch(cmach, idx) character, intent(in) :: cmach integer, intent(in) :: idx double precision x real slamch double precision dlamch external dlamch, slamch intrinsic dble if (idx .eq. 1) then x = dble(slamch(cmach)) write( *, '(a e25.15)') cmach, x else write( *, '(a e25.15)') cmach, dlamch(cmach) end if end subroutine tlamch real function slamch(cmach) character, intent(in) :: cmach real zero, one parameter (zero = 0.0e+0, one = 1.0e0) real sfmin, small intrinsic digits, epsilon, radix, real if (cmach .eq. 'B' .or. cmach .eq. 'b') then slamch = radix(zero) else if (cmach .eq. 'E' .or. cmach .eq. 'e') then slamch = epsilon(zero) else if (cmach .eq. 'L' .or. cmach .eq. 'l') then slamch = maxexponent(zero) else if (cmach .eq. 'M' .or. cmach .eq. 'm') then slamch = minexponent(zero) else if (cmach .eq. 'N' .or. cmach .eq. 'n') then slamch = real(digits(zero)) else if (cmach .eq. 'O' .or. cmach .eq. 'o') then slamch = huge(zero) else if (cmach .eq. 'P' .or. cmach .eq. 'p') then slamch = epsilon(zero) * radix(zero) else if (cmach .eq. 'R' .or. cmach .eq. 'r') then slamch = one else if (cmach .eq. 'S' .or. cmach .eq. 's') then sfmin = tiny(zero) small = one / huge(zero) if (small .ge. sfmin) sfmin = small * (one+epsilon(zero)) slamch = sfmin else if (cmach .eq. 'U' .or. cmach .eq. 'u') then slamch = tiny(zero) else slamch = zero end if return end function slamch double precision function dlamch(cmach) character, intent(in) :: cmach double precision zero, one parameter (zero = 0.0d+0, one = 1.0d0) double precision :: small, sfmin intrinsic digits, dble, epsilon, huge, maxexponent, minexponent intrinsic radix, tiny if (cmach .eq. 'B' .or. cmach .eq. 'b') then dlamch = radix(zero) else if (cmach .eq. 'E' .or. cmach .eq. 'e') then dlamch = epsilon(zero) else if (cmach .eq. 'L' .or. cmach .eq. 'l') then dlamch = maxexponent(zero) else if (cmach .eq. 'M' .or. cmach .eq. 'm') then dlamch = minexponent(zero) else if (cmach .eq. 'N' .or. cmach .eq. 'n') then dlamch = dble(digits(zero)) else if (cmach .eq. 'O' .or. cmach .eq. 'o') then dlamch = huge(zero) else if (cmach .eq. 'P' .or. cmach .eq. 'p') then dlamch = epsilon(zero) * radix(zero) else if (cmach .eq. 'R' .or. cmach .eq. 'r') then dlamch = one else if (cmach .eq. 'S' .or. cmach .eq. 's') then sfmin = tiny(zero) small = one / huge(zero) if (small .ge. sfmin) sfmin = small * (one+epsilon(zero)) dlamch = sfmin else if (cmach .eq. 'U' .or. cmach .eq. 'u') then dlamch = tiny(zero) else dlamch = zero end if return end function dlamch From martin.wiechert at gmx.de Tue Sep 19 11:34:57 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Tue, 19 Sep 2006 17:34:57 +0200 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? Message-ID: <200609191734.57899.martin.wiechert@gmx.de> Hi list, Please forgive my ignorance: Is there any difference between npy_intp and size_t. Aren't both "ints just big enough to be safe with pointer arithmetics even on 64 bit architectures?". Thanks, Martin. From konrad.hinsen at laposte.net Tue Sep 19 12:05:55 2006 From: konrad.hinsen at laposte.net (Konrad Hinsen) Date: Tue, 19 Sep 2006 18:05:55 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: On Sep 19, 2006, at 5:11, Dr. Seth Olsen wrote: > Bill's answer to my inquiry about the problem I'm having with > Collection.findTransformation() (and also, incidentally, with the > dgesvd call in Subspace.getBasis(), has convinced me that I can no > long use MMTK without changing some of the code over to numpy. I > have already been able to determine that invoking > numpy.oldnumeric.alter_code1.convertall() over the MMTK directory > tree is not the answer. Has anyone on either of these lists ever > tried this before and, if so, can it be done (without destroying my > sanity)? Adapting MMTK to NumPy is likely to be a major effort, in particular for the C modules. A lot ot testing would also be required. If anyone wants to tackle this, I'd be happy to see the results. Personally, I don't expect to find time for this soon. MMTK works fine with Numeric 23.x (and probably many other versions), so I don't see a pressing need to change to NumPy. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From faltet at carabos.com Tue Sep 19 12:45:37 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 19 Sep 2006 18:45:37 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: <200609191845.38788.faltet@carabos.com> A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > I note that argsort also produces indexes that are hard to use in the > ndim>1 case. Perhaps it is worth to mention here that I've always liked to have a sort() and argsort() functionality merged in one shot function (or method). Having both separated implies two sorts, while if I'd have such a combo available, the resulting operation would take quite less time. One can easily stablish kind of a pattern for situations where this could happen: in all the 'arg*()' functions. Perhaps I'm wrong in my count, but there appear to be only three of such functions in NumPy, namely, argmax, argmin and argsort. Adding three additional 'combos' doesn't seem a lot to my mind, but it can be just 'too much' for more common sense minds. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Tue Sep 19 12:48:31 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 19 Sep 2006 09:48:31 -0700 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <45101F5F.6080209@noaa.gov> Konrad Hinsen wrote: > MMTK works fine with Numeric 23.x (and probably many other versions), > so I don't see a pressing need to change to NumPy. Pressing is in the eye of the beholder. However: I don't think we should underestimate the negative impact of the Numeric/numarray split on the usability and perception of the community. Also the impact on how much work has been done to accommodate it. If you consider matplotlib alone: Effort to write the "numerix" compatibility layer Effort to support problems people have with incompatibility Effort to build binaries that support all the packages Effort to do extra testing to determine whether a given problem is caused by a different numerix back-end. This really adds up! Because of this, it IS a pressing need to get as much stuff converted as possible. In addition, as I understand it, MMTK was NOT working fine for the OP. this is also the case for a variety of other codes that depend on Numeric. As robust as they (and Numeric) might be, when you need to run something on a new platform (OS-X - Intel comes to mind), or use a new LAPACK, or whatever, there are going to be (and have been) issues that need to be addressed. No one is maintaining Numeric, so it makes much more sense to put your effort into porting to numpy, rather than trying to fix or work around Numeric issues. This is also a great time to get help with porting issues -- see Travis' recent message. -Chris PS: this really highlights the strength of having good unit tests: as far as i can tell, it's really not that much work to do the port -- the work is in the testing. Comprehensive units tests would make that part trivial too. -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Tue Sep 19 13:21:19 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 11:21:19 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191845.38788.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> Message-ID: On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > I note that argsort also produces indexes that are hard to use in the > > ndim>1 case. > > Perhaps it is worth to mention here that I've always liked to have a > sort() > and argsort() functionality merged in one shot function (or method). > Having > both separated implies two sorts, while if I'd have such a combo > available, > the resulting operation would take quite less time. Do you want both the indexes and index sorted array returned, or just sort the array using indexes, i.e., something like a.sort(kind="quicksort", method="indirect") IIRC, the old numeric argsort actually did something like this under the covers but discarded the sorted array. One can easily stablish kind of a pattern for situations where this could > happen: in all the 'arg*()' functions. Perhaps I'm wrong in my count, but > there appear to be only three of such functions in NumPy, namely, argmax, > argmin and argsort. Adding three additional 'combos' doesn't seem a lot to > my > mind, but it can be just 'too much' for more common sense minds. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Tue Sep 19 13:23:57 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 10:23:57 -0700 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? Message-ID: <200609191023.57656.haase@msg.ucsf.edu> Hi, What are the possible values of arr.dtype.kind ? It seems that signed and unsigned are considered to be the same "kind" >>> arr=N.arange(10,dtype=N.uint) >>> arr.dtype.kind 'i' >>> arr.dtype.itemsize 8 (OK - this is just showing off our amd64 linux ;-) ) How can I distinguish signed from unsigned without having to list all possible cases explicitly ? Thanks, Sebastian Haase From charlesr.harris at gmail.com Tue Sep 19 13:28:24 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 11:28:24 -0600 Subject: [Numpy-discussion] Resolution of tickets. Message-ID: Question, How does one mark a ticket as fixed? I don't see this field in the ticket views I get, is there a list of accepted fixers? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Tue Sep 19 13:39:23 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 19 Sep 2006 19:39:23 +0200 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191023.57656.haase@msg.ucsf.edu> Message-ID: Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase > Sent: 19 September 2006 19:24 > To: Discussion of Numerical Python > Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? > > Hi, > What are the possible values of > arr.dtype.kind ? > > It seems that signed and unsigned are considered to be the same "kind" > >>> arr=N.arange(10,dtype=N.uint) > >>> arr.dtype.kind > 'i' > >>> arr.dtype.itemsize > 8 > (OK - this is just showing off our amd64 linux ;-) ) > > How can I distinguish signed from unsigned without having to list all > possible > cases explicitly ? How about sctypes? In [16]: numpy.sctypes.keys() Out[17]: ['int', 'float', 'uint', 'complex', 'others'] So this should work: sometype in numpy.sctypes['uint'] sometype in numpy.sctypes['int'] Cheers, Albert From faltet at carabos.com Tue Sep 19 13:41:55 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 19 Sep 2006 19:41:55 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191845.38788.faltet@carabos.com> Message-ID: <200609191941.56395.faltet@carabos.com> A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > > I note that argsort also produces indexes that are hard to use in the > > > ndim>1 case. > > > > Perhaps it is worth to mention here that I've always liked to have a > > sort() > > and argsort() functionality merged in one shot function (or method). > > Having > > both separated implies two sorts, while if I'd have such a combo > > available, > > the resulting operation would take quite less time. > > Do you want both the indexes and index sorted array returned, or just sort > the array using indexes, i.e., something like > > a.sort(kind="quicksort", method="indirect") Uh, I don't understand what do you mean by "sort the array using indexes", sorry. What I'd like is to get the indexes and the sorted array returned. Or, another option could be getting the sorted indexes returned and a sort of the array done in-place (which is surely faster). Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Tue Sep 19 13:42:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 12:42:50 -0500 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: Message-ID: Charles R Harris wrote: > Question, > > How does one mark a ticket as fixed? I don't see this field in the > ticket views I get, is there a list of accepted fixers? Indeed there is. charris is on that list. charris208 is not. Forget your password? :-) Talk to me offlist about which username you want to continue with. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Sep 19 14:36:31 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:36:31 -0600 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD5BC.9040208@ubuntu.com.au> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> <450FD5BC.9040208@ubuntu.com.au> Message-ID: <451038AF.3010806@ieee.org> William Grant wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > mg wrote: > >> Hi, >> >> I have the same problem with Python-2.5 and Python-2.6a with which i can >> not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of >> Scipy is based on one of these Numpy.) >> I posted my problem yesterday. The answer from Travis was it will be >> corrected in Numpy-1.0rc1 which will come soon. >> >> regards, >> Mathieu. >> > > Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or > after the 28th? > Before. It was supposed to come out this weekend. It will be today or tomorrow. -Travis From oliphant.travis at ieee.org Tue Sep 19 14:37:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:37:40 -0600 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? In-Reply-To: <200609191734.57899.martin.wiechert@gmx.de> References: <200609191734.57899.martin.wiechert@gmx.de> Message-ID: <451038F4.7090704@ieee.org> Martin Wiechert wrote: > Hi list, > > Please forgive my ignorance: Is there any difference between npy_intp and > size_t. Aren't both "ints just big enough to be safe with pointer arithmetics > even on 64 bit architectures?". > size_t is unsigned npy_intp is signed It is basically the same as Py_ssize_t (which is not available until Python 2.5). -Travis From charlesr.harris at gmail.com Tue Sep 19 14:42:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 12:42:23 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191941.56395.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > On 9/19/06, Francesc Altet wrote: > > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > > > I note that argsort also produces indexes that are hard to use in > the > > > > ndim>1 case. > > > > > > Perhaps it is worth to mention here that I've always liked to have a > > > sort() > > > and argsort() functionality merged in one shot function (or method). > > > Having > > > both separated implies two sorts, while if I'd have such a combo > > > available, > > > the resulting operation would take quite less time. > > > > Do you want both the indexes and index sorted array returned, or just > sort > > the array using indexes, i.e., something like > > > > a.sort(kind="quicksort", method="indirect") > > Uh, I don't understand what do you mean by "sort the array using indexes", > sorry. > > What I'd like is to get the indexes and the sorted array returned. Or, > another > option could be getting the sorted indexes returned and a sort of the > array > done in-place (which is surely faster). Don't know about the faster part, it depends on the architecture and the size of the elements being swapped. It is the fact that the new quicksort does 1/2 as many swaps together with the type specific code that makes it faster than the original numpy version. The overhead in the current indirect sorts comes mostly from the indirect references to array elements in the innermost loop and it is quite possible that doing swaps on both the indexes and the array would be a bigger hit than the current argsort followed by a suitable take type function. Lexsort is another function that would benefit from what we are discussing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 14:42:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:42:57 -0600 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <45103A31.60704@ieee.org> Konrad Hinsen wrote: > On Sep 19, 2006, at 5:11, Dr. Seth Olsen wrote: > > >> Bill's answer to my inquiry about the problem I'm having with >> Collection.findTransformation() (and also, incidentally, with the >> dgesvd call in Subspace.getBasis(), has convinced me that I can no >> long use MMTK without changing some of the code over to numpy. I >> have already been able to determine that invoking >> numpy.oldnumeric.alter_code1.convertall() over the MMTK directory >> tree is not the answer. Has anyone on either of these lists ever >> tried this before and, if so, can it be done (without destroying my >> sanity)? >> > > Adapting MMTK to NumPy is likely to be a major effort, in particular > for the C modules. Well, I got both ScientificPython and MMTK to compile and import using the steps outlined on http://www.scipy.org/Porting_to_NumPy in about 1 hour (including time to fix alter_code1 to make the process even easier). C-modules are actually easier to port because the Numeric C-API is totally supported. > A lot ot testing would also be required. If anyone > wants to tackle this, > I'd be happy to see the results. Great. I can totally understand people not having the time, but it is fantastic to hear that the willingness to accept patches is there. I was able to install ScientificPython and MMTK for NumPy on my system using the patches provided on that page. Is there a test suite that can be run? Users of MMTK could really help out here. -Travis From oliphant.travis at ieee.org Tue Sep 19 14:46:32 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:46:32 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191023.57656.haase@msg.ucsf.edu> References: <200609191023.57656.haase@msg.ucsf.edu> Message-ID: <45103B08.3070808@ieee.org> Sebastian Haase wrote: > Hi, > What are the possible values of > arr.dtype.kind ? > > It seems that signed and unsigned are considered to be the same "kind" > >>>> arr=N.arange(10,dtype=N.uint) >>>> arr.dtype.kind >>>> > 'i' > >>>> arr.dtype.itemsize >>>> > 8 > (OK - this is just showing off our amd64 linux ;-) ) > > How can I distinguish signed from unsigned without having to list all possible > cases explicitly ? > > > Hmm.... This is a problem. There is a 'u' kind for unsigned integers. On my system I get 'u' when running the code you just gave. Can anybody on a 64-bit system confirm? -Travis From oliphant.travis at ieee.org Tue Sep 19 14:47:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:47:17 -0600 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: Message-ID: <45103B35.8020205@ieee.org> Charles R Harris wrote: > Question, > > How does one mark a ticket as fixed? I don't see this field in the > ticket views I get, is there a list of accepted fixers? > You have to be logged in to the Trac site. If you have SVN write access you should be able to log in. Then there is a "resolution" section at the very bottom. -Travis From sransom at nrao.edu Tue Sep 19 14:51:16 2006 From: sransom at nrao.edu (Scott Ransom) Date: Tue, 19 Sep 2006 14:51:16 -0400 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <45103B08.3070808@ieee.org> References: <200609191023.57656.haase@msg.ucsf.edu> <45103B08.3070808@ieee.org> Message-ID: <200609191451.16896.sransom@nrao.edu> On Tuesday 19 September 2006 14:46, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > What are the possible values of > > arr.dtype.kind ? > > > > It seems that signed and unsigned are considered to be the same > > "kind" > > > >>>> arr=N.arange(10,dtype=N.uint) > >>>> arr.dtype.kind > > > > 'i' > > > >>>> arr.dtype.itemsize > > > > 8 > > (OK - this is just showing off our amd64 linux ;-) ) > > > > How can I distinguish signed from unsigned without having to list all > > possible cases explicitly ? > > Hmm.... This is a problem. There is a 'u' kind for unsigned > integers. > > On my system I get 'u' when running the code you just gave. > > Can anybody on a 64-bit system confirm? I'm on 64-bit Debian: In [11]: arr=N.arange(10,dtype=N.uint) In [12]: arr.dtype.kind Out[12]: 'u' In [13]: arr.dtype.itemsize Out[13]: 4 In [14]: arr=N.arange(10,dtype=N.long) In [15]: arr.dtype.kind Out[15]: 'i' In [16]: arr.dtype.itemsize Out[16]: 8 Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From sransom at nrao.edu Tue Sep 19 14:54:03 2006 From: sransom at nrao.edu (Scott Ransom) Date: Tue, 19 Sep 2006 14:54:03 -0400 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191451.16896.sransom@nrao.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <45103B08.3070808@ieee.org> <200609191451.16896.sransom@nrao.edu> Message-ID: <200609191454.03405.sransom@nrao.edu> > > Can anybody on a 64-bit system confirm? > > I'm on 64-bit Debian: > > In [11]: arr=N.arange(10,dtype=N.uint) > > In [12]: arr.dtype.kind > Out[12]: 'u' > > In [13]: arr.dtype.itemsize > Out[13]: 4 > > In [14]: arr=N.arange(10,dtype=N.long) > > In [15]: arr.dtype.kind > Out[15]: 'i' > > In [16]: arr.dtype.itemsize > Out[16]: 8 Ack! That was on the wrong machine (32-bit Debian). Here is the 64-bit version: In [2]: arr=N.arange(10,dtype=N.uint) In [3]: arr.dtype.kind Out[3]: 'u' In [4]: arr.dtype.itemsize Out[4]: 8 In [5]: arr=N.arange(10,dtype=N.long) In [6]: arr.dtype.kind Out[6]: 'i' In [7]: arr.dtype.itemsize Out[7]: 8 Sorry about that, Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From rowen at cesmail.net Tue Sep 19 14:59:58 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Tue, 19 Sep 2006 11:59:58 -0700 Subject: [Numpy-discussion] 1.0b5 Installation on OS X References: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> <450EF580.90103@noaa.gov> Message-ID: In article <450EF580.90103 at noaa.gov>, Christopher Barker wrote: > David Andrews wrote: > > > I'm unable to install the 1.0b5 release from the .mpkg on OS X - when > > it comes to the two checkboxes to select the platlib and the scripts, > > both are greyed out. The installer then claims 'Cannot continue: > > Nothing to install'. > > hmmm. It works just fine for me -- just clicking OK as I go merrily along. > > OS version? python version? processor? > > Also, try deleting or renaming any existing install you have from > site-packages, then try again. In addition to that, I suggest also deleting any relevant receipts in /Library/Receipts It is a long shot, but receipt files have occasionally prevented me from installing software, though I confess only when an older version of something. -- Russell From hjn253 at tom.com Sun Sep 24 15:00:36 2006 From: hjn253 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Mon, 25 Sep 2006 03:00:36 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 19 15:06:46 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:06:46 -0600 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: <45103B35.8020205@ieee.org> References: <45103B35.8020205@ieee.org> Message-ID: On 9/19/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > Question, > > > > How does one mark a ticket as fixed? I don't see this field in the > > ticket views I get, is there a list of accepted fixers? > > > You have to be logged in to the Trac site. If you have SVN write access > you should be able to log in. Then there is a "resolution" section at > the very bottom. Ah, that works. I orignally tried to register and discovered that charris was already taken and I didn't know what password to use. I've now deleted the charris208 account. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Sep 19 15:11:30 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 19 Sep 2006 20:11:30 +0100 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: <45103B35.8020205@ieee.org> Message-ID: <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> Hi, > > You have to be logged in to the Trac site. If you have SVN write access > > you should be able to log in. Then there is a "resolution" section at > > the very bottom. > > Ah, that works. I orignally tried to register and discovered that charris > was already taken and I didn't know what password to use. I've now deleted > the charris208 account. Not for me, when logged on with my SVN username. I took Robert's email to mean we needed specific permission (over and above SVN write access) to resolve tickets - is that not the case? Best, Matthew From kwgoodman at gmail.com Tue Sep 19 15:34:16 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 19 Sep 2006 12:34:16 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf Message-ID: In what order would you like argsort to sort the values -inf, nan, inf? In numpy 1.0b1 nan is always left where it began: EXAMPLE 1 >> x matrix([[ inf], [ -inf], [ nan]]) >> x[x.argsort(0),:] matrix([[ -inf], [ inf], [ nan]]) EXAMPLE 2 >> x matrix([[ nan], [ inf], [ -inf]]) >> x[x.argsort(0),:] matrix([[ nan], [ -inf], [ inf]]) I would like nan to be in the middle between -inf and inf. From tim.hochberg at ieee.org Tue Sep 19 15:40:04 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 12:40:04 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: Message-ID: <45104794.2050106@ieee.org> Keith Goodman wrote: > In what order would you like argsort to sort the values -inf, nan, inf? > Ideally, -inf should sort first, inf should sort last and nan should raise an exception if present. -tim > In numpy 1.0b1 nan is always left where it began: > > EXAMPLE 1 > > >>> x >>> > > matrix([[ inf], > [ -inf], > [ nan]]) > >>> x[x.argsort(0),:] >>> > > matrix([[ -inf], > [ inf], > [ nan]]) > > EXAMPLE 2 > > >>> x >>> > > matrix([[ nan], > [ inf], > [ -inf]]) > >>> x[x.argsort(0),:] >>> > > matrix([[ nan], > [ -inf], > [ inf]]) > > > I would like nan to be in the middle between -inf and inf. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Tue Sep 19 15:41:56 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 20 Sep 2006 04:41:56 +0900 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191941.56395.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/20/06, Francesc Altet wrote: > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > > > Do you want both the indexes and index sorted array returned, or just sort > > the array using indexes, i.e., something like > > > > a.sort(kind="quicksort", method="indirect") > > Uh, I don't understand what do you mean by "sort the array using indexes", > sorry. > I think he meant do an argsort first, then use fancy indexing to get the sorted array. For a 1-d array that's just ind = A.argsort() Asorted = A[ind] That should be O(N lg N + N), aka O(N lg N) For A >1-d, you need an indexing expression that's a little more complicated, hence the discussion about making an "extract" function for that purpose. Then you could say: ind = A.argsort(axis=d) Asorted = A.extract(ind,axis=d) and it should just work no matter what 'd' is. --bb From charlesr.harris at gmail.com Tue Sep 19 15:52:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:52:01 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Keith Goodman wrote: > > In what order would you like argsort to sort the values -inf, nan, inf? > > > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. > > -tim That sounds right. Nan can't be compared because, well, it is undefined. For instance: In [73]: a = arange(2) In [74]: a/0 Out[74]: array([0, 0]) In [75]: a/0.0 Out[75]: array([ nan, inf]) In [76]: a/(-0.0) Out[76]: array([ nan, -inf]) I.e., 0.0/0.0 is undefined. But unless the hardware generates the exception I would be loath to introduce checks in the code. Putting a check in the innermost loop of the sorts would cost significant time. But look at the integer division by zero where the IEEE stuff has no relevence. That sure looks like a bug to me. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Sep 19 15:57:22 2006 From: ndarray at mac.com (Sasha) Date: Tue, 19 Sep 2006 15:57:22 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. I disagree. NumPy sort leaving nan's where they are is probably just a side-effect of nans unusual properties (both nanx is always false), but it is a logical choice. Checking for nan will inevitably slow the most common use case. If you want an exception, just check isnan(x).any() before sort. From charlesr.harris at gmail.com Tue Sep 19 15:58:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:58:03 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/19/06, Bill Baxter wrote: > > On 9/20/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > > > > > Do you want both the indexes and index sorted array returned, or just > sort > > > the array using indexes, i.e., something like > > > > > > a.sort(kind="quicksort", method="indirect") > > > > Uh, I don't understand what do you mean by "sort the array using > indexes", > > sorry. > > > > I think he meant do an argsort first, then use fancy indexing to get > the sorted array. > For a 1-d array that's just > > ind = A.argsort() > Asorted = A[ind] > > That should be O(N lg N + N), aka O(N lg N) > For A >1-d, you need an indexing expression that's a little more > complicated, hence the discussion about making an "extract" function > for that purpose. Then you could say: > > ind = A.argsort(axis=d) > Asorted = A.extract(ind,axis=d) I'm also thinking of the name argtake. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 16:01:50 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 16:01:50 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > Keith Goodman wrote: > > In what order would you like argsort to sort the values -inf, nan, inf? > > > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. > > -tim Mmm. Somebody who's working with NaNs has more or less already decided they don't want to be pestered with exceptions for invalid data. I'd be happy if they wound up at either end, but I'm not sure it's worth hacking up the sort algorithm when a simple isnan() can pull them out. What's happening now is that NaNa are all false, which rather confuses the sorting algorithm. But as long as it doesn't actually *break* (that is, leave some of the non-NaNs incorrectly sorted) I don't care. A. M. Archibald From kwgoodman at gmail.com Tue Sep 19 16:06:40 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 19 Sep 2006 13:06:40 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > > Keith Goodman wrote: > > > In what order would you like argsort to sort the values -inf, nan, inf? > > > > > Ideally, -inf should sort first, inf should sort last and nan should > > raise an exception if present. > > > > -tim > > Mmm. Somebody who's working with NaNs has more or less already decided > they don't want to be pestered with exceptions for invalid data. I'd > be happy if they wound up at either end, but I'm not sure it's worth > hacking up the sort algorithm when a simple isnan() can pull them out. Is there an easy way to use isnan to pull out the nans if the matrix I am sorting has more than one column? From ndarray at mac.com Tue Sep 19 16:13:55 2006 From: ndarray at mac.com (Sasha) Date: Tue, 19 Sep 2006 16:13:55 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Keith Goodman wrote: > Is there an easy way to use isnan to pull out the nans if the matrix I > am sorting has more than one column? > There seems to be a "nan_to_num" function that converts nans to zeros, but I would suggest just using fancy indexing to fill the nans with appropriate values: x[isnan(x)] = inf # if you want nans on the right From tim.hochberg at ieee.org Tue Sep 19 16:21:56 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 13:21:56 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: <45105164.90700@ieee.org> A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > >> Keith Goodman wrote: >> >>> In what order would you like argsort to sort the values -inf, nan, inf? >>> >>> >> Ideally, -inf should sort first, inf should sort last and nan should >> raise an exception if present. >> >> -tim >> > > Mmm. Somebody who's working with NaNs has more or less already decided > they don't want to be pestered with exceptions for invalid data. Do you really think so? In my experience NaNs are nearly always just an indication of a mistake somewhere that didn't get trapped for one reason or another. > I'd > be happy if they wound up at either end, but I'm not sure it's worth > hacking up the sort algorithm when a simple isnan() can pull them out. > Moving them to the end seems to be the worst choice to me. Leaving them alone is fine with me. Or raising an exception would be fine. Or doing one or the other depending on the error mode settings would be even better if it is practical. > What's happening now is that NaNa are all false, > which rather confuses the sorting algorithm. But as long as it doesn't > actually *break* (that is, leave some of the non-NaNs incorrectly > sorted) I don't care. > Is that true? Are all of numpy's sorting algorithms robust against nontransitive objects laying around? The answer to that appears to be no. Try running this a couple of times to see what I mean: n = 10 for i in range(10): for kind in 'quicksort', 'heapsort', 'mergesort': a = rand(n) b = a.copy() a[n//2] = nan a.sort(kind=kind) b.sort(kind=kind) assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) The values don't correctly cross the inserted NaN and the sort is incorrect. -tim From charlesr.harris at gmail.com Tue Sep 19 16:56:00 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 14:56:00 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105164.90700@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg wrote: > > > >> Keith Goodman wrote: > >> > >>> In what order would you like argsort to sort the values -inf, nan, > inf? > >>> > >>> > >> Ideally, -inf should sort first, inf should sort last and nan should > >> raise an exception if present. > >> > >> -tim > >> > > > > Mmm. Somebody who's working with NaNs has more or less already decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always just an > indication of a mistake somewhere that didn't get trapped for one reason > or another. > > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull them out. > > > Moving them to the end seems to be the worst choice to me. Leaving them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. > > > What's happening now is that NaNa are all false, > > which rather confuses the sorting algorithm. But as long as it doesn't > > actually *break* (that is, leave some of the non-NaNs incorrectly > > sorted) I don't care. > > > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > > n = 10 > for i in range(10): > for kind in 'quicksort', 'heapsort', 'mergesort': > a = rand(n) > b = a.copy() > a[n//2] = nan > a.sort(kind=kind) > b.sort(kind=kind) > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > The values don't correctly cross the inserted NaN and the sort is > incorrect. Except for heapsort those are doing insertion sorts because n is so small. Heapsort is trickier to understand because of the way the heap is formed and values pulled off. I'll look into that. I bet searchsorted gets messed up by nan's. Do you think it worthwhile to worry about it? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 17:04:49 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:04:49 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: <45105B71.7090102@ieee.org> Charles R Harris wrote: > > > On 9/19/06, *Tim Hochberg* > wrote: > > A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg > wrote: > > > >> Keith Goodman wrote: > >> > >>> In what order would you like argsort to sort the values -inf, > nan, inf? > >>> > >>> > >> Ideally, -inf should sort first, inf should sort last and nan > should > >> raise an exception if present. > >> > >> -tim > >> > > > > Mmm. Somebody who's working with NaNs has more or less already > decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always > just an > indication of a mistake somewhere that didn't get trapped for one > reason > or another. > > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull > them out. > > > Moving them to the end seems to be the worst choice to me. Leaving > them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. > > > What's happening now is that NaNa are all > false, > > which rather confuses the sorting algorithm. But as long as it > doesn't > > actually *break* (that is, leave some of the non-NaNs incorrectly > > sorted) I don't care. > > > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > > n = 10 > for i in range(10): > for kind in 'quicksort', 'heapsort', 'mergesort': > a = rand(n) > b = a.copy() > a[n//2] = nan > a.sort(kind=kind) > b.sort(kind=kind) > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > The values don't correctly cross the inserted NaN and the sort is > incorrect. > > > Except for heapsort those are doing insertion sorts because n is so > small. Heapsort is trickier to understand because of the way the heap > is formed and values pulled off. I'm not sure where the breakpoint is, but I was seeing failures for all three sort types with N as high as 10000. I suspect that they're all broken in the presence of NaNs. I further suspect you'd need some punishingly slow n**2 algorithm to be robust in the presence of NaNs. > I'll look into that. I bet searchsorted gets messed up by nan's. Do > you think it worthwhile to worry about it? No. But that's because it's my contention is that any sequence with NaNs in it is *not sorted* and thus isn't suitable input for searchsorted. -tim From peridot.faceted at gmail.com Tue Sep 19 17:09:14 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:09:14 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105164.90700@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > A. M. Archibald wrote: > > Mmm. Somebody who's working with NaNs has more or less already decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always just an > indication of a mistake somewhere that didn't get trapped for one reason > or another. Well, I said that because for an image porcessing project I was doing, the easiest thing to do with certain troublesome pixels was to fill in NaNs, and then at the end replace the NaNs with sensible values. It seems as if the point of NaNs is to allow you to keep working with those numbers that make sense while ingoring those that don't. If you wanted exceptions, why not get them as soon as the first NaN would have been generated? > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull them out. > > > Moving them to the end seems to be the worst choice to me. Leaving them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. I was just thinking in terms of easy removal. > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > The values don't correctly cross the inserted NaN and the sort is incorrect. You're quite right: when NaNs are present in the array, sorting and then removing them does not yield a sorted array. For example, mergesort just output [ 2. 4. 6. 9. nan 0. 1. 3. 5. 7. 8. ] The other two are no better (and arguably worse). Python's built-in sort() for lists has the same problem. This is definitely a bug, and the best way to fix it is not clear to me - perhaps sort() needs to always do any(isnan(A)) before starting to sort. I don't really like raising an exception, but sort() isn't really very meaningful with NaNs in the array. The only other option I can think of is to somehow remove them, sort without, and reintroduce them at the end, which is going to be a nightmare when sorting a single axis of a large array. Or, I suppose, sort() could simply fill the array with NaNs; I'm sure users will love that. A. M. Archibald From peridot.faceted at gmail.com Tue Sep 19 17:15:24 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:15:24 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105B71.7090102@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > I'm not sure where the breakpoint is, but I was seeing failures for all > three sort types with N as high as 10000. I suspect that they're all > broken in the presence of NaNs. I further suspect you'd need some > punishingly slow n**2 algorithm to be robust in the presence of NaNs. Not at all. Just temporarily make NaNs compare greater than any other floating-point value and use quicksort (say). You can even do this for python lists without much trouble. That's actually a viable suggestion for numpy's sorting, although it would be kind of ugly to implement: do a quick any(isnan(A)), and if not, use the fast stock sorting algorithms; if there is a NaN somewhere in the array, use a version of the sort that has a tweaked comparison function so the NaNs wind up at the end and are easy to trim off. But the current situation, silently returning arrays in which the non-NaNs are unsorted, is really bad. A. M. Archibald From robert.kern at gmail.com Tue Sep 19 17:18:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 16:18:10 -0500 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> References: <45103B35.8020205@ieee.org> <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> Message-ID: Matthew Brett wrote: > Hi, > >>> You have to be logged in to the Trac site. If you have SVN write access >>> you should be able to log in. Then there is a "resolution" section at >>> the very bottom. >> Ah, that works. I orignally tried to register and discovered that charris >> was already taken and I didn't know what password to use. I've now deleted >> the charris208 account. > > Not for me, when logged on with my SVN username. I took Robert's > email to mean we needed specific permission (over and above SVN write > access) to resolve tickets - is that not the case? It used to be the case that the subversion password database was used as the Trac's password database. This is no longer the case since we need to require registration by non-developers. *Operationally*, people added to the subversion write list should also be be provided with appropriate Trac privileges, but this did not happen in your case. I will fix that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 19 17:21:13 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 15:21:13 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105B71.7090102@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Charles R Harris wrote: > > > > > > On 9/19/06, *Tim Hochberg* > > wrote: > > > > A. M. Archibald wrote: > > > On 19/09/06, Tim Hochberg > > wrote: > > > > > >> Keith Goodman wrote: > > >> > > >>> In what order would you like argsort to sort the values -inf, > > nan, inf? > > >>> > > >>> > > >> Ideally, -inf should sort first, inf should sort last and nan > > should > > >> raise an exception if present. > > >> > > >> -tim > > >> > > > > > > Mmm. Somebody who's working with NaNs has more or less already > > decided > > > they don't want to be pestered with exceptions for invalid data. > > Do you really think so? In my experience NaNs are nearly always > > just an > > indication of a mistake somewhere that didn't get trapped for one > > reason > > or another. > > > > > I'd > > > be happy if they wound up at either end, but I'm not sure it's > worth > > > hacking up the sort algorithm when a simple isnan() can pull > > them out. > > > > > Moving them to the end seems to be the worst choice to me. Leaving > > them > > alone is fine with me. Or raising an exception would be fine. Or > doing > > one or the other depending on the error mode settings would be even > > better if it is practical. > > > > > What's happening now is that NaNa are all > > false, > > > which rather confuses the sorting algorithm. But as long as it > > doesn't > > > actually *break* (that is, leave some of the non-NaNs incorrectly > > > sorted) I don't care. > > > > > Is that true? Are all of numpy's sorting algorithms robust against > > nontransitive objects laying around? The answer to that appears to > be > > no. Try running this a couple of times to see what I mean: > > > > n = 10 > > for i in range(10): > > for kind in 'quicksort', 'heapsort', 'mergesort': > > a = rand(n) > > b = a.copy() > > a[n//2] = nan > > a.sort(kind=kind) > > b.sort(kind=kind) > > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > > > > The values don't correctly cross the inserted NaN and the sort is > > incorrect. > > > > > > Except for heapsort those are doing insertion sorts because n is so > > small. Heapsort is trickier to understand because of the way the heap > > is formed and values pulled off. > I'm not sure where the breakpoint is, but I was seeing failures for all > three sort types with N as high as 10000. I suspect that they're all > broken in the presence of NaNs. I further suspect you'd need some > punishingly slow n**2 algorithm to be robust in the presence of NaNs. Quicksort and Mergesort are divide and conquer types. When the pieces get below a certain length they finish up with insertion sort as it is more efficient for small bits. In numpy the breakover points are 15 and 20 respectively. I believe microsoft did a project on this and ended up with a number somewhere around 7, but I didn't see that when doing the tuning for numarray way back when. Not that I did a *lot* of careful tuning work ;) > I'll look into that. I bet searchsorted gets messed up by nan's. Do > > you think it worthwhile to worry about it? > > No. But that's because it's my contention is that any sequence with NaNs > in it is *not sorted* and thus isn't suitable input for searchsorted. If this sort of thing can cause unexpected errors I wonder if it would be worth it to have a global debugging flag that essentially causes isnan to be called before any function applications. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 17:28:35 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:28:35 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 19/09/06, Charles R Harris wrote: > If this sort of thing can cause unexpected errors I wonder if it would be > worth it to have a global debugging flag that essentially causes isnan to > be called before any function applications. That sounds very like the IEEE floating-point flags, which would be extremely useful to have, and which are being wored on, IIRC. A. M. Archibald From tim.hochberg at ieee.org Tue Sep 19 17:37:12 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:37:12 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <45106308.7010302@ieee.org> A. M. Archibald wrote: > On 19/09/06, Charles R Harris wrote: > > >> If this sort of thing can cause unexpected errors I wonder if it would be >> worth it to have a global debugging flag that essentially causes isnan to >> be called before any function applications. >> > > That sounds very like the IEEE floating-point flags, which would be > extremely useful to have, and which are being wored on, IIRC. > Are you referring to numpy.geterr / seterr? These exist, but they don't seem to be working right as of 1.0b4 (my installation is a little out of date). -tim From tim.hochberg at ieee.org Tue Sep 19 17:40:21 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:40:21 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <451063C5.20201@ieee.org> A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > > >> I'm not sure where the breakpoint is, but I was seeing failures for all >> three sort types with N as high as 10000. I suspect that they're all >> broken in the presence of NaNs. I further suspect you'd need some >> punishingly slow n**2 algorithm to be robust in the presence of NaNs. >> > > Not at all. Just temporarily make NaNs compare greater than any other > floating-point value and use quicksort (say). You can even do this for > python lists without much trouble. > I misspoke. What I meant here was keeping the behavior that people think that we already have but don't: NaNs stay in place and everything is sorted around them. And even that's not true, since you could just record where the NaNs are, remove them, sort and put them back. What I was really getting at was, that I'm guessing, and it's just a guess, that (a) none of the fast sorting algorithms do anything sensible unless special cased and (b) one could come up with a naive n**2 sort that does do something sensible without special casing (where sensible means leave the NaNs alone). > That's actually a viable suggestion for numpy's sorting, although it > would be kind of ugly to implement: do a quick any(isnan(A)), and if > not, use the fast stock sorting algorithms; if there is a NaN > somewhere in the array, use a version of the sort that has a tweaked > comparison function so the NaNs wind up at the end and are easy to > trim off. > > But the current situation, silently returning arrays in which the > non-NaNs are unsorted, is really bad. > If your going to do isnan anyway, why not just raise an exception. An array with NaNs in it can't be sorted by any common sense definition of sorting. Any treatment of NaNs is going to be arbitrary, so we might as well make the user specify what they want. "In the face of ambiguity, refuse the temptation to guess" and all that. My favorite solution would be to make sort respect the invalid mode of seterr/geterr. However at the moment it doesn't seem to (in beta4 at least) but neither does add or multiply so those probably need to be looked at again.... -tim From strawman at astraw.com Tue Sep 19 17:44:53 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 19 Sep 2006 14:44:53 -0700 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FB705.7010002@ubuntu-au.org> References: <450FB705.7010002@ubuntu-au.org> Message-ID: <451064D5.60806@astraw.com> William Grant wrote: >Hi, > >I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it >currently cannot happen as numpy doesn't build with Python 2.5. I note >that changeset 3109 >(http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to >give 2.5 compatibility, but it is those bits which seem to break it. The >end of the build log with the errors can be found at >http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. > >Has anybody got any ideas on how to fix this? A number of Ubuntu users >want scipy 0.5.1, but that can't happen while it won't build with Python >2.5. > > AFAIK a number of Ubuntu Dapper users are happily using my .debs with Python 2.4 available at debs.astraw.com. This includes scipy 0.5.2.dev2197 Where are you getting that Ubuntu requires Python 2.5? From oliphant at ee.byu.edu Tue Sep 19 17:48:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 15:48:18 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <451063C5.20201@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> Message-ID: <451065A2.8040406@ee.byu.edu> Tim Hochberg wrote: >A. M. Archibald wrote: > > >>On 19/09/06, Tim Hochberg wrote: >> >> >> >> >>>I'm not sure where the breakpoint is, but I was seeing failures for all >>>three sort types with N as high as 10000. I suspect that they're all >>>broken in the presence of NaNs. I further suspect you'd need some >>>punishingly slow n**2 algorithm to be robust in the presence of NaNs. >>> >>> >>> >>Not at all. Just temporarily make NaNs compare greater than any other >>floating-point value and use quicksort (say). You can even do this for >>python lists without much trouble. >> >> >> >I misspoke. What I meant here was keeping the behavior that people think >that we already have but don't: NaNs stay in place and everything is >sorted around them. And even that's not true, since you could just >record where the NaNs are, remove them, sort and put them back. What I >was really getting at was, that I'm guessing, and it's just a guess, >that (a) none of the fast sorting algorithms do anything sensible unless >special cased and (b) one could come up with a naive n**2 sort that does >do something sensible without special casing (where sensible means leave >the NaNs alone). > > >>That's actually a viable suggestion for numpy's sorting, although it >>would be kind of ugly to implement: do a quick any(isnan(A)), and if >>not, use the fast stock sorting algorithms; if there is a NaN >>somewhere in the array, use a version of the sort that has a tweaked >>comparison function so the NaNs wind up at the end and are easy to >>trim off. >> >>But the current situation, silently returning arrays in which the >>non-NaNs are unsorted, is really bad. >> >> >> >If your going to do isnan anyway, why not just raise an exception. An >array with NaNs in it can't be sorted by any common sense definition of >sorting. Any treatment of NaNs is going to be arbitrary, so we might as >well make the user specify what they want. "In the face of ambiguity, >refuse the temptation to guess" and all that. > >My favorite solution would be to make sort respect the invalid mode of >seterr/geterr. However at the moment it doesn't seem to (in beta4 at >least) but neither does add or multiply so those probably need to be >looked at again.... > > The geterr/seterr stuff changes how IEEE hardware flags are handled in ufuncs. Currently they are not even looked at elsewhere. Are you saying that add and multiply don't respect the invalid flag? If not, then this might be hardware related. Does the IEEE invalid hardware flag get raised on multiplication by nan or only on creation of nan? All the seterr/geterr stuff relies on the hardware flags. We don't do any other checking. -Travis From william.grant at ubuntu.com.au Tue Sep 19 17:54:16 2006 From: william.grant at ubuntu.com.au (William Grant) Date: Wed, 20 Sep 2006 07:54:16 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <451064D5.60806@astraw.com> References: <450FB705.7010002@ubuntu-au.org> <451064D5.60806@astraw.com> Message-ID: <45106708.3090906@ubuntu.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andrew Straw wrote: > William Grant wrote: > >> Hi, >> >> I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it >> currently cannot happen as numpy doesn't build with Python 2.5. I note >> that changeset 3109 >> (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to >> give 2.5 compatibility, but it is those bits which seem to break it. The >> end of the build log with the errors can be found at >> http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. >> >> Has anybody got any ideas on how to fix this? A number of Ubuntu users >> want scipy 0.5.1, but that can't happen while it won't build with Python >> 2.5. >> >> > AFAIK a number of Ubuntu Dapper users are happily using my .debs with > Python 2.4 available at debs.astraw.com. This includes scipy 0.5.2.dev2197 > > Where are you getting that Ubuntu requires Python 2.5? Ubuntu 6.10 (currently in development) includes Python 2.5. William. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFFEGcIrbS+qZ0dQCURAhsLAJwLwZvYabv6IUOWHlmvIaRnQmk69QCeJfOt xFZp00DLb+/PWtbVjUg99gY= =D2zA -----END PGP SIGNATURE----- From charlesr.harris at gmail.com Tue Sep 19 18:00:18 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 16:00:18 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Charles R Harris wrote: > > > If this sort of thing can cause unexpected errors I wonder if it would > be > > worth it to have a global debugging flag that essentially causes isnan > to > > be called before any function applications. > > That sounds very like the IEEE floating-point flags, which would be > extremely useful to have, and which are being wored on, IIRC. Thinking a bit, keeping the values in place isn't easy. Mergesort isn't fixable because values can be moved in front of the nan before it is ever looked at. Nor can it be easily set up to leave all the nans at one end because both a < nan and nan < a return false. Quicksort might be doable with some checks. I mean, what if the selected pivot is a nan? The median of three version used also needs thinking about. Hmm. But I think it is the insertion sort that is messing up the order in mergesort as now nothing will move past the nan even if it has to. That could be fixed, but the nan's would still move around. I think the best thing to do is punt unless the hardware can be set to do something. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 18:12:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 15:12:08 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <451065A2.8040406@ee.byu.edu> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> Message-ID: <45106B38.6010504@ieee.org> Travis Oliphant wrote: > Tim Hochberg wrote: > > >> A. M. Archibald wrote: >> >> >> >>> On 19/09/06, Tim Hochberg wrote: >>> >>> >>> >>> >>> >>>> I'm not sure where the breakpoint is, but I was seeing failures for all >>>> three sort types with N as high as 10000. I suspect that they're all >>>> broken in the presence of NaNs. I further suspect you'd need some >>>> punishingly slow n**2 algorithm to be robust in the presence of NaNs. >>>> >>>> >>>> >>>> >>> Not at all. Just temporarily make NaNs compare greater than any other >>> floating-point value and use quicksort (say). You can even do this for >>> python lists without much trouble. >>> >>> >>> >>> >> I misspoke. What I meant here was keeping the behavior that people think >> that we already have but don't: NaNs stay in place and everything is >> sorted around them. And even that's not true, since you could just >> record where the NaNs are, remove them, sort and put them back. What I >> was really getting at was, that I'm guessing, and it's just a guess, >> that (a) none of the fast sorting algorithms do anything sensible unless >> special cased and (b) one could come up with a naive n**2 sort that does >> do something sensible without special casing (where sensible means leave >> the NaNs alone). >> >> >> >>> That's actually a viable suggestion for numpy's sorting, although it >>> would be kind of ugly to implement: do a quick any(isnan(A)), and if >>> not, use the fast stock sorting algorithms; if there is a NaN >>> somewhere in the array, use a version of the sort that has a tweaked >>> comparison function so the NaNs wind up at the end and are easy to >>> trim off. >>> >>> But the current situation, silently returning arrays in which the >>> non-NaNs are unsorted, is really bad. >>> >>> >>> >>> >> If your going to do isnan anyway, why not just raise an exception. An >> array with NaNs in it can't be sorted by any common sense definition of >> sorting. Any treatment of NaNs is going to be arbitrary, so we might as >> well make the user specify what they want. "In the face of ambiguity, >> refuse the temptation to guess" and all that. >> >> My favorite solution would be to make sort respect the invalid mode of >> seterr/geterr. However at the moment it doesn't seem to (in beta4 at >> least) but neither does add or multiply so those probably need to be >> looked at again.... >> >> >> > The geterr/seterr stuff changes how IEEE hardware flags are handled in > ufuncs. Currently they are not even looked at elsewhere. Are you > saying that add and multiply don't respect the invalid flag? If not, > then this might be hardware related. Does the IEEE invalid hardware > flag get raised on multiplication by nan or only on creation of nan? It looks like I jumped to conclusions here. I was expecting that with invalid set to 'raise' that an array that (+-*operations on nans would raise an exception. It appears that is incorrect -- only operations that create nans from non-nans will trigger this as you suspected. Similarly, I expected that over='raise' would cause inf+something to raise an exception. Again, not true; this raises an exception only when a new inf is created from non infs. At least on my box. Interesting. And a little surprising. An interesting tidbit (in an array) inf/inf will raise invalid, but nan/nan will not, it just returns nan. Fun, fun fun! > > All the seterr/geterr stuff relies on the hardware flags. We don't do > any other checking. > Yeah. And just by itself this isn't going to do anything for sort since comparing nans will not set any flags, it's just that the result will be problematic.If one wanted to use these flags for this one would have to use/abuse the result of geterr to trigger an isnan test at the beginning of sort and then warn, raise or call as appropriate. It would probably also not be unreasonable to punt and document sort as failing in the presence of nans. -tim From haase at msg.ucsf.edu Tue Sep 19 18:23:44 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 15:23:44 -0700 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191454.03405.sransom@nrao.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> Message-ID: <200609191523.45253.haase@msg.ucsf.edu> OK - I'm really sorry !! I also get 'u' -- I had a typo there ... But what is the complete list of kind values ? -Sebastian On Tuesday 19 September 2006 11:54, Scott Ransom wrote: > > > Can anybody on a 64-bit system confirm? > > > > I'm on 64-bit Debian: > > > > In [11]: arr=N.arange(10,dtype=N.uint) > > > > In [12]: arr.dtype.kind > > Out[12]: 'u' > > > > In [13]: arr.dtype.itemsize > > Out[13]: 4 > > > > In [14]: arr=N.arange(10,dtype=N.long) > > > > In [15]: arr.dtype.kind > > Out[15]: 'i' > > > > In [16]: arr.dtype.itemsize > > Out[16]: 8 > > Ack! That was on the wrong machine (32-bit Debian). Here is the 64-bit > version: > > In [2]: arr=N.arange(10,dtype=N.uint) > > In [3]: arr.dtype.kind > Out[3]: 'u' > > In [4]: arr.dtype.itemsize > Out[4]: 8 > > In [5]: arr=N.arange(10,dtype=N.long) > > In [6]: arr.dtype.kind > Out[6]: 'i' > > In [7]: arr.dtype.itemsize > Out[7]: 8 > > Sorry about that, > > Scott From haase at msg.ucsf.edu Tue Sep 19 18:33:23 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 15:33:23 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float Message-ID: <200609191533.24097.haase@msg.ucsf.edu> Hello all, I just had someone from my lab coming to my desk saying: "My god - SciPy is really stupid .... An array with only positive numbers claims to have a negative mean !! "? I was asking about this before ... the reason was of course that her array was of dtype int32 and had many large values to cause an overflow (wrap around) . Now that the default for axis is None (for all functions having an axis argument), can we please change dtype to default to float64 !? It is really a very confusing and shocking result to get a negative mean on all positive values. It has been stated here before that numpy should target the "scientist" rather than the programmers ... I would argue that mean() almost always requires the precision of "double" (float64) to produce usable results. Please consider this change before the 1.0 release ... Thanks, Sebastian Haase From oliphant at ee.byu.edu Tue Sep 19 18:44:49 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 16:44:49 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191523.45253.haase@msg.ucsf.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> Message-ID: <451072E1.7030608@ee.byu.edu> Sebastian Haase wrote: >OK - I'm really sorry !! >I also get 'u' -- I had a typo there ... > >But what is the complete list of kind values ? > > It's in the array interface specification: http://numpy.scipy.org/array_interface.shtml -Travis From oliphant at ee.byu.edu Tue Sep 19 18:48:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 16:48:05 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191533.24097.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> Message-ID: <451073A5.4060204@ee.byu.edu> Sebastian Haase wrote: >Hello all, >I just had someone from my lab coming to my desk saying: >"My god - SciPy is really stupid .... >An array with only positive numbers claims to have a negative mean !! "? > > > >I was asking about this before ... the reason was of course that her array was >of dtype int32 and had many large values to cause an overflow (wrap >around) . > >Now that the default for axis is None (for all functions having an axis >argument), >can we please change dtype to default to float64 !? > > The default is float64 now (as long as you are not using numpy.oldnumeric). I suppose more appropriately, we could reduce over float for integer data-types when calculating the mean as well (since a floating point is returned anyway). -Travis From charlesr.harris at gmail.com Tue Sep 19 18:55:57 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 16:55:57 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45106B38.6010504@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > Tim Hochberg wrote: > > > > > >> A. M. Archibald wrote: > >> > >> > >> > >>> On 19/09/06, Tim Hochberg wrote: > >>> > >>> > >>> > >>> > >>> > >>>> I'm not sure where the breakpoint is, but I was seeing failures for > all > >>>> three sort types with N as high as 10000. I suspect that they're all > >>>> broken in the presence of NaNs. I further suspect you'd need some > >>>> punishingly slow n**2 algorithm to be robust in the presence of NaNs. > >>>> > >>>> > >>>> > >>>> > >>> Not at all. Just temporarily make NaNs compare greater than any other > >>> floating-point value and use quicksort (say). You can even do this for > >>> python lists without much trouble. > >>> > >>> > >>> > >>> > >> I misspoke. What I meant here was keeping the behavior that people > think > >> that we already have but don't: NaNs stay in place and everything is > >> sorted around them. And even that's not true, since you could just > >> record where the NaNs are, remove them, sort and put them back. What I > >> was really getting at was, that I'm guessing, and it's just a guess, > >> that (a) none of the fast sorting algorithms do anything sensible > unless > >> special cased and (b) one could come up with a naive n**2 sort that > does > >> do something sensible without special casing (where sensible means > leave > >> the NaNs alone). > >> > >> > >> > >>> That's actually a viable suggestion for numpy's sorting, although it > >>> would be kind of ugly to implement: do a quick any(isnan(A)), and if > >>> not, use the fast stock sorting algorithms; if there is a NaN > >>> somewhere in the array, use a version of the sort that has a tweaked > >>> comparison function so the NaNs wind up at the end and are easy to > >>> trim off. > >>> > >>> But the current situation, silently returning arrays in which the > >>> non-NaNs are unsorted, is really bad. > >>> > >>> > >>> > >>> > >> If your going to do isnan anyway, why not just raise an exception. An > >> array with NaNs in it can't be sorted by any common sense definition of > >> sorting. Any treatment of NaNs is going to be arbitrary, so we might as > >> well make the user specify what they want. "In the face of ambiguity, > >> refuse the temptation to guess" and all that. > >> > >> My favorite solution would be to make sort respect the invalid mode of > >> seterr/geterr. However at the moment it doesn't seem to (in beta4 at > >> least) but neither does add or multiply so those probably need to be > >> looked at again.... > >> > >> > >> > > The geterr/seterr stuff changes how IEEE hardware flags are handled in > > ufuncs. Currently they are not even looked at elsewhere. Are you > > saying that add and multiply don't respect the invalid flag? If not, > > then this might be hardware related. Does the IEEE invalid hardware > > flag get raised on multiplication by nan or only on creation of nan? > It looks like I jumped to conclusions here. I was expecting that with > invalid set to 'raise' that an array that (+-*operations on nans would > raise an exception. It appears that is incorrect -- only operations that > create nans from non-nans will trigger this as you suspected. Similarly, > I expected that over='raise' would cause inf+something to raise an > exception. Again, not true; this raises an exception only when a new inf > is created from non infs. At least on my box. > > Interesting. And a little surprising. > > An interesting tidbit (in an array) inf/inf will raise invalid, but > nan/nan will not, it just returns nan. Fun, fun fun! > > > > All the seterr/geterr stuff relies on the hardware flags. We don't do > > any other checking. > > > > Yeah. And just by itself this isn't going to do anything for sort since > comparing nans will not set any flags, it's just that the result will be > problematic.If one wanted to use these flags for this one would have to > use/abuse the result of geterr to trigger an isnan test at the beginning > of sort and then warn, raise or call as appropriate. > > It would probably also not be unreasonable to punt and document sort as > failing in the presence of nans. For floats we could use something like: lessthan(a,b) := a < b || (a == nan && b != nan) Which would put all the nans at one end and might not add too much overhead. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Tue Sep 19 18:59:07 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 20 Sep 2006 00:59:07 +0200 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? In-Reply-To: <451038F4.7090704@ieee.org> References: <200609191734.57899.martin.wiechert@gmx.de> <451038F4.7090704@ieee.org> Message-ID: <200609200059.07746.martin.wiechert@gmx.de> On Tuesday 19 September 2006 20:37, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > Please forgive my ignorance: Is there any difference between npy_intp and > > size_t. Aren't both "ints just big enough to be safe with pointer > > arithmetics even on 64 bit architectures?". > > size_t is unsigned > npy_intp is signed > (!) Thanks again, Travis. > It is basically the same as Py_ssize_t (which is not available until > Python 2.5). Out now! http://www.python.org/2.5 > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From charlesr.harris at gmail.com Tue Sep 19 19:10:26 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 17:10:26 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. Message-ID: Travis, Is this intentional? In [77]: arange(5, dtype=int)/0 Out[77]: array([0, 0, 0, 0, 0]) It looks deliberate because all zeros are returned, but it might be better if it raised an exception. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Tue Sep 19 19:13:54 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:13:54 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: References: Message-ID: <451079B2.60805@ee.byu.edu> Charles R Harris wrote: > Travis, > > Is this intentional? > > In [77]: arange(5, dtype=int)/0 > Out[77]: array([0, 0, 0, 0, 0]) > > It looks deliberate because all zeros are returned, but it might be > better if it raised an exception. It is deliberate. Numarray introduced it (the only difference being that by default NumPy has division-by-zero erros turned off). It's tied to the way floating-point division-by zero is handled. There is a valid argument for having a separate integer-division flag so that you can raise exceptions for integer-division but not for floating-point division. I'm open to that change for 1.0rc1 -Travis From oliphant at ee.byu.edu Tue Sep 19 19:16:32 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:16:32 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: <45107A50.4010502@ee.byu.edu> Charles R Harris wrote: > > > On 9/18/06, *Bill Baxter* > wrote: > > On 9/19/06, Charles R Harris > wrote: > > On 9/18/06, Bill Baxter > wrote: > > > I find myself often wanting both the max and the argmax of an > array. > > > (And same for the other arg* functions) > > > > You have to do something like > > > a = rand(10,5) > > > imax = a.argmax(axis=0) > > > vmax = a[(imax, range(5))] > > > > > I don't generally like overloading return values, the function > starts to > > lose its definition and becomes a bit baroque where simply > changing a > > keyword value can destroy the viability of the following code. > > Agreed. Seems like the only justification is if you get multiple > results from one calculation but only rarely want the extra values. > It doesn't make sense to always return them, but it's also not worth > making a totally different function. > > > > But I can see the utility of what you want. Hmm, this problem > is not unique to argmax. > > Maybe what we need is a general way to extract values, something > like > > > > extract(a, imax, axis=0) > > > > to go along with all the single axis functions. > > Yes, I think that would be easier to remember. > > It should also work for the axis=None case. > imax = a.argmax(axis=None) > v = extract(a, imax, axis=None) > > > It shouldn't be too difficult to jig something up given all the > example code. I can do that, but I would like more input first. The > questions I have are these. > > 1) Should it be done? > 2) Should it be a method? (functions being somewhat deprecated) > 3) What name should it have? > > I think Travis will have to weigh in on this. IIRC, he felt that the > number of methods was getting out of hand. I can support adding a *function* that does both. It can't be named extract (that already exists). There should be one for all the "arg"-like functions. If somebody doesn't add it before 1.0 final, it can wait for 1.0.1 -Travis From oliphant at ee.byu.edu Tue Sep 19 19:53:32 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:53:32 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: References: Message-ID: <451082FC.2060404@ee.byu.edu> Charles R Harris wrote: > Travis, > > Is this intentional? > > In [77]: arange(5, dtype=int)/0 > Out[77]: array([0, 0, 0, 0, 0]) > > It looks deliberate because all zeros are returned, but it might be > better if it raised an exception. As mentioned before we translate integer division errors into floating-point erros and use the same hardware trapping to trap them if the user requests it. Simulating and "integer-division-by-zero" hardware flag is not trivial as we would have to manage context switching ourselves. So, at least for 1.0, integer and floating-point division by zero are going to be handled the same. -Travis From haase at msg.ucsf.edu Tue Sep 19 19:59:08 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 16:59:08 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451073A5.4060204@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> Message-ID: <200609191659.08788.haase@msg.ucsf.edu> On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > Sebastian Haase wrote: > >can we please change dtype to default to float64 !? > > The default is float64 now (as long as you are not using > numpy.oldnumeric). > > I suppose more appropriately, we could reduce over float for integer > data-types when calculating the mean as well (since a floating point is > returned anyway). > Is now mean() always "reducing over" float64 ? The svn note """Log: Fix mean, std, and var methods so that they reduce over double data-type with integer inputs. """ makes it sound that a float32 input is stays float32 ? For mean calculation this might introduce large errors - I usually would require double-precision for *any* input type ... (don't know how to say this for complex types !? Are here real and imag treated separately / independently ?) Thanks, Sebastian Haase From oliphant at ee.byu.edu Tue Sep 19 20:17:47 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 18:17:47 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191659.08788.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> Message-ID: <451088AB.4050107@ee.byu.edu> Sebastian Haase wrote: >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > > >>Sebastian Haase wrote: >> >> > > > >>>can we please change dtype to default to float64 !? >>> >>> >>The default is float64 now (as long as you are not using >>numpy.oldnumeric). >> >>I suppose more appropriately, we could reduce over float for integer >>data-types when calculating the mean as well (since a floating point is >>returned anyway). >> >> >> > >Is now mean() always "reducing over" float64 ? >The svn note """Log: >Fix mean, std, and var methods so that they reduce over double data-type with >integer inputs. >""" >makes it sound that a float32 input is stays float32 ? > > Yes, that is true. Only integer inputs are changed because you are going to get a floating point output anyway. >For mean calculation this might introduce large errors - I usually would >require double-precision for *any* input type ... > > Of course. The system is not fool-proof. I hesitate to arbitrarily change this. The advantage of using single-precision calculation is that it is faster. We do rely on the user who expressly requests these things to be aware of the difficulties. >(don't know how to say this for complex types !? Are here real and imag >treated separately / independently ?) > > There is a complex add performed at a low-level as two separate adds. The addition is performed in the precision requested. -Travis From robert.kern at gmail.com Tue Sep 19 20:18:52 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 19:18:52 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191659.08788.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > (don't know how to say this for complex types !? Are here real and imag > treated separately / independently ?) Yes. For mean(), there's really no alternative. Scalar variance is not a well-defined concept for complex numbers, but treating the real and imaginary parts separately is a sensible and (partially) informative thing to do. Simply applying the formula for estimating variance for real numbers to complex numbers (i.e. change "x" to "z") is a meaningless operation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 19 20:37:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 18:37:01 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451088AB.4050107@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: On 9/19/06, Travis Oliphant wrote: > > Sebastian Haase wrote: > > >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > > > > > >>Sebastian Haase wrote: > >> > >> > > > > > > > >>>can we please change dtype to default to float64 !? > >>> > >>> > >>The default is float64 now (as long as you are not using > >>numpy.oldnumeric). > >> > >>I suppose more appropriately, we could reduce over float for integer > >>data-types when calculating the mean as well (since a floating point is > >>returned anyway). > >> > >> > >> > > > >Is now mean() always "reducing over" float64 ? > >The svn note """Log: > >Fix mean, std, and var methods so that they reduce over double data-type > with > >integer inputs. > >""" > >makes it sound that a float32 input is stays float32 ? > > > > > Yes, that is true. Only integer inputs are changed because you are > going to get a floating point output anyway. > > >For mean calculation this might introduce large errors - I usually would > >require double-precision for *any* input type ... > > > > > Of course. The system is not fool-proof. I hesitate to arbitrarily > change this. The advantage of using single-precision calculation is > that it is faster. We do rely on the user who expressly requests these > things to be aware of the difficulties. Speed depends on the achitecture. Float is a trifle slower than double on my Athlon64, but faster on PPC750. I don't know about other machines. I think there is a good argument for accumlating in double and converting to float for output if required. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 20:41:30 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 20:41:30 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 19/09/06, Charles R Harris wrote: > > > > For floats we could use something like: > > lessthan(a,b) := a < b || (a == nan && b != nan) > > Which would put all the nans at one end and might not add too much overhead. You could put an any(isnan()) out front and run this slower version only if there are any NaNs (also, you can't use == for NaNs, you have to use C isNaN). But I'm starting to see the wisdom in simply throwing an exception, since sorting is not well-defined with NaNs. A. M. Archibald From haase at msg.ucsf.edu Tue Sep 19 20:44:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 17:44:43 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451088AB.4050107@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: <200609191744.43847.haase@msg.ucsf.edu> On Tuesday 19 September 2006 17:17, Travis Oliphant wrote: > Sebastian Haase wrote: > >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > >>Sebastian Haase wrote: > > > > > > > >>>can we please change dtype to default to float64 !? > >> > >>The default is float64 now (as long as you are not using > >>numpy.oldnumeric). > >> > >>I suppose more appropriately, we could reduce over float for integer > >>data-types when calculating the mean as well (since a floating point is > >>returned anyway). > > > >Is now mean() always "reducing over" float64 ? > >The svn note """Log: > >Fix mean, std, and var methods so that they reduce over double data-type > > with integer inputs. > >""" > >makes it sound that a float32 input is stays float32 ? > > Yes, that is true. Only integer inputs are changed because you are > going to get a floating point output anyway. > > >For mean calculation this might introduce large errors - I usually would > >require double-precision for *any* input type ... > > Of course. The system is not fool-proof. I hesitate to arbitrarily > change this. The advantage of using single-precision calculation is > that it is faster. We do rely on the user who expressly requests these > things to be aware of the difficulties. I still would argue that getting a "good" (smaller rounding errors) answer should be the default -- if speed is wanted, then *that* could be still specified by explicitly using dtype=float32 (which would also be a possible choice for int32 input) . In image processing we always want means to be calculated in float64 even though input data is always float32 (if not uint16). Also it is simpler to say "float64 is the default" (full stop.) - instead "float64 is the default unless you have float32" Thanks, Sebastian Haase From charlesr.harris at gmail.com Tue Sep 19 20:55:17 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 18:55:17 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Charles R Harris wrote: > > > > > > > > For floats we could use something like: > > > > lessthan(a,b) := a < b || (a == nan && b != nan) > > > > Which would put all the nans at one end and might not add too much > overhead. > > You could put an any(isnan()) out front and run this slower version > only if there are any NaNs (also, you can't use == for NaNs, you have > to use C isNaN). But I'm starting to see the wisdom in simply throwing > an exception, since sorting is not well-defined with NaNs. Looks like mergesort can be modified to sort around the NaNs without too much trouble if there is a good isnan function available: just cause the pointers to skip over them. I see that most of the isnan stuff seems to be in the ufunc source and isn't terribly simple. Could be broken out into a separate include, I suppose. I still wonder if it is worth the trouble. As to raising an exception, I seem to recall reading somewhere that exception code tends to be expensive, I haven't done any benchmarks myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From NetBankNotification at cba.com.au Tue Sep 19 21:42:13 2006 From: NetBankNotification at cba.com.au (Commonwealth Bank Groups) Date: Tue, 19 Sep 2006 21:42:13 -0400 Subject: [Numpy-discussion] Security Alert Message-ID: An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 21:45:36 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 18:45:36 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: <45109D40.7020008@ieee.org> Charles R Harris wrote: > > > On 9/19/06, *A. M. Archibald* > wrote: > > On 19/09/06, Charles R Harris > wrote: > > > > > > > > For floats we could use something like: > > > > lessthan(a,b) := a < b || (a == nan && b != nan) > I believe this would have to be some sort of isnan macros since everything compares not equal to nan. I forget the correct spelling at the moment though, > > > > Which would put all the nans at one end and might not add too > much overhead. > > You could put an any(isnan()) out front and run this slower version > only if there are any NaNs (also, you can't use == for NaNs, you have > to use C isNaN). But I'm starting to see the wisdom in simply throwing > an exception, since sorting is not well-defined with NaNs. > > > Looks like mergesort can be modified to sort around the NaNs without > too much trouble if there is a good isnan function available: just > cause the pointers to skip over them. I see that most of the isnan > stuff seems to be in the ufunc source and isn't terribly simple. Could > be broken out into a separate include, I suppose. > > I still wonder if it is worth the trouble. As to raising an exception, > I seem to recall reading somewhere that exception code tends to be > expensive, I haven't done any benchmarks myself. I'm still somewhat mystified by the desire to move the nans to one end of the sorted object. I see two scenarios: 1. The user is not expecting to have nans in the data. In this case moving the nans to end is not helpful. The resulting array is still not sorted in the sense that a[i-1]<=a[i]<=a[i+1] does not hold and thus is likely to break code that relies on the array being sorted. The most prominent example of which is searchsorted. In this case you really want to raise an exception if possible since no good will come from letting the code continue to run. In this case the time in involved in throwing and catching an exception is irrelevant. 2. The user *is* expecting to have nans in the data. This is presumably the case that the sorting-nans-to-the-end idea is aimed at. So far at least the suggested use has been to sort and then strip the nans. I suggest that a better approach is to test for and strip the nans before the sort. For example: # a is an array that may have some nans # you can do this more pithily, but I'm aiming to minimize isnan calls # note that this *sometimes* makes a copy. nanmask = isnan(a) if sometrue(nanmask): a = a[not nanmask] a.sort() #..... I presume that isnan is somewhat more expensive than the basic '<' operator. In the proposed sort to end version we need N*log(N) isnan calls versus just N in the above case. The sort to end case probably won't look any cleaner than the above either since you still need to count the nans to determine how many to strip. Perhaps there's some use for the sort to end behaviour that I'm missing, but the raise an exception behaviour sure looks a lot more appealing to me. Here's a strawman proposal: Sort the array. Then examine numpy.geterr()['invalid']. If it is not 'ignore', then check examine sometrue(isnan(thearray)). If the latter is true then raise and error, issue a warning or call the error reporting functioni as appropriate. Note that we always sort the array to be consistent with the behaviour of the ufuncs that proceed even when they end up raising an exception. -tim From peridot.faceted at gmail.com Tue Sep 19 22:09:33 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 22:09:33 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45109D40.7020008@ieee.org> References: <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > I'm still somewhat mystified by the desire to move the nans to one end > of the sorted object. I see two scenarios: It's mostly to have something to do with them other than throw an exception. Leaving them in place while the rest of the array is reshuffled requires a lot of work and isn't particularly better. I mostly presented it as an alternative to throwing an exception. Throwing a Python exception now seems like the most reasonable idea. A. M. Archibald From oliphant.travis at ieee.org Tue Sep 19 22:31:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 20:31:07 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: <4510A7EB.8000207@ieee.org> Charles R Harris wrote: > > Speed depends on the achitecture. Float is a trifle slower than double > on my Athlon64, but faster on PPC750. I don't know about other > machines. I think there is a good argument for accumlating in double > and converting to float for output if required. Yes there is. It's just not what NumPy ever does so it would be an exception in this case and would need a more convincing argument in my opinion. You can always specify the accumulation type yourself with the dtype argument. We are only talking about what the default should be. -Travis From oliphant.travis at ieee.org Tue Sep 19 22:33:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 20:33:52 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191744.43847.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> Message-ID: <4510A890.5090807@ieee.org> Sebastian Haase wrote: > I still would argue that getting a "good" (smaller rounding errors) answer > should be the default -- if speed is wanted, then *that* could be still > specified by explicitly using dtype=float32 (which would also be a possible > choice for int32 input) . > So you are arguing for using long double then.... ;-) > In image processing we always want means to be calculated in float64 even > though input data is always float32 (if not uint16). > > Also it is simpler to say "float64 is the default" (full stop.) - instead > > "float64 is the default unless you have float32" > "the type you have is the default except for integers". Do you really want float64 to be the default for float96? Unless we are going to use long double as the default, then I'm not convinced that we should special-case the "double" type. -Travis From charlesr.harris at gmail.com Tue Sep 19 22:42:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 20:42:52 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg wrote: > > > I'm still somewhat mystified by the desire to move the nans to one end > > of the sorted object. I see two scenarios: > > It's mostly to have something to do with them other than throw an > exception. Leaving them in place while the rest of the array is > reshuffled requires a lot of work and isn't particularly better. I > mostly presented it as an alternative to throwing an exception. > > Throwing a Python exception now seems like the most reasonable idea. Well, mergesort can be adapted without a lot of work. Could be used to sort masked arrays too, not that I know why anyone would want that, but then again, I haven't used masked arrays. Agreed, throwing some sort of error still seems the simplest thing to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Tue Sep 19 23:07:37 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 20:07:37 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510A890.5090807@ieee.org> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> Message-ID: <4510B079.5010209@msg.ucsf.edu> Travis Oliphant wrote: > Sebastian Haase wrote: >> I still would argue that getting a "good" (smaller rounding errors) answer >> should be the default -- if speed is wanted, then *that* could be still >> specified by explicitly using dtype=float32 (which would also be a possible >> choice for int32 input) . >> > So you are arguing for using long double then.... ;-) > >> In image processing we always want means to be calculated in float64 even >> though input data is always float32 (if not uint16). >> >> Also it is simpler to say "float64 is the default" (full stop.) - instead >> >> "float64 is the default unless you have float32" >> > "the type you have is the default except for integers". Do you really > want float64 to be the default for float96? > > Unless we are going to use long double as the default, then I'm not > convinced that we should special-case the "double" type. > I guess I'm not really aware of the float96 type ... Is that a "machine type" on any system ? I always thought that -- e.g. coming from C -- double is "as good as it gets"... Who uses float96 ? I heard somewhere that (some) CPUs use 80bits internally when calculating 64bit double-precision... Is this not going into some academic argument !? For all I know, calculating mean()s (and sum()s, ...) is always done in double precision -- never in single precision, even when the data is in float32. Having float32 be the default for float32 data is just requiring more typing, and more explaining ... it would compromise numpy usability as a day-to-day replacement for other systems. Sorry, if I'm being ignorant ... - Sebastian From charlesr.harris at gmail.com Tue Sep 19 23:15:29 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 21:15:29 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510B079.5010209@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Sebastian Haase wrote: > > Travis Oliphant wrote: > > Sebastian Haase wrote: > >> I still would argue that getting a "good" (smaller rounding errors) > answer > >> should be the default -- if speed is wanted, then *that* could be still > >> specified by explicitly using dtype=float32 (which would also be a > possible > >> choice for int32 input) . > >> > > So you are arguing for using long double then.... ;-) > > > >> In image processing we always want means to be calculated in float64 > even > >> though input data is always float32 (if not uint16). > >> > >> Also it is simpler to say "float64 is the default" (full stop.) - > instead > >> > >> "float64 is the default unless you have float32" > >> > > "the type you have is the default except for integers". Do you really > > want float64 to be the default for float96? > > > > Unless we are going to use long double as the default, then I'm not > > convinced that we should special-case the "double" type. > > > I guess I'm not really aware of the float96 type ... > Is that a "machine type" on any system ? I always thought that -- e.g. > coming from C -- double is "as good as it gets"... > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > internally when calculating 64bit double-precision... > > Is this not going into some academic argument !? > For all I know, calculating mean()s (and sum()s, ...) is always done in > double precision -- never in single precision, even when the data is in > float32. > > Having float32 be the default for float32 data is just requiring more > typing, and more explaining ... it would compromise numpy usability as > a day-to-day replacement for other systems. > > Sorry, if I'm being ignorant ... I'm going to side with Travis here. It is only a default and easily overridden. And yes, there are precisions greater than double. I was using quad precision back in the eighties on a VAX for some inherently ill conditioned problems. And on my machine long double is 12 bytes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Tue Sep 19 23:24:47 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 20 Sep 2006 12:24:47 +0900 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/20/06, Charles R Harris wrote: > > I guess I'm not really aware of the float96 type ... > > Is that a "machine type" on any system ? I always thought that -- e.g . > > coming from C -- double is "as good as it gets"... > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > internally when calculating 64bit double-precision... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was using > quad precision back in the eighties on a VAX for some inherently ill > conditioned problems. And on my machine long double is 12 bytes. > And on Intel chips the internal fp precision is 80bits. The D programming language even exposes this 80-bit floating point type to the user. http://www.digitalmars.com/d/type.html http://www.digitalmars.com/d/faq.html#real --bb From haase at msg.ucsf.edu Tue Sep 19 23:39:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 20:39:04 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: <4510B7D8.4010708@msg.ucsf.edu> Charles R Harris wrote: > > > On 9/19/06, *Sebastian Haase* > wrote: > > Travis Oliphant wrote: > > Sebastian Haase wrote: > >> I still would argue that getting a "good" (smaller rounding > errors) answer > >> should be the default -- if speed is wanted, then *that* could > be still > >> specified by explicitly using dtype=float32 (which would also > be a possible > >> choice for int32 input) . > >> > > So you are arguing for using long double then.... ;-) > > > >> In image processing we always want means to be calculated in > float64 even > >> though input data is always float32 (if not uint16). > >> > >> Also it is simpler to say "float64 is the default" (full stop.) > - instead > >> > >> "float64 is the default unless you have float32" > >> > > "the type you have is the default except for integers". Do you > really > > want float64 to be the default for float96? > > > > Unless we are going to use long double as the default, then I'm not > > convinced that we should special-case the "double" type. > > > I guess I'm not really aware of the float96 type ... > Is that a "machine type" on any system ? I always thought that -- e.g . > coming from C -- double is "as good as it gets"... > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > internally when calculating 64bit double-precision... > > Is this not going into some academic argument !? > For all I know, calculating mean()s (and sum()s, ...) is always done in > double precision -- never in single precision, even when the data is in > float32. > > Having float32 be the default for float32 data is just requiring more > typing, and more explaining ... it would compromise numpy usability as > a day-to-day replacement for other systems. > > Sorry, if I'm being ignorant ... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was > using quad precision back in the eighties on a VAX for some inherently > ill conditioned problems. And on my machine long double is 12 bytes. > > Chuck > I just did a web search for "long double" http://www.google.com/search?q=%22long+double%22 and it does not look like there is much agreement on what that is - see also http://en.wikipedia.org/wiki/Long_double I really think that float96 *is* a special case - but computing mean()s and var()s in float32 would be "bad science". I hope I'm not alone in seeing numpy a great "interactive platform" to do evaluate data... I know that having too much knowledge of the details often makes one forget what the "newcomers" will do and expect. We are only talking about people that will a) work with single-precision data (e.g. large scale-image analysis) and who b) will tend to "just use the default" (dtype) --- How else can I say this: these people will just assume that arr.mean() *is* the mean of arr. -Sebastian From charlesr.harris at gmail.com Tue Sep 19 23:42:58 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 21:42:58 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Charles R Harris wrote: > > > > On 9/19/06, Sebastian Haase wrote: > > > > Travis Oliphant wrote: > > > Sebastian Haase wrote: > > >> I still would argue that getting a "good" (smaller rounding errors) > > answer > > >> should be the default -- if speed is wanted, then *that* could be > > still > > >> specified by explicitly using dtype=float32 (which would also be a > > possible > > >> choice for int32 input) . > > >> > > > So you are arguing for using long double then.... ;-) > > > > > >> In image processing we always want means to be calculated in float64 > > even > > >> though input data is always float32 (if not uint16). > > >> > > >> Also it is simpler to say "float64 is the default" (full stop.) - > > instead > > >> > > >> "float64 is the default unless you have float32" > > >> > > > "the type you have is the default except for integers". Do you really > > > want float64 to be the default for float96? > > > > > > Unless we are going to use long double as the default, then I'm not > > > convinced that we should special-case the "double" type. > > > > > I guess I'm not really aware of the float96 type ... > > Is that a "machine type" on any system ? I always thought that -- e.g . > > coming from C -- double is "as good as it gets"... > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > internally when calculating 64bit double-precision... > > > > Is this not going into some academic argument !? > > For all I know, calculating mean()s (and sum()s, ...) is always done in > > double precision -- never in single precision, even when the data is in > > float32. > > > > Having float32 be the default for float32 data is just requiring more > > typing, and more explaining ... it would compromise numpy usability as > > a day-to-day replacement for other systems. > > > > Sorry, if I'm being ignorant ... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was using > quad precision back in the eighties on a VAX for some inherently ill > conditioned problems. And on my machine long double is 12 bytes. > Here is the 754r (revision) spec: http://en.wikipedia.org/wiki/IEEE_754r It includes quads (128 bits) and half precision (16 bits) floats. I believe the latter are used for some imaging stuff, radar for instance, and are also available in some high end GPUs from Nvidia and other companies. The 80 bit numbers you refer to were defined as extended precision in the original 754 spec and were mostly intended for temporaries in internal FPU computations. They have various alignment requirements for efficient use, which is why they show up as 96 bits (4 byte alignment) and sometimes 128 bits (8 byte alignment). So actually, float128 would not always distinquish between extended precision and quad precision. I see more work for Travis in the future ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 23:53:26 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 23:53:26 -0400 Subject: [Numpy-discussion] ufunc.reduce and conversion Message-ID: Hi, What are the rules for datatype conversion in ufuncs? Does ufunc(a,b) always yield the smallest type big enough to represent both a and b? What is the datatype of ufunc.reduce(a)? I ask because I was startled by the following behaviour: >>> a = array([1,1],uint8); print a.dtype; print maximum.reduce(a).dtype; '|u1' ' float, possibly only for add.reduce). Thanks, A. M. Archibald From robert.kern at gmail.com Wed Sep 20 00:10:26 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 23:10:26 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510B7D8.4010708@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > I know that having too much knowledge of the details often makes one > forget what the "newcomers" will do and expect. Please be more careful with such accusations. Repeated frequently, they can become quite insulting. > We are only talking > about people that will a) work with single-precision data (e.g. large > scale-image analysis) and who b) will tend to "just use the default" > (dtype) --- How else can I say this: these people will just assume that > arr.mean() *is* the mean of arr. I don't understand what you mean, here. arr.mean() is almost never *the* mean of arr. Double precision can't fix that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Sep 20 00:18:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 22:18:53 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Charles R Harris wrote: > > > > On 9/19/06, Charles R Harris wrote: > > > > > > > > On 9/19/06, Sebastian Haase < haase at msg.ucsf.edu> wrote: > > > > > > Travis Oliphant wrote: > > > > Sebastian Haase wrote: > > > >> I still would argue that getting a "good" (smaller rounding errors) > > > answer > > > >> should be the default -- if speed is wanted, then *that* could be > > > still > > > >> specified by explicitly using dtype=float32 (which would also be a > > > possible > > > >> choice for int32 input) . > > > >> > > > > So you are arguing for using long double then.... ;-) > > > > > > > >> In image processing we always want means to be calculated in > > > float64 even > > > >> though input data is always float32 (if not uint16). > > > >> > > > >> Also it is simpler to say "float64 is the default" (full stop.) - > > > instead > > > >> > > > >> "float64 is the default unless you have float32" > > > >> > > > > "the type you have is the default except for integers". Do you > > > really > > > > want float64 to be the default for float96? > > > > > > > > Unless we are going to use long double as the default, then I'm not > > > > convinced that we should special-case the "double" type. > > > > > > > I guess I'm not really aware of the float96 type ... > > > Is that a "machine type" on any system ? I always thought that -- e.g. > > > coming from C -- double is "as good as it gets"... > > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > > internally when calculating 64bit double-precision... > > > > > > Is this not going into some academic argument !? > > > For all I know, calculating mean()s (and sum()s, ...) is always done > > > in > > > double precision -- never in single precision, even when the data is > > > in > > > float32. > > > > > > Having float32 be the default for float32 data is just requiring more > > > typing, and more explaining ... it would compromise numpy usability > > > as > > > a day-to-day replacement for other systems. > > > > > > Sorry, if I'm being ignorant ... > > > > > > I'm going to side with Travis here. It is only a default and easily > > overridden. And yes, there are precisions greater than double. I was using > > quad precision back in the eighties on a VAX for some inherently ill > > conditioned problems. And on my machine long double is 12 bytes. > > > > Here is the 754r (revision) spec: http://en.wikipedia.org/wiki/IEEE_754r > > It includes quads (128 bits) and half precision (16 bits) floats. I > believe the latter are used for some imaging stuff, radar for instance, and > are also available in some high end GPUs from Nvidia and other companies. > The 80 bit numbers you refer to were defined as extended precision in the > original 754 spec and were mostly intended for temporaries in internal FPU > computations. They have various alignment requirements for efficient use, > which is why they show up as 96 bits (4 byte alignment) and sometimes 128 > bits (8 byte alignment). So actually, float128 would not always distinquish > between extended precision and quad precision. I see more work for Travis > in the future ;) > I just checked this out. On amd64 32 bit linux gives 12 bytes for long double, 64 bit linux gives 16 bytes for long doubles, but they both have 64 bit mantissas, i.e., they are both 80 bit extended precision. Those sizes are the defaults and can be overridden by compiler flags. Anyway, we may need some way to tell the difference between float128 and quads since they will both have the same length on 64 bit architectures. But that is a problem for the future. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Sep 20 00:53:16 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 21:53:16 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> Message-ID: <4510C93C.8030902@msg.ucsf.edu> Robert Kern wrote: > Sebastian Haase wrote: >> I know that having too much knowledge of the details often makes one >> forget what the "newcomers" will do and expect. > > Please be more careful with such accusations. Repeated frequently, they can > become quite insulting. > I did not mean to insult anyone - what I meant was, that I'm for numpy becoming an easy platform to use. I have spend and enjoyed part the last four years developing and evangelizing Python as an alternative to Matlab and C/Fortran based image analysis environment. I often find myself arguing for good support of the single precision data format. So I find it actually somewhat ironic to see myself arguing now for wanting float64 over float32 ;-) >> We are only talking >> about people that will a) work with single-precision data (e.g. large >> scale-image analysis) and who b) will tend to "just use the default" >> (dtype) --- How else can I say this: these people will just assume that >> arr.mean() *is* the mean of arr. > > I don't understand what you mean, here. arr.mean() is almost never *the* mean of > arr. Double precision can't fix that. > This was not supposed to be a scientific statement -- I'm (again) thinking of our students that not always appreciate the full complexity of computational numerics and data types and such. The best I can hope for is a "sound" default for most (practical) cases... I still think that 80bit vs. 128bit vs 96bit is rather academic for most people ... most people seem to only use float64 and then there are some that use float32 (like us) ... Cheers, Sebastian From konrad.hinsen at laposte.net Wed Sep 20 03:41:23 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Wed, 20 Sep 2006 09:41:23 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: <45101F5F.6080209@noaa.gov> References: <45101F5F.6080209@noaa.gov> Message-ID: <3BB688EB-A95D-464F-8AAC-674CA4E5ECB5@laposte.net> On 19.09.2006, at 18:48, Christopher Barker wrote: > Konrad Hinsen wrote: >> MMTK works fine with Numeric 23.x (and probably many other versions), >> so I don't see a pressing need to change to NumPy. > > Pressing is in the eye of the beholder. Obviously. It also depends on the context in which one develops or uses software. For me, a pressing need is to finish the two publications I am working on. Next come various administrational tasks that have a deadline. On the third place, there's work on a new research project that I started recently. Software development is at best on position number 4, but in that category my personal priorities are adding more unit tests and reworking the MMTK documentation system, the old one being unusable because various pieces of code it relied on are no longer supported. As with many other scientific software projects, MMTK development is completely unfunded and even hardly recognized by my employer's evaluation committees. Any work on MMTK that does not help me in a research project can therefore not be a priority for me. > However: I don't think we should underestimate the negative impact of > the Numeric/numarray split on the usability and perception of the > community. Also the impact on how much work has been done to > accommodate it. If you consider matplotlib alone: I completely agree. The existence of a third choice (NumPy) just makes it worse. For client code like mine there is little chance to escape from the split issues. Even if I had the resources to adapt all my code to NumPy immediately, I'd still have to support Numeric because that's what everyone is using at the moment, and many users can't or won't switch immediately. Since the APIs are not fully compatible, I either have to support two versions in parallel, or introduce my own compatibility layer. > In addition, as I understand it, MMTK was NOT working fine for the OP. The issues he had were already solved, he just had to apply the solutions (i.e. reinstall using a more recent version and appropriate compilation options). > As robust as they (and Numeric) might be, when you need to run > something on a new platform (OS-X - Intel comes to mind), or use a new > LAPACK, or whatever, there are going to be (and have been) issues that > need to be addressed. No one is maintaining Numeric, so it makes much > more sense to put your effort into porting to numpy, rather than > trying > to fix or work around Numeric issues. In the long run, I agree. But on the time scale on which is what my work conditions force me to plan, it is less work for me to provide patches for Numeric as the need arises. > PS: this really highlights the strength of having good unit tests: as > far as i can tell, it's really not that much work to do the port -- > the > work is in the testing. Comprehensive units tests would make that part > trivial too. Yes, testing is the bigger chunk of the work. And yes, unit tests are nice to have - but they don't write themselves, unfortunately. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From konrad.hinsen at laposte.net Wed Sep 20 03:45:27 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Wed, 20 Sep 2006 09:45:27 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: <45103A31.60704@ieee.org> References: <45103A31.60704@ieee.org> Message-ID: On 19.09.2006, at 20:42, Travis Oliphant wrote: > Well, I got both ScientificPython and MMTK to compile and import > using > the steps outlined on http://www.scipy.org/Porting_to_NumPy in > about 1 > hour (including time to fix alter_code1 to make the process even > easier). Could you please send me those versions? I'd happily put them on my Web site for volunteers to test. > I was able to install ScientificPython and MMTK for NumPy on my system > using the patches provided on that page. Is there a test suite that > can be run? Not much yet, unfortunately. There is a minuscule test suite for ScientificPython in the latest release, and an even more miniscule one for MMTK that I didn't even publish yet because it doesn't test more than that everything can be imported. > Users of MMTK could really help out here. I hope so! Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From faltet at carabos.com Wed Sep 20 03:52:26 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 09:52:26 +0200 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping Message-ID: <200609200952.28046.faltet@carabos.com> Hi, I'm sending a message here because discussing about this in the bug tracker is not very comfortable. This my last try before giving up, so don't be afraid ;-) In bug #283 (http://projects.scipy.org/scipy/numpy/ticket/283) I complained about the fact that a numpy.int32 is being mapped in NumPy to NPY_LONG enumerated type and I think I failed to explain well why I think this is a bad thing. Now, I'll try to expose an (real life) example, in the hope that things will make clearer. Realize that you are coding a C extension that receives NumPy arrays for saving them on-disk for a later retrieval. Realize also that an user is using your extension on a 32-bit platform. If she pass to this extension an array of type 'int32', and the extension tries to read the enumerated type (using array.dtype.num), it will get NPY_LONG. So, the extension use this code (NPY_LONG) to save the type (together with data) on-disk. Now, she send this data file to a teammate that works on a 64-bit machine, and tries to read the data using the same extension. The extension would see that the data is NPY_LONG type and would try to deserialize interpreting data elements as being as 64-bit integer (this is the size of a NPY_LONG in 64-bit platforms), and this is clearly wrong. Besides this, if for making your C extension you are using a C library that is meant to save data in a platform-independent (say, HDF5), then, having a NPY_LONG will not automatically say which C library datatype maps to, because it only have datatypes that are of a definite size in all platforms. So, this is a second problem. Of course there are workarounds for this, but my impression is that they can be avoided with a more sensible mapping between NumPy Python types and NumPy enumerated types, like: numpy.int32 --> NPY_INT numpy.int64 --> NPY_LONGLONG numpy.int_ --> NPY_LONG in all platforms, avoiding the current situation of ambiguous mapping between platforms. Sorry for being so persistent, but I think the issue is worth it. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Wed Sep 20 04:01:18 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 20 Sep 2006 03:01:18 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510C93C.8030902@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > Robert Kern wrote: >> Sebastian Haase wrote: >>> I know that having too much knowledge of the details often makes one >>> forget what the "newcomers" will do and expect. >> Please be more careful with such accusations. Repeated frequently, they can >> become quite insulting. >> > I did not mean to insult anyone - what I meant was, that I'm for numpy > becoming an easy platform to use. I have spend and enjoyed part the last > four years developing and evangelizing Python as an alternative to > Matlab and C/Fortran based image analysis environment. I often find > myself arguing for good support of the single precision data format. So > I find it actually somewhat ironic to see myself arguing now for wanting > float64 over float32 ;-) No one is doubting that you want numpy to be easy to use. Please don't doubt that the rest of us want otherwise. However, the fact that you *want* numpy to be easy to use does not mean that your suggestions *will* make numpy easy to use. We haven't forgotten what newcomers will do; to the contrary, we are quite aware that new users need consistent behavior in order to learn how to use a system. Adding another special case in how dtypes implicitly convert to one another will impede new users being able to understand the whole system. See A. M. Archibald's question in the thread "ufunc.reduce and conversion" for an example. In our judgement this is a worse outcome than notational convenience for float32 users, who already need to be aware of the effects of their precision choice. Each of us can come to different conclusions in good faith without one of us forgetting the new user experience. Let me offer a third path: the algorithms used for .mean() and .var() are substandard. There are much better incremental algorithms that entirely avoid the need to accumulate such large (and therefore precision-losing) intermediate values. The algorithms look like the following for 1D arrays in Python: def mean(a): m = a[0] for i in range(1, len(a)): m += (a[i] - m) / (i + 1) return m def var(a): m = a[0] t = a.dtype.type(0) for i in range(1, len(a)): q = a[i] - m r = q / (i+1) m += r t += i * q * r t /= len(a) return t Alternatively, from Knuth: def var_knuth(a): m = a.dtype.type(0) variance = a.dtype.type(0) for i in range(len(a)): delta = a[i] - m m += delta / (i+1) variance += delta * (a[i] - m) variance /= len(a) return variance If you will code up implementations of these for ndarray.mean() and ndarray.var(), I will check them in and then float32 arrays will avoid most of the catastrophes that the current implementations run into. >>> We are only talking >>> about people that will a) work with single-precision data (e.g. large >>> scale-image analysis) and who b) will tend to "just use the default" >>> (dtype) --- How else can I say this: these people will just assume that >>> arr.mean() *is* the mean of arr. >> I don't understand what you mean, here. arr.mean() is almost never *the* mean of >> arr. Double precision can't fix that. >> > This was not supposed to be a scientific statement -- I'm (again) > thinking of our students that not always appreciate the full complexity > of computational numerics and data types and such. They need to appreciate the complexity of computational numerics if they are going to do numerical computation. Double precision does not make it any simpler. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Wed Sep 20 06:00:20 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 12:00:20 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191941.56395.faltet@carabos.com> Message-ID: <200609201200.21492.faltet@carabos.com> A Dimarts 19 Setembre 2006 21:41, Bill Baxter va escriure: > I think he meant do an argsort first, then use fancy indexing to get > the sorted array. > For a 1-d array that's just > > ind = A.argsort() > Asorted = A[ind] > > That should be O(N lg N + N), aka O(N lg N) I see. Thanks. OTOH, maybe your estimations are right, but the effect of the constants in these O(whatever) estimations can be surprisingly high: In [3]: from timeit import Timer In [4]: Timer("b=numpy.argsort(a);c=a[b]", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[4]: [1.6653108596801758, 1.670341968536377, 1.6632120609283447] In [5]: Timer("b=numpy.argsort(a);c=numpy.sort(a)", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[5]: [1.6533238887786865, 1.6272940635681152, 1.6253311634063721] In [6]: Timer("b=numpy.argsort(a);a.sort();c=a", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[6]: [0.95492100715637207, 0.90312504768371582, 0.90426898002624512] so, it seems that argsorting first and fancy indexing later on is the most expensive procedure for relatively large arrays (100000). Interestingly, figures above seems to indicate that in-place sort is stunningly fast: In [7]: Timer("a.sort()","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[7]: [0.32840394973754883, 0.2746579647064209, 0.2770991325378418] and much faster indeed than fancy indexing In [8]: Timer("b[a]","import numpy; a=numpy.arange(100000,-1,-1);b=a.copy()").repeat(3,100) Out[8]: [0.79876089096069336, 0.74172186851501465, 0.74209499359130859] i.e. in-place sort seems 2.7x faster than fancy indexing (at least for these datasets). Mmmm, with this, I really ponder if a combo that makes the argsort() and sort() in one shot really makes any sense, at least from the point of view of speed: In [10]: Timer("b=numpy.argsort(a);a.sort();c=a","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[10]: [0.98506593704223633, 0.89880609512329102, 0.89982390403747559] In [11]: Timer("b=numpy.argsort(a)","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[11]: [0.92959284782409668, 0.85385990142822266, 0.87773990631103516] So, it seems that doing an in-place sort() immediately after an argsort() operation is very efficient (cache effects here?), and would avoid the need of the combo function (from the point of view of efficiency, I repeat). Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Wed Sep 20 06:48:28 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 04:48:28 -0600 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping In-Reply-To: <200609200952.28046.faltet@carabos.com> References: <200609200952.28046.faltet@carabos.com> Message-ID: <45111C7C.2060700@ieee.org> Francesc Altet wrote: > Hi, > > I'm sending a message here because discussing about this in the bug tracker is > not very comfortable. This my last try before giving up, so don't be > afraid ;-) > > In bug #283 (http://projects.scipy.org/scipy/numpy/ticket/283) I complained > about the fact that a numpy.int32 is being mapped in NumPy to NPY_LONG > enumerated type and I think I failed to explain well why I think this is a > bad thing. Now, I'll try to expose an (real life) example, in the hope that > things will make clearer. > > Realize that you are coding a C extension that receives NumPy arrays for > saving them on-disk for a later retrieval. Realize also that an user is using > your extension on a 32-bit platform. If she pass to this extension an array > of type 'int32', and the extension tries to read the enumerated type (using > array.dtype.num), it will get NPY_LONG. > So, the extension use this code > (NPY_LONG) to save the type (together with data) on-disk. Now, she send this > data file to a teammate that works on a 64-bit machine, and tries to read the > data using the same extension. The extension would see that the data is > NPY_LONG type and would try to deserialize interpreting data elements as > being as 64-bit integer (this is the size of a NPY_LONG in 64-bit platforms), > and this is clearly wrong. > > In my view, this "real-life" example points to a flaw in the coding design that will not be fixed by altering what numpy.int32 maps to under the covers. It is wrong to use a code for the platform c data-type (NPY_LONG) as a key to understand data written to disk. This is and always has been a bad idea. No matter what we do with numpy.int32 this can cause problems. Just because a lot of platforms think an int is 32-bits does not mean all of them do. C gives you no such guarantee. Notice that pickling of NumPy arrays does not store the "enumerated type" as the code. Instead it stores the data-type object (which itself pickles using the kind and element size so that the correct data-type object can be reconstructed on the other end --- if it is available at all). Thus, you should not be storing the enumerated type but instead something like the kind and element-size. > Besides this, if for making your C extension you are using a C library that is > meant to save data in a platform-independent (say, HDF5), then, having a > NPY_LONG will not automatically say which C library datatype maps to, because > it only have datatypes that are of a definite size in all platforms. So, this > is a second problem. > > Making sure you get the correct data-type is why there are NPY_INT32 and NPY_INT64 enumerated types. You can't code using NPY_LONG and expect it will give you the same sizes when moving from 32-bit and 64-bit platforms. That's a problem that has been fixed with the bitwidth types. I don't understand why you are using the enumerated types at all in this circumstance. > Of course there are workarounds for this, but my impression is that they can > be avoided with a more sensible mapping between NumPy Python types and NumPy > enumerated types, like: > > numpy.int32 --> NPY_INT > numpy.int64 --> NPY_LONGLONG > numpy.int_ --> NPY_LONG > > in all platforms, avoiding the current situation of ambiguous mapping between > platforms. > The problem is that C gives us this ambiguous mapping. You are asking us to pretend it isn't there because it "simplifies" a hypothetical case so that poor coding practice can be allowed to work in a special case. I'm not convinced. This persists the myth that C data-types have a defined length. This is not guaranteed. The current system defines data-types with a guaranteed length. Yes, there is ambiguity as to which is "the" underlying c-type on certain platforms, but if you are running into trouble with the difference, then you need to change how you are coding because you would run into trouble on some combination of platforms even if we made the change. Basically, you are asking to make a major change, and at this point I'm very hesitant to make such a change without a clear and pressing need for it. Your hypothetical example does not rise to the level of "clear and pressing need." In fact, I see your proposal as a step backwards. Now, it is true that we could change the default type that gets first grab at int32 to be int (instead of the current long) --- I could see arguments for that. But, since the choice is ambiguous and the Python integer type is the c-type long, I let long get first dibs on everything as this seemed to work better for code I was wrapping in the past. I don't see any point in changing this choice now and risk code breakage, especially when your argument is that it would let users think that a c int is always 32-bits. Best regards, -Travis From oliphant.travis at ieee.org Wed Sep 20 06:59:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 04:59:42 -0600 Subject: [Numpy-discussion] ufunc.reduce and conversion In-Reply-To: References: Message-ID: <45111F1E.9080900@ieee.org> A. M. Archibald wrote: > Hi, > > What are the rules for datatype conversion in ufuncs? Does ufunc(a,b) > always yield the smallest type big enough to represent both a and b? > What is the datatype of ufunc.reduce(a)? > This is an unintended consequence of making add.reduce() reduce over at least a ("long"). I've fixed the code so that only add.reduce and multiply.reduce alter the default reducing data-type to be long. All other cases use the data-type of the array as the default. Regarding your other question on data-type conversion in ufuncs: 1) If you specify an output array, then the result will be cast to the output array data-type. 2) The actual computation takes place using a data-type that all (non-scalar) inputs can be cast to safely (with the exception that we assume that long long integers can be "safely" cast to "doubles" even though this is not technically true). -Travis From zdm105 at tom.com Sat Sep 23 07:22:22 2006 From: zdm105 at tom.com (tpm) Date: Sat, 23 Sep 2006 19:22:22 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From zdm105 at tom.com Sat Sep 23 07:31:53 2006 From: zdm105 at tom.com (tpm) Date: Sat, 23 Sep 2006 19:31:53 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From rlw at stsci.edu Wed Sep 20 08:09:57 2006 From: rlw at stsci.edu (Rick White) Date: Wed, 20 Sep 2006 08:09:57 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45109D40.7020008@ieee.org> References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: <07857523-4E43-4DDA-8A4C-1A5A96130CBD@stsci.edu> On Sep 19, 2006, at 9:45 PM, Tim Hochberg wrote: > Perhaps there's some use for the sort to end behaviour that I'm > missing, > but the raise an exception behaviour sure looks a lot more > appealing to me. FYI, in IDL the NaN values wind up at the end of the sorted array. That's true despite the fact that IDL does respect all the comparison properties of NaNs (i.e. Value>NaN, Value Here's a strawman proposal: > > Sort the array. Then examine numpy.geterr()['invalid']. If it > is not > 'ignore', then check examine sometrue(isnan(thearray)). If the > latter is true then raise and error, issue a warning or call the > error reporting functioni as appropriate. Note that we always sort > the array to be consistent with the behaviour of the ufuncs that > proceed even when they end up raising an exception. Here's another proposal: Make a first pass through the array, replacing NaN values with Inf and counting the NaNs ("nancount"). Raise an exception at this point if NaNs are not supposed to be allowed. Otherwise sort the array, and then as the last step replace the trailing NaNcount values with NaN. It seems to me that this would give predictable results while respecting the exception flags at little extra cost. Rick From gnata at obs.univ-lyon1.fr Wed Sep 20 09:15:40 2006 From: gnata at obs.univ-lyon1.fr (Xavier Gnata) Date: Wed, 20 Sep 2006 15:15:40 +0200 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: <45113EFC.106@obs.univ-lyon1.fr> IMHO, the only correct way to handle this case is to raise an exception. It does not make sense to compare NaN and "real" numbers. It could be very confusing not to raise an exception. Xavier. > On 19/09/06, Tim Hochberg wrote: > >> A. M. Archibald wrote: >> >>> Mmm. Somebody who's working with NaNs has more or less already decided >>> they don't want to be pestered with exceptions for invalid data. >>> >> Do you really think so? In my experience NaNs are nearly always just an >> indication of a mistake somewhere that didn't get trapped for one reason >> or another. >> > > Well, I said that because for an image porcessing project I was doing, > the easiest thing to do with certain troublesome pixels was to fill in > NaNs, and then at the end replace the NaNs with sensible values. It > seems as if the point of NaNs is to allow you to keep working with > those numbers that make sense while ingoring those that don't. If you > wanted exceptions, why not get them as soon as the first NaN would > have been generated? > > >>> I'd >>> be happy if they wound up at either end, but I'm not sure it's worth >>> hacking up the sort algorithm when a simple isnan() can pull them out. >>> >>> >> Moving them to the end seems to be the worst choice to me. Leaving them >> alone is fine with me. Or raising an exception would be fine. Or doing >> one or the other depending on the error mode settings would be even >> better if it is practical. >> > > I was just thinking in terms of easy removal. > > >> Is that true? Are all of numpy's sorting algorithms robust against >> nontransitive objects laying around? The answer to that appears to be >> no. Try running this a couple of times to see what I mean: >> > > >> The values don't correctly cross the inserted NaN and the sort is incorrect. >> > > You're quite right: when NaNs are present in the array, sorting and > then removing them does not yield a sorted array. For example, > mergesort just output > [ 2. 4. 6. 9. nan 0. 1. > 3. 5. 7. 8. ] > > The other two are no better (and arguably worse). Python's built-in > sort() for lists has the same problem. > > This is definitely a bug, and the best way to fix it is not clear to > me - perhaps sort() needs to always do any(isnan(A)) before starting > to sort. I don't really like raising an exception, but sort() isn't > really very meaningful with NaNs in the array. The only other option I > can think of is to somehow remove them, sort without, and reintroduce > them at the end, which is going to be a nightmare when sorting a > single axis of a large array. Or, I suppose, sort() could simply fill > the array with NaNs; I'm sure users will love that. > > A. M. Archibald > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- ############################################ Xavier Gnata CRAL - Observatoire de Lyon 9, avenue Charles Andr? 69561 Saint Genis Laval cedex Phone: +33 4 78 86 85 28 Fax: +33 4 78 86 83 86 E-mail: gnata at obs.univ-lyon1.fr ############################################ From faltet at carabos.com Wed Sep 20 10:34:35 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 16:34:35 +0200 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping In-Reply-To: <45111C7C.2060700@ieee.org> References: <200609200952.28046.faltet@carabos.com> <45111C7C.2060700@ieee.org> Message-ID: <200609201634.36232.faltet@carabos.com> A Dimecres 20 Setembre 2006 12:48, Travis Oliphant va escriure: > Making sure you get the correct data-type is why there are NPY_INT32 and > NPY_INT64 enumerated types. You can't code using NPY_LONG and expect > it will give you the same sizes when moving from 32-bit and 64-bit > platforms. That's a problem that has been fixed with the bitwidth > types. I don't understand why you are using the enumerated types at all > in this circumstance. Ooops. I didn't know that NPY_INT32 and NPY_INT64 were there. I think this solves all my problems. In fact, you were proposing this from the very beginning, but I was confused because I was hoping to find NPY_INT32 and NPY_INT64 in NPY_TYPES enumerated and I didn't find it there. I didn't realized that NPY_INT32 and NPY_INT64 were defined outside NPY_TYPES as platform independent constants. Blame on me. Sorry about any inconveniences and thanks once more for your patience! -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Wed Sep 20 10:55:08 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 20 Sep 2006 16:55:08 +0200 Subject: [Numpy-discussion] immutable arrays Message-ID: <200609201655.08552.martin.wiechert@gmx.de> Hi list, I just stumbled accross NPY_WRITEABLE flag. Now I'd like to know if there are ways either from Python or C to make an array temporarily immutable. Thanks, Martin. From haase at msg.ucsf.edu Wed Sep 20 11:47:03 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 20 Sep 2006 08:47:03 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <45116277.7010201@msg.ucsf.edu> Robert Kern wrote: >> This was not supposed to be a scientific statement -- I'm (again) >> thinking of our students that not always appreciate the full >> complexity >> of computational numerics and data types and such. > > They need to appreciate the complexity of computational numerics if > they are going to do numerical computation. Double precision does not > make it any simpler. This is were we differ. > We haven't forgotten what newcomers will do; to the contrary, we are > quite aware > that new users need consistent behavior in order to learn how to use a > system. > Adding another special case in how dtypes implicitly convert to one > another will > impede new users being able to understand the whole system. All I'm proposing could be summarized in: mean(), sum(), var() ... produce output of dtype float64 (except for input float96 which produces float96) A comment on this is also that for these operations the input type/precision is almost not related to the resulting output precision -- the int case makes that already clear. (This is different for e.g. min() or max() ) The proposed alternative implementations seem to have one or more multiplication (or division) for each value -- this might be noticeably slower ... Regards, Sebastian From tim.hochberg at ieee.org Wed Sep 20 12:08:15 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 09:08:15 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <4511676F.6000909@ieee.org> Robert Kern wrote: > Sebastian Haase wrote: > >> Robert Kern wrote: >> >>> Sebastian Haase wrote: >>> >>>> I know that having too much knowledge of the details often makes one >>>> forget what the "newcomers" will do and expect. >>>> >>> Please be more careful with such accusations. Repeated frequently, they can >>> become quite insulting. >>> >>> >> I did not mean to insult anyone - what I meant was, that I'm for numpy >> becoming an easy platform to use. I have spend and enjoyed part the last >> four years developing and evangelizing Python as an alternative to >> Matlab and C/Fortran based image analysis environment. I often find >> myself arguing for good support of the single precision data format. So >> I find it actually somewhat ironic to see myself arguing now for wanting >> float64 over float32 ;-) >> > > No one is doubting that you want numpy to be easy to use. Please don't doubt > that the rest of us want otherwise. However, the fact that you *want* numpy to > be easy to use does not mean that your suggestions *will* make numpy easy to use. > > We haven't forgotten what newcomers will do; to the contrary, we are quite aware > that new users need consistent behavior in order to learn how to use a system. > Adding another special case in how dtypes implicitly convert to one another will > impede new users being able to understand the whole system. See A. M. > Archibald's question in the thread "ufunc.reduce and conversion" for an example. > In our judgement this is a worse outcome than notational convenience for float32 > users, who already need to be aware of the effects of their precision choice. > Each of us can come to different conclusions in good faith without one of us > forgetting the new user experience. > > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. The algorithms look like the following for 1D arrays in Python: > > def mean(a): > m = a[0] > for i in range(1, len(a)): > m += (a[i] - m) / (i + 1) > return m > > def var(a): > m = a[0] > t = a.dtype.type(0) > for i in range(1, len(a)): > q = a[i] - m > r = q / (i+1) > m += r > t += i * q * r > t /= len(a) > return t > > Alternatively, from Knuth: > > def var_knuth(a): > m = a.dtype.type(0) > variance = a.dtype.type(0) > for i in range(len(a)): > delta = a[i] - m > m += delta / (i+1) > variance += delta * (a[i] - m) > variance /= len(a) > return variance > > If you will code up implementations of these for ndarray.mean() and > ndarray.var(), I will check them in and then float32 arrays will avoid most of > the catastrophes that the current implementations run into. > +1 > >>>> We are only talking >>>> about people that will a) work with single-precision data (e.g. large >>>> scale-image analysis) and who b) will tend to "just use the default" >>>> (dtype) --- How else can I say this: these people will just assume that >>>> arr.mean() *is* the mean of arr. >>>> >>> I don't understand what you mean, here. arr.mean() is almost never *the* mean of >>> arr. Double precision can't fix that. >>> >>> >> This was not supposed to be a scientific statement -- I'm (again) >> thinking of our students that not always appreciate the full complexity >> of computational numerics and data types and such. >> > > They need to appreciate the complexity of computational numerics if they are > going to do numerical computation. Double precision does not make it any simpler. > > From eric at enthought.com Wed Sep 20 12:29:28 2006 From: eric at enthought.com (eric) Date: Wed, 20 Sep 2006 11:29:28 -0500 Subject: [Numpy-discussion] ANN: Job postings -- Enthought, Inc. Message-ID: <45116C68.1070706@enthought.com> Hey group, We are growing again and have multiple positions open here at Enthought. I'd love to recruit more people out of the Scipy/numpy community. I think many of you would find the work interesting. You can look at our career page for more info: http://www.enthought.com/careers.htm thanks! eric From cookedm at physics.mcmaster.ca Wed Sep 20 12:34:17 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 20 Sep 2006 12:34:17 -0400 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. The algorithms look like the following for 1D arrays in Python: > > def mean(a): > m = a[0] > for i in range(1, len(a)): > m += (a[i] - m) / (i + 1) > return m This isn't really going to be any better than using a simple sum. It'll also be slower (a division per iteration). You do avoid accumulating large sums, but then doing the division a[i]/len(a) and adding that will do the same. Now, if you want to avoid losing precision, you want to use a better summation technique, like compensated (or Kahan) summation: def mean(a): s = e = a.dtype.type(0) for i in range(0, len(a)): temp = s y = a[i] + e s = temp + y e = (temp - s) + y return s / len(a) Some numerical experiments in Maple using 5-digit precision show that your mean is maybe a bit better in some cases, but can also be much worse, than sum(a)/len(a), but both are quite poor in comparision to the Kahan summation. (We could probably use a fast implementation of Kahan summation in addition to a.sum()) > def var(a): > m = a[0] > t = a.dtype.type(0) > for i in range(1, len(a)): > q = a[i] - m > r = q / (i+1) > m += r > t += i * q * r > t /= len(a) > return t > > Alternatively, from Knuth: > > def var_knuth(a): > m = a.dtype.type(0) > variance = a.dtype.type(0) > for i in range(len(a)): > delta = a[i] - m > m += delta / (i+1) > variance += delta * (a[i] - m) > variance /= len(a) > return variance These formulas are good when you can only do one pass over the data (like in a calculator where you don't store all the data points), but are slightly worse than doing two passes. Kahan summation would probably also be good here too. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Wed Sep 20 12:56:53 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 09:56:53 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510C93C.8030902@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <451172D5.5070101@noaa.gov> Sebastian Haase wrote: > The best I can hope for is a "sound" default for most (practical) cases... > I still think that 80bit vs. 128bit vs 96bit is rather academic for most > people ... most people seem to only use float64 and then there are some > that use float32 (like us) ... I fully agree with Sebastian here. As Travis pointed out, "all we are talking about is the default". The default should follow the principle of least surprise for the less-knowledgeable users. Personally, I try to always use doubles, unless I have a real reason not to. The recent change of default types for zeros et al. will help. clearly, there is a problem to say the default accumulator for *all* types is double, as you wouldn't want to downcast float128s, even if they are obscure. However, is it that hard to say that the default accumulator will have *at least* the precision of double? Robert Kern wrote: > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. This, of course, is an even better solution, unless there are substantial performance impacts. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Sep 20 12:59:00 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 20 Sep 2006 10:59:00 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: On 9/20/06, David M. Cooke wrote: > > On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > > Let me offer a third path: the algorithms used for .mean() and .var() > are > > substandard. There are much better incremental algorithms that entirely > avoid > > the need to accumulate such large (and therefore precision-losing) > intermediate > > values. The algorithms look like the following for 1D arrays in Python: > > > > def mean(a): > > m = a[0] > > for i in range(1, len(a)): > > m += (a[i] - m) / (i + 1) > > return m > > This isn't really going to be any better than using a simple sum. > It'll also be slower (a division per iteration). You do avoid > accumulating large sums, but then doing the division a[i]/len(a) and > adding that will do the same. > > Now, if you want to avoid losing precision, you want to use a better > summation technique, like compensated (or Kahan) summation: > > def mean(a): > s = e = a.dtype.type(0) > for i in range(0, len(a)): > temp = s > y = a[i] + e > s = temp + y > e = (temp - s) + y > return s / len(a) A variant of this can also be used to generate the sin/cos for fourier transforms using repeated complex multiplication. It works amazingly well. But I suspect accumulating in higher precision would be just as good and faster if the hardware supports it. Divide and conquer, like an fft where only the constant coefficient is computed, is also a good bet but requires lg(n) passes over the data and extra storage. Some numerical experiments in Maple using 5-digit precision show that > your mean is maybe a bit better in some cases, but can also be much > worse, than sum(a)/len(a), but both are quite poor in comparision to the > Kahan summation. > > (We could probably use a fast implementation of Kahan summation in > addition to a.sum()) > > > def var(a): > > m = a[0] > > t = a.dtype.type(0) > > for i in range(1, len(a)): > > q = a[i] - m > > r = q / (i+1) > > m += r > > t += i * q * r > > t /= len(a) > > return t > > > > Alternatively, from Knuth: > > > > def var_knuth(a): > > m = a.dtype.type(0) > > variance = a.dtype.type(0) > > for i in range(len(a)): > > delta = a[i] - m > > m += delta / (i+1) > > variance += delta * (a[i] - m) > > variance /= len(a) > > return variance > > These formulas are good when you can only do one pass over the data > (like in a calculator where you don't store all the data points), but > are slightly worse than doing two passes. Kahan summation would probably > also be good here too. I think we should leave things as they are. Choosing the right precision for accumulating sums is absolutely fundamental, I always think about it when programming such loops because it is always a potential gotcha. Making the defaults higher precision just kicks the can down the road where the unwary are still likely to trip over it at some point. Perhaps these special tricks for special cases could be included in scipy somewhere. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 20 13:22:16 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 20 Sep 2006 12:22:16 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: David M. Cooke wrote: > On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> Let me offer a third path: the algorithms used for .mean() and .var() are >> substandard. There are much better incremental algorithms that entirely avoid >> the need to accumulate such large (and therefore precision-losing) intermediate >> values. The algorithms look like the following for 1D arrays in Python: >> >> def mean(a): >> m = a[0] >> for i in range(1, len(a)): >> m += (a[i] - m) / (i + 1) >> return m > > This isn't really going to be any better than using a simple sum. > It'll also be slower (a division per iteration). With one exception, every test that I've thrown at it shows that it's better for float32. That exception is uniformly spaced arrays, like linspace(). > You do avoid > accumulating large sums, but then doing the division a[i]/len(a) and > adding that will do the same. Okay, this is true. > Now, if you want to avoid losing precision, you want to use a better > summation technique, like compensated (or Kahan) summation: > > def mean(a): > s = e = a.dtype.type(0) > for i in range(0, len(a)): > temp = s > y = a[i] + e > s = temp + y > e = (temp - s) + y > return s / len(a) > > Some numerical experiments in Maple using 5-digit precision show that > your mean is maybe a bit better in some cases, but can also be much > worse, than sum(a)/len(a), but both are quite poor in comparision to the > Kahan summation. > > (We could probably use a fast implementation of Kahan summation in > addition to a.sum()) +1 >> def var(a): >> m = a[0] >> t = a.dtype.type(0) >> for i in range(1, len(a)): >> q = a[i] - m >> r = q / (i+1) >> m += r >> t += i * q * r >> t /= len(a) >> return t >> >> Alternatively, from Knuth: >> >> def var_knuth(a): >> m = a.dtype.type(0) >> variance = a.dtype.type(0) >> for i in range(len(a)): >> delta = a[i] - m >> m += delta / (i+1) >> variance += delta * (a[i] - m) >> variance /= len(a) >> return variance > > These formulas are good when you can only do one pass over the data > (like in a calculator where you don't store all the data points), but > are slightly worse than doing two passes. Kahan summation would probably > also be good here too. Again, my tests show otherwise for float32. I'll condense my ipython log into a module for everyone's perusal. It's possible that the Kahan summation of the squared residuals will work better than the current two-pass algorithm and the implementations I give above. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Sep 20 14:18:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 12:18:19 -0600 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <200609201655.08552.martin.wiechert@gmx.de> References: <200609201655.08552.martin.wiechert@gmx.de> Message-ID: <451185EB.9080507@ieee.org> Martin Wiechert wrote: > Hi list, > > I just stumbled accross NPY_WRITEABLE flag. > Now I'd like to know if there are ways either from Python or C to make an > array temporarily immutable. > Just setting the flag Python: make immutable: a.flags.writeable = False make mutable again: a.flags.writeable = True C: make immutable: a->flags &= ~NPY_WRITEABLE make mutable again: a->flags |= NPY_WRITEABLE In C you can play with immutability all you want. In Python you can only make something writeable if you either 1) own the data or 2) the object that owns the data is itself "writeable" -Travis From tim.hochberg at ieee.org Wed Sep 20 14:53:41 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 11:53:41 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <45118E35.90207@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > Here's version of mean based on the Kahan sum, including the compensation term that seems to be consistently much better than builtin mean It seems to be typically 4 orders or magnitude closer to the "correct" answer than the builtin mean and 2 orders of magnitude better than just naively using the Kahan sum. I'm using the sum performed with dtype=int64 as the "correct" value. def rawKahanSum(values): # Stolen from mathworld if not input: return 0.0 total = values[0] c = type(total)(0.0) # A running compensation for lost low-order bits. for x in values[1:]: y = x - c # So far, so good: c is zero. t = total + y # Alas, total is big, y small, so low-order digits of y are lost. c = (t - total) - y # (t - total) recovers the high-order part of y; subtracting y recovers -(low part of y) total = t # Algebriacally, c should always be zero. Beware eagerly optimising compilers! # Next time around, the lost low part will be added to y in a fresh attempt. return total, c def kahanSum(values): total, c = rawKahanSum(values) return total def mean(values): total, c = rawKahanSum(values) n = float(len(values)) return total / n - c / n for i in range(100): data = random.random([10000]).astype(float32) baseline = data.mean(dtype=float64) delta_builtin_mean = baseline - data.mean() delta_compensated_mean = baseline - mean(data) delta_kahan_mean = baseline - (kahanSum(data) / len(data)) if not abs(delta_builtin_mean) >= abs(delta_kahan_mean) >= abs(delta_compensated_mean): print ">>>", print delta_builtin_mean, delta_kahan_mean, delta_compensated_mean -tim > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > > From tim.hochberg at ieee.org Wed Sep 20 15:02:38 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 12:02:38 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45118E35.90207@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <45118E35.90207@ieee.org> Message-ID: <4511904E.2090002@ieee.org> [Sorry, this version should have less munged formatting since I clipped the comments. Oh, and the Kahan sum algorithm was grabbed from wikipedia, not mathworld] Tim Hochberg wrote: > Robert Kern wrote: > >> David M. Cooke wrote: >> >> >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>> >>> >>>> Let me offer a third path: the algorithms used for .mean() and .var() are >>>> substandard. There are much better incremental algorithms that entirely avoid >>>> the need to accumulate such large (and therefore precision-losing) intermediate >>>> values. The algorithms look like the following for 1D arrays in Python: >>>> >>>> def mean(a): >>>> m = a[0] >>>> for i in range(1, len(a)): >>>> m += (a[i] - m) / (i + 1) >>>> return m >>>> >>>> >>> This isn't really going to be any better than using a simple sum. >>> It'll also be slower (a division per iteration). >>> >>> >> With one exception, every test that I've thrown at it shows that it's better for >> float32. That exception is uniformly spaced arrays, like linspace(). >> >> > Here's version of mean based on the Kahan sum, including the > compensation term that seems to be consistently much better than builtin > mean It seems to be typically 4 orders or magnitude closer to the > "correct" answer than the builtin mean and 2 orders of magnitude better > than just naively using the Kahan sum. I'm using the sum performed with > dtype=int64 as the "correct" value. > > > def rawKahanSum(values): if not input: return 0.0 total = values[0] c = type(total)(0.0) for x in values[1:]: y = x - c t = total + y c = (t - total) - y total = t return total, c def kahanSum(values): total, c = rawKahanSum(values) return total def mean(values): total, c = rawKahanSum(values) n = float(len(values)) return total / n - c / n for i in range(100): data = random.random([10000]).astype(float32) baseline = data.mean(dtype=float64) delta_builtin_mean = baseline - data.mean() delta_compensated_mean = baseline - mean(data) delta_kahan_mean = baseline - (kahanSum(data) / len(data)) if not abs(delta_builtin_mean) >= abs(delta_kahan_mean) >= abs(delta_compensated_mean): print ">>>", print delta_builtin_mean, delta_kahan_mean, delta_compensated_mean > -tim > >> > You do avoid >> > accumulating large sums, but then doing the division a[i]/len(a) and >> > adding that will do the same. >> >> Okay, this is true. >> >> >> >>> Now, if you want to avoid losing precision, you want to use a better >>> summation technique, like compensated (or Kahan) summation: >>> >>> def mean(a): >>> s = e = a.dtype.type(0) >>> for i in range(0, len(a)): >>> temp = s >>> y = a[i] + e >>> s = temp + y >>> e = (temp - s) + y >>> return s / len(a) >>> >>> Some numerical experiments in Maple using 5-digit precision show that >>> your mean is maybe a bit better in some cases, but can also be much >>> worse, than sum(a)/len(a), but both are quite poor in comparision to the >>> Kahan summation. >>> >>> (We could probably use a fast implementation of Kahan summation in >>> addition to a.sum()) >>> >>> >> +1 >> >> >> >>>> def var(a): >>>> m = a[0] >>>> t = a.dtype.type(0) >>>> for i in range(1, len(a)): >>>> q = a[i] - m >>>> r = q / (i+1) >>>> m += r >>>> t += i * q * r >>>> t /= len(a) >>>> return t >>>> >>>> Alternatively, from Knuth: >>>> >>>> def var_knuth(a): >>>> m = a.dtype.type(0) >>>> variance = a.dtype.type(0) >>>> for i in range(len(a)): >>>> delta = a[i] - m >>>> m += delta / (i+1) >>>> variance += delta * (a[i] - m) >>>> variance /= len(a) >>>> return variance >>>> >>>> >>> These formulas are good when you can only do one pass over the data >>> (like in a calculator where you don't store all the data points), but >>> are slightly worse than doing two passes. Kahan summation would probably >>> also be good here too. >>> >>> >> Again, my tests show otherwise for float32. I'll condense my ipython log into a >> module for everyone's perusal. It's possible that the Kahan summation of the >> squared residuals will work better than the current two-pass algorithm and the >> implementations I give above. >> >> >> > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Wed Sep 20 15:12:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 13:12:06 -0600 Subject: [Numpy-discussion] I've just tagged the tree for 1.0rc1 Message-ID: <45119286.3050704@ieee.org> There is now a 1.0rc1 tag on the NumPy SVN tree. I've confirmed it builds and passes all tests on my Linux box for Python2.3-Python2.5 -Travis From tim.hochberg at ieee.org Wed Sep 20 15:41:22 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 12:41:22 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <45119962.4020502@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> On the flip side, it doesn't take a very large array (somewhere in the vicinity of 10,000 values in my experience) before memory bandwidth becomes a limiting factor. In that region a two pass algorithm could well be twice as slow as a single pass algorithm even if the former is somewhat simpler in terms of numeric operations. -tim > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > From myeates at jpl.nasa.gov Wed Sep 20 16:41:55 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 20 Sep 2006 13:41:55 -0700 Subject: [Numpy-discussion] I've just tagged the tree for 1.0rc1 In-Reply-To: <45119286.3050704@ieee.org> References: <45119286.3050704@ieee.org> Message-ID: <4511A793.2080102@jpl.nasa.gov> I don't see a windows binary. Will this be added? Mathew Travis Oliphant wrote: > There is now a 1.0rc1 tag on the NumPy SVN tree. I've confirmed it > builds and passes all tests on my Linux box for Python2.3-Python2.5 > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From alerandazzo at 1-computerdesks.com Wed Sep 20 18:34:20 2006 From: alerandazzo at 1-computerdesks.com (Katherine Goldberg) Date: Wed, 20 Sep 2006 21:34:20 -0060 Subject: [Numpy-discussion] If you don't want to receive $5000 per month then don't open that letter Message-ID: Please read this letter attentively! Financial company is in search of representatives in the US. - Age: from 18 till 60 years - Have a skill to communicate and access to the Internet. - You can speak english language above average - You need to know how to use excel and word. You will earn from 1500 up to 3000 USD per week, working only some hours per day. You can work part time or full time. If you are interested in our offer send the following data to our e-mail usa at euroimperial2.com - Your full name - Your contact e-mail - Your phone number If you have any questions. Feel free to ask them From kwgoodman at gmail.com Wed Sep 20 17:46:54 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 20 Sep 2006 14:46:54 -0700 Subject: [Numpy-discussion] If you don't want to receive $5000 per month then don't open that letter In-Reply-To: References: Message-ID: I like the salutation of the letter below: "Please read this letter attentively!" Why does so much spam make it through the sourceforge filters? Or maybe they only let through the very good deals. "You will earn from 1500 up to 3000 USD per week, working only some hours per day." On Wed, 20 Sep 2006 21:34:20 -0060, Katherine Goldberg wrote: > Please read this letter attentively! > > Financial company is in search of representatives in the US. > > - Age: from 18 till 60 years > - Have a skill to communicate and access to the Internet. > - You can speak english language above average > - You need to know how to use excel and word. > > You will earn from 1500 up to 3000 USD per week, working only some hours per day. You can work part time or full time. > > If you are interested in our offer send the following data to our e-mail usa at euroimperial2.com > > - Your full name > - Your contact e-mail > - Your phone number > > If you have any questions. Feel free to ask them > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Wed Sep 20 18:17:58 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 15:17:58 -0700 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: <451079B2.60805@ee.byu.edu> References: <451079B2.60805@ee.byu.edu> Message-ID: <4511BE16.7040500@noaa.gov> Travis Oliphant wrote: >> In [77]: arange(5, dtype=int)/0 >> Out[77]: array([0, 0, 0, 0, 0]) > It is deliberate. Numarray introduced it (the only difference being > that by default NumPy has division-by-zero erros turned off). It's tied > to the way floating-point division-by zero is handled. There is a > valid argument for having a separate integer-division flag so that you > can raise exceptions for integer-division but not for floating-point > division. I'm open to that change for 1.0rc1 +1 (+inf) There is a BIG difference between getting an inf with a floating point computation and getting a 0 with an integer one! Also, the default integer-division flag should be set to raise an exception. It sure would be nice to have special values for integers too.... Travis Oliphant wrote: > Simulating and "integer-division-by-zero" > hardware flag is not trivial as we would have to manage context > switching ourselves. So, at least for 1.0, integer and floating-point > division by zero are going to be handled the same. Darn. Oh well. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Sep 20 18:18:27 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 15:18:27 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <4511BE33.4080708@noaa.gov> Charles R Harris wrote: > Thinking a bit, keeping the values in place isn't easy. Why the heck would "in place" be desirable for sorted data anyway? I understand that it means that if there is a NaN in the nth position before sorting, there will be one in the nth position after sorting. However, I see absolutely no reason at all why that would be useful (or any more useful than putting them anywhere else) A couple years ago, there was a long debate on this list about whether numpy should pass -inf, NaN, and +inf through all the ufuncs without error. there were two schools of thought: 1) They indicate a problem, the programmer should know about hat problem as soon as it occurs, not at the end of the computation, many steps later, when they might get presented with nothing but NaNs. 2) The whole point of "vector" computation is that you can act on a whole bunch of numbers at once. If only subset of those numbers are invalid, why stop the process. Raising an error when a single number has a problem defeats the purpose of vector operations. It seems that numpy has settled on school of thought (2), at least by default. That being the case, it should apply to sorting also. If it does, then that means no exception will be raised, but it makes no difference where the heck the NaNs end up in the sorted array, as long as everything else is in order. NaN means exactly what it's called: it's not a number, so it doesn't matter what you do with them, as long as they are preserved and don't mess up other things. Let the coder decide what they want to so with them, and when they want to do it. Personally, I'd prefer that they all ended up at the beginning or end after sorting, but it really doesn't much matter. That being said, if it's impossible to do a efficient sort with NaNs mixed in, then we'll just have to live with it. It really would be best if an exception was raised if the non-NaN values are not going to be sorted correctly -- that really would surprise people! > It would probably also not be unreasonable to punt and document sort > as failing in the presence of nans. That would be one of the worst options, but may be the only one available. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From davidgrant at gmail.com Wed Sep 20 19:41:56 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 20 Sep 2006 16:41:56 -0700 Subject: [Numpy-discussion] change of default dtype Message-ID: I noticed that the default dtype for the "ones" and "zeros" functions is now float, whereas it used to be int. Should this be noted at http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does break code (if you are using the array you creating with ones or zeros as indices into another array). -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Sep 20 20:03:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 21 Sep 2006 09:03:58 +0900 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: Message-ID: I think that's supposed to be covered by this line: "The default array data-type is now float64 (c double precision) instead of c integer." But yeh, I agree. It's definitely not obvious what that means in terms of concrete API changes. --bb On 9/21/06, David Grant wrote: > I noticed that the default dtype for the "ones" and "zeros" functions is now > float, whereas it used to be int. Should this be noted at > http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does > break code (if you are using the array you creating with ones or zeros as > indices into another array). > > -- > David Grant > http://www.davidgrant.ca > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From strawman at astraw.com Wed Sep 20 20:39:43 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed, 20 Sep 2006 17:39:43 -0700 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: Message-ID: <4511DF4F.9070206@astraw.com> It is a wiki, and contributions are absolutely welcome, so please go ahead and change it to be more clear. Bill Baxter wrote: >I think that's supposed to be covered by this line: >"The default array data-type is now float64 (c double precision) >instead of c integer." > >But yeh, I agree. It's definitely not obvious what that means in >terms of concrete API changes. > >--bb > >On 9/21/06, David Grant wrote: > > >>I noticed that the default dtype for the "ones" and "zeros" functions is now >>float, whereas it used to be int. Should this be noted at >>http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does >>break code (if you are using the array you creating with ones or zeros as >>indices into another array). >> >>-- >>David Grant >>http://www.davidgrant.ca >>------------------------------------------------------------------------- >>Take Surveys. Earn Cash. Influence the Future of IT >>Join SourceForge.net's Techsay panel and you'll get the chance to share your >>opinions on IT & business topics through brief surveys -- and earn cash >>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> >> > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share your >opinions on IT & business topics through brief surveys -- and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From kwgoodman at gmail.com Wed Sep 20 23:47:02 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 20 Sep 2006 20:47:02 -0700 Subject: [Numpy-discussion] nan adds lots of whitespace Message-ID: NaN adds a lot of whitespace in the representation of a matrix. Without NaN: matrix([[1, 2], [3, 4]]) With NaN: matrix([[ nan, 2. ], [ 3. , 4. ]]) There's enough room for the wikipedia entry for nan. From wbaxter at gmail.com Thu Sep 21 00:00:35 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 21 Sep 2006 13:00:35 +0900 Subject: [Numpy-discussion] change of default dtype In-Reply-To: <4511DF4F.9070206@astraw.com> References: <4511DF4F.9070206@astraw.com> Message-ID: Hey Andrew, point taken, but I think it would be better if someone who actually knows the full extent of the change made the edit. I know zeros and ones changed. Did anything else? Anyway, I'm surprised the release notes page is publicly editable. --bb On 9/21/06, Andrew Straw wrote: > It is a wiki, and contributions are absolutely welcome, so please go > ahead and change it to be more clear. > > Bill Baxter wrote: > > >I think that's supposed to be covered by this line: > >"The default array data-type is now float64 (c double precision) > >instead of c integer." > > > >But yeh, I agree. It's definitely not obvious what that means in > >terms of concrete API changes. > > > >--bb > > > >On 9/21/06, David Grant wrote: > > > > > >>I noticed that the default dtype for the "ones" and "zeros" functions is now > >>float, whereas it used to be int. Should this be noted at > >>http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does > >>break code (if you are using the array you creating with ones or zeros as > >>indices into another array). > >> > >>-- > >>David Grant > >>http://www.davidgrant.ca > >> > >> From gh743 at tom.com Tue Sep 26 03:39:39 2006 From: gh743 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Tue, 26 Sep 2006 15:39:39 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Thu Sep 21 04:04:57 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Thu, 21 Sep 2006 10:04:57 +0200 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <451185EB.9080507@ieee.org> References: <200609201655.08552.martin.wiechert@gmx.de> <451185EB.9080507@ieee.org> Message-ID: <200609211004.57370.martin.wiechert@gmx.de> Thanks Travis. Do I understand correctly that the only way to be really safe is to make a copy and not to export a reference to it? Because anybody having a reference to the owner of the data can override the flag? Cheers, Martin On Wednesday 20 September 2006 20:18, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > I just stumbled accross NPY_WRITEABLE flag. > > Now I'd like to know if there are ways either from Python or C to make an > > array temporarily immutable. > > Just setting the flag > > Python: > > make immutable: > a.flags.writeable = False > > make mutable again: > a.flags.writeable = True > > > C: > > make immutable: > a->flags &= ~NPY_WRITEABLE > > make mutable again: > a->flags |= NPY_WRITEABLE > > > In C you can play with immutability all you want. In Python you can > only make something writeable if you either 1) own the data or 2) the > object that owns the data is itself "writeable" > > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Peter.Bienstman at ugent.be Thu Sep 21 07:28:02 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Thu, 21 Sep 2006 13:28:02 +0200 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 Message-ID: <200609211328.02710.Peter.Bienstman@ugent.be> Hi, I just installed rc1 on an AMD64 machine. but I get this error message when trying to import it: Python 2.4.3 (#1, Sep 21 2006, 13:06:42) [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/numpy/__init__.py", line 36, in ? import core File "/usr/lib64/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 191, in ? _add_aliases() File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 169, in _add_aliases base, bit, char = bitname(typeobj) File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 119, in bitname char = base[0] IndexError: string index out of range Thanks! Peter From lroubeyrie at limair.asso.fr Thu Sep 21 10:16:58 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Thu, 21 Sep 2006 16:16:58 +0200 Subject: [Numpy-discussion] Question about recarray Message-ID: <200609211616.59129.lroubeyrie@limair.asso.fr> Hi all, Is it possible to put masked values into recarrays, I need a array with heterogenous types of datas (datetime objects in the first col, all others are float) but with missing values in some records. For the moment, I don't find any solution for that. I have tried with arrays of dtype=object, but I have problem when I want to compute min, max, ... with an error like: TypeError: function not supported for these types, and can't coerce safely to supported types. thanks -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From charlesr.harris at gmail.com Thu Sep 21 10:33:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 08:33:34 -0600 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 In-Reply-To: <200609211328.02710.Peter.Bienstman@ugent.be> References: <200609211328.02710.Peter.Bienstman@ugent.be> Message-ID: On 9/21/06, Peter Bienstman wrote: > > Hi, > > I just installed rc1 on an AMD64 machine. but I get this error message > when > trying to import it: > > Python 2.4.3 (#1, Sep 21 2006, 13:06:42) > [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > Traceback (most recent call last): I don't see this running the latest from svn on AMD64 here. Not sayin' there might not be a problem with rc1, I just don't see it with my sources. Python 2.4.3 (#1, Jun 13 2006, 11:46:22) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.0.dev3202' >>> numpy.version.os.uname() ('Linux', 'tethys', '2.6.17-1.2187_FC5', '#1 SMP Mon Sep 11 01:16:59 EDT 2006', 'x86_64') If you are building on Gentoo maybe you could delete the build directory (and maybe the numpy site package) and rebuild. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 21 12:05:58 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 10:05:58 -0600 Subject: [Numpy-discussion] Tests and code documentation Message-ID: Travis, A few questions. 1) I can't find any systematic code testing units, although there seem to be tests for regressions and such. Is there a place we should be putting such tests? 2) Any plans for code documentation? I documented some of my stuff with doxygen markups and wonder if we should include a Doxyfile as part of the package. 3) Would you consider breaking out the Converters into a separate .c file for inclusion? The code generator seems to take care of the ordering. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Sep 21 12:18:05 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 21 Sep 2006 09:18:05 -0700 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <200609210918.05413.haase@msg.ucsf.edu> On Thursday 21 September 2006 09:05, Charles R Harris wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem to > be tests for regressions and such. Is there a place we should be putting > such tests? > > 2) Any plans for code documentation? I documented some of my stuff with > doxygen markups and wonder if we should include a Doxyfile as part of the > package. Are able to use doxygen for Python code ? I thought it only worked for C (and alike) ? > > 3) Would you consider breaking out the Converters into a separate .c file > for inclusion? The code generator seems to take care of the ordering. > > Chuck From oliphant.travis at ieee.org Thu Sep 21 12:24:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:24:36 -0600 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <200609211004.57370.martin.wiechert@gmx.de> References: <200609201655.08552.martin.wiechert@gmx.de> <451185EB.9080507@ieee.org> <200609211004.57370.martin.wiechert@gmx.de> Message-ID: <4512BCC4.3000708@ieee.org> Martin Wiechert wrote: > Thanks Travis. > > Do I understand correctly that the only way to be really safe is to make a > copy and not to export a reference to it? > Because anybody having a reference to the owner of the data can override the > flag? > No, that's not quite correct. Of course in C, anybody can do anything they want to the flags. In Python, only the owner of the object itself can change the writeable flag once it is set to False. So, if you only return a "view" of the array (a.view()) then the Python user will not be able to change the flags. Example: a = array([1,2,3]) a.flags.writeable = False b = a.view() b.flags.writeable = True # raises an error. c = a c.flags.writeable = True # can be done because c is a direct alias to a. Hopefully, that explains the situation a bit better. -Travis From oliphant.travis at ieee.org Thu Sep 21 12:30:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:30:40 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <4512BE30.7070405@ieee.org> Charles R Harris wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem > to be tests for regressions and such. Is there a place we should be > putting such tests? All tests are placed under the tests directory of the corresponding sub-package. They will only be picked up by .test(level < 10) if the file is named test_. .test(level>10) should pick up all test files. If you want to name something different but still have it run at a test level < 10, then you need to run the test from one of the other test files that will be picked up (test_regression.py and test_unicode.py are doing that for example). > > 2) Any plans for code documentation? I documented some of my stuff > with doxygen markups and wonder if we should include a Doxyfile as > part of the package. I'm not familiar with Doxygen, but would welcome any improvements to the code documentation. > > 3) Would you consider breaking out the Converters into a separate .c > file for inclusion? The code generator seems to take care of the ordering. You are right that it doesn't matter which order the API subroutines are placed. I'm not opposed to more breaking up of the .c files, as long as it is clear where things will be located. The #include strategy is necessary to get it all in one Python module, but having smaller .c files usually makes for faster editing. It's the arrayobject.c file that is "too-large" IMHO, however. That's where I would look for ways to break it up. The iterobject and the data-type object could be taken out, for example. -Travis From lcordier at point45.com Thu Sep 21 12:32:40 2006 From: lcordier at point45.com (Louis Cordier) Date: Thu, 21 Sep 2006 18:32:40 +0200 (SAST) Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <200609210918.05413.haase@msg.ucsf.edu> References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: > Are able to use doxygen for Python code ? I thought it only worked for C (and > alike) ? There is an ugly-hack :) http://i31www.ira.uka.de/~baas/pydoxy/ But I wouldn't recommend using it, rather stick with Epydoc. -- Louis Cordier cell: +27721472305 Point45 Entertainment (Pty) Ltd. http://www.point45.org From charlesr.harris at gmail.com Thu Sep 21 12:42:59 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 10:42:59 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <200609210918.05413.haase@msg.ucsf.edu> References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: On 9/21/06, Sebastian Haase wrote: > > On Thursday 21 September 2006 09:05, Charles R Harris wrote: > > Travis, > > > > A few questions. > > > > 1) I can't find any systematic code testing units, although there seem > to > > be tests for regressions and such. Is there a place we should be putting > > such tests? > > > > 2) Any plans for code documentation? I documented some of my stuff with > > doxygen markups and wonder if we should include a Doxyfile as part of > the > > package. > > Are able to use doxygen for Python code ? I thought it only worked for C > (and > alike) ? IIRC correctly, it now does Python too. Let's see... here is an example ## Documentation for this module. # # More details. ## Documentation for a function. # # More details. def func(): pass Looks like ## replaces the /** Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Sep 21 12:43:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:43:44 -0600 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609211616.59129.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> Message-ID: <4512C140.8070805@ieee.org> Lionel Roubeyrie wrote: > Hi all, > Is it possible to put masked values into recarrays, I need a array with > heterogenous types of datas (datetime objects in the first col, all others > are float) but with missing values in some records. For the moment, I don't > find any solution for that. Either use "nans" or "inf" for missing values or use the masked array object with a complex data-type. You don't need to use a recarray object to get "records". Any array can have "records". Therefore, you can have a masked array of "records" by creating an array with the appropriate data-type. It may also be possible to use a recarray as the "array" for the masked array object becuase the recarray is a sub-class of the array. > I have tried with arrays of dtype=object, but I > have problem when I want to compute min, max, ... with an error like: > TypeError: function not supported for these types, and can't coerce safely to > supported types. > It looks like the max and min functions are not supported for Object arrays. import numpy as N N.maximum.types does not include Object arrays. It probably should. -Travis From matthew.brett at gmail.com Thu Sep 21 12:52:49 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 21 Sep 2006 17:52:49 +0100 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <451072E1.7030608@ee.byu.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> <451072E1.7030608@ee.byu.edu> Message-ID: <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> Hi, > It's in the array interface specification: > > http://numpy.scipy.org/array_interface.shtml I was interested in the 't' (bitfield) type - is there an example of usage somewhere? In [13]: dtype('t8') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/mb312/python/ TypeError: data type not understood Best, Matthew From oliphant.travis at ieee.org Thu Sep 21 12:59:32 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:59:32 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> <451072E1.7030608@ee.byu.edu> <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> Message-ID: <4512C4F4.80605@ieee.org> Matthew Brett wrote: > Hi, > > >> It's in the array interface specification: >> >> http://numpy.scipy.org/array_interface.shtml >> > > I was interested in the 't' (bitfield) type - is there an example of > usage somewhere? > No, It's not implemented in NumPy. It's just part of the array interface specification for completeness. -Travis From oliphant.travis at ieee.org Thu Sep 21 13:01:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 11:01:07 -0600 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609211616.59129.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> Message-ID: <4512C553.6090609@ieee.org> Lionel Roubeyrie wrote: > find any solution for that. I have tried with arrays of dtype=object, but I > have problem when I want to compute min, max, ... with an error like: > TypeError: function not supported for these types, and can't coerce safely to > supported types. > I just added support for min and max methods of object arrays, by adding support for Object arrays to the minimum and maximum functions. -Travis From tim.hochberg at ieee.org Thu Sep 21 13:56:38 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 10:56:38 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <4512D256.2080604@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > This is what my tests show as well var_knuth outperformed any simple two pass algorithm I could come up with, even ones using Kahan sums. Interestingly, for 1D arrays the built in float32 variance performs better than it should. After a bit of twiddling around I discovered that it actually does most of it's calculations in float64. It uses a two pass calculation, the result of mean is a scalar, and in the process of converting that back to an array we end up with float64 values. Or something like that; I was mostly reverse engineering the sequence of events from the results. -tim From tim.hochberg at ieee.org Thu Sep 21 14:34:42 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 11:34:42 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4512D256.2080604@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> Message-ID: <4512DB42.7080509@ieee.org> Tim Hochberg wrote: > Robert Kern wrote: > >> David M. Cooke wrote: >> >> >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>> >>> >>>> Let me offer a third path: the algorithms used for .mean() and .var() are >>>> substandard. There are much better incremental algorithms that entirely avoid >>>> the need to accumulate such large (and therefore precision-losing) intermediate >>>> values. The algorithms look like the following for 1D arrays in Python: >>>> >>>> def mean(a): >>>> m = a[0] >>>> for i in range(1, len(a)): >>>> m += (a[i] - m) / (i + 1) >>>> return m >>>> >>>> >>> This isn't really going to be any better than using a simple sum. >>> It'll also be slower (a division per iteration). >>> >>> >> With one exception, every test that I've thrown at it shows that it's better for >> float32. That exception is uniformly spaced arrays, like linspace(). >> >> > You do avoid >> > accumulating large sums, but then doing the division a[i]/len(a) and >> > adding that will do the same. >> >> Okay, this is true. >> >> >> >>> Now, if you want to avoid losing precision, you want to use a better >>> summation technique, like compensated (or Kahan) summation: >>> >>> def mean(a): >>> s = e = a.dtype.type(0) >>> for i in range(0, len(a)): >>> temp = s >>> y = a[i] + e >>> s = temp + y >>> e = (temp - s) + y >>> return s / len(a) >>> >>> Some numerical experiments in Maple using 5-digit precision show that >>> your mean is maybe a bit better in some cases, but can also be much >>> worse, than sum(a)/len(a), but both are quite poor in comparision to the >>> Kahan summation. >>> >>> (We could probably use a fast implementation of Kahan summation in >>> addition to a.sum()) >>> >>> >> +1 >> >> >> >>>> def var(a): >>>> m = a[0] >>>> t = a.dtype.type(0) >>>> for i in range(1, len(a)): >>>> q = a[i] - m >>>> r = q / (i+1) >>>> m += r >>>> t += i * q * r >>>> t /= len(a) >>>> return t >>>> >>>> Alternatively, from Knuth: >>>> >>>> def var_knuth(a): >>>> m = a.dtype.type(0) >>>> variance = a.dtype.type(0) >>>> for i in range(len(a)): >>>> delta = a[i] - m >>>> m += delta / (i+1) >>>> variance += delta * (a[i] - m) >>>> variance /= len(a) >>>> return variance >>>> >>>> >>> These formulas are good when you can only do one pass over the data >>> (like in a calculator where you don't store all the data points), but >>> are slightly worse than doing two passes. Kahan summation would probably >>> also be good here too. >>> >>> >> Again, my tests show otherwise for float32. I'll condense my ipython log into a >> module for everyone's perusal. It's possible that the Kahan summation of the >> squared residuals will work better than the current two-pass algorithm and the >> implementations I give above. >> >> > This is what my tests show as well var_knuth outperformed any simple two > pass algorithm I could come up with, even ones using Kahan sums. > Interestingly, for 1D arrays the built in float32 variance performs > better than it should. After a bit of twiddling around I discovered that > it actually does most of it's calculations in float64. It uses a two > pass calculation, the result of mean is a scalar, and in the process of > converting that back to an array we end up with float64 values. Or > something like that; I was mostly reverse engineering the sequence of > events from the results. > Here's a simple of example of how var is a little wacky. A shape-[N] array will give you a different result than a shape-[1,N] array. The reason is clear -- in the second case the mean is not a scalar so there isn't the inadvertent promotion to float64, but it's still odd. >>> data = (1000*(random.random([10000]) - 0.1)).astype(float32) >>> print data.var() - data.reshape([1, -1]).var(-1) [ 0.1171875] I'm going to go ahead and attach a module containing the versions of mean, var, etc that I've been playing with in case someone wants to mess with them. Some were stolen from traffic on this list, for others I grabbed the algorithms from wikipedia or equivalent. -tim > -tim > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy_statistics.py URL: From lists.steve at arachnedesign.net Thu Sep 21 14:45:29 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 21 Sep 2006 14:45:29 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> > Are able to use doxygen for Python code ? I thought it only worked > for C (and > alike) ? > > IIRC correctly, it now does Python too. Let's see... here is an > example > ## Documentation for this module. > # > # More details. > > ## Documentation for a function. > # > # More details. > def func(): > pass > Looks like ## replaces the /** I never found it (although I haven't looked too hard), but I always thought there was an official way to document python code -- minimally to put the documentation in the docstring following the function definition: def func(..): """One liner. Continue docs -- some type of reStructredText style """ pas Isn't that the same docstring that ipython uses to bring up help, when you do: In [1]: myobject.some_func? So .. I guess I'm wondering why we want to break from the standard? -steve From cookedm at physics.mcmaster.ca Thu Sep 21 15:20:42 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 21 Sep 2006 15:20:42 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> On Thu, 21 Sep 2006 10:05:58 -0600 "Charles R Harris" wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem to be > tests for regressions and such. Is there a place we should be putting such > tests? > > 2) Any plans for code documentation? I documented some of my stuff with > doxygen markups and wonder if we should include a Doxyfile as part of the > package. We don't have much of a defined standard for docs. Personally, I wouldn't use doxygen: what I've seen for Python versions are hacks, whose output looks like C++, and which requires markup that's not like commonly-used conventions in Python (\brief, for instance). Foremost for Python doc strings, I think, is that it look ok when using pydoc or similar (ipython's ?, for instance). That means a minimal amount of markup. Someone previously mentioned including cross-references; I think that's a good idea. A 'See also' line, for instance. Examples are good too, especially if there's been disputes on the interpretation of the command :-) For the C code, documentation is autogenerated from the /** ... API */ comments that determine which functions are part of the C API. This are put into files multiarray_api.txt and ufunc_api.txt (in the include/ directory). The files are in reST format, so the comments should/could be. At some point I've got to through and add more :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From robert.kern at gmail.com Thu Sep 21 15:24:33 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 21 Sep 2006 14:24:33 -0500 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> References: <200609210918.05413.haase@msg.ucsf.edu> <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: Steve Lianoglou wrote: > So .. I guess I'm wondering why we want to break from the standard? We don't as far as Python code goes. The code that Chuck added Doxygen-style comments to was C code. I presume he was simply answering Sebastian's question rather than suggesting we use Doxygen for Python code, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Sep 21 15:47:40 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 13:47:40 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu> <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: Hi, On 9/21/06, Robert Kern wrote: > > Steve Lianoglou wrote: > > So .. I guess I'm wondering why we want to break from the standard? > > We don't as far as Python code goes. The code that Chuck added > Doxygen-style > comments to was C code. I presume he was simply answering Sebastian's > question > rather than suggesting we use Doxygen for Python code, too. Exactly. I also don't think the Python hack description applies to doxygen any longer. As to the oddness of \param or @param, here is an example from Epydoc using Epytext @type m: number @param m: The slope of the line. @type b: number @param b: The y intercept of the line. The X{y intercept} of a Looks like they borrowed something there ;) The main advantage of epydoc vs doxygen seems to be that you can use the markup inside the normal python docstring without having to make a separate comment block. Or would that be a disadvantage? Then again, I've been thinking of moving the python function docstrings into the add_newdocs.py file so everything is together in one spot and that would separate the Python docstrings from the functions anyway. I'll fool around with doxygen a bit and see what it does. The C code is the code that most needs documentation in any case. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ytz576 at 163.com Sun Sep 24 17:41:37 2006 From: ytz576 at 163.com (tpm) Date: Mon, 25 Sep 2006 05:41:37 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Thu Sep 21 17:59:37 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 21 Sep 2006 17:59:37 -0400 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4512DB42.7080509@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> Message-ID: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> On Thu, 21 Sep 2006 11:34:42 -0700 Tim Hochberg wrote: > Tim Hochberg wrote: > > Robert Kern wrote: > > > >> David M. Cooke wrote: > >> > >> > >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>> > >>> > >>>> Let me offer a third path: the algorithms used for .mean() and .var() > >>>> are substandard. There are much better incremental algorithms that > >>>> entirely avoid the need to accumulate such large (and therefore > >>>> precision-losing) intermediate values. The algorithms look like the > >>>> following for 1D arrays in Python: > >>>> > >>>> def mean(a): > >>>> m = a[0] > >>>> for i in range(1, len(a)): > >>>> m += (a[i] - m) / (i + 1) > >>>> return m > >>>> > >>>> > >>> This isn't really going to be any better than using a simple sum. > >>> It'll also be slower (a division per iteration). > >>> > >>> > >> With one exception, every test that I've thrown at it shows that it's > >> better for float32. That exception is uniformly spaced arrays, like > >> linspace(). > >> > >> > You do avoid > >> > accumulating large sums, but then doing the division a[i]/len(a) and > >> > adding that will do the same. > >> > >> Okay, this is true. > >> > >> > >> > >>> Now, if you want to avoid losing precision, you want to use a better > >>> summation technique, like compensated (or Kahan) summation: > >>> > >>> def mean(a): > >>> s = e = a.dtype.type(0) > >>> for i in range(0, len(a)): > >>> temp = s > >>> y = a[i] + e > >>> s = temp + y > >>> e = (temp - s) + y > >>> return s / len(a) > >> > >>>> def var(a): > >>>> m = a[0] > >>>> t = a.dtype.type(0) > >>>> for i in range(1, len(a)): > >>>> q = a[i] - m > >>>> r = q / (i+1) > >>>> m += r > >>>> t += i * q * r > >>>> t /= len(a) > >>>> return t > >>>> > >>>> Alternatively, from Knuth: > >>>> > >>>> def var_knuth(a): > >>>> m = a.dtype.type(0) > >>>> variance = a.dtype.type(0) > >>>> for i in range(len(a)): > >>>> delta = a[i] - m > >>>> m += delta / (i+1) > >>>> variance += delta * (a[i] - m) > >>>> variance /= len(a) > >>>> return variance > > I'm going to go ahead and attach a module containing the versions of > mean, var, etc that I've been playing with in case someone wants to mess > with them. Some were stolen from traffic on this list, for others I > grabbed the algorithms from wikipedia or equivalent. I looked into this a bit more. I checked float32 (single precision) and float64 (double precision), using long doubles (float96) for the "exact" results. This is based on your code. Results are compared using abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the range of [-100, 900] First, the mean. In float32, the Kahan summation in single precision is better by about 2 orders of magnitude than simple summation. However, accumulating the sum in double precision is better by about 9 orders of magnitude than simple summation (7 orders more than Kahan). In float64, Kahan summation is the way to go, by 2 orders of magnitude. For the variance, in float32, Knuth's method is *no better* than the two-pass method. Tim's code does an implicit conversion of intermediate results to float64, which is why he saw a much better result. The two-pass method using Kahan summation (again, in single precision), is better by about 2 orders of magnitude. There is practically no difference when using a double-precision accumulator amongst the techniques: they're all about 9 orders of magnitude better than single-precision two-pass. In float64, Kahan summation is again better than the rest, by about 2 orders of magnitude. I've put my adaptation of Tim's code, and box-and-whisker plots of the results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ Conclusions: - If you're going to calculate everything in single precision, use Kahan summation. Using it in double-precision also helps. - If you can use a double-precision accumulator, it's much better than any of the techniques in single-precision only. - for speed+precision in the variance, either use Kahan summation in single precision with the two-pass method, or use double precision with simple summation with the two-pass method. Knuth buys you nothing, except slower code :-) After 1.0 is out, we should look at doing one of the above. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Thu Sep 21 18:07:44 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 21 Sep 2006 16:07:44 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <45130D30.8070806@ee.byu.edu> David M. Cooke wrote: > >Conclusions: > >- If you're going to calculate everything in single precision, use Kahan >summation. Using it in double-precision also helps. >- If you can use a double-precision accumulator, it's much better than any of >the techniques in single-precision only. > >- for speed+precision in the variance, either use Kahan summation in single >precision with the two-pass method, or use double precision with simple >summation with the two-pass method. Knuth buys you nothing, except slower >code :-) > >After 1.0 is out, we should look at doing one of the above. > > +1 From Martin.Wiechert at mpimf-heidelberg.mpg.de Thu Sep 21 17:55:09 2006 From: Martin.Wiechert at mpimf-heidelberg.mpg.de (Martin Wiechert) Date: Thu, 21 Sep 2006 23:55:09 +0200 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <4512BCC4.3000708@ieee.org> References: <200609201655.08552.martin.wiechert@gmx.de> <200609211004.57370.martin.wiechert@gmx.de> <4512BCC4.3000708@ieee.org> Message-ID: <200609212355.09682.wiechert@mpimf-heidelberg.mpg.de> On Thursday 21 September 2006 18:24, Travis Oliphant wrote: > Martin Wiechert wrote: > > Thanks Travis. > > > > Do I understand correctly that the only way to be really safe is to make > > a copy and not to export a reference to it? > > Because anybody having a reference to the owner of the data can override > > the flag? > > No, that's not quite correct. Of course in C, anybody can do anything > they want to the flags. > > In Python, only the owner of the object itself can change the writeable > flag once it is set to False. So, if you only return a "view" of the > array (a.view()) then the Python user will not be able to change the > flags. > > Example: > > a = array([1,2,3]) > a.flags.writeable = False > > b = a.view() > > b.flags.writeable = True # raises an error. > > c = a > c.flags.writeable = True # can be done because c is a direct alias to a. > > Hopefully, that explains the situation a bit better. > It does. Thanks Travis. > -Travis > > > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- AV scanned by FortiGate From tim.hochberg at ieee.org Thu Sep 21 18:28:18 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 15:28:18 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <45131202.6000105@ieee.org> David M. Cooke wrote: > On Thu, 21 Sep 2006 11:34:42 -0700 > Tim Hochberg wrote: > > >> Tim Hochberg wrote: >> >>> Robert Kern wrote: >>> >>> >>>> David M. Cooke wrote: >>>> >>>> >>>> >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>> >>>>> >>>>> >>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>> are substandard. There are much better incremental algorithms that >>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>> following for 1D arrays in Python: >>>>>> >>>>>> def mean(a): >>>>>> m = a[0] >>>>>> for i in range(1, len(a)): >>>>>> m += (a[i] - m) / (i + 1) >>>>>> return m >>>>>> >>>>>> >>>>>> >>>>> This isn't really going to be any better than using a simple sum. >>>>> It'll also be slower (a division per iteration). >>>>> >>>>> >>>>> >>>> With one exception, every test that I've thrown at it shows that it's >>>> better for float32. That exception is uniformly spaced arrays, like >>>> linspace(). >>>> >>>> > You do avoid >>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>> > adding that will do the same. >>>> >>>> Okay, this is true. >>>> >>>> >>>> >>>> >>>>> Now, if you want to avoid losing precision, you want to use a better >>>>> summation technique, like compensated (or Kahan) summation: >>>>> >>>>> def mean(a): >>>>> s = e = a.dtype.type(0) >>>>> for i in range(0, len(a)): >>>>> temp = s >>>>> y = a[i] + e >>>>> s = temp + y >>>>> e = (temp - s) + y >>>>> return s / len(a) >>>>> >>>> >>>> >>>>>> def var(a): >>>>>> m = a[0] >>>>>> t = a.dtype.type(0) >>>>>> for i in range(1, len(a)): >>>>>> q = a[i] - m >>>>>> r = q / (i+1) >>>>>> m += r >>>>>> t += i * q * r >>>>>> t /= len(a) >>>>>> return t >>>>>> >>>>>> Alternatively, from Knuth: >>>>>> >>>>>> def var_knuth(a): >>>>>> m = a.dtype.type(0) >>>>>> variance = a.dtype.type(0) >>>>>> for i in range(len(a)): >>>>>> delta = a[i] - m >>>>>> m += delta / (i+1) >>>>>> variance += delta * (a[i] - m) >>>>>> variance /= len(a) >>>>>> return variance >>>>>> >> I'm going to go ahead and attach a module containing the versions of >> mean, var, etc that I've been playing with in case someone wants to mess >> with them. Some were stolen from traffic on this list, for others I >> grabbed the algorithms from wikipedia or equivalent. >> > > I looked into this a bit more. I checked float32 (single precision) and > float64 (double precision), using long doubles (float96) for the "exact" > results. This is based on your code. Results are compared using > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the > range of [-100, 900] > > First, the mean. In float32, the Kahan summation in single precision is > better by about 2 orders of magnitude than simple summation. However, > accumulating the sum in double precision is better by about 9 orders of > magnitude than simple summation (7 orders more than Kahan). > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > For the variance, in float32, Knuth's method is *no better* than the two-pass > method. Tim's code does an implicit conversion of intermediate results to > float64, which is why he saw a much better result. Doh! And I fixed that same problem in the mean implementation earlier too. I was astounded by how good knuth was doing, but not astounded enough apparently. Does it seem weird to anyone else that in: numpy_scalar python_scalar the precision ends up being controlled by the python scalar? I would expect the numpy_scalar to control the resulting precision just like numpy arrays do in similar circumstances. Perhaps the egg on my face is just clouding my vision though. > The two-pass method using > Kahan summation (again, in single precision), is better by about 2 orders of > magnitude. There is practically no difference when using a double-precision > accumulator amongst the techniques: they're all about 9 orders of magnitude > better than single-precision two-pass. > > In float64, Kahan summation is again better than the rest, by about 2 orders > of magnitude. > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > Conclusions: > > - If you're going to calculate everything in single precision, use Kahan > summation. Using it in double-precision also helps. > - If you can use a double-precision accumulator, it's much better than any of > the techniques in single-precision only. > > - for speed+precision in the variance, either use Kahan summation in single > precision with the two-pass method, or use double precision with simple > summation with the two-pass method. Knuth buys you nothing, except slower > code :-) > The two pass methods are definitely more accurate. I won't be convinced on the speed front till I see comparable C implementations slug it out. That may well mean never in practice. However, I expect that somewhere around 10,000 items, the cache will overflow and memory bandwidth will become the bottleneck. At that point the extra operations of Knuth won't matter as much as making two passes through the array and Knuth will win on speed. Of course the accuracy is pretty bad at single precision, so the possible, theoretical speed advantage at large sizes probably doesn't matter. -tim > After 1.0 is out, we should look at doing one of the above. > +1 From tim.hochberg at ieee.org Thu Sep 21 18:56:47 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 15:56:47 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <451318AF.2090205@ieee.org> David M. Cooke wrote: > On Thu, 21 Sep 2006 11:34:42 -0700 > Tim Hochberg wrote: > > >> Tim Hochberg wrote: >> >>> Robert Kern wrote: >>> >>> >>>> David M. Cooke wrote: >>>> >>>> >>>> >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>> >>>>> >>>>> >>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>> are substandard. There are much better incremental algorithms that >>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>> following for 1D arrays in Python: >>>>>> >>>>>> def mean(a): >>>>>> m = a[0] >>>>>> for i in range(1, len(a)): >>>>>> m += (a[i] - m) / (i + 1) >>>>>> return m >>>>>> >>>>>> >>>>>> >>>>> This isn't really going to be any better than using a simple sum. >>>>> It'll also be slower (a division per iteration). >>>>> >>>>> >>>>> >>>> With one exception, every test that I've thrown at it shows that it's >>>> better for float32. That exception is uniformly spaced arrays, like >>>> linspace(). >>>> >>>> > You do avoid >>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>> > adding that will do the same. >>>> >>>> Okay, this is true. >>>> >>>> >>>> >>>> >>>>> Now, if you want to avoid losing precision, you want to use a better >>>>> summation technique, like compensated (or Kahan) summation: >>>>> >>>>> def mean(a): >>>>> s = e = a.dtype.type(0) >>>>> for i in range(0, len(a)): >>>>> temp = s >>>>> y = a[i] + e >>>>> s = temp + y >>>>> e = (temp - s) + y >>>>> return s / len(a) >>>>> >>>> >>>> >>>>>> def var(a): >>>>>> m = a[0] >>>>>> t = a.dtype.type(0) >>>>>> for i in range(1, len(a)): >>>>>> q = a[i] - m >>>>>> r = q / (i+1) >>>>>> m += r >>>>>> t += i * q * r >>>>>> t /= len(a) >>>>>> return t >>>>>> >>>>>> Alternatively, from Knuth: >>>>>> >>>>>> def var_knuth(a): >>>>>> m = a.dtype.type(0) >>>>>> variance = a.dtype.type(0) >>>>>> for i in range(len(a)): >>>>>> delta = a[i] - m >>>>>> m += delta / (i+1) >>>>>> variance += delta * (a[i] - m) >>>>>> variance /= len(a) >>>>>> return variance >>>>>> >> I'm going to go ahead and attach a module containing the versions of >> mean, var, etc that I've been playing with in case someone wants to mess >> with them. Some were stolen from traffic on this list, for others I >> grabbed the algorithms from wikipedia or equivalent. >> > > I looked into this a bit more. I checked float32 (single precision) and > float64 (double precision), using long doubles (float96) for the "exact" > results. This is based on your code. Results are compared using > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the > range of [-100, 900] > > First, the mean. In float32, the Kahan summation in single precision is > better by about 2 orders of magnitude than simple summation. However, > accumulating the sum in double precision is better by about 9 orders of > magnitude than simple summation (7 orders more than Kahan). > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > For the variance, in float32, Knuth's method is *no better* than the two-pass > method. Tim's code does an implicit conversion of intermediate results to > float64, which is why he saw a much better result. The two-pass method using > Kahan summation (again, in single precision), is better by about 2 orders of > magnitude. There is practically no difference when using a double-precision > accumulator amongst the techniques: they're all about 9 orders of magnitude > better than single-precision two-pass. > > In float64, Kahan summation is again better than the rest, by about 2 orders > of magnitude. > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > Conclusions: > > - If you're going to calculate everything in single precision, use Kahan > summation. Using it in double-precision also helps. > - If you can use a double-precision accumulator, it's much better than any of > the techniques in single-precision only. > > - for speed+precision in the variance, either use Kahan summation in single > precision with the two-pass method, or use double precision with simple > summation with the two-pass method. Knuth buys you nothing, except slower > code :-) > > After 1.0 is out, we should look at doing one of the above. > One more little tidbit; it appears possible to "fix up" Knuth's algorithm so that it's comparable in accuracy to the two pass Kahan version by doing Kahan summation while accumulating the variance. Testing on this was far from thorough, but in the tests I did it nearly always produced identical results to kahan. Of course this is even slower than the original Knuth version, but it's interesting anyway: # This is probably messier than it need to be # but I'm out of time for today... def var_knuth2(values, dtype=default_prec): """var(values) -> variance of values computed using Knuth's one pass algorithm""" m = variance = mc = vc = dtype(0) for i, x in enumerate(values): delta = values[i] - m y = delta / dtype(i+1) + mc t = m + y mc = y - (t - m) m = t y = delta * (x - m) + vc t = variance + y vc = y - (t - variance) variance = t assert type(variance) == dtype variance /= dtype(len(values)) return variance From ytz576 at 163.com Sun Sep 24 18:59:58 2006 From: ytz576 at 163.com (tpm) Date: Mon, 25 Sep 2006 06:59:58 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From ckkart at hoc.net Thu Sep 21 19:18:34 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Thu, 21 Sep 2006 23:18:34 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy_1=2E0rc1_bdist=5Frpm_fails?= Message-ID: Hi, on linux I get an error when trying to build a rpm package from numpy 1.0rc1: building extension "numpy.core.umath" sources adding 'build/src.linux-i686-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.4/numpy/core/__ufunc_api.h' to sources. creating build/src.linux-i686-2.4/src conv_template:> build/src.linux-i686-2.4/src/umathmodule.c error: src/umathmodule.c.src: No such file or directory error: Bad exit status from /home/ck/testarea/rpm/tmp/rpm-tmp.68597 (%build) RPM build errors: Bad exit status from /home/ck/testarea/rpm/tmp/rpm-tmp.68597 (%build) error: command 'rpmbuild' failed with exit status 1 Christian From haase at msg.ucsf.edu Thu Sep 21 19:16:44 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 21 Sep 2006 16:16:44 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45131202.6000105@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> Message-ID: <200609211616.44773.haase@msg.ucsf.edu> On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > David M. Cooke wrote: > > On Thu, 21 Sep 2006 11:34:42 -0700 > > > > Tim Hochberg wrote: > >> Tim Hochberg wrote: > >>> Robert Kern wrote: > >>>> David M. Cooke wrote: > >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>> Let me offer a third path: the algorithms used for .mean() and > >>>>>> .var() are substandard. There are much better incremental algorithms > >>>>>> that entirely avoid the need to accumulate such large (and therefore > >>>>>> precision-losing) intermediate values. The algorithms look like the > >>>>>> following for 1D arrays in Python: > >>>>>> > >>>>>> def mean(a): > >>>>>> m = a[0] > >>>>>> for i in range(1, len(a)): > >>>>>> m += (a[i] - m) / (i + 1) > >>>>>> return m > >>>>> > >>>>> This isn't really going to be any better than using a simple sum. > >>>>> It'll also be slower (a division per iteration). > >>>> > >>>> With one exception, every test that I've thrown at it shows that it's > >>>> better for float32. That exception is uniformly spaced arrays, like > >>>> linspace(). > >>>> > >>>> > You do avoid > >>>> > accumulating large sums, but then doing the division a[i]/len(a) > >>>> > and adding that will do the same. > >>>> > >>>> Okay, this is true. > >>>> > >>>>> Now, if you want to avoid losing precision, you want to use a better > >>>>> summation technique, like compensated (or Kahan) summation: > >>>>> > >>>>> def mean(a): > >>>>> s = e = a.dtype.type(0) > >>>>> for i in range(0, len(a)): > >>>>> temp = s > >>>>> y = a[i] + e > >>>>> s = temp + y > >>>>> e = (temp - s) + y > >>>>> return s / len(a) > >>>>> > >>>>>> def var(a): > >>>>>> m = a[0] > >>>>>> t = a.dtype.type(0) > >>>>>> for i in range(1, len(a)): > >>>>>> q = a[i] - m > >>>>>> r = q / (i+1) > >>>>>> m += r > >>>>>> t += i * q * r > >>>>>> t /= len(a) > >>>>>> return t > >>>>>> > >>>>>> Alternatively, from Knuth: > >>>>>> > >>>>>> def var_knuth(a): > >>>>>> m = a.dtype.type(0) > >>>>>> variance = a.dtype.type(0) > >>>>>> for i in range(len(a)): > >>>>>> delta = a[i] - m > >>>>>> m += delta / (i+1) > >>>>>> variance += delta * (a[i] - m) > >>>>>> variance /= len(a) > >>>>>> return variance > >> > >> I'm going to go ahead and attach a module containing the versions of > >> mean, var, etc that I've been playing with in case someone wants to mess > >> with them. Some were stolen from traffic on this list, for others I > >> grabbed the algorithms from wikipedia or equivalent. > > > > I looked into this a bit more. I checked float32 (single precision) and > > float64 (double precision), using long doubles (float96) for the "exact" > > results. This is based on your code. Results are compared using > > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in > > the range of [-100, 900] > > > > First, the mean. In float32, the Kahan summation in single precision is > > better by about 2 orders of magnitude than simple summation. However, > > accumulating the sum in double precision is better by about 9 orders of > > magnitude than simple summation (7 orders more than Kahan). > > > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > > > For the variance, in float32, Knuth's method is *no better* than the > > two-pass method. Tim's code does an implicit conversion of intermediate > > results to float64, which is why he saw a much better result. > > Doh! And I fixed that same problem in the mean implementation earlier > too. I was astounded by how good knuth was doing, but not astounded > enough apparently. > > Does it seem weird to anyone else that in: > numpy_scalar python_scalar > the precision ends up being controlled by the python scalar? I would > expect the numpy_scalar to control the resulting precision just like > numpy arrays do in similar circumstances. Perhaps the egg on my face is > just clouding my vision though. > > > The two-pass method using > > Kahan summation (again, in single precision), is better by about 2 orders > > of magnitude. There is practically no difference when using a > > double-precision accumulator amongst the techniques: they're all about 9 > > orders of magnitude better than single-precision two-pass. > > > > In float64, Kahan summation is again better than the rest, by about 2 > > orders of magnitude. > > > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > > > Conclusions: > > > > - If you're going to calculate everything in single precision, use Kahan > > summation. Using it in double-precision also helps. > > - If you can use a double-precision accumulator, it's much better than > > any of the techniques in single-precision only. > > > > - for speed+precision in the variance, either use Kahan summation in > > single precision with the two-pass method, or use double precision with > > simple summation with the two-pass method. Knuth buys you nothing, except > > slower code :-) > > The two pass methods are definitely more accurate. I won't be convinced > on the speed front till I see comparable C implementations slug it out. > That may well mean never in practice. However, I expect that somewhere > around 10,000 items, the cache will overflow and memory bandwidth will > become the bottleneck. At that point the extra operations of Knuth won't > matter as much as making two passes through the array and Knuth will win > on speed. Of course the accuracy is pretty bad at single precision, so > the possible, theoretical speed advantage at large sizes probably > doesn't matter. > > -tim > > > After 1.0 is out, we should look at doing one of the above. > > +1 > Hi, I don't understand much of these "fancy algorithms", but does the above mean that after 1.0 is released the float32 functions will catch up in terms of precision precision ("to some extend") with using dtype=float64 ? - Sebastian Haase From davidgrant at gmail.com Thu Sep 21 19:23:54 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 21 Sep 2006 16:23:54 -0700 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: <4511DF4F.9070206@astraw.com> Message-ID: On 9/20/06, Bill Baxter wrote: > > Hey Andrew, point taken, but I think it would be better if someone who > actually knows the full extent of the change made the edit. I know > zeros and ones changed. Did anything else? > > Anyway, I'm surprised the release notes page is publicly editable. I'm glad that it is editable. I hate wikis that are only editable by a select few. Defeats the purpose (or at least does not maximize the capability of a wiki). -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Thu Sep 21 19:58:35 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 21 Sep 2006 19:58:35 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> References: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> Message-ID: On Thu, 21 Sep 2006, "David M. Cooke" apparently wrote: > Foremost for Python doc strings, I think, is that it look > ok when using pydoc or similar (ipython's ?, for > instance). That means a minimal amount of markup. IMO reStructuredText is very natural for documentation, and it is nicely handled by epydoc. fwiw, Alan Isaac From aisaac at american.edu Thu Sep 21 19:58:31 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 21 Sep 2006 19:58:31 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu><1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: On Thu, 21 Sep 2006, Charles R Harris apparently wrote: > As to the oddness of \param or @param, here is an example from > Epydoc using Epytext > @type m: number > @param m: The slope of the line. > @type b: number > @param b: The y intercept of the line. Compare to definition list style for consolidated field lists in section 5.1 of http://epydoc.sourceforge.net/fields.html#rst which is much more elegant, IMO. Cheers, Alan Isaac From pgmdevlist at gmail.com Thu Sep 21 20:35:02 2006 From: pgmdevlist at gmail.com (P GM) Date: Thu, 21 Sep 2006 20:35:02 -0400 Subject: [Numpy-discussion] putmask/take ? Message-ID: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> Folks, I'm running into the following problem with putmask on take. >>> import numpy >>> x = N.arange(12.) >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>> i = N.nonzero(m)[0] >>> w = N.array([-1, -2, -3, -4.]) >>> x.putmask(w,m) >>> x.take(i) >>> N.allclose(x.take(i),w) False I'm wondering ifit is intentional, or if it's a problem on my build (1.0b5), or if somebody experiences that as well. Thanks a lot for your input. P. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Sep 21 21:20:33 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 10:20:33 +0900 Subject: [Numpy-discussion] matrixmultiply moved (release notes?) Message-ID: Apparently numpy.matrixmultiply got moved into numpy.oldnumeric.matrixmultiply at some point (or rather ceased to be imported into the numpy namespace). Is there any list of all such methods that got banished? This would be nice to have in the release notes. --bb From ckkart at hoc.net Thu Sep 21 22:14:28 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Fri, 22 Sep 2006 02:14:28 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed Message-ID: Hi, from 1.0b1 to 1.0rc1 the default behaviour of take seems to have changed when omitting the axis argument: In [13]: a = reshape(arange(12),(3,4)) In [14]: take(a,[2,3]) Out[14]: array([2, 3]) In [15]: take(a,[2,3],1) Out[15]: array([[ 2, 3], [ 6, 7], [10, 11]]) Is this intended? Christian From wbaxter at gmail.com Thu Sep 21 22:34:49 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 11:34:49 +0900 Subject: [Numpy-discussion] general version of repmat? Message-ID: Is there some way to get the equivalent of repmat() for ndim == 1 and ndim >2. For ndim == 1, repmat always returns a 2-d array, instead of remaining 1-d. For ndim >2, repmat just doesn't work. Maybe we could add a 'reparray', with the signature: reparray(A, repeats, axis=None) where repeats is a scalar or a sequence. If 'repeats' is a scalar then the matrix is duplicated along 'axis' that many times. If 'repeats' is a sequence of length N, then A is duplicated repeats[i] times along axis[i]. If axis is None then it is assumed to be (0,1,2...N). Er that's not quite complete, because it doesn't specify what happens when you reparray an array to a higher dimension, like a 1-d to a 3-d. Like reparray([1,2], (2,2,2)). I guess the axis parameter could have some 'newaxis' entries to accomodate that. --bb From wbaxter at gmail.com Thu Sep 21 22:37:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 11:37:21 +0900 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: Message-ID: Yep, check the release notes: http://www.scipy.org/ReleaseNotes/NumPy_1.0 search for 'take' on that page to find out what others have changed as well. --bb On 9/22/06, Christian Kristukat wrote: > Hi, > from 1.0b1 to 1.0rc1 the default behaviour of take seems to have changed when > omitting the axis argument: > > In [13]: a = reshape(arange(12),(3,4)) > > In [14]: take(a,[2,3]) > Out[14]: array([2, 3]) > > In [15]: take(a,[2,3],1) > Out[15]: > array([[ 2, 3], > [ 6, 7], > [10, 11]]) > > Is this intended? > > Christian > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ckkart at hoc.net Thu Sep 21 23:45:43 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Fri, 22 Sep 2006 03:45:43 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed References: Message-ID: Bill Baxter gmail.com> writes: > > Yep, check the release notes: > http://www.scipy.org/ReleaseNotes/NumPy_1.0 > search for 'take' on that page to find out what others have changed as well. > --bb Ok. Does axis=None then mean, that take(a, ind) operates on the flattened array? This it at least what it seem to be. I noticed that the ufunc behaves differently. a.take(ind) and a.take(ind, axis=0) behave the same, so the default argument to axis is 0 rather than None. Christian From pgmdevlist at gmail.com Thu Sep 21 19:40:39 2006 From: pgmdevlist at gmail.com (PGM) Date: Thu, 21 Sep 2006 19:40:39 -0400 Subject: [Numpy-discussion] Putmask/take ? Message-ID: <200609211940.40154.pgmdevlist@gmail.com> Folks, I'm running into the following problem with putmask on take. >>> import numpy >>> x = N.arange(12.) >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>> i = N.nonzero(m)[0] >>> w = N.array([-1, -2, -3, -4.]) >>> x.putmask(w,m) >>> x.take(i) >>> N.allclose(x.take(i),w) False I'm wondering ifit is intentional, or if it's a problem on my build (1.0b5), or if somebody experienced it as well. Thanks a lot for your input. P. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Bienstman at ugent.be Fri Sep 22 00:56:51 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Fri, 22 Sep 2006 06:56:51 +0200 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 In-Reply-To: References: Message-ID: <200609220656.54764.Peter.Bienstman@ugent.be> Cleaning out and rebuilding did the trick! Thanks, Peter On Thursday 21 September 2006 18:33, numpy-discussion-request at lists.sourceforge.net wrote: > Subject: Re: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 > > > I don't see this running the latest from svn on AMD64 here. Not sayin' > there might not be a problem with rc1, I just don't see it with my sources. > > Python 2.4.3 (#1, Jun 13 2006, 11:46:22) > [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy > >>> numpy.version.version > > '1.0.dev3202' > > >>> numpy.version.os.uname() > > ('Linux', 'tethys', '2.6.17-1.2187_FC5', '#1 SMP Mon Sep 11 01:16:59 EDT > 2006', 'x86_64') > > If you are building on Gentoo maybe you could delete the build directory > (and maybe the numpy site package) and rebuild. > > Chuck. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From stefan at sun.ac.za Fri Sep 22 03:02:51 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 09:02:51 +0200 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <200609211940.40154.pgmdevlist@gmail.com> References: <200609211940.40154.pgmdevlist@gmail.com> Message-ID: <20060922070250.GA15593@mentat.za.net> Hi P., On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > I'm running into the following problem with putmask on take. > > >>> import numpy > >>> x = N.arange(12.) > >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>> i = N.nonzero(m)[0] > >>> w = N.array([-1, -2, -3, -4.]) > >>> x.putmask(w,m) > >>> x.take(i) > >>> N.allclose(x.take(i),w) According to the putmask docstring: a.putmask(values, mask) sets a.flat[n] = v[n] for each n where mask.flat[n] is true. v can be scalar. This would mean that 'w' is not of the right length. Would the following do what you want? import numpy as N m = N.array([1,0,0,0,0,0,1,0,0,1,0,1],dtype=bool) w = N.array([-1,-2,-3,-4]) x[m] = w Regards St?fan From stefan at sun.ac.za Fri Sep 22 03:13:02 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 09:13:02 +0200 Subject: [Numpy-discussion] putmask/take ? In-Reply-To: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> References: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> Message-ID: <20060922071302.GA17679@mentat.za.net> On Thu, Sep 21, 2006 at 08:35:02PM -0400, P GM wrote: > Folks, > I'm running into the following problem with putmask on take. > > >>> import numpy > >>> x = N.arange(12.) > >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>> i = N.nonzero (m)[0] > >>> w = N.array([-1, -2, -3, -4.]) > >>> x.putmask(w,m) You can also use N.place: Similar to putmask arr[mask] = vals but the 1D array vals has the same number of elements as the non-zero values of mask. Inverse of extract. > >>> x.take(i) > >>> N.allclose(x.take(i),w) > False Regards St?fan From robert.kern at gmail.com Fri Sep 22 03:17:57 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 02:17:57 -0500 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <20060922070250.GA15593@mentat.za.net> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> Message-ID: Stefan van der Walt wrote: > Hi P., > > On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > >> I'm running into the following problem with putmask on take. >> >>>>> import numpy >>>>> x = N.arange(12.) >>>>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>>>> i = N.nonzero(m)[0] >>>>> w = N.array([-1, -2, -3, -4.]) >>>>> x.putmask(w,m) >>>>> x.take(i) >>>>> N.allclose(x.take(i),w) > > According to the putmask docstring: > > a.putmask(values, mask) sets a.flat[n] = v[n] for each n where > mask.flat[n] is true. v can be scalar. > > This would mean that 'w' is not of the right length. There are 4 true values in m and 4 values in w. What's the wrong length? For the sake of clarity: In [1]: from numpy import * In [3]: m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] In [4]: i = nonzero(m)[0] In [5]: i Out[5]: array([ 0, 6, 9, 11]) In [6]: w = array([-1, -2, -3, -4.]) In [7]: x = arange(12.) In [8]: x.putmask(w, m) In [9]: x Out[9]: array([ -1., 1., 2., 3., 4., 5., -3., 7., 8., -2., 10., -4.]) In [17]: x[array(m, dtype=bool)] = w In [18]: x Out[18]: array([ -1., 1., 2., 3., 4., 5., -2., 7., 8., -3., 10., -4.]) Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. It's pretty clear that this is a bug in .putmask(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lroubeyrie at limair.asso.fr Fri Sep 22 03:34:31 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Fri, 22 Sep 2006 09:34:31 +0200 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <4512C553.6090609@ieee.org> References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> Message-ID: <200609220934.31217.lroubeyrie@limair.asso.fr> Le jeudi 21 septembre 2006 19:01, Travis Oliphant a ?crit?: > Lionel Roubeyrie wrote: > > find any solution for that. I have tried with arrays of dtype=object, but > > I have problem when I want to compute min, max, ... with an error like: > > TypeError: function not supported for these types, and can't coerce > > safely to supported types. > > I just added support for min and max methods of object arrays, by adding > support for Object arrays to the minimum and maximum functions. > > -Travis Hello travis, good news, and thanks for your last comment. However, using nans give some errors with scipy.stats: lionel52>t=array([1,2,nan,4]) lionel53>stats.nanmean(t) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /home/lionel/ /usr/lib/python2.4/site-packages/scipy/stats/stats.py in nanmean(x, axis) 258 259 # XXX: this line is quite clearly wrong --> 260 n = N-sum(isnan(x),axis) 261 putmask(x,isnan(x),0) 262 return stats.mean(x,axis)/factor NameError: global name 'N' is not defined > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From robert.kern at gmail.com Fri Sep 22 04:06:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 03:06:36 -0500 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609220934.31217.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> <200609220934.31217.lroubeyrie@limair.asso.fr> Message-ID: Lionel Roubeyrie wrote: > good news, and thanks for your last comment. However, using nans give some > errors with scipy.stats: > lionel52>t=array([1,2,nan,4]) > > lionel53>stats.nanmean(t) > --------------------------------------------------------------------------- > exceptions.NameError Traceback (most recent > call last) > > /home/lionel/ > > /usr/lib/python2.4/site-packages/scipy/stats/stats.py in nanmean(x, axis) > 258 > 259 # XXX: this line is quite clearly wrong > --> 260 n = N-sum(isnan(x),axis) > 261 putmask(x,isnan(x),0) > 262 return stats.mean(x,axis)/factor > > NameError: global name 'N' is not defined It's a bug in nanmean() as the comment immediately preceding it mentions. I don't know who put it in, but I noticed it and couldn't figure out what it intended to do (or didn't have to time to try). Ah, it's Travis's fault. So he can fix it. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Sep 22 04:21:57 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 10:21:57 +0200 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> Message-ID: <20060922082157.GA21884@mentat.za.net> On Fri, Sep 22, 2006 at 02:17:57AM -0500, Robert Kern wrote: > Stefan van der Walt wrote: > > Hi P., > > > > On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > > > >> I'm running into the following problem with putmask on take. > >> > >>>>> import numpy > >>>>> x = N.arange(12.) > >>>>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>>>> i = N.nonzero(m)[0] > >>>>> w = N.array([-1, -2, -3, -4.]) > >>>>> x.putmask(w,m) > >>>>> x.take(i) > >>>>> N.allclose(x.take(i),w) > > > > According to the putmask docstring: > > > > a.putmask(values, mask) sets a.flat[n] = v[n] for each n where > > mask.flat[n] is true. v can be scalar. > > > > This would mean that 'w' is not of the right length. > > There are 4 true values in m and 4 values in w. What's the wrong length? The way I read the docstring, you use putmask like this: In [4]: x = N.array([1,2,3,4]) In [5]: x.putmask([4,3,2,1],[1,0,0,1]) In [6]: x Out[6]: array([4, 2, 3, 1]) > For the sake of clarity: > > > In [1]: from numpy import * > > In [3]: m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > > In [4]: i = nonzero(m)[0] > > In [5]: i > Out[5]: array([ 0, 6, 9, 11]) > > In [6]: w = array([-1, -2, -3, -4.]) > > In [7]: x = arange(12.) > > In [8]: x.putmask(w, m) > > In [9]: x > Out[9]: > array([ -1., 1., 2., 3., 4., 5., -3., 7., 8., -2., 10., > -4.]) > > In [17]: x[array(m, dtype=bool)] = w > > In [18]: x > Out[18]: > array([ -1., 1., 2., 3., 4., 5., -2., 7., 8., -3., 10., > -4.]) > > > Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. > It's pretty clear that this is a bug in .putmask(). Based purely on what I read in the docstring, I would expect the above to do x[0] = w[0] x[6] = w[6] x[9] = w[9] x[11] = w[11] Since w is of length 4, you'll probably get indices modulo 4: w[6] == w[2] == -3 w[9] == w[1] == -2 w[11] == w[3] == -4 Which seems to explain what you are seeing. Regards St?fan From Svein-Erik.Hamran at ffi.no Fri Sep 22 05:07:03 2006 From: Svein-Erik.Hamran at ffi.no (Svein-Erik.Hamran at ffi.no) Date: Fri, 22 Sep 2006 11:07:03 +0200 Subject: [Numpy-discussion] reading large files Message-ID: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> I would like to read files >2Gbyte. From earlier posting I believed it should be possible with python 2.5. I am getting MemoryError when trying to read a file larger than 2Gb. I am using python2.5 and numpy1.0rc1. Any solution to the problem? SEH -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwe239 at tom.com Wed Sep 27 07:40:32 2006 From: qwe239 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Wed, 27 Sep 2006 19:40:32 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU6t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Sep 22 08:06:17 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 21:06:17 +0900 Subject: [Numpy-discussion] reading large files In-Reply-To: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> References: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> Message-ID: Do you also have a 64-bit processor? Just checking since you didn't mention it. --bb On 9/22/06, Svein-Erik.Hamran at ffi.no wrote: > > > > > I would like to read files >2Gbyte. From earlier posting I believed it > should be possible with python 2.5. > > I am getting MemoryError when trying to read a file larger than 2Gb. > > I am using python2.5 and numpy1.0rc1. > > Any solution to the problem? > > > > SEH > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Fri Sep 22 08:47:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 21:47:28 +0900 Subject: [Numpy-discussion] Rationale for atleast_3d Message-ID: 26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why atleast_3d acts the way it does: http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d He doesn't seem to have gotten any answers. And now I'm wondering the same thing. Anyone have any idea? --bb From tim.hochberg at ieee.org Fri Sep 22 09:08:39 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 06:08:39 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45131202.6000105@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> Message-ID: <4513E057.7070906@ieee.org> Tim Hochberg wrote: > David M. Cooke wrote: > >> On Thu, 21 Sep 2006 11:34:42 -0700 >> Tim Hochberg wrote: >> >> >> >>> Tim Hochberg wrote: >>> >>> >>>> Robert Kern wrote: >>>> >>>> >>>> >>>>> David M. Cooke wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>>> are substandard. There are much better incremental algorithms that >>>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>>> following for 1D arrays in Python: >>>>>>> >>>>>>> def mean(a): >>>>>>> m = a[0] >>>>>>> for i in range(1, len(a)): >>>>>>> m += (a[i] - m) / (i + 1) >>>>>>> return m >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> This isn't really going to be any better than using a simple sum. >>>>>> It'll also be slower (a division per iteration). >>>>>> >>>>>> >>>>>> >>>>>> >>>>> With one exception, every test that I've thrown at it shows that it's >>>>> better for float32. That exception is uniformly spaced arrays, like >>>>> linspace(). >>>>> >>>>> > You do avoid >>>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>>> > adding that will do the same. >>>>> >>>>> Okay, this is true. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Now, if you want to avoid losing precision, you want to use a better >>>>>> summation technique, like compensated (or Kahan) summation: >>>>>> >>>>>> def mean(a): >>>>>> s = e = a.dtype.type(0) >>>>>> for i in range(0, len(a)): >>>>>> temp = s >>>>>> y = a[i] + e >>>>>> s = temp + y >>>>>> e = (temp - s) + y >>>>>> return s / len(a) >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>>>> def var(a): >>>>>>> m = a[0] >>>>>>> t = a.dtype.type(0) >>>>>>> for i in range(1, len(a)): >>>>>>> q = a[i] - m >>>>>>> r = q / (i+1) >>>>>>> m += r >>>>>>> t += i * q * r >>>>>>> t /= len(a) >>>>>>> return t >>>>>>> >>>>>>> Alternatively, from Knuth: >>>>>>> >>>>>>> def var_knuth(a): >>>>>>> m = a.dtype.type(0) >>>>>>> variance = a.dtype.type(0) >>>>>>> for i in range(len(a)): >>>>>>> delta = a[i] - m >>>>>>> m += delta / (i+1) >>>>>>> variance += delta * (a[i] - m) >>>>>>> variance /= len(a) >>>>>>> return variance >>>>>>> >>>>>>> >>> I'm going to go ahead and attach a module containing the versions of >>> mean, var, etc that I've been playing with in case someone wants to mess >>> with them. Some were stolen from traffic on this list, for others I >>> grabbed the algorithms from wikipedia or equivalent. >>> >>> >> I looked into this a bit more. I checked float32 (single precision) and >> float64 (double precision), using long doubles (float96) for the "exact" >> results. This is based on your code. Results are compared using >> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the >> range of [-100, 900] >> >> First, the mean. In float32, the Kahan summation in single precision is >> better by about 2 orders of magnitude than simple summation. However, >> accumulating the sum in double precision is better by about 9 orders of >> magnitude than simple summation (7 orders more than Kahan). >> >> In float64, Kahan summation is the way to go, by 2 orders of magnitude. >> >> For the variance, in float32, Knuth's method is *no better* than the two-pass >> method. Tim's code does an implicit conversion of intermediate results to >> float64, which is why he saw a much better result. >> > Doh! And I fixed that same problem in the mean implementation earlier > too. I was astounded by how good knuth was doing, but not astounded > enough apparently. > > Does it seem weird to anyone else that in: > numpy_scalar python_scalar > the precision ends up being controlled by the python scalar? I would > expect the numpy_scalar to control the resulting precision just like > numpy arrays do in similar circumstances. Perhaps the egg on my face is > just clouding my vision though. > > >> The two-pass method using >> Kahan summation (again, in single precision), is better by about 2 orders of >> magnitude. There is practically no difference when using a double-precision >> accumulator amongst the techniques: they're all about 9 orders of magnitude >> better than single-precision two-pass. >> >> In float64, Kahan summation is again better than the rest, by about 2 orders >> of magnitude. >> >> I've put my adaptation of Tim's code, and box-and-whisker plots of the >> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ >> >> Conclusions: >> >> - If you're going to calculate everything in single precision, use Kahan >> summation. Using it in double-precision also helps. >> - If you can use a double-precision accumulator, it's much better than any of >> the techniques in single-precision only. >> >> - for speed+precision in the variance, either use Kahan summation in single >> precision with the two-pass method, or use double precision with simple >> summation with the two-pass method. Knuth buys you nothing, except slower >> code :-) >> >> > The two pass methods are definitely more accurate. I won't be convinced > on the speed front till I see comparable C implementations slug it out. > That may well mean never in practice. However, I expect that somewhere > around 10,000 items, the cache will overflow and memory bandwidth will > become the bottleneck. At that point the extra operations of Knuth won't > matter as much as making two passes through the array and Knuth will win > on speed. OK, let me take this back. I looked at this speed effect a little more and the effect is smaller than I remember. For example, "a+=a" slows down by about a factor of 2.5 on my box between 10,000 and 100,0000 elements. That's not insignificant, but assuming (naively) that this means that memory effects account to the equivalent of 1.5 add operations, this doesn't come close to closing to gap between Knuth and the two pass approach. The other thing is that while a+=a and similar operations show this behaviour, a.sum() and add.reduce(a) do not, hinting that perhaps it's only writing to memory that is a bottleneck. Or perhaps hinting that my mental model of what's going on here is badly flawed. So, +1 for me on Kahan summation for computing means and two pass with Kahan summation for variances. It would probably be nice to expose the Kahan sum and maybe even the raw_kahan_sum somewhere. I can see use for the latter in summing a bunch of disparate matrices with high precision. I'm on the fence on using the array dtype for the accumulator dtype versus always using at least double precision for the accumulator. The former is easier to explain and is probably faster, but the latter is a lot more accuracy for basically free. It depends on how the relative speeds shake out I suppose. If the speed of using float64 is comparable to that of using float32, we might as well. One thing I'm not on the fence about is the return type: it should always match the input type, or dtype if that is specified. Otherwise one gets mysterious, gratuitous upcasting. On the subject of upcasting, I'm still skeptical of the behavior of mixed numpy-scalar and python-scalar ops. Since numpy-scalars are generally the results of indexing operations, not literals, I think that they should behave like arrays for purposes of determining the resulting precision, not like python-scalars. Otherwise situations arise where the rank of the input array determine the kind of output (float32 versus float64 for example) since if the array is 1D indexing into the array yields numpy-scalars which then get promoted when they interact with python-scalars.However if the array is 2D, indexing yields a vector, which is not promoted when interacting with python scalars and the precision remains fixed. -tim > Of course the accuracy is pretty bad at single precision, so > the possible, theoretical speed advantage at large sizes probably > doesn't matter. > > -tim > > >> After 1.0 is out, we should look at doing one of the above. >> >> > > +1 > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > m bnb From wbaxter at gmail.com Fri Sep 22 09:31:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 22:31:26 +0900 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: Ok, here's my best shot at a generalized repmat: def reparray(A, tup): if numpy.isscalar(tup): tup = (tup,) d = len(tup) A = numpy.array(A,copy=False,subok=True,ndmin=d) for i,k in enumerate(tup): A = numpy.concatenate([A]*k, axis=i) return A Can anyone improve on this? The only way I could think to seriously improve it would be if there were an N-way concatenate() function in the C-API. Each call to concatenate() involves a potentially big allocation and memcpy of the data. And here's a docstring for it: """Repeat an array the number of times given in the integer tuple, tup. Similar to repmat, but works for arrays of any dimension. reparray(A,(m,n)) is equivalent to repmat(A,m,n) If tup has length d, the result will have dimension of max(d, A.ndim). If tup is scalar it is treated as a 1-tuple. If A.ndim < d, A is promoted to be d-dimensional by prepending new axes. So a shape (3,) array is promoted to (1,3) for 2-D replication, or shape (1,1,3) for 3-D replication. If this is not the desired behavior, promote A to d-dimensions manually before calling this function. If d < A.ndim, ndim is effectively promoted to A.ndim by appending 1's to tup. Examples: >>> a = array([0,1,2]) >>> reparray(a,2) array([0, 1, 2, 0, 1, 2]) >>> reparray(a,(1,2)) array([[0, 1, 2, 0, 1, 2]]) >>> reparray(a,(2,2)) array([[0, 1, 2, 0, 1, 2], [0, 1, 2, 0, 1, 2]]) >>> reparray(a,(2,1,2)) array([[[0, 1, 2, 0, 1, 2]], [[0, 1, 2, 0, 1, 2]]]) See Also: repmat, repeat """ From aisaac at american.edu Fri Sep 22 09:53:32 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 22 Sep 2006 09:53:32 -0400 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: On Fri, 22 Sep 2006, Bill Baxter apparently wrote: > Ok, here's my best shot at a generalized repmat: Sorry to always repeat this suggestion when it comes to repmat, but I think the whole approach is wrong. A repmat should be a separate object type, which behaves like the described matrix but has only one copy of the repetitive data. Cheers, Alan Isaac From wbaxter at gmail.com Fri Sep 22 10:41:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 23:41:36 +0900 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: On 9/22/06, Alan G Isaac wrote: > On Fri, 22 Sep 2006, Bill Baxter apparently wrote: > > Ok, here's my best shot at a generalized repmat: > > Sorry to always repeat this suggestion when > it comes to repmat, but I think the whole approach > is wrong. A repmat should be a separate object type, > which behaves like the described matrix but has only one > copy of the repetitive data. That may be true for some cases. But I usually start modifying the data I create right after a repmat. It wouldn't help in that case. So unless you're really making a lot of large repmats of arrays that never change, or for use as temp variables, I can't see a separate class being that much of a win, compared with the complexity of implementing and maintaining it (think "fancy indexing"). YMMV. However, repmat seems to be far less commonly needed in numpy than in Matlab. I think that's mostly thanks to the broadcasting rules, which already create a sort of implicit repmat of the input in many common cases. --bb From charlesr.harris at gmail.com Fri Sep 22 10:43:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 22 Sep 2006 08:43:53 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4513E057.7070906@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <4513E057.7070906@ieee.org> Message-ID: Hi, On 9/22/06, Tim Hochberg wrote: > > Tim Hochberg wrote: > > David M. Cooke wrote: > > > >> On Thu, 21 Sep 2006 11:34:42 -0700 > >> Tim Hochberg wrote: > >> > >> > >> > >>> Tim Hochberg wrote: > >>> > >>> > >>>> Robert Kern wrote: > >>>> > >>>> > >>>> > >>>>> David M. Cooke wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Let me offer a third path: the algorithms used for .mean() and > .var() > >>>>>>> are substandard. There are much better incremental algorithms that > >>>>>>> entirely avoid the need to accumulate such large (and therefore > >>>>>>> precision-losing) intermediate values. The algorithms look like > the > >>>>>>> following for 1D arrays in Python: > >>>>>>> > >>>>>>> def mean(a): > >>>>>>> m = a[0] > >>>>>>> for i in range(1, len(a)): > >>>>>>> m += (a[i] - m) / (i + 1) > >>>>>>> return m > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> This isn't really going to be any better than using a simple sum. > >>>>>> It'll also be slower (a division per iteration). > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> With one exception, every test that I've thrown at it shows that > it's > >>>>> better for float32. That exception is uniformly spaced arrays, like > >>>>> linspace(). > >>>>> > >>>>> > You do avoid > >>>>> > accumulating large sums, but then doing the division a[i]/len(a) > and > >>>>> > adding that will do the same. > >>>>> > >>>>> Okay, this is true. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> Now, if you want to avoid losing precision, you want to use a > better > >>>>>> summation technique, like compensated (or Kahan) summation: > >>>>>> > >>>>>> def mean(a): > >>>>>> s = e = a.dtype.type(0) > >>>>>> for i in range(0, len(a)): > >>>>>> temp = s > >>>>>> y = a[i] + e > >>>>>> s = temp + y > >>>>>> e = (temp - s) + y > >>>>>> return s / len(a) > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>>>> def var(a): > >>>>>>> m = a[0] > >>>>>>> t = a.dtype.type(0) > >>>>>>> for i in range(1, len(a)): > >>>>>>> q = a[i] - m > >>>>>>> r = q / (i+1) > >>>>>>> m += r > >>>>>>> t += i * q * r > >>>>>>> t /= len(a) > >>>>>>> return t > >>>>>>> > >>>>>>> Alternatively, from Knuth: > >>>>>>> > >>>>>>> def var_knuth(a): > >>>>>>> m = a.dtype.type(0) > >>>>>>> variance = a.dtype.type(0) > >>>>>>> for i in range(len(a)): > >>>>>>> delta = a[i] - m > >>>>>>> m += delta / (i+1) > >>>>>>> variance += delta * (a[i] - m) > >>>>>>> variance /= len(a) > >>>>>>> return variance > >>>>>>> > >>>>>>> > >>> I'm going to go ahead and attach a module containing the versions of > >>> mean, var, etc that I've been playing with in case someone wants to > mess > >>> with them. Some were stolen from traffic on this list, for others I > >>> grabbed the algorithms from wikipedia or equivalent. > >>> > >>> > >> I looked into this a bit more. I checked float32 (single precision) and > >> float64 (double precision), using long doubles (float96) for the > "exact" > >> results. This is based on your code. Results are compared using > >> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values > in the > >> range of [-100, 900] > >> > >> First, the mean. In float32, the Kahan summation in single precision is > >> better by about 2 orders of magnitude than simple summation. However, > >> accumulating the sum in double precision is better by about 9 orders of > >> magnitude than simple summation (7 orders more than Kahan). > >> > >> In float64, Kahan summation is the way to go, by 2 orders of magnitude. > >> > >> For the variance, in float32, Knuth's method is *no better* than the > two-pass > >> method. Tim's code does an implicit conversion of intermediate results > to > >> float64, which is why he saw a much better result. > >> > > Doh! And I fixed that same problem in the mean implementation earlier > > too. I was astounded by how good knuth was doing, but not astounded > > enough apparently. > > > > Does it seem weird to anyone else that in: > > numpy_scalar python_scalar > > the precision ends up being controlled by the python scalar? I would > > expect the numpy_scalar to control the resulting precision just like > > numpy arrays do in similar circumstances. Perhaps the egg on my face is > > just clouding my vision though. > > > > > >> The two-pass method using > >> Kahan summation (again, in single precision), is better by about 2 > orders of > >> magnitude. There is practically no difference when using a > double-precision > >> accumulator amongst the techniques: they're all about 9 orders of > magnitude > >> better than single-precision two-pass. > >> > >> In float64, Kahan summation is again better than the rest, by about 2 > orders > >> of magnitude. > >> > >> I've put my adaptation of Tim's code, and box-and-whisker plots of the > >> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > >> > >> Conclusions: > >> > >> - If you're going to calculate everything in single precision, use > Kahan > >> summation. Using it in double-precision also helps. > >> - If you can use a double-precision accumulator, it's much better than > any of > >> the techniques in single-precision only. > >> > >> - for speed+precision in the variance, either use Kahan summation in > single > >> precision with the two-pass method, or use double precision with simple > >> summation with the two-pass method. Knuth buys you nothing, except > slower > >> code :-) > >> > >> > > The two pass methods are definitely more accurate. I won't be convinced > > on the speed front till I see comparable C implementations slug it out. > > That may well mean never in practice. However, I expect that somewhere > > around 10,000 items, the cache will overflow and memory bandwidth will > > become the bottleneck. At that point the extra operations of Knuth won't > > matter as much as making two passes through the array and Knuth will win > > on speed. > OK, let me take this back. I looked at this speed effect a little more > and the effect is smaller than I remember. For example, "a+=a" slows > down by about a factor of 2.5 on my box between 10,000 and 100,0000 > elements. That's not insignificant, but assuming (naively) that this > means that memory effects account to the equivalent of 1.5 add > operations, this doesn't come close to closing to gap between Knuth and > the two pass approach. The other thing is that while a+=a and similar > operations show this behaviour, a.sum() and add.reduce(a) do not, > hinting that perhaps it's only writing to memory that is a bottleneck. > Or perhaps hinting that my mental model of what's going on here is badly > flawed. > > So, +1 for me on Kahan summation for computing means and two pass with > Kahan summation for variances. It would probably be nice to expose the > Kahan sum and maybe even the raw_kahan_sum somewhere. I can see use for > the latter in summing a bunch of disparate matrices with high precision. > > I'm on the fence on using the array dtype for the accumulator dtype > versus always using at least double precision for the accumulator. I keep going back and forth on this myself. So I ask myself what I do in practice. In practice, I accumulate in doubles, sometimes extended precision. I only think of accumulating in floats when the number of items is small and computation might be noticeably faster -- and these days it seldom is. The days of the Vax and a 2x speed difference are gone, now we generally have FPUs whose base type is double with float tacked on as an afterthought. The only lesson from those days that remains pertinent is: never, ever, do division. So in practice, I would go with doubles. The only good reason to use floats for floats is a sort of pure consistency. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fakeyq at broadlandguards.com Fri Sep 22 10:37:33 2006 From: fakeyq at broadlandguards.com (Hal Giles) Date: Fri, 22 Sep 2006 16:37:33 +0200 Subject: [Numpy-discussion] intellectual federation Message-ID: <000501c6de55$9fadc64f$5b2a4e57@cjjgtx> Feisalintroduced us one by one, and Auda with a measured word seemed toregister each person. I said I was a simple man who disliked servants about him. The walls each sidewere of regular bands of sandstone, streaked red in many shades. Jealousy, superadded to inefficiency, would be the outcome. The walls each sidewere of regular bands of sandstone, streaked red in many shades. They expected him next day; so wepassed two nights in this strange-coloured, echoing place. At first it was two and two, but theothers joined, till they were six abreast. Each required to be consulted on the details of ourgoing, and where and when we should halt. The edges of the cliffs about us were clipped strangely,like fantastic parapets. The fluency had a lack of grammar, which made my talk aperpetual adventure for my hearers. Therefore it seemed fit to the boyMohammed to run races. Sheslipped, so that he crashed off and broke an arm. The Arab war was geographical, and the Turkish Army an accident. Thirdly, that their virtue lay in depth, not inface. NASIR, AUDA, AND I SET OFF TOGETHER ON THE LONG RIDE. They expected him next day; so wepassed two nights in this strange-coloured, echoing place. The aeroplanes from it had flown uphere and were established. Wadi Osman to-day was less irregular in course, and broadened slowly. Everybody took the offer, and the camels set offin a mob. The aeroplanes from it had flown uphere and were established. Therefore it seemed fit to the boyMohammed to run races. Of the number of dead Turks he could give no account: they didnot enter the register. There entered a tall, strong figure, with a haggard face,passionate and tragic. Or was it merely that long ago we had seen freshgrass growing in the spring? The last part of the marchwas after dark, and when we stopped, Arslan was missing. We were now on the limestoneof the Shefa crest. If his performancewas one-half his desire, we should be prosperous and fortunate. So I judged mywork in Wadi Ais sufficiently done, and well done. Everybody took the offer, and the camels set offin a mob. In prison inEgypt he would cost us food and guards. A short hour later we stopped at the tents of a wife of Dakhil-Allah,for a meal. All he could do for my sake wasto let Daud share the ordained sentence. Mohammed el Kadhi was ourguide, with six Juheina. I was eating my Lords breadwith Turkish teeth! When innocent they were hot andunashamed. If his performancewas one-half his desire, we should be prosperous and fortunate. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mockingbird.gif Type: image/gif Size: 11589 bytes Desc: not available URL: From tim.hochberg at ieee.org Fri Sep 22 12:06:10 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 09:06:10 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609211616.44773.haase@msg.ucsf.edu> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> Message-ID: <451409F2.8010805@ieee.org> Sebastian Haase wrote: > On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > >> David M. Cooke wrote: >> >>> On Thu, 21 Sep 2006 11:34:42 -0700 >>> >>> Tim Hochberg wrote: >>> >>>> Tim Hochberg wrote: >>>> >>>>> Robert Kern wrote: >>>>> >>>>>> David M. Cooke wrote: >>>>>> >>>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>>>> >>>>>>>> Let me offer a third path: the algorithms used for .mean() and >>>>>>>> .var() are substandard. There are much better incremental algorithms >>>>>>>> that entirely avoid the need to accumulate such large (and therefore >>>>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>>>> following for 1D arrays in Python: >>>>>>>> >>>>>>>> def mean(a): >>>>>>>> m = a[0] >>>>>>>> for i in range(1, len(a)): >>>>>>>> m += (a[i] - m) / (i + 1) >>>>>>>> return m >>>>>>>> >>>>>>> This isn't really going to be any better than using a simple sum. >>>>>>> It'll also be slower (a division per iteration). >>>>>>> >>>>>> With one exception, every test that I've thrown at it shows that it's >>>>>> better for float32. That exception is uniformly spaced arrays, like >>>>>> linspace(). >>>>>> >>>>>> > You do avoid >>>>>> > accumulating large sums, but then doing the division a[i]/len(a) >>>>>> > and adding that will do the same. >>>>>> >>>>>> Okay, this is true. >>>>>> >>>>>> >>>>>>> Now, if you want to avoid losing precision, you want to use a better >>>>>>> summation technique, like compensated (or Kahan) summation: >>>>>>> >>>>>>> def mean(a): >>>>>>> s = e = a.dtype.type(0) >>>>>>> for i in range(0, len(a)): >>>>>>> temp = s >>>>>>> y = a[i] + e >>>>>>> s = temp + y >>>>>>> e = (temp - s) + y >>>>>>> return s / len(a) >>>>>>> >>>>>>> >>>>>>>> def var(a): >>>>>>>> m = a[0] >>>>>>>> t = a.dtype.type(0) >>>>>>>> for i in range(1, len(a)): >>>>>>>> q = a[i] - m >>>>>>>> r = q / (i+1) >>>>>>>> m += r >>>>>>>> t += i * q * r >>>>>>>> t /= len(a) >>>>>>>> return t >>>>>>>> >>>>>>>> Alternatively, from Knuth: >>>>>>>> >>>>>>>> def var_knuth(a): >>>>>>>> m = a.dtype.type(0) >>>>>>>> variance = a.dtype.type(0) >>>>>>>> for i in range(len(a)): >>>>>>>> delta = a[i] - m >>>>>>>> m += delta / (i+1) >>>>>>>> variance += delta * (a[i] - m) >>>>>>>> variance /= len(a) >>>>>>>> return variance >>>>>>>> >>>> I'm going to go ahead and attach a module containing the versions of >>>> mean, var, etc that I've been playing with in case someone wants to mess >>>> with them. Some were stolen from traffic on this list, for others I >>>> grabbed the algorithms from wikipedia or equivalent. >>>> >>> I looked into this a bit more. I checked float32 (single precision) and >>> float64 (double precision), using long doubles (float96) for the "exact" >>> results. This is based on your code. Results are compared using >>> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in >>> the range of [-100, 900] >>> >>> First, the mean. In float32, the Kahan summation in single precision is >>> better by about 2 orders of magnitude than simple summation. However, >>> accumulating the sum in double precision is better by about 9 orders of >>> magnitude than simple summation (7 orders more than Kahan). >>> >>> In float64, Kahan summation is the way to go, by 2 orders of magnitude. >>> >>> For the variance, in float32, Knuth's method is *no better* than the >>> two-pass method. Tim's code does an implicit conversion of intermediate >>> results to float64, which is why he saw a much better result. >>> >> Doh! And I fixed that same problem in the mean implementation earlier >> too. I was astounded by how good knuth was doing, but not astounded >> enough apparently. >> >> Does it seem weird to anyone else that in: >> numpy_scalar python_scalar >> the precision ends up being controlled by the python scalar? I would >> expect the numpy_scalar to control the resulting precision just like >> numpy arrays do in similar circumstances. Perhaps the egg on my face is >> just clouding my vision though. >> >> >>> The two-pass method using >>> Kahan summation (again, in single precision), is better by about 2 orders >>> of magnitude. There is practically no difference when using a >>> double-precision accumulator amongst the techniques: they're all about 9 >>> orders of magnitude better than single-precision two-pass. >>> >>> In float64, Kahan summation is again better than the rest, by about 2 >>> orders of magnitude. >>> >>> I've put my adaptation of Tim's code, and box-and-whisker plots of the >>> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ >>> >>> Conclusions: >>> >>> - If you're going to calculate everything in single precision, use Kahan >>> summation. Using it in double-precision also helps. >>> - If you can use a double-precision accumulator, it's much better than >>> any of the techniques in single-precision only. >>> >>> - for speed+precision in the variance, either use Kahan summation in >>> single precision with the two-pass method, or use double precision with >>> simple summation with the two-pass method. Knuth buys you nothing, except >>> slower code :-) >>> >> The two pass methods are definitely more accurate. I won't be convinced >> on the speed front till I see comparable C implementations slug it out. >> That may well mean never in practice. However, I expect that somewhere >> around 10,000 items, the cache will overflow and memory bandwidth will >> become the bottleneck. At that point the extra operations of Knuth won't >> matter as much as making two passes through the array and Knuth will win >> on speed. Of course the accuracy is pretty bad at single precision, so >> the possible, theoretical speed advantage at large sizes probably >> doesn't matter. >> >> -tim >> >> >>> After 1.0 is out, we should look at doing one of the above. >>> >> +1 >> >> > Hi, > I don't understand much of these "fancy algorithms", but does the above mean > that > after 1.0 is released the float32 functions will catch up in terms of > precision precision ("to some extend") with using dtype=float64 ? > It sounds like there is some consensus to do something to improve the precision after 1.0 comes out. I don't think the details are entirely nailed down, but it sounds like some combination of using Kahan summation and increasing the minimum precision of the accumulator is likely for sum, mean, var and std. -tim From Chris.Barker at noaa.gov Fri Sep 22 12:34:42 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 22 Sep 2006 09:34:42 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4513E057.7070906@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <4513E057.7070906@ieee.org> Message-ID: <451410A2.6060508@noaa.gov> Tim Hochberg wrote: > It would probably be nice to expose the > Kahan sum and maybe even the raw_kahan_sum somewhere. What about using it for .sum() by default? What is the speed hit anyway? In any case, having it available would be nice. > I'm on the fence on using the array dtype for the accumulator dtype > versus always using at least double precision for the accumulator. The > former is easier to explain and is probably faster, but the latter is a > lot more accuracy for basically free. I don't think the difficulty of explanation is a big issue at all -- I'm having a really hard time imagining someone getting confused and/or disappointed that their single precision calculation didn't overflow or was more accurate than expected. Anyone that did, would know enough to understand the explanation. In general, users expect things to "just work". They only dig into the details when something goes wrong, like the mean of a bunch of positive integers coming out as negative (wasn't that the OP's example?). The fewer such instance we have, the fewer questions we have. > speeds shake out I suppose. If the speed of using float64 is comparable > to that of using float32, we might as well. Only testing will tell, but my experience is that with double precision FPUs, doubles are just as fast as floats unless you're dealing with enough memory to make a difference in caching. In this case, only the accumulator is double, so that wouldn't be an issue. I suppose the float to double conversion could take some time though... > One thing I'm not on the > fence about is the return type: it should always match the input type, > or dtype if that is specified. +1 > Since numpy-scalars are > generally the results of indexing operations, not literals, I think that > they should behave like arrays for purposes of determining the resulting > precision, not like python-scalars. +1 >> Of course the accuracy is pretty bad at single precision, so >> the possible, theoretical speed advantage at large sizes probably >> doesn't matter. good point. the larger the array -- the more accuracy matters. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Sep 22 13:13:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 11:13:42 -0600 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: Message-ID: <451419C6.1040909@ieee.org> Christian Kristukat wrote: > Bill Baxter gmail.com> writes: > > >> Yep, check the release notes: >> http://www.scipy.org/ReleaseNotes/NumPy_1.0 >> search for 'take' on that page to find out what others have changed as well. >> --bb >> > > Ok. Does axis=None then mean, that take(a, ind) operates on the flattened array? > This it at least what it seem to be. I noticed that the ufunc behaves > differently. a.take(ind) and a.take(ind, axis=0) behave the same, so the default > argument to axis is 0 rather than None. > What do you mean. There is no "ufunc" take. There is a function take that just calls the method. The default arguments for all functions that match methods are the same as the methods (which means axis=None). However, in oldnumeric (which pylab imports by the way), the default axes are the same as they were in Numeric. Also, if you have a 1-d array, then the axis argument doesn't make any difference. Please clarify what you are saying to be sure we don't have a bug floating around. -Travis From oliphant.travis at ieee.org Fri Sep 22 13:24:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 11:24:00 -0600 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <20060922082157.GA21884@mentat.za.net> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> <20060922082157.GA21884@mentat.za.net> Message-ID: <45141C30.60504@ieee.org> Stefan van der Walt wrote: > On Fri, Sep 22, 2006 at 02:17:57AM -0500, Robert Kern wrote: > >>> According to the putmask docstring: >>> >>> a.putmask(values, mask) sets a.flat[n] = v[n] for each n where >>> mask.flat[n] is true. v can be scalar. >>> >>> This would mean that 'w' is not of the right length. >>> >> There are 4 true values in m and 4 values in w. What's the wrong >> > length? > > The way I read the docstring, you use putmask like this: > > In [4]: x = N.array([1,2,3,4]) > > In [5]: x.putmask([4,3,2,1],[1,0,0,1]) > > In [6]: x > Out[6]: array([4, 2, 3, 1]) > > >> >> Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. >> It's pretty clear that this is a bug in .putmask(). >> > > Based purely on what I read in the docstring, I would expect the above to do > > x[0] = w[0] > x[6] = w[6] > x[9] = w[9] > x[11] = w[11] > > Since w is of length 4, you'll probably get indices modulo 4: > > w[6] == w[2] == -3 > w[9] == w[1] == -2 > w[11] == w[3] == -4 > > Which seems to explain what you are seeing. > Yes, this does explain what you are seeing. It is the behavior of Numeric's putmask (where this method came from). It does seem counter-intuitive, and I'm not sure what to do with it. In some sense putmask should behave the same as x[m] = w. But, on the other-hand, was anybody actually using the "modular" indexing "feature" of "putmask". Here are our options: 1) "fix-it" and risk breaking code for people who used putmask and the modular indexing "feature," 2) Get rid of it as a method (and keep it as a function so that oldnumeric can use it.) 3) Keep everything the way it is. -Travis From sransom at nrao.edu Fri Sep 22 13:36:48 2006 From: sransom at nrao.edu (Scott Ransom) Date: Fri, 22 Sep 2006 13:36:48 -0400 Subject: [Numpy-discussion] bug in oldnumeric module Message-ID: <200609221336.48144.sransom@nrao.edu> argmin is currently defined as using the argmax method! From numpy/oldnumeric/functions.py def argmin(x, axis=-1): return N.argmax(x, axis) Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From ellisonbg.net at gmail.com Fri Sep 22 13:41:33 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 22 Sep 2006 11:41:33 -0600 Subject: [Numpy-discussion] Always using scientific notation to print Message-ID: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> I want to be able to print an array in scientific notation. I have seen the set_printoptions() functions, but it doesn't really have an option for *always* using scientific notation. Can this be done? How? Thanks Brian From charlesr.harris at gmail.com Fri Sep 22 14:14:22 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 22 Sep 2006 12:14:22 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451409F2.8010805@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> <451409F2.8010805@ieee.org> Message-ID: On 9/22/06, Tim Hochberg wrote: > > Sebastian Haase wrote: > > On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > > > >> David M. Cooke wrote: > >> > >>> On Thu, 21 Sep 2006 11:34:42 -0700 > >>> > >>> Tim Hochberg wrote: > >>> > >>>> Tim Hochberg wrote: > >>>> > >>>>> Robert Kern wrote: > >>>>> > >>>>>> David M. Cooke wrote: > >>>>>> > >>>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>>> > >>>>>>>> Let me offer a third path: the algorithms used for .mean() and > >>>>>>>> .var() are substandard. There are much better incremental > algorithms > >>>>>>>> that entirely avoid the need to accumulate such large (and > therefore > >>>>>>>> precision-losing) intermediate values. The algorithms look like > the > >>>>>>>> following for 1D arrays in Python: > >>>>>>>> > >>>>>>>> def mean(a): > >>>>>>>> m = a[0] > >>>>>>>> for i in range(1, len(a)): > >>>>>>>> m += (a[i] - m) / (i + 1) > >>>>>>>> return m > >>>>>>>> > >>>>>>> This isn't really going to be any better than using a simple sum. > >>>>>>> It'll also be slower (a division per iteration). > >>>>>>> > >>>>>> With one exception, every test that I've thrown at it shows that > it's > >>>>>> better for float32. That exception is uniformly spaced arrays, like > >>>>>> linspace(). > >>>>>> > >>>>>> > You do avoid > >>>>>> > accumulating large sums, but then doing the division a[i]/len(a) > >>>>>> > and adding that will do the same. > >>>>>> > >>>>>> Okay, this is true. > >>>>>> > >>>>>> > >>>>>>> Now, if you want to avoid losing precision, you want to use a > better > >>>>>>> summation technique, like compensated (or Kahan) summation: > >>>>>>> > >>>>>>> def mean(a): > >>>>>>> s = e = a.dtype.type(0) > >>>>>>> for i in range(0, len(a)): > >>>>>>> temp = s > >>>>>>> y = a[i] + e > >>>>>>> s = temp + y > >>>>>>> e = (temp - s) + y > >>>>>>> return s / len(a) > >>>>>>> > >>>>>>> > >>>>>>>> def var(a): > >>>>>>>> m = a[0] > >>>>>>>> t = a.dtype.type(0) > >>>>>>>> for i in range(1, len(a)): > >>>>>>>> q = a[i] - m > >>>>>>>> r = q / (i+1) > >>>>>>>> m += r > >>>>>>>> t += i * q * r > >>>>>>>> t /= len(a) > >>>>>>>> return t > >>>>>>>> > >>>>>>>> Alternatively, from Knuth: > >>>>>>>> > >>>>>>>> def var_knuth(a): > >>>>>>>> m = a.dtype.type(0) > >>>>>>>> variance = a.dtype.type(0) > >>>>>>>> for i in range(len(a)): > >>>>>>>> delta = a[i] - m > >>>>>>>> m += delta / (i+1) > >>>>>>>> variance += delta * (a[i] - m) > >>>>>>>> variance /= len(a) > >>>>>>>> return variance > >>>>>>>> > >>>> I'm going to go ahead and attach a module containing the versions of > >>>> mean, var, etc that I've been playing with in case someone wants to > mess > >>>> with them. Some were stolen from traffic on this list, for others I > >>>> grabbed the algorithms from wikipedia or equivalent. > >>>> > >>> I looked into this a bit more. I checked float32 (single precision) > and > >>> float64 (double precision), using long doubles (float96) for the > "exact" > >>> results. This is based on your code. Results are compared using > >>> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values > in > >>> the range of [-100, 900] > >>> > >>> First, the mean. In float32, the Kahan summation in single precision > is > >>> better by about 2 orders of magnitude than simple summation. However, > >>> accumulating the sum in double precision is better by about 9 orders > of > >>> magnitude than simple summation (7 orders more than Kahan). > >>> > >>> In float64, Kahan summation is the way to go, by 2 orders of > magnitude. > >>> > >>> For the variance, in float32, Knuth's method is *no better* than the > >>> two-pass method. Tim's code does an implicit conversion of > intermediate > >>> results to float64, which is why he saw a much better result. > >>> > >> Doh! And I fixed that same problem in the mean implementation earlier > >> too. I was astounded by how good knuth was doing, but not astounded > >> enough apparently. > >> > >> Does it seem weird to anyone else that in: > >> numpy_scalar python_scalar > >> the precision ends up being controlled by the python scalar? I would > >> expect the numpy_scalar to control the resulting precision just like > >> numpy arrays do in similar circumstances. Perhaps the egg on my face is > >> just clouding my vision though. > >> > >> > >>> The two-pass method using > >>> Kahan summation (again, in single precision), is better by about 2 > orders > >>> of magnitude. There is practically no difference when using a > >>> double-precision accumulator amongst the techniques: they're all about > 9 > >>> orders of magnitude better than single-precision two-pass. > >>> > >>> In float64, Kahan summation is again better than the rest, by about 2 > >>> orders of magnitude. > >>> > >>> I've put my adaptation of Tim's code, and box-and-whisker plots of the > >>> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > >>> > >>> Conclusions: > >>> > >>> - If you're going to calculate everything in single precision, use > Kahan > >>> summation. Using it in double-precision also helps. > >>> - If you can use a double-precision accumulator, it's much better than > >>> any of the techniques in single-precision only. > >>> > >>> - for speed+precision in the variance, either use Kahan summation in > >>> single precision with the two-pass method, or use double precision > with > >>> simple summation with the two-pass method. Knuth buys you nothing, > except > >>> slower code :-) > >>> > >> The two pass methods are definitely more accurate. I won't be convinced > >> on the speed front till I see comparable C implementations slug it out. > >> That may well mean never in practice. However, I expect that somewhere > >> around 10,000 items, the cache will overflow and memory bandwidth will > >> become the bottleneck. At that point the extra operations of Knuth > won't > >> matter as much as making two passes through the array and Knuth will > win > >> on speed. Of course the accuracy is pretty bad at single precision, so > >> the possible, theoretical speed advantage at large sizes probably > >> doesn't matter. > >> > >> -tim > >> > >> > >>> After 1.0 is out, we should look at doing one of the above. > >>> > >> +1 > >> > >> > > Hi, > > I don't understand much of these "fancy algorithms", but does the above > mean > > that > > after 1.0 is released the float32 functions will catch up in terms of > > precision precision ("to some extend") with using dtype=float64 ? > > > It sounds like there is some consensus to do something to improve the > precision after 1.0 comes out. I don't think the details are entirely > nailed down, but it sounds like some combination of using Kahan > summation and increasing the minimum precision of the accumulator is > likely for sum, mean, var and std. I would go with double as the default except when the base precision is greater. Higher precisions should be available as a check, i.e., if the results look funny use Kahan or extended to see if roundoff is a problem. I think Kahan should be available because it adds precision to *any* base precision, even extended or quad, so there is always something to check against. But I don't think Kahan should be the default because of the speed hit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 22 14:27:49 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 13:27:49 -0500 Subject: [Numpy-discussion] bug in oldnumeric module In-Reply-To: <200609221336.48144.sransom@nrao.edu> References: <200609221336.48144.sransom@nrao.edu> Message-ID: Scott Ransom wrote: > argmin is currently defined as using the argmax method! Please check out the latest source from SVN. I fixed this a few days ago. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Sep 22 14:30:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 13:30:06 -0500 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> Message-ID: Brian Granger wrote: > I want to be able to print an array in scientific notation. > > I have seen the set_printoptions() functions, but it doesn't really > have an option for *always* using scientific notation. > > Can this be done? How? You can write a function that formats arrays how you like them and then tell ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Fri Sep 22 14:44:16 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 22 Sep 2006 12:44:16 -0600 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> Message-ID: <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> > You can write a function that formats arrays how you like them and then tell > ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). That seems to be a little low level for most users. Would it be hard to have the possibility of specifying a format string? Brian From pgmdevlist at gmail.com Fri Sep 22 03:35:41 2006 From: pgmdevlist at gmail.com (PGM) Date: Fri, 22 Sep 2006 03:35:41 -0400 Subject: [Numpy-discussion] putmask/take ? In-Reply-To: <20060922071302.GA17679@mentat.za.net> References: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> <20060922071302.GA17679@mentat.za.net> Message-ID: <200609220335.42110.pgmdevlist@gmail.com> Stefan, Thanks for your suggestions, but that won't do for what I'm working on : I need to get putmask working, or at least knowing it doesnt'. Robert, thanks for your input. The function putmask doesn't work either. Oh, thinking about it: would it be possible to have the same order of arguments in the function and the method ? Compare putmask(array, mask, values) with array.putmask(values, mask) The same comment applies to put: put(array, indices, values) vs array.put(values, indices, mode) That's a tad confusing... From oliphant at ee.byu.edu Fri Sep 22 14:55:45 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 12:55:45 -0600 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> Message-ID: <451431B1.2020004@ee.byu.edu> Brian Granger wrote: >>You can write a function that formats arrays how you like them and then tell >>ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). >> >> > >That seems to be a little low level for most users. Would it be hard >to have the possibility of specifying a format string? > > No, that woudn't be hard. If you can wait several weeks, add a ticket, or better, dig in to the current printing function and add it to the set_printoptions code. -Travis From tim.hochberg at ieee.org Fri Sep 22 15:18:07 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 12:18:07 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> <451409F2.8010805@ieee.org> Message-ID: <451436EF.1070005@ieee.org> Charles R Harris wrote: [CHOP] > On 9/22/06, *Tim Hochberg* > wrote: > > > It sounds like there is some consensus to do something to improve the > precision after 1.0 comes out. I don't think the details are entirely > nailed down, but it sounds like some combination of using Kahan > summation and increasing the minimum precision of the accumulator is > likely for sum, mean, var and std. > > > I would go with double as the default except when the base precision > is greater. Higher precisions should be available as a check, i.e., if > the results look funny use Kahan or extended to see if roundoff is a > problem. I think Kahan should be available because it adds precision > to *any* base precision, even extended or quad, so there is always > something to check against. But I don't think Kahan should be the > default because of the speed hit. > Note that add.reduce will be available for doing simple-fast-reductions, so I consider some speed hit as acceptable. I'll be interested to see what the actual speed difference between Kahan and straight summation actually are. It may also be possible to unroll the core loop cleverly so that multiple ops can be scheduled on the FPU in parallel. It doesn't look like naive unrolling will work since each iteration depends on the values of the previous one. However, it probably doesn't matter much where the low order bits get added back in, so there's some flexibility in how the loop gets unrolled. -tim PS. Sorry about the untrimmed posts. From oliphant at ee.byu.edu Fri Sep 22 15:29:08 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 13:29:08 -0600 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <45141C30.60504@ieee.org> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> <20060922082157.GA21884@mentat.za.net> <45141C30.60504@ieee.org> Message-ID: <45143984.4060607@ee.byu.edu> Travis Oliphant wrote: >Yes, this does explain what you are seeing. It is the behavior of >Numeric's putmask (where this method came from). It does seem >counter-intuitive, and I'm not sure what to do with it. In some sense >putmask should behave the same as x[m] = w. But, on the other-hand, >was anybody actually using the "modular" indexing "feature" of "putmask". > >Here are our options: > >1) "fix-it" and risk breaking code for people who used putmask and the >modular indexing "feature," >2) Get rid of it as a method (and keep it as a function so that >oldnumeric can use it.) >3) Keep everything the way it is. > > O.K. putmask is keeping the same behavior (it is intentional behavior and has been for a long time). However, because of the confusion, it is being removed as a method (I know this is late, but it's a little-used method and was only added by NumPy). Putmask will be a function. Also, the remaining put method will have it's arguments switched to match the function. IIRC the original switch was made to accomodate masked arrays methods of the same name. But, this does not really help anyway since arr.put for a masked array doesn't even take an indicies argument, so confusing users who don't even use masked arrays seems pointless. Scream now if you think I'm being unreasonable. This will be in 1.0rc2 (yes we will need rc2) The original poster has still not explained why a[mask] = values does not work suitably. -Travis From mike.ressler at alum.mit.edu Fri Sep 22 15:37:37 2006 From: mike.ressler at alum.mit.edu (Mike Ressler) Date: Fri, 22 Sep 2006 12:37:37 -0700 Subject: [Numpy-discussion] reading large files In-Reply-To: References: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> Message-ID: <268febdf0609221237y5ccc1a3cj7bed84eadac44503@mail.gmail.com> On 9/22/06, Bill Baxter wrote: > > Do you also have a 64-bit processor? Just checking since you didn't > mention it. > --bb > > On 9/22/06, Svein-Erik.Hamran at ffi.no wrote: > > I would like to read files >2Gbyte. From earlier posting I believed it > > should be possible with python 2.5. > > > > I am getting MemoryError when trying to read a file larger than 2Gb. > > > > I am using python2.5 and numpy1.0rc1. > Also, how much memory do you have? You will have to use memory mapping as well if the file size is greater than your memory size. But you should be hopeful. I have been working with > 10 GB files for some time using beta versions of Python 2.5 and numpy-1.0 on a 64-bit AMD machine with 4 GB of memory. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 22 15:54:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 14:54:04 -0500 Subject: [Numpy-discussion] Question about recarray In-Reply-To: References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> <200609220934.31217.lroubeyrie@limair.asso.fr> Message-ID: Robert Kern wrote: > Ah, it's Travis's fault. So he can fix it. :-) And lo, it was fixed. Amen. http://projects.scipy.org/scipy/scipy/changeset/2217 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Sep 22 15:57:10 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 13:57:10 -0600 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: References: Message-ID: <45144016.4060504@ee.byu.edu> Bill Baxter wrote: >26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why >atleast_3d acts the way it does: >http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d > >He doesn't seem to have gotten any answers. And now I'm wondering the >same thing. Anyone have any idea? > > This function came from scipy and was written by somebody at Enthought. I was hoping they would respond. The behavior of atleast_3d does make sense in the context of atleast_2d and thinking of 3-d arrays as "stacks" of 2-d arrays where the stacks are in the last dimension. atleast_2d converts 1-d arrays to 1xN arrays atleast_3d converts 1-d arrays to 1xNx1 arrays so that they can be "stacked" in the last dimension. I agree that this isn't consistent with the general notion of "pre-pending" 1's to increase the dimensionality of the array. However, array(a, copy=False, ndmin=3) will always produce arrays with a 1 at the begining. So at_least3d is convenient if you like to think of 3-d arrays of stacks of 2-d arrays where the last axis is the "stacking" dimension. -Travis From robert.kern at gmail.com Fri Sep 22 17:14:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 16:14:47 -0500 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <45144016.4060504@ee.byu.edu> References: <45144016.4060504@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Bill Baxter wrote: > >> 26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why >> atleast_3d acts the way it does: >> http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d >> >> He doesn't seem to have gotten any answers. And now I'm wondering the >> same thing. Anyone have any idea? >> > This function came from scipy and was written by somebody at Enthought. > I was hoping they would respond. That would have either been from Eric or Travis V way back in the day. Both are out of the office today. The "3D arrays as stacks of 2D arrays" is probably as good an explanation as any. Much of Numeric (e.g. the axis dot() reduces over) was predicated on that kind of logic. There is probably some kind of consistency argument with dstack() and atleast_2d(), i.e. In [46]: a = arange(5) In [47]: atleast_3d(a) Out[47]: array([[[0], [1], [2], [3], [4]]]) In [48]: dstack([a, a]) Out[48]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) In [49]: dstack(map(atleast_2d, [a, a])) Out[49]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) In [50]: dstack(map(atleast_3d, [a, a])) Out[50]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) That's the problem with consistency arguments; there are so many things one could be consistent with! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Fri Sep 22 17:50:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 22 Sep 2006 14:50:04 -0700 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <45144016.4060504@ee.byu.edu> References: <45144016.4060504@ee.byu.edu> Message-ID: <200609221450.04587.haase@msg.ucsf.edu> On Friday 22 September 2006 12:57, Travis Oliphant wrote: > Bill Baxter wrote: > >26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why > >atleast_3d acts the way it does: > >http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atle > >ast+3d > > > >He doesn't seem to have gotten any answers. And now I'm wondering the > >same thing. Anyone have any idea? > > This function came from scipy and was written by somebody at Enthought. > I was hoping they would respond. The behavior of atleast_3d does make > sense in the context of atleast_2d and thinking of 3-d arrays as > "stacks" of 2-d arrays where the stacks are in the last dimension. > > atleast_2d converts 1-d arrays to 1xN arrays > > atleast_3d converts 1-d arrays to 1xNx1 arrays so that they can be > "stacked" in the last dimension. I agree that this isn't consistent > with the general notion of "pre-pending" 1's to increase the > dimensionality of the array. > > However, array(a, copy=False, ndmin=3) will always produce arrays with > a 1 at the begining. So at_least3d is convenient if you like to think > of 3-d arrays of stacks of 2-d arrays where the last axis is the > "stacking" dimension. > I'm working with "stacks of 2d arrays" -- but I was always under the impression that -- since the last axis is the "fastest" -- "stacks of " should stack in the first dimension -- not the last --so that the members of the stack still remain contiguous is memory. Is there a general consensus on how one should look at this ? Or are there multiple incompatible view -- maybe coming from different fields -- ? -Sebastian From ckkart at hoc.net Fri Sep 22 22:00:04 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Sat, 23 Sep 2006 02:00:04 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed References: <451419C6.1040909@ieee.org> Message-ID: Travis Oliphant ieee.org> writes: > > Christian Kristukat wrote: > > Bill Baxter gmail.com> writes: > > > > > >> Yep, check the release notes: > >> http://www.scipy.org/ReleaseNotes/NumPy_1.0 > >> search for 'take' on that page to find out what others have changed as well. > >> --bb > >> > > > > Ok. Does axis=None then mean, that take(a, ind) operates on the > > flattened array? > > This it at least what it seem to be. I noticed that the ufunc behaves > > differently. a.take(ind) and a.take(ind, axis=0) behave the same, so > > the default > > argument to axis is 0 rather than None. > > > > What do you mean. There is no "ufunc" take. There is a function take > that just calls the method. The default arguments for all functions Sorry, I never really read about what are ufuncs. I thought those are class methods of the ndarray objects... Anyway, I was refering to the following difference: In [7]: a Out[7]: array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11]]) In [8]: a.take([0]) Out[8]: array([[0, 1, 2, 3, 4, 5]]) In [9]: take(a,[0]) Out[9]: array([0]) To be sure I understood: Does axis=None then mean, that take operates on the flattened array? Christian From oliphant.travis at ieee.org Fri Sep 22 23:31:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 21:31:55 -0600 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: <451419C6.1040909@ieee.org> Message-ID: <4514AAAB.3010906@ieee.org> Christian Kristukat wrote: >>> Ok. Does axis=None then mean, that take(a, ind) operates on the >>> flattened array? >>> Yes, that is correct. > Sorry, I never really read about what are ufuncs. I thought those are class > methods of the ndarray objects... Anyway, I was refering to the following > difference: > > In [7]: a > Out[7]: > array([[ 0, 1, 2, 3, 4, 5], > [ 6, 7, 8, 9, 10, 11]]) > > In [8]: a.take([0]) > Out[8]: array([[0, 1, 2, 3, 4, 5]]) > > In [9]: take(a,[0]) > Out[9]: array([0]) > Doh!. That is a bug. take(a,[0]) is correct a.take([0]) is not correct. -Travis From wbaxter at gmail.com Fri Sep 22 23:53:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 23 Sep 2006 12:53:02 +0900 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <200609221450.04587.haase@msg.ucsf.edu> References: <45144016.4060504@ee.byu.edu> <200609221450.04587.haase@msg.ucsf.edu> Message-ID: Thanks for the explanations, folks, I thought that might be the case. On 9/23/06, Sebastian Haase wrote: > I'm working with "stacks of 2d arrays" -- but I was always under the > impression that -- since the last axis is the "fastest" -- "stacks of > " should stack in the first dimension -- not the last --so that > the members of the stack still remain contiguous is memory. I also don't see why you'd want to stack matrices along the last axis in numpy. C-style memory order is one reason not to, but printing as well seems to favor stacking matrices along the first (0th) axis instead of the last. > > Is there a general consensus on how one should look at this ? Or are there > multiple incompatible view -- maybe coming from different fields -- ? I've beed stacking along the first axis myself. Perhaps the original folks who implemented atleast_3d were coming from Matlab, where stacking is generally done along the last axis. But everything in Matlab is Fortran order under the hood so it makes sense there. Anyway, glad to know it was just the implementer's idiosyncracy rather than some deep secret of the universe that I wasn't aware of. I sometimes wonder what sort of overhead is induced by numpy using primarily C-style while many of the underlying algorithms we use are in Fortran. I suspect f2py has to do a lot of copying to get things in the right order for making the calls to Fortran. --bb From matthew.brett at gmail.com Sat Sep 23 07:36:58 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 23 Sep 2006 12:36:58 +0100 Subject: [Numpy-discussion] dtype truth value Message-ID: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> Hi, Forgive my ignorance, but why is this? In [1]:from numpy import * In [2]:if not dtype(' References: Message-ID: Here's a new version of 'reparray'. I tested the previous version against 'repmat' for the 2d case and found it to be as much as 3x slower in some instances. Ick. So I redid it using the approach repmat uses -- reshape() and repeat() rather than concatenate(). Now it's very nearly as fast as repmat for the 2-d case. def reparray(A, tup): if numpy.isscalar(tup): tup = (tup,) d = len(tup) c = numpy.array(A,copy=False,subok=True,ndmin=d) shape = list(c.shape) n = c.size for i, nrep in enumerate(tup): if nrep!=1: c = c.reshape(-1,n).repeat(nrep,0) dim_in = shape[i] dim_out = dim_in*nrep shape[i] = dim_out n /= dim_in return c.reshape(shape) The full file with tests and timing code is attached. One thing I noticed while doing this was that repmat doesn't preserve subclasses. Which is funny considering 'mat' is part of the name and it only works on 2-d arrays. Using asanyarray() instead of asarray() in the first line should fix that. --Bill -------------- next part -------------- A non-text attachment was scrubbed... Name: play.py Type: text/x-python Size: 5486 bytes Desc: not available URL: From wvtyix at buyevent.com Sat Sep 23 12:43:36 2006 From: wvtyix at buyevent.com (Mat Foster) Date: Sat, 23 Sep 2006 16:43:36 -0000 Subject: [Numpy-discussion] wade puma Message-ID: <001401c624fa$f418fd13$9490d551@nawres> And bonus news, the story is already being translated to the big screen. I come down, with some vested interest here, I suppose, on the side of eccentric vision. ML: The Lord Soho series held some rather dark and acrid views of your fellow Englishmen. However, as a study in social behavior, I think its a neat site. I put the work into it and it stayed in the bestseller list for three months. You tune into us to hear us chat with science fiction and fantasy authors primarily. The problem, as Ive suggested, is a general one. However, as a study in social behavior, I think its a neat site. My piece is entitled The Last Frequency, and it was a lot of fun to do. Lorrie and Sheila talk about subjectivity and required reading list and interview Kathe Koja. got produced because my editor fell in love with the idea. If youve got the time and the inclination, please do so. For many underpublished authors, obscurity remains the number one obstacle to overcome. Swarming bees are usually incredibly docile, at least for a few hours. Yes, hes a friend of mine, and Im pimping the hell out of the book. And at such times she is as uncompromisingly physical as she is real. The problem, as Ive suggested, is a general one. The novella is also available to read online at Robs website. Gotta say I wont be complaining about this weather. No hard feelings, even if you knock the tar out of me. Create your own superhero or supervillian I wrote it because the fans wanted another Rincewind novel. And it looks like he just might have a taker. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pixel.gif Type: image/gif Size: 7900 bytes Desc: not available URL: From oliphant.travis at ieee.org Sat Sep 23 14:42:09 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 23 Sep 2006 12:42:09 -0600 Subject: [Numpy-discussion] dtype truth value In-Reply-To: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> References: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> Message-ID: <45158001.5080101@ieee.org> Matthew Brett wrote: > Hi, > > Forgive my ignorance, but why is this? > > In [1]:from numpy import * > > In [2]:if not dtype(' ...: print 'Truth value is no' > ...: > ...: > Truth value is no > Truth value of user-built Python objects is tested by looking at: 1) __nonzero__ method is called to determine truth value. 2) sequence or mapping behavior (then the length is used. If the length is greater than 0, then True, otherwise it's False. For data-type objects, there is no __nonzero__ method, but it does have "mapping" behavior so that fields can be extracted from a data-type using mapping notation. The "length" of the data-type is the number of defined fields. Therefore, if the data-type has no defined fields, it's length is 0 and it's truth value is "False". So, you can think of the test as if dtype(...): print "Data-type has fields:" else: print "Data-type does not have fields:" -Travis From matthew.brett at gmail.com Sat Sep 23 15:15:57 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 23 Sep 2006 20:15:57 +0100 Subject: [Numpy-discussion] dtype truth value In-Reply-To: <45158001.5080101@ieee.org> References: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> <45158001.5080101@ieee.org> Message-ID: <1e2af89e0609231215p456fbef2m5fe9e42c9060fb5d@mail.gmail.com> > So, you can think of the test as > > if dtype(...): > print "Data-type has fields:" > else: > print "Data-type does not have fields:" Thank you as ever for your very clear and helpful explanation, Best, Matthew From amcmorl at gmail.com Sun Sep 24 19:32:21 2006 From: amcmorl at gmail.com (Angus McMorland) Date: Mon, 25 Sep 2006 11:32:21 +1200 Subject: [Numpy-discussion] Shift or rotate command? Message-ID: Hi all, Other data languages that I have worked with have a routine for shifting data along axes, with wrapping. In IDL it's called 'shift', and works such that print, a 0 1 2 3 4 5 6 7 8 9 10 11 print, shift(a, 2, 0) 2 3 0 1 6 7 4 5 10 11 8 9 print, shift(a, 2, 1) 10 11 8 9 2 3 0 1 6 7 4 5 In pdl (pdl.perl.org) the equivalent routine is called rotate. Is there a version of this in numpy, or can it be easily achieved using existing technology - I could do it with multiple sliced assignment statements but that seems very clunky and likely slow. I've looked through the examples and the numpy book but without success. Angus. -- AJC McMorland, PhD Student Physiology, University of Auckland Armourer, Auckland University Fencing Secretary, Fencing North Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sun Sep 24 20:44:25 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 09:44:25 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Howdy Angus, Yeh, that does seem like a hole in the API. Travis added a rollaxis() but there's still no simple way to roll the elements themselves. I took a look at numpy.fft.fftshift, which is a function that has to do a similar thing. It does it by concatenating aranges and then doing a take(). Presumably Pearu knew what he was doing when he wrote that, so we can assume this is probably close to the best possible. :-) From that idea here's a function that implements roll(). def roll(y,shift,axis): """Roll the elements in the array by 'shift' positions along the given axis.""" from numpy import asanyarray,concatenate,arange y = asanyarray(y) n = y.shape[axis] shift %= n # does the right thing for negative shifts, too return y.take(concatenate((arange(shift,n),arange(shift))), axis) Performance is only very slightly worse (<1%) using return y.take(r_[shift:n, :shift], axis) as the last line, which is maybe slightly more readable if you're down wit' the r_. I prefer the name 'roll' because: a) there's already rollaxis with conceptually similar behavior and b) 'shift' is used in other languages (perl comes to mind) sometimes to indicate shifting off the end, possibly filling in with zeros or just leaving the array shorter. c) 'rotate' seems potentially confusing given the existence of the 'rot90' function already. --bb On 9/25/06, Angus McMorland wrote: > Hi all, > > Other data languages that I have worked with have a routine for shifting > data along axes, with wrapping. > In IDL it's called 'shift', and works such that > print, a > 0 1 2 3 > 4 5 6 7 > 8 9 10 11 > > print, shift(a, 2, 0) > 2 3 0 1 > 6 7 4 5 > 10 11 8 9 > > print, shift(a, 2, 1) > 10 11 8 9 > 2 3 0 1 > 6 7 4 5 > > In pdl (pdl.perl.org) the equivalent routine is called rotate. Is there a > version of this in numpy, or can it be easily achieved using existing > technology - I could do it with multiple sliced assignment statements but > that seems very clunky and likely slow. I've looked through the examples and > the numpy book but without success. > > Angus. From charlesr.harris at gmail.com Sun Sep 24 20:51:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 24 Sep 2006 18:51:23 -0600 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: On 9/24/06, Bill Baxter wrote: > > Howdy Angus, > Yeh, that does seem like a hole in the API. Travis added a rollaxis() > but there's still no simple way to roll the elements themselves. > > I took a look at numpy.fft.fftshift, which is a function that has to > do a similar thing. It does it by concatenating aranges and then > doing a take(). Presumably Pearu knew what he was doing when he wrote > that, so we can assume this is probably close to the best possible. > :-) From that idea here's a function that implements roll(). > > def roll(y,shift,axis): > """Roll the elements in the array by 'shift' positions along the > given axis.""" > from numpy import asanyarray,concatenate,arange > y = asanyarray(y) > n = y.shape[axis] > shift %= n # does the right thing for negative shifts, too > return y.take(concatenate((arange(shift,n),arange(shift))), axis) It is possible to do a shift inplace using two reflections implemented with swaps. This works because two reflections is the same as a rotating twice the distance between the centers of the reflections. I don't know if it is worth implementing this, however. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sun Sep 24 21:12:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 10:12:43 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Went ahead and added an enhancement request: http://projects.scipy.org/scipy/numpy/ticket/293 This is something I've wanted in the past too. --bb On 9/25/06, Charles R Harris wrote: > > > On 9/24/06, Bill Baxter wrote: > > Howdy Angus, > > Yeh, that does seem like a hole in the API. Travis added a rollaxis() > > but there's still no simple way to roll the elements themselves. > > > > I took a look at numpy.fft.fftshift, which is a function that has to > > do a similar thing. It does it by concatenating aranges and then > > doing a take(). Presumably Pearu knew what he was doing when he wrote > > that, so we can assume this is probably close to the best possible. > > :-) From that idea here's a function that implements roll(). > > > > def roll(y,shift,axis): > > """Roll the elements in the array by 'shift' positions along the > > given axis.""" > > from numpy import asanyarray,concatenate,arange > > y = asanyarray(y) > > n = y.shape[axis] > > shift %= n # does the right thing for negative shifts, too > > return > y.take(concatenate((arange(shift,n),arange(shift))), axis) > > It is possible to do a shift inplace using two reflections implemented with > swaps. This works because two reflections is the same as a rotating twice > the distance between the centers of the reflections. I don't know if it is > worth implementing this, however. > > Chuck From aisaac at american.edu Sun Sep 24 21:41:22 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 24 Sep 2006 21:41:22 -0400 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: On Mon, 25 Sep 2006, Bill Baxter apparently wrote: > Went ahead and added an enhancement request: > http://projects.scipy.org/scipy/numpy/ticket/293 > This is something I've wanted in the past too. GAUSS draws a useful distinction between "shifting" and "rotating": Works roughly like this for the 2D case: #rotater: rotate row elements # Format: y = rotater(x,r) # rotater(x,r) # Input: x RxK array # rotateby size R integer array, or integer (rotation amounts) # Output: y RxK array: # rows rotated by rotateby #shiftr: shift row elements and fill with fv # Format: y = shiftr(x,shiftby,fv) # Input: x RxC array # shiftby Rx1 array or scalar (shift amounts) # fv Rx1 array or scalar (fill values) # Output: y RxC array: # rows shifted by shiftby # rows filled with fill fwiw, Alan Isaac From wbaxter at gmail.com Sun Sep 24 22:04:15 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 11:04:15 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Hmm. Yes maybe a shift w/fill would be useful too. I can't recall needing such a thing, but it could probably also be implemented easily in a way similar to roll() above. A multi-axis roll might also be nice. Could just allow roll's shiftby and axis args to be tuples. Looking closer at the existing 'rollaxis', numpy.rollaxis(a, axis, start=0) if a.shape is (3,4,5,6) rollaxis(a, 3, 1).shape is (3,6,4,5) rollaxis(a, 2, 0).shape is (5,3,4,6) rollaxis(a, 1, 3).shape is (3,5,4,6) rollaxis(a, 1, 4).shape is (3,5,6,4) it occurs to me that what it is actually doing is not what we've been calling 'rolling'. It's just a move, really (remove value from one place and re-insert in another). So perhaps the name should be 'moveaxis' instead? --bb On 9/25/06, Alan G Isaac wrote: > On Mon, 25 Sep 2006, Bill Baxter apparently wrote: > > Went ahead and added an enhancement request: > > http://projects.scipy.org/scipy/numpy/ticket/293 > > This is something I've wanted in the past too. > > > GAUSS draws a useful distinction between "shifting" and > "rotating": > > Works roughly like this for the 2D case: > > #rotater: rotate row elements > # Format: y = rotater(x,r) > # rotater(x,r) > # Input: x RxK array > # rotateby size R integer array, or integer (rotation amounts) > # Output: y RxK array: > # rows rotated by rotateby > > #shiftr: shift row elements and fill with fv > # Format: y = shiftr(x,shiftby,fv) > # Input: x RxC array > # shiftby Rx1 array or scalar (shift amounts) > # fv Rx1 array or scalar (fill values) > # Output: y RxC array: > # rows shifted by shiftby > # rows filled with fill > > fwiw, > Alan Isaac From amcmorl at gmail.com Mon Sep 25 00:37:35 2006 From: amcmorl at gmail.com (Angus McMorland) Date: Mon, 25 Sep 2006 16:37:35 +1200 Subject: [Numpy-discussion] eval shortcomings? Message-ID: Hi all, Can someone explain why the following occurs? a = numpy.zeros((100)) b = numpy.ones((10)) a[20:30] = b # okay eval('a[50:60] = b') # raises SyntaxError: invalid syntax Is there some line mangling that the interpretor does that eval doesn't do? I see there's room for global and local parameters to eval. Would setting one of those be the answer? Cheers, Angus. -- AJC McMorland, PhD Student Physiology, University of Auckland Armourer, Auckland University Fencing Secretary, Fencing North Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Sep 25 00:40:32 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 25 Sep 2006 00:40:32 -0400 Subject: [Numpy-discussion] eval shortcomings? In-Reply-To: References: Message-ID: On 25/09/06, Angus McMorland wrote: > Hi all, > > Can someone explain why the following occurs? > > a = numpy.zeros((100)) > b = numpy.ones((10)) > a[20:30] = b # okay > eval('a[50:60] = b') # raises SyntaxError: invalid syntax > > Is there some line mangling that the interpretor does that eval doesn't do? No. Eval evaluates expressions, that is, formulas producing a value. "a=b" does not produce a value, so you are obtaining the same error you would if you'd written if a=b: ... The way you run code that doesn't return a value is with "exec". A. M. Archibald From ivilata at carabos.com Mon Sep 25 08:00:50 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 25 Sep 2006 14:00:50 +0200 Subject: [Numpy-discussion] Fixes to numexpr string support Message-ID: <4517C4F2.6040103@carabos.com> Hi all, these are just a couple of small fixes to the string support in Numexpr, and a test case for the string copy operation. For the base patches: 1.http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg01551.html 2.http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg02261.html Cheers, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr-str-fix.diff Type: text/x-patch Size: 1910 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: OpenPGP digital signature URL: From chanley at stsci.edu Mon Sep 25 08:37:46 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 25 Sep 2006 08:37:46 -0400 Subject: [Numpy-discussion] please restore the way dtype prints to original version Message-ID: <4517CD9A.8090103@stsci.edu> Hi, Change set 3213 changed the data type printing with an array from something like dtype=int64 to dtype='int64'. Although this is a small cosmetic change it has broken all of the doctests I have written for numpy code. I expect that I am not the only person this change has caught. Please restore the previous behavior until after the 1.0 release at least. Thank you, Chris From mihoto. at yahoo.co.jp Mon Sep 25 10:10:21 2006 From: mihoto. at yahoo.co.jp (=?iso-2022-jp?B?bWlobw==?=) Date: Mon, 25 Sep 2006 14:10:21 -0000 Subject: [Numpy-discussion] =?iso-2022-jp?b?GyRCTDVOQRsoQg==?= Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?09 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From matthew.brett at gmail.com Mon Sep 25 10:35:14 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 25 Sep 2006 15:35:14 +0100 Subject: [Numpy-discussion] Downcasting to smallest storage Message-ID: <1e2af89e0609250735n6968e9bcsfc1fb49cda13c21@mail.gmail.com> Hi, I am sorry if I have missed anything obvious here, but is there a fast simple way to downcast an array to the smallest storage that hold array data within a specified precision - e.g. a = array([1.0]) small_a = fantasy_function(a, rtol=1.0000000000000001e-05, atol=1e-08 ) b = array([1.2]) small_b = fantasy_function(b, tol=1.0000000000000001e-05, atol=1e-08) where a.dtype becomes, say, uint8, and b.dtype becomes float32? Thanks a lot, Matthew From faltet at carabos.com Mon Sep 25 11:23:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 17:23:04 +0200 Subject: [Numpy-discussion] Negative division and numpy scalars Message-ID: <1159197785.3971.10.camel@localhost.localdomain> Hi, I have been bitten by a subtlety in numpy scalar divisions. The next exposes the issue: >>> -1/20 -1 >>> Numeric.array([-1])[0] / Numeric.array([20])[0] -1 >>> numarray.array([-1])[0] / numarray.array([20])[0] -1 >>> numpy.array([-1])[0] / numpy.array([20])[0] 0 After some digging, I've found that Python and C follow different conventions for doing this negative division: Python do a floor of the result, while C truncates it. As numpy scalar operations seems to be implemented in C, it seems that it follows the C convention and truncates the result. In fact, I like this behaviour of NumPy scalars (at least, when I'm aware of it!), but I thought it would be nice to warn other people about that. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Sep 25 12:28:34 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 25 Sep 2006 10:28:34 -0600 Subject: [Numpy-discussion] please restore the way dtype prints to original version In-Reply-To: <4517CD9A.8090103@stsci.edu> References: <4517CD9A.8090103@stsci.edu> Message-ID: <451803B2.1060402@ieee.org> Christopher Hanley wrote: > Hi, > > Change set 3213 changed the data type printing with an array from > something like dtype=int64 to dtype='int64'. Although this is a small > cosmetic change it has broken all of the doctests I have written for > numpy code. I was changing the way dtypes print and this was an unexpected consequence. It has been restored. Please try r3215 -Travis From faltet at carabos.com Mon Sep 25 12:47:38 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 18:47:38 +0200 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? Message-ID: <1159202858.3971.14.camel@localhost.localdomain> Hi, Anybody know if there is a map between NumPy types and Numeric typecodes? Something like 'typecodes' for numarray: >>> numarray.typecode {'UInt64': 'U', 'Int32': 'i', 'Int16': 's', 'Float64': 'd', 'Object': 'O', 'UInt8': 'b', 'UInt32': 'u', 'Complex64': 'D', 'UInt16': 'w', 'Bool': 'B', 'Complex32': 'F', 'Int64': 'N', 'Int8': '1', 'Float32': 'f'} Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Mon Sep 25 13:08:59 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 25 Sep 2006 11:08:59 -0600 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <1159202858.3971.14.camel@localhost.localdomain> References: <1159202858.3971.14.camel@localhost.localdomain> Message-ID: <45180D2B.6000200@ee.byu.edu> Francesc Altet wrote: >Hi, > >Anybody know if there is a map between NumPy types and Numeric >typecodes? Something like 'typecodes' for numarray: > > How about dtype(obj).char? -Travis From faltet at carabos.com Mon Sep 25 13:18:14 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 19:18:14 +0200 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <45180D2B.6000200@ee.byu.edu> References: <1159202858.3971.14.camel@localhost.localdomain> <45180D2B.6000200@ee.byu.edu> Message-ID: <1159204695.3971.18.camel@localhost.localdomain> El dl 25 de 09 del 2006 a les 11:08 -0600, en/na Travis Oliphant va escriure: > Francesc Altet wrote: > > >Hi, > > > >Anybody know if there is a map between NumPy types and Numeric > >typecodes? Something like 'typecodes' for numarray: > > > > > How about > > dtype(obj).char? This doesn't work for many types: >>> Numeric.array([1], typecode=numpy.dtype('int32').char) array([1]) this is fine, but: >>> Numeric.array([1], typecode=numpy.dtype('int16').char) Traceback (most recent call last): File "", line 1, in ? TypeError: typecode argument must be a valid type. >>> Numeric.array([1], typecode=numpy.dtype('int64').char) Traceback (most recent call last): File "", line 1, in ? TypeError: typecode argument must be a valid type. Anyway, this is not very important as I can do my own map internally. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Mon Sep 25 13:41:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 25 Sep 2006 11:41:53 -0600 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <1159204695.3971.18.camel@localhost.localdomain> References: <1159202858.3971.14.camel@localhost.localdomain> <45180D2B.6000200@ee.byu.edu> <1159204695.3971.18.camel@localhost.localdomain> Message-ID: <451814E1.4070506@ee.byu.edu> Francesc Altet wrote: >El dl 25 de 09 del 2006 a les 11:08 -0600, en/na Travis Oliphant va >escriure: > > >>Francesc Altet wrote: >> >> >> >>>Hi, >>> >>>Anybody know if there is a map between NumPy types and Numeric >>>typecodes? Something like 'typecodes' for numarray: >>> >>> Oh, you mean actual Numeric typecodes, not Numeric-like typecodes :-) dtype(obj).char will not work for the Numeric typecodes that changed, set up a dictionary-like object which uses dtype(obj).char on all but the ones that changed is my suggestion. See the core/numerictypes.py module for dictionary-like objects. Perhaps this would be a good thing to add to numpy/oldnumeric 'b' --> 'B' '1' --> 'b' 's' --> 'h' 'w' --> 'H' 'u' --> 'I' -Travis From listservs at mac.com Mon Sep 25 14:30:25 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 25 Sep 2006 14:30:25 -0400 Subject: [Numpy-discussion] Numpy RC1 build blows up Message-ID: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> I've just tried building RC1 on OSX 10.4 (Intel), and it fails almost immediately. This did not fail in the past with beta5 or from SVN: Osoyoos:~/Development/numpy-1.0rc1 chris$ python setup.py build Running from numpy source directory. F2PY Version 2_3198 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3'] running build running config_fc running build_src building py_modules sources creating build creating build/src.darwin-8.7.1-i386-2.4 creating build/src.darwin-8.7.1-i386-2.4/numpy creating build/src.darwin-8.7.1-i386-2.4/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src.darwin-8.7.1-i386-2.4/numpy/core Generating build/src.darwin-8.7.1-i386-2.4/numpy/core/config.h customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fPIC -fno-common -dynamic -DNDEBUG -g -O3 -Wall - Wstrict-prototypes compile options: '-I/Library/Frameworks/Python.framework/Versions/2.4/ include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/Library/ Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: _configtest.c gcc: installation problem, cannot exec `cc1': No such file or directory gcc: installation problem, cannot exec `cc1': No such file or directory failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ core.py", line 174, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 87, in run self.build_sources() File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 50, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From robert.kern at gmail.com Mon Sep 25 14:57:48 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 25 Sep 2006 13:57:48 -0500 Subject: [Numpy-discussion] Numpy RC1 build blows up In-Reply-To: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> References: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> Message-ID: listservs at mac.com wrote: > I've just tried building RC1 on OSX 10.4 (Intel), and it fails almost > immediately. This did not fail in the past with beta5 or from SVN: It looks like a problem on your end: > gcc: installation problem, cannot exec `cc1': No such file or directory Can you build anything else with gcc? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From listservs at mac.com Mon Sep 25 15:05:27 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 25 Sep 2006 15:05:27 -0400 Subject: [Numpy-discussion] Numpy RC1 build blows up -- IGNORE In-Reply-To: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> References: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> Message-ID: <4BB90E4B-83D0-4AE4-B767-BCAF2E0CA54B@mac.com> Ignore this query -- silly mistake on my part. Sorry. On Sep 25, 2006, at 2:30 PM, listservs at mac.com wrote: > I've just tried building RC1 on OSX 10.4 (Intel), and it fails > almost immediately. This did not fail in the past with beta5 or > from SVN: > > Osoyoos:~/Development/numpy-1.0rc1 chris$ python setup.py build > Running from numpy source directory. > F2PY Version 2_3198 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ > vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3'] > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src.darwin-8.7.1-i386-2.4 > creating build/src.darwin-8.7.1-i386-2.4/numpy > creating build/src.darwin-8.7.1-i386-2.4/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src.darwin-8.7.1-i386-2.4/numpy/core > Generating build/src.darwin-8.7.1-i386-2.4/numpy/core/config.h > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > Could not locate executable g77 > Could not locate executable f77 > Could not locate executable f95 > customize GnuFCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler using config > C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fPIC -fno-common -dynamic -DNDEBUG -g -O3 - > Wall -Wstrict-prototypes > > compile options: '-I/Library/Frameworks/Python.framework/Versions/ > 2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/ > Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: _configtest.c > gcc: installation problem, cannot exec `cc1': No such file or > directory > gcc: installation problem, cannot exec `cc1': No such file or > directory > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 89, in ? > setup_package() > File "setup.py", line 82, in setup_package > configuration=configuration ) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > core.py", line 174, in setup > return old_setup(**new_attr) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/core.py", line 149, in setup > dist.run_commands() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 946, in run_commands > self.run_command(cmd) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/command/build.py", line 112, in run > self.run_command(cmd_name) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 87, in run > self.build_sources() > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 106, in build_sources > self.build_extension_sources(ext) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 212, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 270, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 50, in generate_config_h > raise "ERROR: Failed to test configuration" > ERROR: Failed to test configuration > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From Peter.Bienstman at ugent.be Tue Sep 26 04:20:33 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Tue, 26 Sep 2006 10:20:33 +0200 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 Message-ID: <200609261020.36232.Peter.Bienstman@ugent.be> Hi, Not sure if this is a numpy or a matplotlib issue, but I can't seem to get matplotlib-0.87.5 to work with numpy1.0rc: Python 2.4.3 (#1, Sep 21 2006, 13:06:42) [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pylab import * Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/pylab.py", line 1, in ? from matplotlib.pylab import * File "/usr/lib/python2.4/site-packages/matplotlib/pylab.py", line 196, in ? import cm File "/usr/lib/python2.4/site-packages/matplotlib/cm.py", line 5, in ? import colors File "/usr/lib/python2.4/site-packages/matplotlib/colors.py", line 33, in ? from numerix import array, arange, take, put, Float, Int, where, \ File "/usr/lib/python2.4/site-packages/matplotlib/numerix/__init__.py", line 74, in ? Matrix = matrix NameError: name 'matrix' is not defined This is on an AMD64 platform. I tried removing the build directories of both packages, and reinstalling, but that didn't work. I also posted this to the matplotlib list, but got no reply. Thanks! Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From wbaxter at gmail.com Tue Sep 26 04:29:52 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 26 Sep 2006 17:29:52 +0900 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 In-Reply-To: <200609261020.36232.Peter.Bienstman@ugent.be> References: <200609261020.36232.Peter.Bienstman@ugent.be> Message-ID: Are you sure it's matplotlib-0.87.5? Line 74 in my c:/Python24/Lib/site-packages/matplotlib/numerix/ is "from numpy import *" and not "Matrix = matrix" as in your stackdump. You can check the site-packages/matplotlib/__init__.py file for the __version__ variable to make sure. Mine's got: __version__ = '0.87.5' __revision__ = '$Revision: 2761 $' __date__ = '$Date: 2006-09-05 09:37:04 -0400 (Tue, 05 Sep 2006) $' --bb On 9/26/06, Peter Bienstman wrote: > Hi, > > Not sure if this is a numpy or a matplotlib issue, but I can't seem to get > matplotlib-0.87.5 to work with numpy1.0rc: > > Python 2.4.3 (#1, Sep 21 2006, 13:06:42) > [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from pylab import * > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/pylab.py", line 1, in ? > from matplotlib.pylab import * > File "/usr/lib/python2.4/site-packages/matplotlib/pylab.py", line 196, in ? > import cm > File "/usr/lib/python2.4/site-packages/matplotlib/cm.py", line 5, in ? > import colors > File "/usr/lib/python2.4/site-packages/matplotlib/colors.py", line 33, in ? > from numerix import array, arange, take, put, Float, Int, where, \ > File "/usr/lib/python2.4/site-packages/matplotlib/numerix/__init__.py", line > 74, in ? > Matrix = matrix > NameError: name 'matrix' is not defined > > This is on an AMD64 platform. I tried removing the build directories of both > packages, and reinstalling, but that didn't work. > > I also posted this to the matplotlib list, but got no reply. > > Thanks! > > Peter > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From crabmeatalcohol at royalityinn.com Tue Sep 26 08:28:52 2006 From: crabmeatalcohol at royalityinn.com (accordion) Date: Tue, 26 Sep 2006 20:28:52 +0800 Subject: [Numpy-discussion] This one is shooting skywards Message-ID: <06778917688355.B093544A88@83AYZY3> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: chiang.gif Type: image/gif Size: 15925 bytes Desc: not available URL: From Peter.Bienstman at ugent.be Tue Sep 26 08:44:24 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Tue, 26 Sep 2006 14:44:24 +0200 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 In-Reply-To: References: Message-ID: <200609261444.24388.Peter.Bienstman@ugent.be> On Tuesday 26 September 2006 14:24, numpy-discussion-request at lists.sourceforge.net wrote: > Are you sure it's matplotlib-0.87.5? > Line 74 in my c:/Python24/Lib/site-packages/matplotlib/numerix/ > is "from numpy import *" > and not "Matrix = matrix" as in your stackdump. > > You can check the site-packages/matplotlib/__init__.py file for the > __version__ variable to make sure. Mine's got: > __version__ = '0.87.5' > __revision__ = '$Revision: 2761 $' > __date__ = '$Date: 2006-09-05 09:37:04 -0400 (Tue, 05 Sep 2006) $' Hmm, turns out deleting the build directory and doing a 'setup.py install' is not enough. When I delete the matplotlib directory from site-packages it does work. Thanks! Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From gnurser at googlemail.com Tue Sep 26 10:10:12 2006 From: gnurser at googlemail.com (George Nurser) Date: Tue, 26 Sep 2006 15:10:12 +0100 Subject: [Numpy-discussion] reload and f2py Message-ID: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> I'm running Python 2.3.5 with recent SVN f2py. Suppose I import an extension I have built with f2py. Then, if I edit the fortran and recompile the extension, I cannot use reload to use the modified version within the same Python session. I believe this is an problem with Python, that reload doesn't work with externally compiled extensions. Is this something that can be changed in future versions of Python, or is it an inherent problem? For editing and recompiling within an ipython session is quite a good way of getting the fortran code to work, leaving aside any problems in doing the f2py interfacing. --George Nurser. From klemm at phys.ethz.ch Tue Sep 26 10:26:23 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Tue, 26 Sep 2006 16:26:23 +0200 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas Message-ID: Hello, I try to build numpy 1.0rc1 with blas and lapack installed in another place than /usr/lib. I compiled blas and lapack according to the instructions on the scipy webpage and installed them under /scratch/python2.4/lib. I set the environment variables BLAS=/scratch/python2.4/lib/libfblas.a and LAPACK=/scratch/python2.4/lib/libflapack.a however, if I run python setup.py config in the numpy source directory, it appears only to pick up the libraries installed under /usr/lib, which are incomplete. What else can I do to help numpy finding the right libraries? Regards, Hanno -- Hanno Klemm klemm at phys.ethz.ch From hetland at tamu.edu Tue Sep 26 11:08:37 2006 From: hetland at tamu.edu (Rob Hetland) Date: Tue, 26 Sep 2006 10:08:37 -0500 Subject: [Numpy-discussion] numpy/scipy/mpl on Mac OS X with python 2.5 successful compilation Message-ID: I successfully compiled numpy/scipy/mpl on my Intel mac (OS X 10.4.7 with gfortran) with the latest Universal MacPython 2.5. Everything went through without a hitch. I've been waiting for this for a while now (between desire for ctypes, and macpython 2.4 readline issues), and I am very happy to have it up and running. Thanks to everyone who makes this happen, -Rob ---- Rob Hetland, Associate Professor Dept. of Oceanography, Texas A&M University http://pong.tamu.edu/~rob phone: 979-458-0096, fax: 979-845-6331 From faltet at carabos.com Tue Sep 26 12:35:52 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 26 Sep 2006 18:35:52 +0200 Subject: [Numpy-discussion] typeDict vs sctypeDict Message-ID: <200609261835.53563.faltet@carabos.com> Hi, I've just noted this: In [12]: numpy.typeNA is numpy.sctypeNA Out[12]: True In [13]: numpy.typeDict is numpy.sctypeDict Out[13]: True Why these aliases? Ara sc* replacing its not-sc-prefix counterparts? I'd like to know this so that I can avoid names that could be deprecated in a future. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From charlesr.harris at gmail.com Tue Sep 26 12:36:18 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Sep 2006 10:36:18 -0600 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas In-Reply-To: References: Message-ID: On 9/26/06, Hanno Klemm wrote: > > > Hello, > > I try to build numpy 1.0rc1 with blas and lapack installed in another > place than /usr/lib. I compiled blas and lapack according to the > instructions on the scipy webpage and installed them under > /scratch/python2.4/lib. > > I set the environment variables > BLAS=/scratch/python2.4/lib/libfblas.a and > LAPACK=/scratch/python2.4/lib/libflapack.a Try putting stuff in the site.cfg file. At one point I needed the following: [atlas] library_dirs = /usr/lib64/atlas atlas_libs = lapack, blas, cblas, atlas Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Sep 26 12:42:46 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 26 Sep 2006 18:42:46 +0200 Subject: [Numpy-discussion] Default values in scalar constructors Message-ID: <200609261842.46548.faltet@carabos.com> Hi, Is there any reason why the next works for some types: In [26]: numpy.int32() Out[26]: 0 In [27]: numpy.float64() Out[27]: 0.0 but don't for the next others? In [28]: numpy.int16() --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : function takes exactly 1 argument (0 given) In [29]: numpy.float32() --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : function takes exactly 1 argument (0 given) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From fperez.net at gmail.com Tue Sep 26 12:46:14 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 26 Sep 2006 10:46:14 -0600 Subject: [Numpy-discussion] reload and f2py In-Reply-To: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> References: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> Message-ID: On 9/26/06, George Nurser wrote: > I'm running Python 2.3.5 with recent SVN f2py. > > Suppose I import an extension I have built with f2py. Then, if I edit > the fortran and recompile the extension, I cannot use reload to use > the modified version within the same Python session. > > I believe this is an problem with Python, that reload doesn't work > with externally compiled extensions. As far as I know, you are correct. > Is this something that can be changed in future versions of Python, or > is it an inherent problem? For editing and recompiling within an > ipython session is quite a good way of getting the fortran code to > work, leaving aside any problems in doing the f2py interfacing. Unfortunately due to this limitation, I just restart ipython when I need to reload extension code. I'd be thrilled to know if there is any easy workaround. It's worth noting that weave is in this regard extremly ipython-friendly: since the name of the extension it loads is a hash of the source, /any/ change to the source results in a newly named extension being built and loaded. So while you get a bit of memory bloat by keeping the old, unused extensions around, the net interactive effect is that 'reloading works' for weave-generated code. One more reason why I think weave is the best thing since toasted bagels. Cheers, f From oliphant at ee.byu.edu Tue Sep 26 15:46:31 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:46:31 -0600 Subject: [Numpy-discussion] typeDict vs sctypeDict In-Reply-To: <200609261835.53563.faltet@carabos.com> References: <200609261835.53563.faltet@carabos.com> Message-ID: <45198397.3050903@ee.byu.edu> Francesc Altet wrote: >Hi, > >I've just noted this: > >In [12]: numpy.typeNA is numpy.sctypeNA >Out[12]: True > >In [13]: numpy.typeDict is numpy.sctypeDict >Out[13]: True > >Why these aliases? Ara sc* replacing its not-sc-prefix counterparts? >I'd like to know this so that I can avoid names that could be deprecated in a >future. > > Yes, The problem is that the type-dictionary is really just a scalar type dictionary. It was created back when scalar-types were synonomous with data-types. When the data-types became a separate object, then the naming was wrong. -Travis From oliphant at ee.byu.edu Tue Sep 26 15:48:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:48:00 -0600 Subject: [Numpy-discussion] Default values in scalar constructors In-Reply-To: <200609261842.46548.faltet@carabos.com> References: <200609261842.46548.faltet@carabos.com> Message-ID: <451983F0.8050904@ee.byu.edu> Francesc Altet wrote: >Hi, > >Is there any reason why the next works for some types: > >In [26]: numpy.int32() >Out[26]: 0 > >In [27]: numpy.float64() >Out[27]: 0.0 > >but don't for the next others? > >In [28]: numpy.int16() >--------------------------------------------------------------------------- > Traceback (most recent call last) > >/home/faltet/python.nobackup/numpy/ in () > >: function takes exactly 1 argument (0 given) > > > I suppose because int() and float() work. I really didn't know that. If that is the case, then we should have all the scalar objects return default values. Please file a ticket. -Travis From oliphant at ee.byu.edu Tue Sep 26 15:49:28 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:49:28 -0600 Subject: [Numpy-discussion] reload and f2py In-Reply-To: References: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> Message-ID: <45198448.1010500@ee.byu.edu> Fernando Perez wrote: >On 9/26/06, George Nurser wrote: > > >>I'm running Python 2.3.5 with recent SVN f2py. >> >>Suppose I import an extension I have built with f2py. Then, if I edit >>the fortran and recompile the extension, I cannot use reload to use >>the modified version within the same Python session. >> >>I believe this is an problem with Python, that reload doesn't work >>with externally compiled extensions. >> >> > >As far as I know, you are correct. > > On many systems there is an "unload" command similar to a shared-library load command which Python uses to load shared libraries. But, this is not performed by Python. I've wondered for awhile if a suitable use of such a command in an extension module would allow "reloading" of shared libraries. -Travis From nvf at MIT.EDU Wed Sep 27 00:07:04 2006 From: nvf at MIT.EDU (Nickolas V Fotopoulos) Date: Wed, 27 Sep 2006 00:07:04 -0400 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 Message-ID: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Dear numpy pros, I have a problem and I don't know whether it's from local setup intricacies (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN checkout). I observe the following: >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 31 tests for numpy.core.numerictypes Found 32 tests for numpy.linalg Found 13 tests for numpy.core.umath Found 4 tests for numpy.core.scalarmath Found 9 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 166 tests for numpy.core.multiarray Found 3 tests for numpy.fft.helper Found 36 tests for numpy.core.ma Found 1 tests for numpy.lib.ufunclike Found 12 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 4 tests for numpy.ctypeslib Found 40 tests for numpy.lib.function_base Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ........Segmentation fault In gdb: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 46912496134400 (LWP 25798)] VOID_setitem (op=Variable "op" is not available. ) at numpy/core/src/arraytypes.inc.src:522 522 if (res < 0) break; Version information: [nvf at ldas-pcdev1 nvf]$ uname -m x86_64 [nvf at ldas-pcdev1 nvf]$ python Python 2.5 (r25:51908, Sep 26 2006, 19:06:29) [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> numpy.__version__ '1.0.dev3219' Any idea what might be wrong? One obvious thing to check was the linear algebra, since ACML is suspect. numpy.linalg.eig worked fine for my simple test. I don't know enough to debug further. Thanks, Nick From robert.kern at gmail.com Wed Sep 27 00:31:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 26 Sep 2006 23:31:13 -0500 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> References: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Message-ID: Nickolas V Fotopoulos wrote: > Dear numpy pros, > > I have a problem and I don't know whether it's from local setup intricacies > (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN > checkout). I observe the following: > >>>> import numpy >>>> numpy.test() > ........Segmentation fault Please run numpy.test(10,10). That will print out the name of the test before running it. Thus we will find out which test is segfaulting. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Sep 27 01:03:22 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Sep 2006 23:03:22 -0600 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> References: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Message-ID: On 9/26/06, Nickolas V Fotopoulos wrote: > > Dear numpy pros, > > I have a problem and I don't know whether it's from local setup > intricacies > (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN > checkout). I observe the following: > > > > Any idea what might be wrong? One obvious thing to check was the linear > algebra, since ACML is suspect. numpy.linalg.eig worked fine for my > simple > test. I don't know enough to debug further. Is this your first build, or have you had success before? If the latter what changed? If you have done previous builds it is also a good idea to delete the build directory and, sometimes, the old package in site-packages whenever you have a problem to see if it goes away. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Sep 27 03:48:47 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 09:48:47 +0200 Subject: [Numpy-discussion] More about data types in NumPy Message-ID: <1159343327.3972.5.camel@localhost.localdomain> Hello, Sorry for being insistent, but I recognize that I'm having a bad time with NumPy data type rational. Is there an explanation for this?: >>> numpy.dtype('i4').type >>> numpy.dtype('int32').type >>> numpy.dtype('i4').type == numpy.dtype('int32').type True So far so good, but is the next the intended behaviour? >>> numpy.typeDict['i4'] >>> numpy.typeDict['int32'] >>> numpy.typeDict['i4'] == numpy.typeDict['int32'] False -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From zwqjdf at xnet.hr Wed Sep 27 08:47:42 2006 From: zwqjdf at xnet.hr (ofert) Date: Wed, 27 Sep 2006 10:47:42 -0200 Subject: [Numpy-discussion] Certified Message-ID: <000f01c6e211$9814cfe0$b84c8b53@fugi> expert help with search. rodziny Uniksa dziau spka. zarwno przygod jak bardziej szczegowe powicone wiata. trendy urzdzenia ktre zmieni nasza WiFi nowe standardy Przegld MP dyskiem twardym zobacz Ankieta Jakie zmiany stronie enter.pl uwaasz potrzebne Szata graficzna Nowe dziay adowania strony Wicej artykuw Mniej Nowe dziay adowania strony Wicej artykuw Mniej reklam Kursy online Zmiana nawigacji wyniki Najnowsze wtki forum: Konta nowymi MB poczt WWW spamu wirusw POP SMTP obsuga poczty przez extra Uwagi odnonie prosimy kierowa vbc.pl Copyright by Vogel Burda Sp. o.o. Wszystkie latest messages emailed to Alerts. Terms of Service Privacy Policy copy Magazyn ENTER Konkursy Ksiki Ankiety Redakcja Forum Kcik Reklama Sownik Wirusy BIOS praktyce Linux spka kiosk archiwum aktualny numer enter special pomoc Usugi sieci Rejestruj domen Personal GB Business Server Patronaty medialne wrzenia UOKiK: przeciwko eCard wyjani czy obcianie kosztami tylko wybranych zgodne prawem... wicej gtgt TP bdzie odwoywa Nowa strona Erachunek Plusie UKE: sprzeciw wobec Rozmw rabatem Privacy Policy copy Magazyn ENTER Konkursy Ksiki Ankiety Redakcja Forum Kcik Reklama Sownik Wirusy BIOS praktyce Linux spka kiosk archiwum aktualny numer enter special pomoc Usugi sieci Rejestruj domen Personal GB Business Server Patronaty medialne wrzenia UOKiK: przeciwko eCard wyjani czy obcianie kosztami tylko wybranych zgodne prawem... wicej gtgt TP bdzie odwoywa Nowa strona Erachunek Plusie UKE: sprzeciw wobec Rozmw rabatem tp Gemius SA Jeeli tworzysz swoj wasn lub prostu pragniesz zapozna tematyk pisania aplikacji zajrzyj naszego Kcika HTML oraz podrcznik PHP zbir porad wiele innych. standardy Przegld MP dyskiem twardym zobacz Ankieta Jakie zmiany stronie enter.pl uwaasz potrzebne Szata graficzna Nowe dziay adowania strony Wicej artykuw Mniej reklam Kursy online Zmiana nawigacji wyniki Najnowsze wtki forum: Konta nowymi MB poczt WWW spamu wirusw POP SMTP obsuga poczty przez Try different keywords. general fewer your on can try Answers for expert help with search. Get the -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: zobacz.gif Type: image/gif Size: 11394 bytes Desc: not available URL: From naomi at yahoo.co.jp Wed Sep 27 04:51:00 2006 From: naomi at yahoo.co.jp (=?iso-2022-jp?B?bmFvbWk=?=) Date: Wed, 27 Sep 2006 08:51:00 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/07 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/07 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/07 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?07 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/07 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From jg307 at cam.ac.uk Wed Sep 27 05:01:09 2006 From: jg307 at cam.ac.uk (James Graham) Date: Wed, 27 Sep 2006 10:01:09 +0100 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451A3DD5.1020608@cam.ac.uk> Francesc Altet wrote: > So far so good, but is the next the intended behaviour? > >>>> numpy.typeDict['i4'] > >>>> numpy.typeDict['int32'] > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] > False > > By way of comparison, I get (Linux/AMD 64): In [1]: import numpy In [2]: numpy.typeDict['i4'] == numpy.typeDict['int32'] Out[2]: True In [3]: numpy.typeDict['i4'] is numpy.typeDict['int32'] Out[3]: True In [4]: numpy.__version__ Out[4]: '1.0b5' -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead From faltet at carabos.com Wed Sep 27 07:22:37 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 13:22:37 +0200 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <451A3DD5.1020608@cam.ac.uk> References: <1159343327.3972.5.camel@localhost.localdomain> <451A3DD5.1020608@cam.ac.uk> Message-ID: <1159356157.3972.11.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 10:01 +0100, en/na James Graham va escriure: > Francesc Altet wrote: > > So far so good, but is the next the intended behaviour? > > > >>>> numpy.typeDict['i4'] > > > >>>> numpy.typeDict['int32'] > > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] > > False > > > > > By way of comparison, I get (Linux/AMD 64): > > In [1]: import numpy > > In [2]: numpy.typeDict['i4'] == numpy.typeDict['int32'] > Out[2]: True > > In [3]: numpy.typeDict['i4'] is numpy.typeDict['int32'] > Out[3]: True > > In [4]: numpy.__version__ > Out[4]: '1.0b5' Yeah, but I'm on Linux/AMD32. I'd bet that in your machine you will get something like: In [2]: numpy.typeDict['i8'] == numpy.typeDict['int64'] Out[2]: False -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From klemm at phys.ethz.ch Wed Sep 27 08:54:36 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Wed, 27 Sep 2006 14:54:36 +0200 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas In-Reply-To: References: , Message-ID: Thanks for the hint, that solved it! Hanno Charles R Harris said: > ------=_Part_15501_7322175.1159288578594 > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > Content-Transfer-Encoding: 7bit > Content-Disposition: inline > > On 9/26/06, Hanno Klemm wrote: > > > > > > Hello, > > > > I try to build numpy 1.0rc1 with blas and lapack installed in another > > place than /usr/lib. I compiled blas and lapack according to the > > instructions on the scipy webpage and installed them under > > /scratch/python2.4/lib. > > > > I set the environment variables > > BLAS=/scratch/python2.4/lib/libfblas.a and > > LAPACK=/scratch/python2.4/lib/libflapack.a > > > Try putting stuff in the site.cfg file. At one point I needed the following: > > [atlas] > library_dirs = /usr/lib64/atlas > atlas_libs = lapack, blas, cblas, atlas > > Chuck From nvf at MIT.EDU Wed Sep 27 12:33:10 2006 From: nvf at MIT.EDU (Nickolas V Fotopoulos) Date: Wed, 27 Sep 2006 12:33:10 -0400 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: References: Message-ID: <20060927123310.uqivdtr0izkkkowo@webmail.mit.edu> On Tuesday, September 26, Robert Kern wrote: > Please run numpy.test(10,10). That will print out the name of the test before > running it. Thus we will find out which test is segfaulting. Robert, The last line is: Check creation from list of list of tuplesSegmentation fault This matches what gdb said. I wouldn't think there any BLAS or LAPACK calls during array creation, so that would rule out ACML as the culprit. Any quick ideas of where specifically to look? Chuck, I had a rather different setup working. I changed too many variables (python 2.4, no ACML, pre-1.0beta SVN checkout to python 2.5, ACML, yesterday SVN checkout) to be sure what is breaking. I'll regress a bit and see if I can find out what's wrong. I always remove the build directory before reinstalling, but I generally don't remove the site-packages/numpy directory. I'll start doing that. I'm fairly sure I'm not picking up 2.4's site-packages. Oh, another thing is that I had trouble with the gnu (g77) version of the ACML libraries, so I forced gfortran usage with .pydistutils.cfg. Should that break things? Thanks, Nick From oliphant.travis at ieee.org Wed Sep 27 12:35:51 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 10:35:51 -0600 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451AA867.2030709@ieee.org> Francesc Altet wrote: > Hello, > > Sorry for being insistent, but I recognize that I'm having a bad time > with NumPy data type rational. Is there an explanation for this?: > > >>>> numpy.dtype('i4').type >>>> > > >>>> numpy.dtype('int32').type >>>> > > >>>> numpy.dtype('i4').type == numpy.dtype('int32').type >>>> > True > > So far so good, but is the next the intended behaviour? > > >>>> numpy.typeDict['i4'] >>>> > > >>>> numpy.typeDict['int32'] >>>> > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] >>>> No, this isn't correct behavior. This time you've caught an actual problem :-) The typeDict (actually the sctypeDict --- scalar-type-dictionary returns a scalar type given a string not a data-type object) is only used if no other conversion can be found for the object. It used to be much more useful before the data-type objects were formalized last year. -Travis From oliphant.travis at ieee.org Wed Sep 27 13:01:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 11:01:44 -0600 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451AAE78.8080906@ieee.org> Francesc Altet wrote: > Hello, > > Sorry for being insistent, but I recognize that I'm having a bad time > with NumPy data type rational. Is there an explanation for this?: > > Your actually talking about the array scalar types not the data-type objects. But, more to the point.... >>>> numpy.typeDict['i4'] >>>> > > >>>> numpy.typeDict['int32'] >>>> > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] >>>> > False > > On my 32-bit system I get: numpy.sctypeDict['i4'] is numpy.sctypeDict['int32'] True Hmm..... I don't know why you are getting a different result, but perhaps it has to do with the fact that the character alias ('i4') is not getting set at the same type as 'int32'. I just fixed that. That should fix the problem. -travis From faltet at carabos.com Wed Sep 27 13:29:25 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 19:29:25 +0200 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <451AAE78.8080906@ieee.org> References: <1159343327.3972.5.camel@localhost.localdomain> <451AAE78.8080906@ieee.org> Message-ID: <1159378165.3972.14.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 11:01 -0600, en/na Travis Oliphant va escriure: > Francesc Altet wrote: > > Hello, > > > > Sorry for being insistent, but I recognize that I'm having a bad time > > with NumPy data type rational. Is there an explanation for this?: > > > > > Your actually talking about the array scalar types not the data-type > objects. Yes, that's right. It's just that I didn't get used to the correct jargon :-( > On my 32-bit system I get: > > numpy.sctypeDict['i4'] is numpy.sctypeDict['int32'] > > True > > > Hmm..... I don't know why you are getting a different result, but > perhaps it has to do with the fact that the character alias ('i4') is > not getting set at the same type as 'int32'. I just fixed that. That > should fix the problem. Yup. I can confirm that the problem is solved in my machine as well. Thanks! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From haase at msg.ucsf.edu Wed Sep 27 19:32:16 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 27 Sep 2006 16:32:16 -0700 Subject: [Numpy-discussion] should I replace asarray with asanyarray in my code ? Message-ID: <200609271632.16689.haase@msg.ucsf.edu> Hi, This is a vaguely formulated question ... When I work with memmap'ed files/arrays I have a derived class that adds special attributes to the array class (referring to the MRC image file format used in medical / microscopy imaging) What are the pros and cons for asarray() vs. asanyarray() One obvious con for asanyarray is that its longer and asarray is what I have been using for the last few years ;-) Thanks, Sebastian From oliphant.travis at ieee.org Wed Sep 27 21:04:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 19:04:50 -0600 Subject: [Numpy-discussion] should I replace asarray with asanyarray in my code ? In-Reply-To: <200609271632.16689.haase@msg.ucsf.edu> References: <200609271632.16689.haase@msg.ucsf.edu> Message-ID: <451B1FB2.6030504@ieee.org> Sebastian Haase wrote: > Hi, > This is a vaguely formulated question ... > When I work with memmap'ed files/arrays I have a derived class > that adds special attributes to the array class (referring to the MRC image > file format used in medical / microscopy imaging) > > What are the pros and cons for asarray() vs. asanyarray() > > One obvious con for asanyarray is that its longer and asarray is what I have > been using for the last few years ;-) > asarray() guarantees you have a base-class array. Thus, you are not going to be thwarted by an re-definitions of infix operators, or other changed methods or attributes which you might use in your routine. asanyarray() allows a simple way of making sure your function returns any sub-class so that, for example, matrices are passed seamlessly through your function (matrix in and matrix out). However, a big drawback of asanyarray is that you must be sure that the way your function is written will not get confused by how a sub-class may overwrite the array methods and attributes. This significantly limits the application of asanyarray in my mind, as it is pretty difficult to predict what a sub-class *might* do to it's methods (including the special methods implementing the infix operators). A better way to write a function that passes any sub-class is to use asarray() so you are sure of the behavior of all methods and "infix" operators and then use the __array_wrap__ method of the actual input arguments (using __array_priority__ to choose between competing input objects). I expect that a decorator that automates this process will be added to NumPy eventually. Several examples have already been posted on this list. After getting the array result, you call the stored __array_wrap__ function which will take a base-class ndarray and return an object of the right Python-type (without copying data if possible). This is how the ufuncs work and why they can take sub-classes (actually anything with an __array__ method) and the same kind of object. -Travis From oliphant.travis at ieee.org Wed Sep 27 22:24:37 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 20:24:37 -0600 Subject: [Numpy-discussion] Question answered incorrectly at NumPy Tutorial Message-ID: <451B3265.8010902@ieee.org> During my NumPy Tutorial at the SciPy conference last month, somebody asked the question about the memory requirements of index arrays that I gave the wrong impression about. Here is the context and the correct response that should alleviate concerns about large cross-product index arrays. I was noting how copy-based (advanced) indexing using index arrays works in multiple-dimensions by creating an array of the same-shape of the input index arrays constructed by selecting the elements indicated by respective elements of the index arrays. If a is 2-d, then a[[10,12,14],[13, 15, 17]] returns a 1-d array with elements [a[10,13], a[12,15], a[14,17]]. This is *not* the cross-product that some would expect. The cross-product can be generated using the ix_ function a[ix_([10,12,14], [13,15,17])] is equivalent to a[[[10,10,10],[12,12,12],[14,14,14]], [[13,15,17],[13,15,17],[13,15,17]]] which will return [[a[10,13] a[10,15], a[10,17]], [a[12,13] a[12,15], a[12,17]], [a[14,13] a[14,15], a[14,17]]] The concern mentioned at the conference was that the cross-product would generate large intermediate index arrays for large input arrays to ix_. At the time, I think I validated the concern. However, the concern is unfounded. This is because the cross product function does not actually create a large intermediate array, but uses the broad-casting implementation of indexing to generate the 2-d indexing array "on-the-fly" (much like ogrid and other tools in NumPy). Notice: ix_([10,12,14], [13,15,17]) (array([[10], [12], [14]]), array([[13, 15, 17]])) The first indexing array is 3x1, while the second is 1x3. The result array will be 3x3, but the 2-d indexing array is never actually stored. Just to set my mind at ease about possible mis-information I spread during the tutorial, and give a little tutorial on advanced indexing. Best, -Travis From oliphant.travis at ieee.org Wed Sep 27 23:17:53 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 21:17:53 -0600 Subject: [Numpy-discussion] Release candidate 2.0 will come out mid-week next week Message-ID: <451B3EE1.1060501@ieee.org> Hi all, I'd like to release numpy 1.0rc2 on about October 5 of next week. Then, the release of numpy 1.0 official should happen on Monday, October 17. Please try and get all fixes and improvements in before then. Backward-incompatible changes are not acceptable at this point (unless they are minor or actually bug-fixes). I think NumPy has been cooking long enough. Any remaining problems can be fixed with maintenance releases. When 1.0 comes out, we will make a 1.0 release branch where bug-fixes should go as well as on the main trunk (I'd love for a way to do that automatically). There are lots of projects that need to start converting to NumPy 1.0 if we are going to finally have a "merged" community. The actual release of NumPy 1.0 will indicate that we are now committing to stability. Thanks to all that have been contributing so much to the project. -Travis From mg.mailing-list at laposte.net Thu Sep 28 02:55:17 2006 From: mg.mailing-list at laposte.net (mg) Date: Thu, 28 Sep 2006 08:55:17 +0200 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? Message-ID: <451B71D5.9000906@laposte.net> Hello, I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine on Linux-x86 32 and 64bit platforms. Now, I try to compile the both distributions on WindowsXP with VisualStudio2003. No problem to compile Python-2.5, but i have some troubles with Numpy-1.0rc1 and I didn't find any help in the provided setup.py. So, does someone can tell me how to do it? Thanks, Mathieu. From oliphant.travis at ieee.org Thu Sep 28 03:11:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 01:11:04 -0600 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451B71D5.9000906@laposte.net> References: <451B71D5.9000906@laposte.net> Message-ID: <451B7588.4000309@ieee.org> mg wrote: > Hello, > > I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine > on Linux-x86 32 and 64bit platforms. > Now, I try to compile the both distributions on WindowsXP with > VisualStudio2003. No problem to compile Python-2.5, but i have some > troubles with Numpy-1.0rc1 and I didn't find any help in the provided > setup.py. So, does someone can tell me how to do it? > > I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). Tim Hochberg once used a microsoft compiler to compile a previous version of NumPy and some things had to be fixed to make it work. I'm not sure if some in-compatibilities have crept in since then or not. But, I'd sure like to resolve it if they have. So, please post what problems you are having. You may be the first person to try a microsoft compiler with Python 2.5 -Travis From olivetti at itc.it Thu Sep 28 04:18:18 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Thu, 28 Sep 2006 10:18:18 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy Message-ID: <451B854A.5070300@itc.it> Hi, I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and LAPACK from sources too and I need to tune compiler flags. Here is the question: which are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: OPTS = -funroll-all-loops -fno-f2c -O3 instead on scipy.org[0] it's suggested: OPTS = -O2 I assume that using -O3 and unrolling loops should have definitely better performance that just -O2 but I'm wondering if the speedy options can be a problem for current numpy release (numpy-1.0rc1). Is it safe to use '-funroll-all-loops -fno-f2c -O3'? Thanks in advance for answers, Emanuele [0]: http://scipy.org/Installing_SciPy P.S.: my GCC is v3.4.5 From olivetti at itc.it Thu Sep 28 04:20:48 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Thu, 28 Sep 2006 10:20:48 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy In-Reply-To: <451B854A.5070300@itc.it> References: <451B854A.5070300@itc.it> Message-ID: <451B85E0.7030706@itc.it> Ops, sorry for the misleding subject: I wrote ATLAS but I meant LAPACK :) Emanuele Olivetti wrote: > Hi, > I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and > LAPACK from sources too and I need to tune compiler flags. Here is the question: which > are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: > OPTS = -funroll-all-loops -fno-f2c -O3 > instead on scipy.org[0] it's suggested: > OPTS = -O2 ... From faltet at carabos.com Thu Sep 28 05:24:15 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 28 Sep 2006 11:24:15 +0200 Subject: [Numpy-discussion] Release candidate 2.0 will come out mid-week next week In-Reply-To: <451B3EE1.1060501@ieee.org> References: <451B3EE1.1060501@ieee.org> Message-ID: <1159435455.4238.12.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 21:17 -0600, en/na Travis Oliphant va escriure: > Hi all, > > I'd like to release numpy 1.0rc2 on about October 5 of next week. > Then, the release of numpy 1.0 official should happen on Monday, October > 17. Please try and get all fixes and improvements in before then. > Backward-incompatible changes are not acceptable at this point (unless > they are minor or actually bug-fixes). I think NumPy has been cooking > long enough. Any remaining problems can be fixed with maintenance > releases. When 1.0 comes out, we will make a 1.0 release branch where > bug-fixes should go as well as on the main trunk (I'd love for a way to > do that automatically). In order to reduce the overhead of commiting bug fixes in both trunk and the 1.0 branch, you may want to delay the making of the 1.0 branch as much as possible. Eventually, when you have to start changes that properly belongs to trunk, then it's time to create the branch, but meanwhile you can save yourself quite a few syncronization work. Anyway, it is my pleasure to help finding bugs for NumPy! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From david at ar.media.kyoto-u.ac.jp Thu Sep 28 05:44:21 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 28 Sep 2006 18:44:21 +0900 Subject: [Numpy-discussion] storage order and concatenate Message-ID: <451B9975.9020904@ar.media.kyoto-u.ac.jp> Hi, I noticed a behaviour which I found counter-intuitive at least when using concatenate. I have a function which takes a numpy array of rank 2 as input, let's say foo(in): a = N.randn((10, 2)) foo(a) To test a ctype implementation of foo against the python version, my test has something like X1 = N.linspace(-2, 2, 10)[:, N.newaxis] X2 = N.linspace(-1, 3, 10)[:, N.newaxis] a = N.concatenate(([X1, X2]), 1) which has Fortran storage (column major order), where as creating a as a = N.zeros((10, 2)) a[:,0] = N.linspace(-2, 2, 10) a[:,1] = N.linspace(-1, 3, 10) has C storage (row major order). What are the rules concerning storage with numpy ? I thought it was always C, except if stated explicitly. I can obviously understand why concatenate gives a Fortran order from an implementation point of view, but this looks kind of strange to me, David From isgpn at kaiserkraft.de Thu Sep 28 05:41:41 2006 From: isgpn at kaiserkraft.de (Neddie Cunningham) Date: Thu, 28 Sep 2006 12:41:41 +0300 Subject: [Numpy-discussion] streetcar Message-ID: <001601c6e2e3$1a471f32$e6e8f758@sqabka> Okay, so not quite as much as those people doing shows every night of the week, but it's still been really busy here. And where is their remembrance? The city becomes awash with media-obsessed types, and it's difficult not to get caught up in the hysteria. Certainly, yesterday I felt really quite fragile, and didn't get much done, except for a quick visit to the supermarket for supplies during the afternoon. I was blown away by some truely amazing singing, and some rather fine acting too. Some of their number are modestly decent musicians, and can actually hold a tune. To help track that proces. She's also quite a creative person, who likes using self-made drawing to illustrate her musings. : Drake Gold Resources Inc. I've been watching a bit of TV, reading a bit, and surfing the net for the rest of time. Even with the usual summer lull, there's still been a steady flow of decent jobs over the last few weeks, which indicates a fairly bouyant employment market. But the others obviously carried on without me, and were quite drunk by the time I met up with them again a few hours later. Luckily, I had arrived a good ten minutes early, and had a great spot right at the front. It felt a bit strange to have the bar open for normal business while filming was in progress, but also interesting for us to be in the midst of all this action. But the web-based review sites don't have that limitation, and so there seems little excuse for not publishing the promised reviews. Since then, I've perked up a bit; that's despite having received another job rejection letter. But even without an amorous advance in your direction, you'll leave with a big grin on your face. But even without an amorous advance in your direction, you'll leave with a big grin on your face. And so, by the time we were thrown out at closing time, I was pissed again. Sure, it'll be great to have a rest. : Drake Gold Resources Inc. She's also quite a creative person, who likes using self-made drawing to illustrate her musings. It's an obsessive practice that I think a lot of amateur Fringe performers get dragged into. In all, I've sung in a total of four different concerts and five different church services in the last month. This departing friend had organised his final farewell drinks in one of Edinburgh's most famous public houses. And that's certainly dented my confidence somewhat. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: persistently.gif Type: image/gif Size: 11893 bytes Desc: not available URL: From stefan at sun.ac.za Thu Sep 28 06:29:27 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 28 Sep 2006 12:29:27 +0200 Subject: [Numpy-discussion] return value of negative power Message-ID: <20060928102927.GA16637@mentat.za.net> Hi all, Currently, the power function returns '0' for negative powers of integers: In [1]: N.power(3,-2) Out[1]: 0 (or, more confusingly) In [1]: N.power(a,b) Out[1]: 0 which is almost certainly not the answer you want. Two possible solutions may be to upcast the input to float before calculation, or to return nan. This would be consistent with a function like sqrt: In [10]: N.sqrt(3) Out[10]: 1.7320508075688772 In [11]: N.sqrt(-3) Out[11]: nan Does anyone have an opinion on whether the change is necessary, and if so, on which one would work best? Regards St?fan From klemm at phys.ethz.ch Thu Sep 28 06:47:55 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Thu, 28 Sep 2006 12:47:55 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy In-Reply-To: <451B85E0.7030706@itc.it> References: <451B854A.5070300@itc.it>, <451B854A.5070300@itc.it> Message-ID: Emanuele, the scipy compiler flags under http://www.scipy.org/Installing_SciPy/BuildingGeneral work well. However, if you happen to have to use gcc 3.2.3 (e.g. often in Redhat Enterprise editions present), you have to turn off optimization, otherwise lapack doesn't build properly. The correct build flags are then OPTS = -m64 -fPIC (at least that's what worked for me) Regards, Hanno Emanuele Olivetti said: > Ops, sorry for the misleding subject: I wrote ATLAS but I meant LAPACK :) > > Emanuele Olivetti wrote: > > Hi, > > I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and > > LAPACK from sources too and I need to tune compiler flags. Here is the question: which > > are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: > > OPTS = -funroll-all-loops -fno-f2c -O3 > > instead on scipy.org[0] it's suggested: > > OPTS = -O2 > ... > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Hanno Klemm klemm at phys.ethz.ch From oliphant.travis at ieee.org Thu Sep 28 07:15:47 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 05:15:47 -0600 Subject: [Numpy-discussion] storage order and concatenate In-Reply-To: <451B9975.9020904@ar.media.kyoto-u.ac.jp> References: <451B9975.9020904@ar.media.kyoto-u.ac.jp> Message-ID: <451BAEE3.8030809@ieee.org> David Cournapeau wrote: > Hi, > > > What are the rules concerning storage with numpy ? The rule is that a numpy array has "strides" which specify how many bytes to skip to get to the next element in the array. That's the internal model. There are no hard and fast rules about storage order. Internally, C-order is as good as Fortran-order (except the array iterator gives special preference to C-order and all functions for which the order can be specified (like zeros) default to C-order). Thus, the storage order is whatever the strides say it is. Now, there are flags that keep track of whether or not the strides agree with the 2 recognized special cases of "Fortran-order" (first-index varies the fastest) or "C-order" (last-index varies the fastest). But, this is only for convenience. Very few functions actually require a specification of storage order. Those that allow it default to "C-order". You can't think of a NumPy array has having a particular storage order unless you explicitly request it. One of the most common ways that Fortran-order arrays show up, for example is when a C-order array is transposed. A transpose operation does nothing except flip the strides (and therefore the flags) of the array. This is what is happening in concatenate (using axis=1) to give you a Fortran-order array. Bascially, code equivalent to the following is being run: concatenate([X1.T, X2.T]).T In the second example, you explicitly create the array (and therefore the strides) as C-order and then fill it (so it doesn't change on you). The first example used array calculations which don't guarantee the storage order. This is all seamless to the user until you have to interface with extension code. Ideally, you write compiled code that deals with strided arrays. If you can't, then you request an array of the required storage-order. By the way, for interfacing with ctypes, check out the ctypeslib.ndpointer class-creation function for flag checking and the require function for automatic conversion of an array to specific requirements. -Travis From eric at enthought.com Thu Sep 28 08:05:21 2006 From: eric at enthought.com (eric) Date: Thu, 28 Sep 2006 07:05:21 -0500 Subject: [Numpy-discussion] lexsort segfault sorting strings Message-ID: <451BBA81.5050904@enthought.com> I've been using the new record arrays and lexsort from numpy quite a lot lately. Very cool stuff. Using the nightly egg for numpy from here (I believe it is up to date...): http://code.enthought.com/enstaller/eggs/numpy-nightly-py2.4-win32.egg I get segfaults when using lexsort on character arrays. A lot of my columns in record arrays are string based, so sorting the arrays based on these columns would be really handy. Here is an example that crashes for me. C:\wrk\mt\trunk\src\lib\mt\statement\tests>python Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug 2 2006, 12:09:59) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import lexsort >>> lst = [1,2,3] >>> lexsort((lst,)) array([0, 1, 2]) >>> lst = ['abc','cde','fgh'] >>> lexsort((lst,)) Do others see this? I've opened a ticket at: http://projects.scipy.org/scipy/numpy/ticket/298 thanks, eric From tim.hochberg at ieee.org Thu Sep 28 09:26:33 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 06:26:33 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451B7588.4000309@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> Message-ID: <451BCD89.9000902@ieee.org> Travis Oliphant wrote: > mg wrote: > >> Hello, >> >> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine >> on Linux-x86 32 and 64bit platforms. >> Now, I try to compile the both distributions on WindowsXP with >> VisualStudio2003. No problem to compile Python-2.5, but i have some >> troubles with Numpy-1.0rc1 and I didn't find any help in the provided >> setup.py. So, does someone can tell me how to do it? >> >> >> > I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). > Tim Hochberg once used a microsoft compiler to compile a previous > version of NumPy and some things had to be fixed to make it work. I'm > not sure if some in-compatibilities have crept in since then or not. > But, I'd sure like to resolve it if they have. > > So, please post what problems you are having. You may be the first > person to try a microsoft compiler with Python 2.5 > It was VS2003 that I used to compile numpy. However, I switched boxes a while ago and I haven't got around to trying to compile on the new box, I've just been using the released builds. So I'm not much help at the moment. Things are clearing up a little bit here, so maybe I can carve some time out to get things set up to compile in the next couple days. If so I'll let you know what I find. -tim > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From mg.mailing-list at laposte.net Thu Sep 28 10:40:54 2006 From: mg.mailing-list at laposte.net (mg) Date: Thu, 28 Sep 2006 16:40:54 +0200 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BCD89.9000902@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> Message-ID: <451BDEF6.7020307@laposte.net> Unfortunately, no Windows-x86 or Windows-x86-64bit Numpy-1.0rc1 installer are available on SourceForge yet. So the only current solution for us is to compile it. Moreover, our generic C++ framework is compiled with VisualStudio on Windows-native and we compile all additions to it with the same compiler, if compilation is needed. Indeed, we have observed that Visual C++ is still the closest to a "default" or "native" compiler for windows platform, like gcc is for linux....well at least that's our experience when using other commercial (or open-source, think python whose provide visual project files) softwares requiring rebuilding/relinking. Use MinGW for Numpy means risk some un-compatibilities between Numpy and our framework (and even Python) or migrate all our development environment from Visual Studio to MinGW. this is not really an option for the near or mean furture.... If some of you have good experiences for linking (static or dynamic) libraries compiled with mixed compilers (especially mingw and visual), knowing that our libraries contains C++, C and fortran code, we could consider this as a temprary option for numpy, but for convenience we would ultimately prefer to use only 1 compiler to avoid a double maintenance of building tools.... Then, I wonder if the compilation of Numpy with Visual-Studio-2003 (or 2005) is scheduled ? For your information, this is the standard output and the standard error of the compilation of Numpy-1.0rc1 with Visual Studio 2003: >>> command line >>> python setup.py build 1>stdout.txt 2>stderr.txt >>> stdout.txt >>> F2PY Version 2_3198 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in c:\Program Files\python\dist\src\lib libraries mkl,vml,guide not found in C:\ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in c:\Program Files\python\dist\src\lib libraries ptf77blas,ptcblas,atlas not found in C:\ NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in c:\Program Files\python\dist\src\lib libraries f77blas,cblas,atlas not found in C:\ NOT AVAILABLE blas_info: libraries blas not found in c:\Program Files\python\dist\src\lib libraries blas not found in C:\ NOT AVAILABLE blas_src_info: NOT AVAILABLE NOT AVAILABLE lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in c:\Program Files\python\dist\src\lib libraries mkl,vml,guide not found in C:\ NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in c:\Program Files\python\dist\src\lib libraries lapack_atlas not found in c:\Program Files\python\dist\src\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in c:\Program Files\python\dist\src\lib libraries lapack_atlas not found in c:\Program Files\python\dist\src\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: libraries lapack not found in c:\Program Files\python\dist\src\lib libraries lapack not found in C:\ NOT AVAILABLE lapack_src_info: NOT AVAILABLE NOT AVAILABLE running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources Generating build\src.win32-2.5\numpy\core\config.h No module named msvccompiler in numpy.distutils, trying from distutils.. 0 Could not locate executable g77 Could not locate executable f77 Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelVisualFCompiler Could not locate executable ifl Executable ifl does not exist customize AbsoftFCompiler Could not locate executable f90 Executable f90 does not exist customize CompaqVisualFCompiler Could not locate executable DF Executable DF does not exist customize IntelItaniumVisualFCompiler Could not locate executable efl Executable efl does not exist customize Gnu95FCompiler Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist customize G95FCompiler Could not locate executable g95 Executable g95 does not exist customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist customize Gnu95FCompiler Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist customize GnuFCompiler using config C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\Program Files\python\dist\src\include -Inumpy\core\src -Inumpy\core\include -Ic:\Program Files\python\dist\src\include -Ic:\Program Files\python\dist\src\PC /Tc_configtest.c /Fo_configtest.obj C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /nologo /INCREMENTAL:NO /LIBPATH:c:\Program Files\python\dist\src\lib /LIBPATH:C:\ _configtest.obj /OUT:_configtest.exe LINK : fatal error LNK1104: cannot open file 'python25.lib' failure. removing: _configtest.c _configtest.obj >>> stderr.txt >>> Running from numpy source directory. C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1296: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1305: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1308: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1205: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1216: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. warnings.warn(LapackNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1219: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. warnings.warn(LapackSrcNotFoundError.__doc__) Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\core.py", line 174, in setup return old_setup(**new_attr) File "c:\Program Files\python\dist\src\lib\distutils\core.py", line 149, in setup dist.run_commands() File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 955, in run_commands self.run_command(cmd) File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 975, in run_command cmd_obj.run() File "c:\Program Files\python\dist\src\lib\distutils\command\build.py", line 112, in run self.run_command(cmd_name) File "c:\Program Files\python\dist\src\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 975, in run_command cmd_obj.run() File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 87, in run self.build_sources() File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 50, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration Thanks a lot for your help. Tim Hochberg wrote: > Travis Oliphant wrote: > >> mg wrote: >> >> >>> Hello, >>> >>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine >>> on Linux-x86 32 and 64bit platforms. >>> Now, I try to compile the both distributions on WindowsXP with >>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>> troubles with Numpy-1.0rc1 and I didn't find any help in the provided >>> setup.py. So, does someone can tell me how to do it? >>> >>> >>> >>> >> I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). >> Tim Hochberg once used a microsoft compiler to compile a previous >> version of NumPy and some things had to be fixed to make it work. I'm >> not sure if some in-compatibilities have crept in since then or not. >> But, I'd sure like to resolve it if they have. >> >> So, please post what problems you are having. You may be the first >> person to try a microsoft compiler with Python 2.5 >> >> > It was VS2003 that I used to compile numpy. However, I switched boxes a > while ago and I haven't got around to trying to compile on the new box, > I've just been using the released builds. So I'm not much help at the > moment. Things are clearing up a little bit here, so maybe I can carve > some time out to get things set up to compile in the next couple days. > If so I'll let you know what I find. > > -tim > > >> -Travis >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From tim.hochberg at ieee.org Thu Sep 28 11:07:43 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 08:07:43 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BCD89.9000902@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> Message-ID: <451BE53F.2060904@ieee.org> Tim Hochberg wrote: > Travis Oliphant wrote: >> mg wrote: >> >>> Hello, >>> >>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work >>> fine on Linux-x86 32 and 64bit platforms. >>> Now, I try to compile the both distributions on WindowsXP with >>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>> troubles with Numpy-1.0rc1 and I didn't find any help in the >>> provided setup.py. So, does someone can tell me how to do it? >>> >>> >> I don't use VisualStudio2003 on Windows to compile NumPy (I use >> mingw). Tim Hochberg once used a microsoft compiler to compile a >> previous version of NumPy and some things had to be fixed to make it >> work. I'm not sure if some in-compatibilities have crept in since >> then or not. But, I'd sure like to resolve it if they have. >> >> So, please post what problems you are having. You may be the first >> person to try a microsoft compiler with Python 2.5 >> > It was VS2003 that I used to compile numpy. However, I switched boxes > a while ago and I haven't got around to trying to compile on the new > box, I've just been using the released builds. So I'm not much help at > the moment. Things are clearing up a little bit here, so maybe I can > carve some time out to get things set up to compile in the next couple > days. If so I'll let you know what I find. FWIW, I just got svn head to compile cleanly against Python *2.4* with VS2003. Later today I will try compiling against 2.5 -tim From tim.hochberg at ieee.org Thu Sep 28 12:32:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 09:32:08 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BE53F.2060904@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> <451BE53F.2060904@ieee.org> Message-ID: <451BF908.8080902@ieee.org> Tim Hochberg wrote: > Tim Hochberg wrote: > >> Travis Oliphant wrote: >> >>> mg wrote: >>> >>> >>>> Hello, >>>> >>>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work >>>> fine on Linux-x86 32 and 64bit platforms. >>>> Now, I try to compile the both distributions on WindowsXP with >>>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>>> troubles with Numpy-1.0rc1 and I didn't find any help in the >>>> provided setup.py. So, does someone can tell me how to do it? >>>> >>>> >>>> >>> I don't use VisualStudio2003 on Windows to compile NumPy (I use >>> mingw). Tim Hochberg once used a microsoft compiler to compile a >>> previous version of NumPy and some things had to be fixed to make it >>> work. I'm not sure if some in-compatibilities have crept in since >>> then or not. But, I'd sure like to resolve it if they have. >>> >>> So, please post what problems you are having. You may be the first >>> person to try a microsoft compiler with Python 2.5 >>> >>> >> It was VS2003 that I used to compile numpy. However, I switched boxes >> a while ago and I haven't got around to trying to compile on the new >> box, I've just been using the released builds. So I'm not much help at >> the moment. Things are clearing up a little bit here, so maybe I can >> carve some time out to get things set up to compile in the next couple >> days. If so I'll let you know what I find. >> > FWIW, I just got svn head to compile cleanly against Python *2.4* with > VS2003. Later today I will try compiling against 2.5 > > OK. SVN head compiles out of the box and passes all tests for me under both Python2.4 and Python2.5. My stderr output includes all of the same warnings about not being able to find ATLAS, BLAS and LAPACK that mg's does, but not "ERROR: Failed to test configuration". The stdout output also looks similar except for the error at the end. (I can send you the entire stderr/stdout output privately if you like). One thing I did notice is that your Python appears to be in a nonstandard location: \Program Files\python\dist\src instead of the standard \Python25, or at least that's where it's looking for things. Perhaps this is confusing either distutils or some of numpy's addons to distutils and causing it to misplace python25.lib. At least that's my best (and only) guess at the moment. -tim From wfspotz at sandia.gov Thu Sep 28 12:43:26 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 28 Sep 2006 10:43:26 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints Message-ID: I am wrapping code using swig and extending it to use numpy. One class method I wrap (let's call it myElements()) returns an array of ints, and I convert it to a numpy array with PyArray_SimpleNew(1,n,'i'); I obtain the data pointer, fill in the values and return it as the method return argument. In python, it is common to want to loop over this array and treat its elements as integers: for row in map.myElements(): matrix.setElements(row, [row-1,row,row+1], [-1.0,2.0,-1.0]) On a 32-bit machine, this has worked fine, but on a 64-bit machine, I get a type error: TypeError: in method 'setElements', argument 2 of type 'int' because row is a . It would be nice if I could get the integer conversion to work automatically under the covers, but I'm not exactly sure how to make that work. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From tim.hochberg at ieee.org Thu Sep 28 13:08:01 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 10:08:01 -0700 Subject: [Numpy-discussion] return value of negative power In-Reply-To: <20060928102927.GA16637@mentat.za.net> References: <20060928102927.GA16637@mentat.za.net> Message-ID: <451C0171.2070406@ieee.org> Stefan van der Walt wrote: > Hi all, > > Currently, the power function returns '0' for negative powers of > integers: > > In [1]: N.power(3,-2) > Out[1]: 0 > > (or, more confusingly) > > In [1]: N.power(a,b) > Out[1]: 0 > > which is almost certainly not the answer you want. Two possible > solutions may be to upcast the input to float before calculation, or > to return nan. > Returning nan seems silly. There really is a result, or rather two possible results on a tad more sensible than the other. If we were going to punt it makes more sense to raise an exception, but I doubt that's necessary. In addition, nan is really a floating point value; if we're going to return a floating point value, not an integer, we might as well return the actual result. > This would be consistent with a function like sqrt: > > In [10]: N.sqrt(3) > Out[10]: 1.7320508075688772 > > In [11]: N.sqrt(-3) > Out[11]: nan > > Does anyone have an opinion on whether the change is necessary, and if > so, on which one would work best? > This is a complicated tangle of worms. First off there's both "a**b" and "power(a, b)". These don't necessarily need to work the same. In fact they already differ somewhat in that a**b does some optimization when b is a small scalar that power does not (I think anyway -- haven't looked at this a while). However, if having them differ in any significant way is likely to be quite confusing, so it should be only be considered if there's some compelling reason to support multiple behaviors here. There is a solution that is, IMO, simple obvious and not quite right. That is to return floats if the b contains any negative numbers while returning integers otherwise. That sounds appealing at first, but will cause confusion and memory blow ups when one suddenly has all of ones arrays become floats because somewhere or other a negative value crept into an exponent. It's fine for the return type to depend on the types of the arguments, it's not so good for it to depends on the values. This restriction gets a little weaker once future division arrives since ints and floats will be closer to indistinguishable, but it still has some force. On the other hand, this is consistent with the rest of Python's behavior. Another path is to just always return floats from power. One down side of this is that we lose the ability to do true integer powers, which is sometimes useful. A second downside is that it introduces a inconsistency with Python's scalar 'x**y'. One way to finesse both of these issues is to make numpy.power consistent with math.pow; that is, it returns floating point values when passed integers. At the same time, make a**b consistent with the python's '**' operator in that any negative exponents trigger a floating point return values. This isn't perfect, but it's as close to a self consistent solution as I can think of. That's my two cents. -tim From oliphant.travis at ieee.org Thu Sep 28 14:03:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 12:03:56 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: References: Message-ID: <451C0E8C.1040803@ieee.org> Bill Spotz wrote: > I am wrapping code using swig and extending it to use numpy. > > One class method I wrap (let's call it myElements()) returns an array > of ints, and I convert it to a numpy array with > > PyArray_SimpleNew(1,n,'i'); > You should probably use NPY_INT instead of 'i' for the type-code. > I obtain the data pointer, fill in the values and return it as the > method return argument. > > In python, it is common to want to loop over this array and treat its > elements as integers: > > for row in map.myElements(): > matrix.setElements(row, [row-1,row,row+1], [-1.0,2.0,-1.0]) > > On a 32-bit machine, this has worked fine, but on a 64-bit machine, I > get a type error: > > TypeError: in method 'setElements', argument 2 of type 'int' > > because row is a . > > It would be nice if I could get the integer conversion to work > automatically under the covers, but I'm not exactly sure how to make > that work. > Yeah, It can be confusing, at first. You just have to make sure you are matching the right c-data-types. I'm not quite sure what the problem here is given your description, because I don't know what setElements expects. My best guess, is that it is related to the fact that a Python int uses the 'long' c-type. Thus, you should very likely be using PyArray_SimpleNew(1, n, NPY_LONG) instead of int so that your integer array always matches what Python is using as integers. The other option is to improve your converter in setElements so that it can understand any of the array scalar integers and not just the default Python integer. The reason this all worked on 32-bit systems is probably the array scalar corresponding to NPY_INT is a sub-class of the Python integer. It can't be on a 64-bit platform because of binary incompatibility of the layout. Hope that helps. -Travis From wfspotz at sandia.gov Thu Sep 28 20:12:00 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 28 Sep 2006 18:12:00 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: <451C0E8C.1040803@ieee.org> References: <451C0E8C.1040803@ieee.org> Message-ID: <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> On Sep 28, 2006, at 12:03 PM, Travis Oliphant wrote: > The other option is to improve your converter in setElements so > that it > can understand any of the array scalar integers and not just the > default > Python integer. I think this may be the best approach. This may be something worthwhile to put in the numpy.i interface file: a set of typemaps that handle a set of basic conversions for those array scalar types for which it makes sense. I'll look into it. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From oliphant at ee.byu.edu Thu Sep 28 21:22:24 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 28 Sep 2006 19:22:24 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> References: <451C0E8C.1040803@ieee.org> <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> Message-ID: <451C7550.4080009@ee.byu.edu> Bill Spotz wrote: >On Sep 28, 2006, at 12:03 PM, Travis Oliphant wrote: > > > >>The other option is to improve your converter in setElements so >>that it >>can understand any of the array scalar integers and not just the >>default >>Python integer. >> >> > >I think this may be the best approach. > >This may be something worthwhile to put in the numpy.i interface >file: a set of typemaps that handle a set of basic conversions for >those array scalar types for which it makes sense. I'll look into it. > > That's a good idea. Notice that there are some routines for making your life easier here. You should look at the tp_int function for the gentype array (it converts scalars to arrays). You call the "__int__" special method of the scalar to convert it to a Python integer. You should first check to see that it is an integer scalar PyArray_IsScalar(obj, Integer) because the "__int__" method coerces to an integer if it is a float (but maybe you want that behavior). There are other functions in the C-API that return the data directly from the scalar --- check them out. The macros in arrayscalar.h are useful. -Travis From david at ar.media.kyoto-u.ac.jp Fri Sep 29 03:36:54 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 29 Sep 2006 16:36:54 +0900 Subject: [Numpy-discussion] storage order and concatenate In-Reply-To: <451BAEE3.8030809@ieee.org> References: <451B9975.9020904@ar.media.kyoto-u.ac.jp> <451BAEE3.8030809@ieee.org> Message-ID: <451CCD16.7050708@ar.media.kyoto-u.ac.jp> Travis Oliphant wrote: > David Cournapeau wrote: > >> Hi, >> >> >> What are the rules concerning storage with numpy ? >> > The rule is that a numpy array has "strides" which specify how many > bytes to skip to get to the next element in the array. That's the > internal model. There are no hard and fast rules about storage order. > Internally, C-order is as good as Fortran-order (except the array > iterator gives special preference to C-order and all functions for which > the order can be specified (like zeros) default to C-order). > Ok, this is again a bad habit from matlab to think in C or F order... > Thus, the storage order is whatever the strides say it is. Now, there > are flags that keep track of whether or not the strides agree with the 2 > recognized special cases of "Fortran-order" (first-index varies the > fastest) or "C-order" (last-index varies the fastest). But, this is > only for convenience. Very few functions actually require a > specification of storage order. Those that allow it default to "C-order". > > You can't think of a NumPy array has having a particular storage order > unless you explicitly request it. One of the most common ways that > Fortran-order arrays show up, for example is when a C-order array is > transposed. A transpose operation does nothing except flip the strides > (and therefore the flags) of the array. This is what is happening in > concatenate (using axis=1) to give you a Fortran-order array. > Bascially, code equivalent to the following is being run: > concatenate([X1.T, X2.T]).T > > In the second example, you explicitly create the array (and therefore > the strides) as C-order and then fill it (so it doesn't change on you). > The first example used array calculations which don't guarantee the > storage order. > > This is all seamless to the user until you have to interface with > extension code. Ideally, you write compiled code that deals with > strided arrays. If you can't, then you request an array of the required > storage-order. > > By the way, for interfacing with ctypes, check out the > ctypeslib.ndpointer class-creation function for flag checking and the > require function for automatic conversion of an array to specific > requirements. > > I tried to to that at first, but I couldn't make the examples of numpy works; after having tried at home with beta versions, it looks like it is a ctype version problem, as it works with ctype 1.0 + numpy 1.0rc1. Thanks for the explanations, this answers most questions I had in mind for numpy internal layout compared to matlab which I am used to. I think this should be in the wiki somewhere; do you mind if I use your email as a basis for the tentative numpy tutorial (memory layout section, maybe) ? David From faltet at carabos.com Fri Sep 29 07:44:34 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 13:44:34 +0200 Subject: [Numpy-discussion] Negative values with unsigned data types problems Message-ID: <200609291344.35863.faltet@carabos.com> Hi, I'm writing this here because the numpy Trac seems down: {{{ Oops... Trac detected an internal error: The Trac Environment needs to be upgraded. Run trac-admin /home/scipy/trac/numpy upgrade" }}} Well, it seems that there are inconsistencies on creating arrays coming from negative values using unsigned integers: In [41]: numpy.asarray([-1], dtype='uint8') Out[41]: array([255], dtype=uint8) In [42]: numpy.asarray([-1], dtype='uint16') Out[42]: array([65535], dtype=uint16) until here it's fine, but: In [43]: numpy.asarray([-1], dtype='uint32') --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () /usr/local/lib/python2.5/site-packages/numpy/core/numeric.py in asarray(a, dtype, order) 129 are converted to base class ndarray. 130 """ --> 131 return array(a, dtype, copy=False, order=order) 132 133 def asanyarray(a, dtype=None, order=None): : long() argument must be a string or a number, not 'list' and the same happens with 'uint64'. My numpy version is 1.0.dev3216 Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Sep 29 11:19:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 17:19:01 +0200 Subject: [Numpy-discussion] Incorrect removal of NULL char in buffers Message-ID: <200609291719.02380.faltet@carabos.com> Hi, I'm trying to build-up numpy arrays coming from buffers, and I'm getting a somewhat unexpected result. First, for numeric values, everything seems ok (i.e. the NULL character is correctly interpretated), and works equally for both numarray and numpy: In [98]: numarray.array("a\x00b"*4, dtype='Float32',shape=3) Out[98]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], type=Float32) In [99]: numpy.ndarray(buffer="a\x00b"*4, dtype='Float32',shape=3) Out[99]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) However, for string values, numpy seems to work in a strange way. The numarray have an expected behaviour, IMO: In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) Out[100]: CharArray(['a', '', 'ba']) but numpy haven't: In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[101]: array([aba, ba, bab], dtype='|S4') i.e. it seems like numpy is striping-off NULL chars before building the object and I don't think this is correct. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Sep 29 11:23:26 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 17:23:26 +0200 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray Message-ID: <200609291723.26896.faltet@carabos.com> Hello, Is the next a bug a feature? In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) In [103]: f4 Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) In [104]: f4[2] = 2 --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : array is not writeable In [105]: f4.flags.writeable = True In [106]: f4[2] = 2 In [107]: f4 Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.00000000e+00], dtype=float32) i.e. in an array built from ndarray, the default is that it has to be read-only? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tim.hochberg at ieee.org Fri Sep 29 12:12:09 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 29 Sep 2006 09:12:09 -0700 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <200609291723.26896.faltet@carabos.com> References: <200609291723.26896.faltet@carabos.com> Message-ID: <451D45D9.6050005@ieee.org> Francesc Altet wrote: > Hello, > > Is the next a bug a feature? > > In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) > > In [103]: f4 > Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], > dtype=float32) > > In [104]: f4[2] = 2 > --------------------------------------------------------------------------- > Traceback (most recent call last) > > /home/faltet/python.nobackup/numpy/ in () > > : array is not writeable > > In [105]: f4.flags.writeable = True > > In [106]: f4[2] = 2 > > In [107]: f4 > Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.00000000e+00], > dtype=float32) > > > i.e. in an array built from ndarray, the default is that it has to be > read-only? > It's not that the it's being built from ndarray, it's that the buffer that you are passing it is read only. In fact, I'd argue that allowing the writeable flag to be set to True in this case is actually a bug. Consider this slightly modified example: >>> a = "12345"*4 >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) >>> f4[2] = 99 Traceback (most recent call last): File "", line 1, in ? RuntimeError: array is not writeable >>> f4.flags.writeable = True >>> a '12345123451234512345' >>> f4.flags.writeable = True >>> f4[2] = 99 >>> a '12345123\x00\x00\xc6B34512345' The original, *immutable* string has been mutated. This could get you into real trouble in certain situations. -tim From faltet at carabos.com Fri Sep 29 12:44:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 18:44:06 +0200 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D45D9.6050005@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> Message-ID: <200609291844.07928.faltet@carabos.com> A Divendres 29 Setembre 2006 18:12, Tim Hochberg va escriure: > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > > Consider this slightly modified example: > >>> a = "12345"*4 > >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) > >>> f4[2] = 99 > > Traceback (most recent call last): > File "", line 1, in ? > RuntimeError: array is not writeable > > >>> f4.flags.writeable = True > >>> a > > '12345123451234512345' > > >>> f4.flags.writeable = True > >>> f4[2] = 99 > >>> a > > '12345123\x00\x00\xc6B34512345' > > The original, *immutable* string has been mutated. This could get you > into real trouble in certain situations. I see. Thanks for the explanation. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Fri Sep 29 12:55:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 29 Sep 2006 10:55:25 -0600 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D45D9.6050005@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> Message-ID: <451D4FFD.6090202@ieee.org> Tim Hochberg wrote: > Francesc Altet wrote: > > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. This is correct. > In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > It's actually intentional. Strings used as buffers are allowed to be writeable. This is an explicit design decision to allow pickles to load without making 2 copies of the memory. The pickled string that Python creates is used as the actual memory for loaded arrays. Now, I suppose it would be possible to still allow this but be more finnicky about when a string-used-as-the-memory can be set writeable (i.e. we are the only reference to it). But, this would be a fragile solution as well. My inclination is to just warn users not to use strings as buffers unless they know what they are doing. The fact that it is read-only by default is enough of a guard against "accidentally" altering a string you didn't want to alter. -Travis From tim.hochberg at ieee.org Fri Sep 29 12:59:06 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 29 Sep 2006 09:59:06 -0700 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D4FFD.6090202@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> <451D4FFD.6090202@ieee.org> Message-ID: <451D50DA.5030901@ieee.org> Travis Oliphant wrote: > Tim Hochberg wrote: > >> Francesc Altet wrote: >> >> It's not that the it's being built from ndarray, it's that the buffer >> that you are passing it is read only. >> > This is correct. > >> In fact, I'd argue that allowing >> the writeable flag to be set to True in this case is actually a bug. >> >> > It's actually intentional. Strings used as buffers are allowed to be > writeable. This is an explicit design decision to allow pickles to load > without making 2 copies of the memory. The pickled string that Python > creates is used as the actual memory for loaded arrays. > > Now, I suppose it would be possible to still allow this but be more > finnicky about when a string-used-as-the-memory can be set writeable > (i.e. we are the only reference to it). But, this would be a fragile > solution as well. > > My inclination is to just warn users not to use strings as buffers > unless they know what they are doing. The fact that it is read-only by > default is enough of a guard against "accidentally" altering a string > you didn't want to alter. > That makes sense. Someday, when string/unicode => bytes/string, we'll be able to use the bytes type for this and it'll be much cleaner. But not for a while I imagine. -tim From robert.kern at gmail.com Fri Sep 29 13:26:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Sep 2006 12:26:35 -0500 Subject: [Numpy-discussion] Negative values with unsigned data types problems In-Reply-To: <200609291344.35863.faltet@carabos.com> References: <200609291344.35863.faltet@carabos.com> Message-ID: Francesc Altet wrote: > Hi, > > I'm writing this here because the numpy Trac seems down: > > {{{ > Oops... > > Trac detected an internal error: The Trac Environment needs to be upgraded. > Run trac-admin /home/scipy/trac/numpy upgrade" > }}} It's back up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Fri Sep 29 14:19:27 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 29 Sep 2006 12:19:27 -0600 Subject: [Numpy-discussion] Incorrect removal of NULL char in buffers In-Reply-To: <200609291719.02380.faltet@carabos.com> References: <200609291719.02380.faltet@carabos.com> Message-ID: <451D63AF.2070106@ieee.org> Francesc Altet wrote: > Hi, > > > However, for string values, numpy seems to work in a strange way. > The numarray have an expected behaviour, IMO: > > In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) > Out[100]: CharArray(['a', '', 'ba']) > > I'm not sure why you think this is "expected." You have non-terminating NULLs in this array and yet they are not printing for you. Just look at the tostring()... > but numpy haven't: > > In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) > Out[101]: > array([aba, ba, bab], > dtype='|S4') > > i.e. it seems like numpy is striping-off NULL chars before building the object > and I don't think this is correct. > Hmmm. I don't see that at all. This is what I get (version of numpy is 1.0.dev3233) In [33]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[33]: array(['a\x00ba', '\x00ba', 'ba\x00b'], dtype='|S4') which to me is very much expected. I.e. only terminating NULLs are stripped off of the strings on printing. I think you are getting different results because string printing used to not include the quotes (which had the side-effect of not printing NULLs in the middle of strings). They are still there, just not showing up in your output. In the end both numarray and numpy have the same data stored internally. It's just a matter of how it is being printed that seems to differ a bit. From my perspective, only NULLs at the end of strings should be stripped off and that is the (current) behavior of NumPy. You are getting different results, because the array-printing for strings was recently updated (to insert the quotes so that it makes more sense). Without these changes, I think the NULLs were being stripped away on printing. In other words, something like print 'a\x00ba' aba used to be happening. -Travis From coatimundi at hotmail.com Fri Sep 29 17:22:21 2006 From: coatimundi at hotmail.com (Kyle Ferrio) Date: Fri, 29 Sep 2006 15:22:21 -0600 Subject: [Numpy-discussion] Q: NumPy (svn) with cygwin build error? Message-ID: Hello. Starting from a freshly installed cygwin, I have built Python 2.5, ATLAS and LAPACK all from source using gcc 3.4.4 (cygming special) for both the C and f77 compilers. Now I am trying to build NumPy from SVN. I have edited site.cfg so that python setup.py config --compiler=mingw32 finds all of the linear algebra packs and completes without error. Then I fire off python setup.py build --compiler=mingw32 and I soon get (while building extension numpy.random.mtrand) _configtest.c:7:2: #error No _WIN32 which is not immediately fatal but definitely a concern. Soon after, building numpy.core.multiarray gives me this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzzled, because /usr/include/sys/select.h does exist. But maybe that's not what pyport.h is actually looking for? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! _________________________________________________________________ Express yourself - download free Windows Live Messenger themes! http://clk.atdmt.com/MSN/go/msnnkwme0020000001msn/direct/01/?href=http://imagine-msn.com/themes/vibe/default.aspx?locale=en-us&source=hmtagline From oliphant at ee.byu.edu Fri Sep 29 18:27:11 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 29 Sep 2006 16:27:11 -0600 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <200609291844.07928.faltet@carabos.com> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> <200609291844.07928.faltet@carabos.com> Message-ID: <451D9DBF.1030303@ee.byu.edu> Francesc Altet wrote: >I see. Thanks for the explanation. > > You deserve the thanks for the great testing of less-traveled corners of NumPy. It's exactly the kind of thing needed to get NumPy ready for release. -Travis From coatimundi at cox.net Fri Sep 29 20:27:27 2006 From: coatimundi at cox.net (Coatimundi) Date: Fri, 29 Sep 2006 17:27:27 -0700 Subject: [Numpy-discussion] Problem building from svn under cygwin Message-ID: <451DB9EF.6030908@cox.net> I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. Configure works: python setup.py configure --compiler=mingw32 Build fails: python setup.py build --compiler=mingw32 While building numpy.random.mtrand I get this: _configtest.c:7:2: #error No _WIN32 and a little bit later the build aborts completely on multiarray with this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzled, because /usr/include/sys/select.h actually does exist on my cygwin installation. But maybe that's not what pyport.h is actually looking for??? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! From robert.kern at gmail.com Fri Sep 29 20:53:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Sep 2006 19:53:25 -0500 Subject: [Numpy-discussion] Problem building from svn under cygwin In-Reply-To: <451DB9EF.6030908@cox.net> References: <451DB9EF.6030908@cox.net> Message-ID: Coatimundi wrote: > I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. > > Configure works: > > python setup.py configure --compiler=mingw32 > > Build fails: > > python setup.py build --compiler=mingw32 What Python are you using? It looks like you built Python 2.5 using cygwin (that is, the UNIX emulation layer). You cannot build your extensions with mingw32 in that case. Don't use --compiler at all and you will be fine. If you are using the regular Python binaries, then you should be using --compiler=mingw32, but you're getting the wrong Python headers in that case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lars.bittrich at googlemail.com Sat Sep 30 08:16:27 2006 From: lars.bittrich at googlemail.com (Lars Bittrich) Date: Sat, 30 Sep 2006 14:16:27 +0200 Subject: [Numpy-discussion] return type diffences of indexed arrays with Intel C++ compiler (Python 2.5) Message-ID: <200609301416.27222.lars.bittrich@googlemail.com> Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: latest svn(1.0.dev3233) with Intel C compiler (Python 2.5): ----------------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 latest svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. A typical error message from scipy.test(10): ====================================================================== ERROR: check_chebyc (scipy.special.tests.test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/lib/python2.5/site-packages/scipy/special/tests/test_basic.py", line 667, in check_chebyc C1 = chebyc(1) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 461, in chebyc p = orthopoly1d(x,w,hn,kn,wfunc=lambda x: 1.0/sqrt(1-x*x/4.0),limits=(-2,2),monic=monic) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 78, in __init__ equiv_weights = [weights[k] / wfunc(roots[k]) for k in range(len(roots))] TypeError: object of type 'numpy.complex128' has no len() Does anyone know how to resolve the problem? Best regards, Lars From lars.bittrich at googlemail.com Fri Sep 29 08:48:30 2006 From: lars.bittrich at googlemail.com (Lars Bittrich) Date: Fri, 29 Sep 2006 14:48:30 +0200 Subject: [Numpy-discussion] return type diffences of indexed arrays with Intel C++ compiler (Python 2.5) Message-ID: <200609291448.30253.lars.bittrich@googlemail.com> Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: latest svn(1.0.dev3233) with Intel C compiler (Python 2.5): ----------------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 latest svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. A typical error message from scipy.test(10): ====================================================================== ERROR: check_chebyc (scipy.special.tests.test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/lib/python2.5/site-packages/scipy/special/tests/test_basic.py", line 667, in check_chebyc C1 = chebyc(1) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 461, in chebyc p = orthopoly1d(x,w,hn,kn,wfunc=lambda x: 1.0/sqrt(1-x*x/4.0),limits=(-2,2),monic=monic) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 78, in __init__ equiv_weights = [weights[k] / wfunc(roots[k]) for k in range(len(roots))] TypeError: object of type 'numpy.complex128' has no len() Does anyone know how to resolve the problem? Best regards, Lars From hjn253 at tom.com Mon Sep 4 16:29:12 2006 From: hjn253 at tom.com (=?GB2312?B?IjnUwjktMTDI1S/Jz7qjIg==?=) Date: Tue, 5 Sep 2006 04:29:12 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU61MvTw0VYQ0VMuMS9+MrQs6HTqs/60+uyxs7xudzA7Q==?= Message-ID: An HTML attachment was scrubbed... URL: From emilee at firstinsurancefunding.com Fri Sep 1 20:18:43 2006 From: emilee at firstinsurancefunding.com (Guendolen Houze) Date: Fri, 1 Sep 2006 17:18:43 -0700 Subject: [Numpy-discussion] iiR Xje Message-ID: <000001c6ce25$597fb010$96f9a8c0@xqyicw> Hi All yo e ur PHA o RRMAC s Y d f irec g tly from the m t anufa a cture j r, Your ch v an b ce to eco i nomiz v e wit o h us http://omgandesunkerion.com r z y floor was smooth and hard, as were the walls when I brushed my fingers What about her? Steengo asked. be a crackup in the old kaserne tonight! -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 2 22:26:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 2 Sep 2006 20:26:42 -0600 Subject: [Numpy-discussion] Keyword added to searchsorted. Message-ID: Hi all, I added the keyword side to the searchsorted method and functions. Documentation: """a.searchsorted(values=v, side='left') -> array of indices. Required Arguments: v -- keys to be searched for in a. Keyword arguments side -- {'left', 'right'}, default('left'). If a is a 1-D array in ascending order, then a.searchsorted(v, side='left') returns an array of indices i such that for each element of values the following holds: a[j] < key <= a[i] for all j < i, If such an index does not exist, a.size() is used. The result is such that if the key were to be inserted in the slot before the index i, then the order of a would be preserved and i would be the smallest index with that property. If a is a 1-D array in ascending order, then a.searchsorted(v, side='right') returns an array of indices i such that for each element of values the following holds: a[j] <= key < a[i] for all j < i, If such an index does not exist, a.size() is used. The result is that if the key were to be inserted in the slot before the index i, then the order of a would be preserved and i would be the largest index with that property. """ I also replaced the old search routine as it was linear time worst case: In [27]: t1 = t.Timer('N.searchsorted(a,1,side="left")','import numpy as N; a = N.ones(1000000)') In [28]: t2 = t.Timer('N.searchsorted(a,1,side="right")','import numpy as N; a = N.ones(1000000)') In [29]: t1.repeat(3,100) Out[29]: [0.5301368236541748, 0.4924161434173584, 0.46317386627197266] In [30]: t2.repeat(3,100) Out[30]: [0.0011379718780517578, 0.00081586837768554688, 0.00083994865417480469] where left was the original routine. It is now noticeably faster in some situations: In [2]: t1 = t.Timer('N.searchsorted(a,1,side="left")','import numpy as N; a = N.ones(1000000)') In [3]: t1.repeat(3,100) Out[3]: [0.00082802772521972656, 0.00077795982360839844, 0.00076913833618164062] I am open to suggestions as to better names for the keyword kind. It also might be worth it to make type-specific routines if anyone commonly uses large lists of keys and large sorted arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Sep 2 22:37:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 02 Sep 2006 21:37:04 -0500 Subject: [Numpy-discussion] Keyword added to searchsorted. In-Reply-To: References: Message-ID: Charles R Harris wrote: > Hi all, > > I added the keyword side to the searchsorted method and functions. Thank you! Just the other day, I was wishing that we had such a thing. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wfspotz at sandia.gov Sun Sep 3 00:47:55 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Sat, 2 Sep 2006 22:47:55 -0600 Subject: [Numpy-discussion] (no subject) Message-ID: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> I think I see a bug in lib.user_array.container class. The __eq__ method is def __eq__(self,other): return self._rc(equal(self.array,other)) the expression equal(self.array,other) will return an ndarray of bools, which is then converted, by way of self._rc(), to whatever the derived class is. In my case, this does not make sense, because an array of bools is not a valid argument to the constructor (actually, the data buffer is accepted, but the results are not reliable). What I want, and it seem most people would want in this situation, is just the array of bools; i.e. don't apply the self._rc() method. Assuming there is agreement that this is the desirable behavior, the same would be true for __lt__, __le__, etc. I will probably override this behavior by defining my own __eq__, etc., in my derived class, just for safety. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From robert.kern at gmail.com Sun Sep 3 01:10:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 03 Sep 2006 00:10:47 -0500 Subject: [Numpy-discussion] user_array.container and __eq__ [was: Re: (no subject)] In-Reply-To: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> References: <2A2009F1-A9F4-4218-8605-CA66040F9158@sandia.gov> Message-ID: Bill Spotz wrote: > I think I see a bug in lib.user_array.container class. The __eq__ > method is > > def __eq__(self,other): return self._rc(equal(self.array,other)) > > the expression equal(self.array,other) will return an ndarray of > bools, which is then converted, by way of self._rc(), to whatever the > derived class is. In my case, this does not make sense, because an > array of bools is not a valid argument to the constructor (actually, > the data buffer is accepted, but the results are not reliable). What > I want, and it seem most people would want in this situation, is just > the array of bools; i.e. don't apply the self._rc() method. No, it's not a bug. It's just the design of that class: always return the same class of object. Of course, since that class only exists to be subclassed, if you need different behavior from those methods, override them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sun Sep 3 14:20:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 12:20:09 -0600 Subject: [Numpy-discussion] diagflat Message-ID: Travis has introduced the function diagflat for creating diagonal arrays. It flattens whatever array is supplied and returns its values as the diagonal of a 2-D array or matrix. As the function is currently undocumented I thought now might be a good time to discuss other possible choices for the name. Here is a quick list that comes to mind diagflat (current name) asdiagonal todiagonal asdiag todiag ... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fernando.Perez at colorado.edu Sun Sep 3 15:15:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun, 03 Sep 2006 13:15:22 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays Message-ID: <44FB29CA.9070808@colorado.edu> Hi all, I'm wondering if the following difference in behavior of object arrays should be considered a bug. Let a and b be: In [21]: a = [0,1] In [22]: b = [ None, None] If we concatenate a with an empty list, it works: In [23]: numpy.concatenate(([],a)) Out[23]: array([0, 1]) But not so for b: In [24]: numpy.concatenate(([],b)) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/fperez/ ValueError: 0-d arrays can't be concatenated This behavior changed recently (it used to work with r2788), and I realize it's probably part of all the reworkings of the object arrays which have been discussed on the list, and all of whose details I have to admit I haven't followed. But this behavior strikes me as a bit inconsistent, since concatenation with a non-empty object array works fine: In [26]: numpy.concatenate(([None],b)) Out[26]: array([None, None, None], dtype=object) This is biting us in some code which keeps object arrays, because when operations of the kind N.concatenate((some_list_of_objects[:nn],other_object_array)) are taken and nn happens to be 0, the code just explodes. In our case, the variable nn is a runtime computed quantity that comes from a numerical algorithm, for which 0 is a perfectly reasonable value. Are we just misusing things and is there a reasonable alternative, or should this be considered a numpy bug? The r2788 behavior was certainly a lot less surprising as far as our code was concerned. I realize that one alternative is to wrap everything into arrays: N.concatenate((N.asarray(some_list_of_objects[:nn]),other_object_array)) Is this the only solution moving forward, or could the previous behavior be restored without breaking other areas of the new code/design? Thanks for any input, f From nwagner at iam.uni-stuttgart.de Sun Sep 3 15:31:53 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 03 Sep 2006 21:31:53 +0200 Subject: [Numpy-discussion] diagflat In-Reply-To: References: Message-ID: On Sun, 3 Sep 2006 12:20:09 -0600 "Charles R Harris" wrote: > Travis has introduced the function diagflat for creating >diagonal arrays. It > flattens whatever array is supplied and returns its >values as the diagonal > of a 2-D array or matrix. As the function is currently >undocumented I > thought now might be a good time to discuss other >possible choices for the > name. Here is a quick list that comes to mind > > diagflat (current name) > asdiagonal > todiagonal > asdiag > todiag > ... > > Chuck +1 for todiag Nils From charlesr.harris at gmail.com Sun Sep 3 16:43:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 14:43:42 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FB29CA.9070808@colorado.edu> References: <44FB29CA.9070808@colorado.edu> Message-ID: On 9/3/06, Fernando Perez wrote: > > Hi all, > > I'm wondering if the following difference in behavior of object arrays > should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated I think it's propably a bug: >>> concatenate((array([]),b)) array([None, None], dtype=object) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Sep 3 17:54:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 03 Sep 2006 16:54:06 -0500 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> Message-ID: Charles R Harris wrote: > On 9/3/06, *Fernando Perez* > wrote: > > Hi all, > > I'm wondering if the following difference in behavior of object > arrays should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > --------------------------------------------------------------------------- > exceptions.ValueError Traceback > (most recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated > > > I think it's propably a bug: > > >>> concatenate((array([]),b)) > array([None, None], dtype=object) Well, if you can fix it without breaking anything else, then it's a bug. However, I would suggest that a rule of thumb for using object arrays is to always be explicit. Never rely on automatic conversion from Python containers to object arrays. Since Python containers are also objects, it is usually ambiguous what the user meant. I kind of liked numarray's choice to move the object array into a separate constructor. I think that gave them some flexibility to choose different syntax and semantics from the generic array() constructor. Since constructing object arrays is so different from constructing numeric arrays, I think that difference is warranted. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From R.Jager at mapperlithography.com Sun Sep 3 19:00:43 2006 From: R.Jager at mapperlithography.com (R.Jager at mapperlithography.com) Date: Mon, 4 Sep 2006 01:00:43 +0200 Subject: [Numpy-discussion] Remco Jager/mapper is out of the office. Message-ID: I will be out of the office starting 09/04/2006 and will not return until 09/25/2006. I will respond to your message when I return. For urgent matters please contact Ton van de Peut: T.vdpeut at mapperlithography.com From haase at msg.ucsf.edu Sun Sep 3 20:03:39 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 03 Sep 2006 17:03:39 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals Message-ID: <44FB6D5B.5090103@msg.ucsf.edu> Hi, I just learn about the existence of round(). Is the following exposing a bug? >>> N.__version__ '1.0b4' >>> N.array([65.0]) [ 65.] >>> _.round(-1) [ 60.] >>> round(65, -1) 70.0 >>> N.array([65]) [65] >>> _.round(-1) [60] >>> N.array([66]) [66] >>> _.round(-1) [60] >>> N.array([66.2]) [ 66.2] >>> _.round(-1) [ 70.] 65 should round *up* to 70. (when decimals is given as -1) In numpy even 66 rounds down, but 66.2 rounds up ... - Sebastian Haase From charlesr.harris at gmail.com Sun Sep 3 20:18:37 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 18:18:37 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FB6D5B.5090103@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> Message-ID: On 9/3/06, Sebastian Haase wrote: > > Hi, > I just learn about the existence of round(). > Is the following exposing a bug? > >>> N.__version__ > '1.0b4' > >>> N.array([65.0]) > [ 65.] > >>> _.round(-1) > [ 60.] > >>> round(65, -1) > 70.0 > >>> N.array([65]) > [65] > >>> _.round(-1) > [60] > >>> N.array([66]) > [66] > >>> _.round(-1) > [60] > >>> N.array([66.2]) > [ 66.2] > >>> _.round(-1) > [ 70.] > > 65 should round *up* to 70. (when decimals is given as -1) > In numpy even 66 rounds down, but 66.2 rounds up ... Numpy rounds to even. >>> around(arange(10)*.5) array([ 0., 0., 1., 2., 2., 2., 3., 4., 4., 4.]) i.e., when at those x.5 boundaries, round to the nearest even number. Always rounding in the same direction biases the numbers when you have gazillions of them. This is most likely of import when the fpu is rounding intermediate results to register length, but numpy does it too. Floats can be weird anyway, .15, for instance, can't be represented exactly as an ieee float. It does tend to throw folks for a loop when they don't keep this in mind. - Sebastian Haase Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 4 00:55:57 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 3 Sep 2006 22:55:57 -0600 Subject: [Numpy-discussion] Random number generators Message-ID: Hi Robert, I am about to get started on some stuff for the random number generators but thought I would run it by you first. I envisage the following: uniform short_doubles -- doubles generated from a single 32 bit random number (advantage: speed) uniform double, short_doubles on the interval (0,1) -- don't touch singularities in functions like log (this is my preferred default) fast_normal -- ziggurat method using single 32 bit random numbers (advantage: speed) fast_exponential -- ziggurat method using single 32 bit random numbers (advantage: speed) MWC8222 random number generator (advantage: speed on some machines, different from mtrand) Except for the last, none conflict with current routines and can be added without a branch. I expect adding MWC8222 might need more extensive work and I will branch for that. So the questions are of utility and naming. I see some utility for myself, otherwise I wouldn't be considering doing the work. OTOH, I already have (C++) routines that I use for these things, so a larger question might be if anyone else sees a use for these. I like speed, but it is not always that important in everyday apps. I see that Pyrex is used for the interface, so I suppose that is one more tool to become familiar with ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Sep 4 01:34:28 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:34:28 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Charles R Harris wrote: > Except for the last, none conflict with current routines and can be added > without a branch. I expect adding MWC8222 might need more extensive work and > I will branch for that. So the questions are of utility and naming. I see > some utility for myself, otherwise I wouldn't be considering doing the work. > OTOH, I already have (C++) routines that I use for these things, so a larger > question might be if anyone else sees a use for these. I like speed, but it > is not always that important in everyday apps. How painful would it be to allow users to drop in their own sources of raw random numbers? A couple of years ago I was trying to analyze some X-ray timing data and we saw some peaks at the end of a long chain of periodicity-detecting software (Fourier transforms, parameter searching, et cetera). To rule out the possibility that they were coming from the random numbers we were using at one stage, I wanted to supply raw random numbers from /dev/random. Unfortunately, that forced me to write my own program to convert uniform random numbers to exponential. Allowing the function that generates raw random bytes to be overwritten would be extremely handy, even if it were painfully slow. In the same project I also noticed it would be nice to be able to (say) do "exponential(2+sin(arange(10)))" to get an array of exponentials with varying parameters. I realize all this is outside the scope of what you were asking. I would have found another pseudorandom number generator that I could just switch to a benefit; the rest of them are less exciting. When you say "32-bit integers" do you really mean 32-bit, or do you mean "machine width"? 32-bit integers may not be faster on 64-bit platforms... A. M. Archibald From haase at msg.ucsf.edu Mon Sep 4 01:42:17 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 03 Sep 2006 22:42:17 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> Message-ID: <44FBBCB9.6050408@msg.ucsf.edu> Charles R Harris wrote: > > > On 9/3/06, *Sebastian Haase* > wrote: > > Hi, > I just learn about the existence of round(). > Is the following exposing a bug? > >>> N.__version__ > '1.0b4' > >>> N.array([65.0]) > [ 65.] > >>> _.round(-1) > [ 60.] > >>> round(65, -1) > 70.0 > >>> N.array([65]) > [65] > >>> _.round(-1) > [60] > >>> N.array([66]) > [66] > >>> _.round(-1) > [60] > >>> N.array([66.2]) > [ 66.2] > >>> _.round(-1) > [ 70.] > > 65 should round *up* to 70. (when decimals is given as -1) > In numpy even 66 rounds down, but 66.2 rounds up ... > > > Numpy rounds to even. > > >>> around(arange(10)*.5) > array([ 0., 0., 1., 2., 2., 2., 3., 4., 4., 4.]) > > i.e., when at those x.5 boundaries, round to the nearest even number. > Always rounding in the same direction biases the numbers when you have > gazillions of them. This is most likely of import when the fpu is > rounding intermediate results to register length, but numpy does it too. > Floats can be weird anyway, .15, for instance, can't be represented > exactly as an ieee float. It does tend to throw folks for a loop when > they don't keep this in mind. > > - Sebastian Haase > > > Chuck Thanks for the reply - but read what the doc says: >>> N.around.__doc__ 'Round 'a' to the given number of decimal places. Rounding behaviour is equivalent to Python. Return 'a' if the array is not floating point. Round both the real and imaginary parts separately if the array is complex. ' it is *not* done this way in Python: >>> round(.5) 1.0 >>> round(1.5) 2.0 >>> round(2.5) 3.0 ( the round obj. method is missing this doc string ) I really think we should stick to what the doc string say - everybody expects x.5 to round up !! look even at this: >>> repr(2.05) '2.0499999999999998' >>> round(2.05, 1) 2.1 (don't ask me how Python does this, but it's "nice" ) Thanks, Sebastian From peridot.faceted at gmail.com Mon Sep 4 01:43:10 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:43:10 -0400 Subject: [Numpy-discussion] diagflat In-Reply-To: References: Message-ID: On 03/09/06, Charles R Harris wrote: > Travis has introduced the function diagflat for creating diagonal arrays. It > flattens whatever array is supplied and returns its values as the diagonal > of a 2-D array or matrix. As the function is currently undocumented I > thought now might be a good time to discuss other possible choices for the > name. Here is a quick list that comes to mind > > diagflat (current name) > asdiagonal > todiagonal > asdiag > todiag > ... I was wishing for that just the other day... I think my vote is for "diagflat", to emphasize the initial flattening. In particular, there's another function I could wish for: given an array, pick an axis, treat the array like an array of vectors along that axis, and replace each vector with the diagonal matrix. So, supposing that it was called todiagonal: >>> A = todiagonal(ones(2,3,4),axis=1) >>> shape(A) (2,3,3,4) >>> A[1,:,:,2] array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) Specifically I find the forcible flattening a bit monstrous. And you can always just do todiagonal(A.flat). No such function seems to exist at the moment, so I'm attaching one; probably not as efficient as it could be with appropriate fancy indexing wizardry. A. M. Archibald -------------- next part -------------- A non-text attachment was scrubbed... Name: diag.py Type: text/x-python Size: 841 bytes Desc: not available URL: From robert.kern at gmail.com Mon Sep 4 01:47:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 00:47:11 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: Charles R Harris wrote: > Hi Robert, > > I am about to get started on some stuff for the random number generators > but thought I would run it by you first. I envisage the following: > > uniform short_doubles -- doubles generated from a single 32 bit random > number (advantage: speed) > uniform double, short_doubles on the interval (0,1) -- don't touch > singularities in functions like log (this is my preferred default) > fast_normal -- ziggurat method using single 32 bit random numbers > (advantage: speed) > fast_exponential -- ziggurat method using single 32 bit random numbers > (advantage: speed) > MWC8222 random number generator (advantage: speed on some machines, > different from mtrand) > > Except for the last, none conflict with current routines and can be > added without a branch. I expect adding MWC8222 might need more > extensive work and I will branch for that. So the questions are of > utility and naming. I see some utility for myself, otherwise I wouldn't > be considering doing the work. OTOH, I already have (C++) routines that > I use for these things, so a larger question might be if anyone else > sees a use for these. I like speed, but it is not always that important > in everyday apps. I would prefer not to expand the API of numpy.random. If it weren't necessary for numpy to provide all of the capabilities that came with Numeric's RandomArray, I wouldn't want numpy.random in there at all. Now, a very productive course of action would be to refactor numpy.random such that the distributions (the first four items on your list fall into this category) and the underlying PRNG (the fifth) are separated from one another such that they can be mixed and matched at runtime. A byproduct of this would expose the C API of both of these in order to be usable by other C extension modules, something that's been asked for about a dozen times now. The five items on your list could be implemented in an extension module distributed in scipy. > I see that Pyrex is used for the interface, so I suppose that is one > more tool to become familiar with ;) Possibly not. Pyrex was getting in the way of exposing a C API the last time I took a stab at it. A possibility that just occurred to me is to make an extension module that *only* exposes the C API and mtrand could be rewritten to use that API. Hmmm. I like that. I can give some guidance about how to proceed and help you navigate the current code, but I'm afraid I don't have much time to actually code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 01:53:40 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 01:53:40 -0400 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FBBCB9.6050408@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: On 04/09/06, Sebastian Haase wrote: > Thanks for the reply - but read what the doc says: > >>> N.around.__doc__ > 'Round 'a' to the given number of decimal places. Rounding > behaviour is equivalent to Python. > > Return 'a' if the array is not floating point. Round both the real > and imaginary parts separately if the array is complex. > ' > > it is *not* done this way in Python: > >>> round(.5) > 1.0 > >>> round(1.5) > 2.0 > >>> round(2.5) > 3.0 > > ( the round obj. method is missing this doc string ) Um, well, the doc is wrong. Numpy should *not* always follow python's lead, and in partciular it's explicit that Numeric's floating point is closer to IEEE floating-point behaviour than python's - compare, for example 1./0. and array(1.)/0. > I really think we should stick to what the doc string say - everybody > expects x.5 to round up !! Not everybody. This (admittedly odd) behaviour wasn't decided on because it was more efficient to implement, it was decided on because a group of very smart numerical analysts agreed that it was the best way to avoid surprising naive users with biased results. (Non-naive users can normally pick from any of several other rounding modes if they want.) A better question to ask is, "Can I change numpy's rounding behaviour for my programs?" (And, more generally, "can I set all the various floating-point options that the IEEE standard and my processor both support?") I don't know the answer to that one, but it does seem to be a goal that numpy is trying for (hence, for example, the difference between numpy float scalars and python floats). A. M. Archibald From robert.kern at gmail.com Mon Sep 4 01:51:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 00:51:19 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: A. M. Archibald wrote: > In the same project I also noticed it would be nice to be able to > (say) do "exponential(2+sin(arange(10)))" to get an array of > exponentials with varying parameters. Travis O. recently added this capability. The distribution parameters (loc, scale, alpha, whatever) are broadcast against each other using the same rules as ufunc parameters. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 02:00:56 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 02:00:56 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Robert Kern wrote: > A. M. Archibald wrote: > > > In the same project I also noticed it would be nice to be able to > > (say) do "exponential(2+sin(arange(10)))" to get an array of > > exponentials with varying parameters. > > Travis O. recently added this capability. The distribution parameters (loc, > scale, alpha, whatever) are broadcast against each other using the same rules as > ufunc parameters. Wonderful! Thank you, Travis O. A. M. Archibald From charlesr.harris at gmail.com Mon Sep 4 02:42:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 00:42:23 -0600 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 9/3/06, Robert Kern wrote: > > Charles R Harris wrote: > > Hi Robert, > > > > I am about to get started on some stuff for the random number generators > > but thought I would run it by you first. I envisage the following: > > > > uniform short_doubles -- doubles generated from a single 32 bit random > > number (advantage: speed) > > uniform double, short_doubles on the interval (0,1) -- don't touch > > singularities in functions like log (this is my preferred default) > > fast_normal -- ziggurat method using single 32 bit random numbers > > (advantage: speed) > > fast_exponential -- ziggurat method using single 32 bit random numbers > > (advantage: speed) > > MWC8222 random number generator (advantage: speed on some machines, > > different from mtrand) > > > > Except for the last, none conflict with current routines and can be > > added without a branch. I expect adding MWC8222 might need more > > extensive work and I will branch for that. So the questions are of > > utility and naming. I see some utility for myself, otherwise I wouldn't > > be considering doing the work. OTOH, I already have (C++) routines that > > I use for these things, so a larger question might be if anyone else > > sees a use for these. I like speed, but it is not always that important > > in everyday apps. > > I would prefer not to expand the API of numpy.random. If it weren't > necessary > for numpy to provide all of the capabilities that came with Numeric's > RandomArray, I wouldn't want numpy.random in there at all. Yes, good point. Now, a very productive course of action would be to refactor numpy.randomsuch > that the distributions (the first four items on your list fall into this > category) and the underlying PRNG (the fifth) are separated from one > another > such that they can be mixed and matched at runtime. A byproduct of this > would > expose the C API of both of these in order to be usable by other C > extension > modules, something that's been asked for about a dozen times now. The five > items > on your list could be implemented in an extension module distributed in > scipy. What sort of api should this be? It occurs to me that there are already 4 sources of random bytes: Initialization: /dev/random (pseudo random, I think) /dev/urandom crypto system on windows Pseudo random generators: mtrand I suppose we could add some cryptologically secure source as well. That indicates to me that one set of random number generators would just be streams of random bytes, possibly in 4 byte chunks. If I were doing this for linux these would all look like file systems, FUSE comes to mind. Another set of functions would transform these into the different distributions. So, how much should stay in numpy? What sort of API are folks asking for? > I see that Pyrex is used for the interface, so I suppose that is one > > more tool to become familiar with ;) > > Possibly not. Pyrex was getting in the way of exposing a C API the last > time I > took a stab at it. A possibility that just occurred to me is to make an > extension module that *only* exposes the C API and mtrand could be > rewritten to > use that API. Hmmm. I like that. Good, I can do without pyrex. I can give some guidance about how to proceed and help you navigate the > current > code, but I'm afraid I don't have much time to actually code. Thanks, that is all I ask. -- > Robert Kern Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 4 04:31:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 03:31:02 -0500 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: Charles R Harris wrote: > What sort of api should this be? It occurs to me that there are already > 4 sources of random bytes: > > Initialization: > > /dev/random (pseudo random, I think) > /dev/urandom > crypto system on windows > > Pseudo random generators: > > mtrand > > I suppose we could add some cryptologically secure source as well. That > indicates to me that one set of random number generators would just be > streams of random bytes, possibly in 4 byte chunks. If I were doing this > for linux these would all look like file systems, FUSE comes to mind. > Another set of functions would transform these into the different > distributions. So, how much should stay in numpy? What sort of API are > folks asking for? It would be good to also support quasi-random generators like Sobol and Halton sequences. They tend to only generate floating point numbers in [0, 1] (or (0, 1) or [0, 1) or (0, 1]; I think it varies). There are probably other PRNGs that only output floating point numbers, but I doubt we want to implement any of them. You can look at the 1.6 version of RandomKit for an implementation of Sobol sequences and ISAAC, a cryptographic PRNG. http://www.jeannot.org/~js/code/index.en.html I'm thinking that the core struct that we're going to pass around will look something like this: struct random_state { void* state; unsigned int (*gen_uint32)(struct random_state*); double (*gen_double)(struct random_state*); } state -- A pointer to the generator-specific struct. gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. Possibly NULL if the generator can not implement such a generator. gen_double -- A function pointer that yields a double-precision number in [0, 1] (possibly omitting one or both of the endpoints depending on the implementation). Possibly this struct needs to be expanded to include function pointers for destroying the state struct and for a copying the state to a new object. The seeding function(s) probably don't need to travel in the struct. We should determine if C++ will work for us, here. Unfortunately, that will force all extension modules that want to use this API to be C++ modules. One other feature of the current implementation of state struct is that a few of the distributions cache information between calls. Notably, the Box-M?ller Gaussian distribution algorithm generates two normal deviates at a time; one is returned, the other is cached until the next call. The binomial distribution computes some intermediate values that are reused if the next call has the same parameters. We should probably store such things separately from the core state struct this time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Sep 4 10:34:36 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 10:34:36 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Robert Kern wrote: > Charles R Harris wrote: > > > What sort of api should this be? It occurs to me that there are already > > 4 sources of random bytes: > > > > Initialization: > > > > /dev/random (pseudo random, I think) /dev/random is (supposed to be) true randomness derived from various sources. It's also fed through a bunch of hashing and CRC functions chosen for convenience, but is probably pretty unbiased. > > /dev/urandom > > crypto system on windows > > > > Pseudo random generators: > > > > mtrand > > > > I suppose we could add some cryptologically secure source as well. That > > indicates to me that one set of random number generators would just be > > streams of random bytes, possibly in 4 byte chunks. If I were doing this > > for linux these would all look like file systems, FUSE comes to mind. > > Another set of functions would transform these into the different > > distributions. So, how much should stay in numpy? What sort of API are > > folks asking for? "Cryptographically secure random number generator" means something different to everybody, and implementing one that everyone is happy with is going to be a colossal pain. Look at Yarrow and its design documents for some idea of the complexities involved. However, a random number generator based on a stream cipher is probably going to be pretty good, if slow, so implementingone if those can't hurt. But I think leaving the possibility open that one could use custom sources of random bytes is a very good idea. There's no particular need that custom ones should be fast, as they're for use when you really want *good* random numbers. > It would be good to also support quasi-random generators like Sobol and Halton > sequences. They tend to only generate floating point numbers in [0, 1] (or (0, > 1) or [0, 1) or (0, 1]; I think it varies). There are probably other PRNGs that > only output floating point numbers, but I doubt we want to implement any of > them. You can look at the 1.6 version of RandomKit for an implementation of > Sobol sequences and ISAAC, a cryptographic PRNG. > > http://www.jeannot.org/~js/code/index.en.html Sobol' sequences may require special care in constructing non-uniform distributions in order to preserve their uniformity properties. If we can simply import ISAAC, it won't hurt, but I'd rather trust (say) AES in counter mode than some custom cryptosystem that hasn't been analyzed as thoroughly. > I'm thinking that the core struct that we're going to pass around will look > something like this: > > struct random_state { > void* state; > unsigned int (*gen_uint32)(struct random_state*); > double (*gen_double)(struct random_state*); > } > > state -- A pointer to the generator-specific struct. > > gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. Possibly > NULL if the generator can not implement such a generator. > > gen_double -- A function pointer that yields a double-precision number in [0, 1] > (possibly omitting one or both of the endpoints depending on the implementation). Are 32-bit numbers really the right least common denominator? There are plenty of 64-bit platforms out there... Given this API, implementing a subclassable class that exports it should satisfy most people's needs for interchangeable generators. A. M. Archibald From pjssilva at ime.usp.br Mon Sep 4 11:08:28 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 12:08:28 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FBBCB9.6050408@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: <1157382509.3087.14.camel@localhost.localdomain> Interesting, I was just reading about the round rule in IEEE standard last Friday. What numpy's "around" function does is called "round to even" (round is take to make the next digit even), instead of "round up". According to "What every computer scientist should know about floating-point arithmetic" (do a Google search), the reason to prefer "round to even" is exemplified by the following result from Reiser and Knuth: Theorem Let x and y be floating-point numbers, and define x0 = x, x1 = (x0 - y) + y, ..., xn = (xn-1 - y) + y. If + and - are exactly rounded using round to even, then either xn = x for all n or xn = x1 for all n >= 1. If you use "round up" the sequence xn can start increasing slowly "for ever", and as the paper says: "...This example suggests that when using the round up rule, computations can gradually drift upward, whereas when using round to even the theorem says this cannot happen." Best, Paulo From charlesr.harris at gmail.com Mon Sep 4 11:30:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 09:30:04 -0600 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 9/4/06, A. M. Archibald wrote: > > On 04/09/06, Robert Kern wrote: > > Charles R Harris wrote: > > > > > What sort of api should this be? It occurs to me that there are > already > > > 4 sources of random bytes: > > > > > > Initialization: > > > > > > /dev/random (pseudo random, I think) > > /dev/random is (supposed to be) true randomness derived from various > sources. It's also fed through a bunch of hashing and CRC functions > chosen for convenience, but is probably pretty unbiased. > > > > /dev/urandom > > > crypto system on windows > > > > > > Pseudo random generators: > > > > > > mtrand > > > > > > I suppose we could add some cryptologically secure source as well. > That > > > indicates to me that one set of random number generators would just be > > > streams of random bytes, possibly in 4 byte chunks. If I were doing > this > > > for linux these would all look like file systems, FUSE comes to mind. > > > Another set of functions would transform these into the different > > > distributions. So, how much should stay in numpy? What sort of API are > > > folks asking for? > > "Cryptographically secure random number generator" means something > different to everybody, and implementing one that everyone is happy > with is going to be a colossal pain. Look at Yarrow and its design > documents for some idea of the complexities involved. > > However, a random number generator based on a stream cipher is > probably going to be pretty good, if slow, so implementingone if those > can't hurt. But I think leaving the possibility open that one could > use custom sources of random bytes is a very good idea. There's no > particular need that custom ones should be fast, as they're for use > when you really want *good* random numbers. I was thinking it might be nice to have one, just to generate seeds if nothing else. This could actually be written in python and use something like the python cryptography toolkit, no need to reinvent the wheel. It might be worth looking at that code just to see how someone else thought through the interface. It's under the Python license, so I need to know if that license is compatible with the BSD license used for numpy. > It would be good to also support quasi-random generators like Sobol and > Halton > > sequences. They tend to only generate floating point numbers in [0, 1] > (or (0, > > 1) or [0, 1) or (0, 1]; I think it varies). There are probably other > PRNGs that > > only output floating point numbers, but I doubt we want to implement any > of > > them. You can look at the 1.6 version of RandomKit for an implementation > of > > Sobol sequences and ISAAC, a cryptographic PRNG. > > > > http://www.jeannot.org/~js/code/index.en.html > > Sobol' sequences may require special care in constructing non-uniform > distributions in order to preserve their uniformity properties. If we > can simply import ISAAC, it won't hurt, but I'd rather trust (say) AES > in counter mode than some custom cryptosystem that hasn't been > analyzed as thoroughly. > > > I'm thinking that the core struct that we're going to pass around will > look > > something like this: > > > > struct random_state { > > void* state; > > unsigned int (*gen_uint32)(struct random_state*); > > double (*gen_double)(struct random_state*); > > } > > > > state -- A pointer to the generator-specific struct. > > > > gen_uint32 -- A function pointer that yields a 32-bit unsigned integer. > Possibly > > NULL if the generator can not implement such a generator. > > > > gen_double -- A function pointer that yields a double-precision number > in [0, 1] > > (possibly omitting one or both of the endpoints depending on the > implementation). > > Are 32-bit numbers really the right least common denominator? There > are plenty of 64-bit platforms out there... MWC8222 runs faster (2x) on some 64 bit platforms because it multiplies two 32 bit numbers and uses the 64 bit product. The mtrand generator really uses a polynomial with coefficients in z_2 and has 624 words worth, an even number, so could probably be adapted to run directly with 64 bit registers although the gain might not be much. Most other generators I know of produce 32 bit numbers. Not that I am an expert on the subject. Given this API, implementing a subclassable class that exports it > should satisfy most people's needs for interchangeable generators. A. M. Archibald Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 4 12:32:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 10:32:52 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <1157382509.3087.14.camel@localhost.localdomain> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> Message-ID: On 9/4/06, Paulo J. S. Silva wrote: > > Interesting, I was just reading about the round rule in IEEE standard > last Friday. > > What numpy's "around" function does is called "round to even" (round is > take to make the next digit even), instead of "round up". According to > "What every computer scientist should know about floating-point > arithmetic" (do a Google search), the reason to prefer "round to even" > is exemplified by the following result from Reiser and Knuth: > > Theorem > > Let x and y be floating-point numbers, and define x0 = x, x1 = (x0 - y) > + y, ..., xn = (xn-1 - y) + y. If + and - are exactly rounded using > round to even, then either xn = x for all n or xn = x1 for all n >= 1. > > If you use "round up" the sequence xn can start increasing slowly "for > ever", and as the paper says: > > "...This example suggests that when using the round up rule, > computations can gradually drift upward, whereas when using round to > even the theorem says this cannot happen." > > Best, > > Paulo However, around does set the sign bit when rounding -.5 to zero: >>> around(-.5).tostring() '\x00\x00\x00\x00\x00\x00\x00\x80' so that >>> around(-.5) -0.0 on the other hand >>> around(-.5) == 0.0 True I didn't know -0.0 was part of the IEEE spec. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Mon Sep 4 13:07:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 14:07:05 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> Message-ID: <1157389625.3087.19.camel@localhost.localdomain> Once again, the information that singed zero is part of IEEE standard is in the paper I cited in my last message. It is very important to be able to compute the sign of an overflowed quantity in expressions like 1/x when x goes to zero. Best, Paulo From peridot.faceted at gmail.com Mon Sep 4 14:17:23 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 4 Sep 2006 14:17:23 -0400 Subject: [Numpy-discussion] Random number generators In-Reply-To: References: Message-ID: On 04/09/06, Charles R Harris wrote: > > > > On 9/4/06, A. M. Archibald wrote: > > On 04/09/06, Robert Kern wrote: > > > Charles R Harris wrote: > > However, a random number generator based on a stream cipher is > > probably going to be pretty good, if slow, so implementingone if those > > can't hurt. But I think leaving the possibility open that one could > > use custom sources of random bytes is a very good idea. There's no > > particular need that custom ones should be fast, as they're for use > > when you really want *good* random numbers. > > > I was thinking it might be nice to have one, just to generate seeds if > nothing else. This could actually be written in python and use something > like the python cryptography toolkit, no need to reinvent the wheel. It > might be worth looking at that code just to see how someone else thought > through the interface. It's under the Python license, so I need to know if > that license is compatible with the BSD license used for numpy. Of the generators they have, only their random pool is of any use for seeding, and even that needs some clever feeding of random data. However, on practically any platofrm this will run on, random bytes based on real entropy can be extracted from the system (/dev/random, windows crypto API, something on the Mac); a uniform API for those would be a handy thing to have. But I can't recommend spending much effort connecting the python crypto stuff to numpy's random number generators; it's the sort of thing application writers can easily cook up by subclassing RandomGenerator (or whatever). A. M. Archibald From haase at msg.ucsf.edu Mon Sep 4 15:03:51 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 04 Sep 2006 12:03:51 -0700 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <1157389625.3087.19.camel@localhost.localdomain> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> Message-ID: <44FC7897.5010005@msg.ucsf.edu> Paulo J. S. Silva wrote: > Once again, the information that singed zero is part of IEEE standard is > in the paper I cited in my last message. > > It is very important to be able to compute the sign of an overflowed > quantity in expressions like 1/x when x goes to zero. > > Best, > > Paulo Hi, This is all very interesting ( and confusing (to me, maybe others) at the same time ...) ... Question 0: Are you sure this is not a bug ? >>> N.array([66]).round(-1) [60] >>> N.array([66.2]).round(-1) [ 70.] Question 1: Was this way of round already in Numeric and /or in numarray ? Question 2: Does this need to be better documented (complete and corrected(!) docstrings - maybe a dedicated wiki page ) !? This is related to "How does Matlab or IDL or others do rounding ?" - This would at least determine how many people would be surprised by this. (I would say that *if* Matlab rounds the same way, we might need less documentation ...) Thanks, Sebastian Haase From fperez.net at gmail.com Mon Sep 4 15:18:45 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 4 Sep 2006 13:18:45 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> Message-ID: On 9/3/06, Robert Kern wrote: > > I think it's propably a bug: > > > > >>> concatenate((array([]),b)) > > array([None, None], dtype=object) > > Well, if you can fix it without breaking anything else, then it's a bug. > > However, I would suggest that a rule of thumb for using object arrays is to > always be explicit. Never rely on automatic conversion from Python containers to > object arrays. Since Python containers are also objects, it is usually ambiguous > what the user meant. > > I kind of liked numarray's choice to move the object array into a separate > constructor. I think that gave them some flexibility to choose different syntax > and semantics from the generic array() constructor. Since constructing object > arrays is so different from constructing numeric arrays, I think that difference > is warranted. This is something that should probably be sorted out before 1.0 is out. IMHO, the current behavior is a bit too full of subtle pitfalls to be a good long-term solution, and I think that predictability trumps convenience for a good API. I know that N.array() is already very complex; perhaps the idea of moving all object array construction into a separate function would be a long-term win. I think that object arrays are actually very important for numpy: they provide the bridge between pure numerical, Fortran-like computing and the richer world of Python datatypes and complex objects. But it's important to acknowledge this bridge character: they connect into a world where the basic assumptions of homogeneity of numpy arrays don't apply anymore. I'd be +1 on forcing this acknowledgement by having a separate N.oarray constructor, accessible via the dtype flag to N.array as well, but /without/ N.array trying to invoke it automatically by guessing the contents of what it was fed. The downside of this approach is that much of the code that 'magically' just works with N.array(foo) today, would now break. I'm becoming almost of the opinion that the code is broken already, it just hasn't failed yet :) Over time, I've become more and more paranoid of what constructors (and factory-type functions that build objects) do with their inputs, how strongly they validate them, and how explicit they require their users to be about their intent. While I'm a big fan of Python's duck-typing, I've also learned (the hard way) that in larger codebases, the place to be very strict about input validation is object constructors. It's easy to let garbage seep into an object by not validating a constructor input, and to later (often MUCH later) have your code bizarrely explode with an unrecognizable exception, because that little piece of garbage was fed to some third-party code which was expecting something else. If I've understood history correctly, some of the motivations behind Enthought's Traits are of a similar nature. For now I've dealt with our private problem that spurred my original posting. But I think this issue is worth clarifying for numpy, before 1.0 paints us into a backwards-compatibility corner with a fairly fundamental datatype and constructor. Regards, f From charlesr.harris at gmail.com Mon Sep 4 15:40:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 13:40:01 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FC7897.5010005@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: On 9/4/06, Sebastian Haase wrote: > > Paulo J. S. Silva wrote: > > Once again, the information that singed zero is part of IEEE standard is > > in the paper I cited in my last message. > > > > It is very important to be able to compute the sign of an overflowed > > quantity in expressions like 1/x when x goes to zero. > > > > Best, > > > > Paulo > Hi, > > This is all very interesting ( and confusing (to me, maybe others) at > the same time ...) ... > > Question 0: Are you sure this is not a bug ? > >>> N.array([66]).round(-1) > [60] That does look like a bug. >>> array([66]).round(-1) array([60]) >>> array([66.]).round(-1) array([ 70.]) I suspect it is related to the integer data type of the first example. The code probably does something like this: round(66/10)*10 and 66/10 == 6 in integer arithmetic. Chuck >>> N.array([66.2]).round(-1) > [ 70.] > > Question 1: Was this way of round already in Numeric and /or in numarray ? Don't recall. Question 2: Does this need to be better documented (complete and > corrected(!) docstrings - maybe a dedicated wiki page ) !? > This is related to "How does Matlab or IDL or others do rounding ?" - > This would at least determine how many people would be surprised by this. > (I would say that *if* Matlab rounds the same way, we might need less > documentation ...) I have improved the document strings and the example at scipy.org. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Mon Sep 4 15:41:16 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Mon, 04 Sep 2006 16:41:16 -0300 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: <44FC7897.5010005@msg.ucsf.edu> References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: <1157398876.3087.35.camel@localhost.localdomain> Ops, I believe you were caught by int versus float here: In [16]:around(66.0, decimals=-1) Out[16]:70.0 In [17]:around(66, decimals=-1) Out[17]:60 Note that in the int case, it seems like round is simply truncation. Paulo From cookedm at physics.mcmaster.ca Mon Sep 4 16:29:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 4 Sep 2006 16:29:06 -0400 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> Message-ID: <20060904162906.7ac14de9@arbutus.physics.mcmaster.ca> On Mon, 4 Sep 2006 01:53:40 -0400 "A. M. Archibald" wrote: > > A better question to ask is, "Can I change numpy's rounding behaviour > for my programs?" (And, more generally, "can I set all the various > floating-point options that the IEEE standard and my processor both > support?") I don't know the answer to that one, but it does seem to be > a goal that numpy is trying for (hence, for example, the difference > between numpy float scalars and python floats). (looks) I think we've missed rounding. And the inexact flag. You can control how overflow, underflow, divide-by-0, and invalid-operand are handled using geterr and seterr, though. Unfortunately, getting/setting the rounding mode is hard to do in a platform-independent way :( gcc has fegetround() and fesetround() (they're part of the C99 standard, I believe). I don't know what to use on other machines (esp. Windows with MSVC), although the Boost Interval library looks like a good place to start: http://boost.cvs.sourceforge.net/boost/boost/boost/numeric/interval/detail/ -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rw679aq02 at sneakemail.com Mon Sep 4 17:27:51 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 14:27:51 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1156926065.27232.269739888@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> Message-ID: <1157405271.21974.270111020@webmail.messagingengine.com> Forgive me if I missed any replies. Since I have seen none, I will rephrase the request. Please demonstrate an irregular array in numpy with time complexity measurement. The shape does not matter so long as it is non-rectangular and includes a complexity measurement. A sparse matrix is conceptually rectangular, so it does not fit the request at all. The question is whether numpy has such support; if not, is it planned. The question is not about the general-purpose Python language. (Although Python demonstrations are of interest if they include the time complexity measure from first principles or experiment.) From charlesr.harris at gmail.com Mon Sep 4 17:53:25 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 15:53:25 -0600 Subject: [Numpy-discussion] bug in round with negative number of decimals In-Reply-To: References: <44FB6D5B.5090103@msg.ucsf.edu> <44FBBCB9.6050408@msg.ucsf.edu> <1157382509.3087.14.camel@localhost.localdomain> <1157389625.3087.19.camel@localhost.localdomain> <44FC7897.5010005@msg.ucsf.edu> Message-ID: On 9/4/06, Charles R Harris wrote: > > > > On 9/4/06, Sebastian Haase wrote: > > > > Paulo J. S. Silva wrote: > > > Once again, the information that singed zero is part of IEEE standard > > is > > > in the paper I cited in my last message. > > > > > > It is very important to be able to compute the sign of an overflowed > > > quantity in expressions like 1/x when x goes to zero. > > > > > > Best, > > > > > > Paulo > > Hi, > > > > This is all very interesting ( and confusing (to me, maybe others) at > > the same time ...) ... > > > > Question 0: Are you sure this is not a bug ? > > >>> N.array([66]).round(-1) > > [60] > > > That does look like a bug. > > >>> array([66]).round(-1) > array([60]) > >>> array([66.]).round(-1) > array([ 70.]) > > I suspect it is related to the integer data type of the first example. The > code probably does something like this: > > round(66/10)*10 > > and 66/10 == 6 in integer arithmetic. > The problem is somewhere in this bit of code: // f will be a double f = PyFloat_FromDouble(power_of_ten(decimals)); if (f==NULL) return NULL; // op1 will be division ret = PyObject_CallFunction(op1, "OOO", a, f, out); if (ret==NULL) goto finish; // round in place. tmp = PyObject_CallFunction(n_ops.rint, "OO", ret, ret); if (tmp == NULL) {Py_DECREF(ret); ret=NULL; goto finish;} I suspect the problem is the division, which has integer 'a', double 'f', and integer 'out'. Integer out is probably the killer. Let's see: # use doubles for output >>> out = zeros(1) >>> array([66]).round(decimals=-1, out=out) array([ 70.]) yep, that's the problem Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 4 17:58:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Sep 2006 16:58:29 -0500 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157405271.21974.270111020@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: rw679aq02 at sneakemail.com wrote: > Forgive me if I missed any replies. Since I have seen none, I will > rephrase the request. > > Please demonstrate an irregular array in numpy with time complexity > measurement. The shape does not matter so long as it is non-rectangular > and includes a complexity measurement. A sparse matrix is conceptually > rectangular, so it does not fit the request at all. > > The question is whether numpy has such support; if not, is it planned. No, and no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Mon Sep 4 18:34:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 04 Sep 2006 16:34:08 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FB29CA.9070808@colorado.edu> References: <44FB29CA.9070808@colorado.edu> Message-ID: <44FCA9E0.4040908@ieee.org> Fernando Perez wrote: > Hi all, > > I'm wondering if the following difference in behavior of object arrays should > be considered a bug. Let a and b be: > > In [21]: a = [0,1] > > In [22]: b = [ None, None] > > If we concatenate a with an empty list, it works: > > In [23]: numpy.concatenate(([],a)) > Out[23]: array([0, 1]) > > But not so for b: > > In [24]: numpy.concatenate(([],b)) > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most recent > call last) > > /home/fperez/ > > ValueError: 0-d arrays can't be concatenated > This is a result of PyArray_FromAny changing when object arrays are explicitly requested (which they are in this case --- although behind the scenes). I decided to revert to the previous behavior and only use the Object_FromNestedLists code when an error occurs and the user explicitly requested an object array. The downside is that you can not place empty lists (or tuples) as objects in an object-array construct. as you could before. Given the trouble people had with the "feature," it seems wise to use it only when previous code would have raised an error. -Travis From rw679aq02 at sneakemail.com Mon Sep 4 18:46:57 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 15:46:57 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: <1157410017.28372.270115201@webmail.messagingengine.com> >> The question is whether numpy has such support; if not, is it planned. > No, and no. Thank you for answering, and I am sorry to hear that. I will be dropping my membership on the scipy-numpy email list shortly. Many systems handle rectangular arrays quite well already, and are more fully developed. It is a common fallacy rectangles are the only shapes one ever needs. Physical geometry is only rarely rectangular, and solution of actual physical problems, and even number-theoretical problem, is a far larger problem domain. From charlesr.harris at gmail.com Mon Sep 4 19:35:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 4 Sep 2006 17:35:04 -0600 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157410017.28372.270115201@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157410017.28372.270115201@webmail.messagingengine.com> Message-ID: On 9/4/06, rw679aq02 at sneakemail.com wrote: > > >> The question is whether numpy has such support; if not, is it planned. > > > No, and no. > > Thank you for answering, and I am sorry to hear that. > > I will be dropping my membership on the scipy-numpy email list shortly. > Many systems handle rectangular arrays quite well already, and are more > fully developed. > > It is a common fallacy rectangles are the only shapes one ever needs. > Physical geometry is only rarely rectangular, and solution of actual > physical problems, and even number-theoretical problem, is a far larger > problem domain. Thanks for blessing us with your exalted presence and superior intellect. You will be missed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Sep 4 19:55:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 04 Sep 2006 17:55:01 -0600 Subject: [Numpy-discussion] Going to cut NumPy 1.0b5 tonight Message-ID: <44FCBCD5.3000407@ieee.org> This is a last reminder that I'm going to cut a release of NumPy 1.0b5 tonight. Please have any fixes and/or problems worked out before then. -Travis From cookedm at physics.mcmaster.ca Mon Sep 4 20:30:45 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 4 Sep 2006 20:30:45 -0400 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157410017.28372.270115201@webmail.messagingengine.com> Message-ID: <20060904203045.37f31874@arbutus.physics.mcmaster.ca> On Mon, 4 Sep 2006 17:35:04 -0600 "Charles R Harris" wrote: > On 9/4/06, rw679aq02 at sneakemail.com wrote: > > > > >> The question is whether numpy has such support; if not, is it planned. > > > > > No, and no. > > > > Thank you for answering, and I am sorry to hear that. > > > > I will be dropping my membership on the scipy-numpy email list shortly. > > Many systems handle rectangular arrays quite well already, and are more > > fully developed. > > > > It is a common fallacy rectangles are the only shapes one ever needs. > > Physical geometry is only rarely rectangular, and solution of actual > > physical problems, and even number-theoretical problem, is a far larger > > problem domain. > > > Thanks for blessing us with your exalted presence and superior intellect. > You will be missed. Well now, that's just snarky. rw679aq02 (if that is indeed your real name!), the reason that numpy will not support irregular "arrays" anytime soon comes down to multiple reasons: 1) It would require a complete rework; better to make a new package. Irregular arrays would require an entirely different approach than regular arrays. 2) While most things are not rectangular, the equations that describe them are, for the most part. Finite-element methods, for instance, use a triangulation of the physical object, and the equations can then be cast as very large set of array equations. 3) I would guess that problems that could be described by irregular arrays could be better recast with a different data structure. There's a saying that you can write Fortran in any language; however, that doesn't mean you should! 4) No one else has asked for them, and the developers don't need them (this is how open source works: scratching one's own itch) 5) If, instead, you're looking for efficient memory representations of of sparse matrices (most elements being 0), then there are various ways to do this. I'm not familiar with them, but scipy has packages to handle sparse matrices. A lot of work happens in this field (those finite-element methods tend to make sparse matrices). Note that you could simulate an irregular array (with a maximum size in the dimensions) using the masked arrays provided by NumPy. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rw679aq02 at sneakemail.com Tue Sep 5 00:27:17 2006 From: rw679aq02 at sneakemail.com (rw679aq02 at sneakemail.com) Date: Mon, 04 Sep 2006 21:27:17 -0700 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157405271.21974.270111020@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> Message-ID: <1157430437.25529.270131342@webmail.messagingengine.com> Exalted presences and superior intellects aside, the point is not hard to get: Motivational examples are everywhere. Think about gridding physical problems expressed in cylindrical or spherical coordinates. The natural slices are not rectangles. You can use rectangular storage but only with O(n^3) waste. More abstract solution spaces of math and physics do not usually lend themselves to rectangular treatments. (I understand finite element techniques and am not referring to those.) Again, rectangular storage is possible only with O(n^d) waste, where commonly d>3. Granted one may overcome these issues with software development effort; that insight begs the question. I am looking for teaching software that already does so. I agree that rectangular storage is easiest for software programmers and hence common. It is not easiest for solving all problems. Students should explore solutiuon spaces in a proper setting. So I just asked what numpy could do in this regard. Now I have the plain answer, and am grateful for it. From cwmoad at gmail.com Tue Sep 5 08:00:26 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Tue, 5 Sep 2006 08:00:26 -0400 Subject: [Numpy-discussion] [matplotlib-devel] Fwd: Going to cut NumPy 1.0b5 tonight In-Reply-To: References: <44FCBCD5.3000407@ieee.org> <6382066a0609041800l504bfbb9t637f6700de9d65a@mail.gmail.com> Message-ID: <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> I also get a compile error when trying to build against the win32-py2.4 release. src\_na_nxutils.c(213) : error C2275: 'PyObject' : illegal use of this type as an expression c:\Python24\include\object.h(104) : see declaration of 'PyObject' src\_na_nxutils.c(213) : error C2065: 'ret' : undeclared identifier src\_na_nxutils.c(215) : warning C4047: 'return' : 'PyObject *' differs in levels of indirection from 'int' Do we need to modify our use of the c-api at all? - Charlie On 9/5/06, Boyd Waters wrote: > Very sorry: here is the initial error: > numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use > in this function) > > > On Sep 5, 2006, at 12:01 AM, Boyd Waters wrote: > > > 1.0b5 > > Fails to compile? > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Matplotlib-devel mailing list > Matplotlib-devel at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > > > > From jdhunter at ace.bsd.uchicago.edu Tue Sep 5 08:04:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue, 05 Sep 2006 07:04:04 -0500 Subject: [Numpy-discussion] [matplotlib-devel] Fwd: Going to cut NumPy 1.0b5 tonight In-Reply-To: <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> ("Charlie Moad"'s message of "Tue, 5 Sep 2006 08:00:26 -0400") References: <44FCBCD5.3000407@ieee.org> <6382066a0609041800l504bfbb9t637f6700de9d65a@mail.gmail.com> <6382066a0609050500n7b16386ep206baadcb535d485@mail.gmail.com> Message-ID: <877j0i7c5n.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Charlie" == Charlie Moad writes: Charlie> I also get a compile error when trying to build against Charlie> the win32-py2.4 release. src\_na_nxutils.c(213) : error Charlie> C2275: 'PyObject' : illegal use of this type as an Charlie> expression c:\Python24\include\object.h(104) : see It looks like I made the typical C++ programmer writing C mistake: putting a type declaration in code where it is first used rather than at the beginning of the function. I just committed a change that should fix this. JDH From matthew.brett at gmail.com Tue Sep 5 09:34:58 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 5 Sep 2006 14:34:58 +0100 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FCA9E0.4040908@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> Message-ID: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Hi, > This is a result of PyArray_FromAny changing when object arrays are > explicitly requested (which they are in this case --- although behind > the scenes). Hmm - I think I am hitting a related bug/feature/surprising change in behavior, which is showing up rather obscurely in a failure of the scipy.io matlab loading tests: http://projects.scipy.org/scipy/scipy/ticket/258 Here's the change I wasn't expecting, present with current SVN: a = arange(2) b = arange(1) c = array([a, b], dtype=object) c -> array([[0, 1], [0, 0]], dtype=object) On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: array([[0], [0 1]], dtype=object) Best, Matthew From charlesr.harris at gmail.com Tue Sep 5 12:40:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 10:40:53 -0600 Subject: [Numpy-discussion] order keyword Message-ID: I am trying to write more document strings for numpy and have a question about the order keyword in tostring. The usual allowed values of this keyword are "C", "F", or None, but in the tostring method there is also the value "Any" which has the same effect as None. I wonder if the "Any" shouldn't be removed as None seems to be the preferred form in other methods. Also, the default value of order in the tostring method is "C" and it seems to me that the principal of least surprise requires None as the default so that the order of the array being converted is the default. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 5 12:44:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 10:44:41 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Message-ID: <44FDA979.6080502@ieee.org> Matthew Brett wrote: > Hi, > > >> This is a result of PyArray_FromAny changing when object arrays are >> explicitly requested (which they are in this case --- although behind >> the scenes). >> > > Hmm - I think I am hitting a related bug/feature/surprising change in > behavior, which is showing up rather obscurely in a failure of the > scipy.io matlab loading tests: > > http://projects.scipy.org/scipy/scipy/ticket/258 > > Here's the change I wasn't expecting, present with current SVN: > > a = arange(2) > b = arange(1) > c = array([a, b], dtype=object) > c > -> > array([[0, 1], > [0, 0]], dtype=object) > > On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: > > array([[0], [0 1]], dtype=object) > Grrr.. Object arrays are very hard to get right. I have no idea why this is happening, but I'll look into it. I think it's the bug that led me to put in the special-case object-array handling in the first place. Now, that special-case object-array handling is only done on an error condition, I need to fix this right and raise an inconsistent shape error. It will probably help with the TypeError messages that are currently raised in this situation with other types as well. -Travis From charlesr.harris at gmail.com Tue Sep 5 13:17:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 11:17:09 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: Message-ID: On 9/5/06, Charles R Harris wrote: > > I am trying to write more document strings for numpy and have a question > about the order keyword in tostring. The usual allowed values of this > keyword are "C", "F", or None, but in the tostring method there is also the > value "Any" which has the same effect as None. I wonder if the "Any" > shouldn't be removed as None seems to be the preferred form in other > methods. Also, the default value of order in the tostring method is "C" and > it seems to me that the principal of least surprise requires None as the > default so that the order of the array being converted is the default. > > Chuck > > What is the actual order of data in memory? Is it always row major? In [38]: array([[1,2],[3,4]], dtype=int8, order="F").tofile('test.txt', sep=" ") In [39]: cat test.txt 1 2 3 4 In [40]: array([[1,2],[3,4]], dtype=int8, order="F").tostring(order="A") Out[40]: '\x01\x03\x02\x04' -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 5 13:02:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 11:02:52 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: Message-ID: <44FDADBC.5030906@ieee.org> Charles R Harris wrote: > I am trying to write more document strings for numpy and have a > question about the order keyword in tostring. The usual allowed values > of this keyword are "C", "F", or None, but in the tostring method > there is also the value "Any" which has the same effect as None. I > wonder if the "Any" shouldn't be removed as None seems to be the > preferred form in other methods. I don't think keeping 'Any' as a keyword here is a problem. > Also, the default value of order in the tostring method is "C" and it > seems to me that the principal of least surprise requires None as the > default so that the order of the array being converted is the default. I've thought this through several times. There may be a few cases where 'Any' is appropriate but the user will be expecting 'C' as the default because that was the only possibility for a long time. It's actually very problematic to switch to 'Any' as the default. You end up with lots of surprises as things start behaving very differently then they used to under Numeric once you transpose the array. -Travis From kortmann at ideaworks.com Tue Sep 5 14:04:11 2006 From: kortmann at ideaworks.com (kortmann at ideaworks.com) Date: Tue, 5 Sep 2006 11:04:11 -0700 (PDT) Subject: [Numpy-discussion] (no subject) Message-ID: <2794.12.216.231.149.1157479451.squirrel@webmail.ideaworks.com> Hey guys I posted this on matplotlibs mailing list > hey guys i got the subversion from the site and I am trying to install it > on windows. > > I changed dir into the matplotlib dir that includes the setup.py file. > > run python setup.py install, and im getting a wierd error. i left the > topmost lines along with the error. has anyone seen anything like this > before? > > building 'matplotlib.enthought.traits.ctraits' extension > creating build\temp.win32-2.4\Release\lib > creating build\temp.win32-2.4\Release\lib\matplotlib > creating build\temp.win32-2.4\Release\lib\matplotlib\enthought > creating build\temp.win32-2.4\Release\lib\matplotlib\enthought\traits > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c > /nologo /Ox > /MD /W3 /GX /DNDEBUG -Ic:\Python24\include -Ic:\Python24\PC > /Tclib/matplotlib/e > nthought/traits/ctraits.c > /Fobuild\temp.win32-2.4\Release\lib/matplotlib/enthoug > ht/traits/ctraits.obj > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /DLL > /nologo > /INCREMENTAL:NO /LIBPATH:c:\Python24\libs /LIBPATH:c:\Python24\PCBuild > /EXPORT: > initctraits > build\temp.win32-2.4\Release\lib/matplotlib/enthought/traits/ctraits > .obj /OUT:build\lib.win32-2.4\matplotlib\enthought\traits\ctraits.pyd > /IMPLIB:bu > ild\temp.win32-2.4\Release\lib/matplotlib/enthought/traits\ctraits.lib > building 'matplotlib.backends._tkagg' extension > C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c > /nologo /Ox > /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc -Iswig > -Iagg23/incl > ude -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetype2 > -Isrc\freetype2 > -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\freetype2 > -Ic:\Pyt > hon24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp > /Fobuild\temp.win32-2.4\Release > \src/_tkagg.obj > _tkagg.cpp > src\_tkagg.cpp(28) : fatal error C1083: Cannot open include file: 'tk.h': > No suc > h file or directory > error: Command ""C:\Program Files\Microsoft Visual Studio .NET > 2003\Vc7\bin\cl.e > xe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. > -Isrc - > Iswig -Iagg23/include -I. -I. -Iwin32_static/include/tcl84\freetype2 > -I.\freetyp > e2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 > -I.\freetype2 -I.\ > freetype2 -Ic:\Python24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp > /Fobuild\temp > .win32-2.4\Release\src/_tkagg.obj" failed with exit status 2 and recieved this response > > You need to install the tcl/tk headers as Darren mentioned. I just > install ActiveTcl and the build should pick up on it no problem. > And then after that I ran into another error that looks like it might be numpy related. So i am posting this here and on Matplotlibs list. The second error is why i am posting here but since some of you use the latest SVN on windows maybe you could offer more general help of how to compile matplotlib on windows I went ahead and installed Active Tcl I removed microsft visual studio 2003 .net from my computer because I never used it, and I did not want it on here in the first place but i had it on here from being an intern over the summer. C:\matplotlib\trunk\matplotlib>c:\Python24\python.exe setup.py install GTK requires pygtk building tkagg 2 4 Building for python24 GTKAgg requires pygtk running install running build running build_py running build_ext No module named msvccompiler in numpy.distutils, trying from distutils.. building 'matplotlib.backends._tkagg' extension C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc -Iswig -Iagg23/incl ude -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetype2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\freetype2 -Ic:\Pyt hon24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp /Fobuild\temp.win32-2.4\Release \src/_tkagg.obj _tkagg.cpp src\_tkagg.cpp(28) : fatal error C1083: Cannot open include file: 'tk.h': No suc h file or directory error: Command ""C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Iwin32_static/include/tcl84 -I. -Isrc - Iswig -Iagg23/include -I. -I. -Iwin32_static/include/tcl84\freetype2 -I.\freetyp e2 -Isrc\freetype2 -Iswig\freetype2 -Iagg23/include\freetype2 -I.\freetype2 -I.\ freetype2 -Ic:\Python24\include -Ic:\Python24\PC /Tpsrc/_tkagg.cpp /Fobuild\temp .win32-2.4\Release\src/_tkagg.obj" failed with exit status 2 This is the error i got after my first try at python setup.py install after it did not work is when i uninstalled .net 2003. and i recieve this error currently. C:\matplotlib\trunk\matplotlib>c:\Python24\python.exe setup.py install GTK requires pygtk building tkagg 2 4 Building for python24 GTKAgg requires pygtk running install running build running build_py running build_ext No module named msvccompiler in numpy.distutils, trying from distutils.. error: The .NET Framework SDK needs to be installed before building extensions f or Python. Has anyone seen this one? I would not be so persistent as to trying to install the current SVN except that I need one of the algorithms in numpy version 1.0b2 and above. I am also going to post this on the numpy mailing list because it says the error is in numpy.distutils. From oliphant.travis at ieee.org Tue Sep 5 13:05:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 05 Sep 2006 11:05:00 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> Message-ID: <44FDAE3C.7000901@ieee.org> Matthew Brett wrote: > Hi, > > >> This is a result of PyArray_FromAny changing when object arrays are >> explicitly requested (which they are in this case --- although behind >> the scenes). >> > > Hmm - I think I am hitting a related bug/feature/surprising change in > behavior, which is showing up rather obscurely in a failure of the > scipy.io matlab loading tests: > > http://projects.scipy.org/scipy/scipy/ticket/258 > > Here's the change I wasn't expecting, present with current SVN: > > a = arange(2) > b = arange(1) > c = array([a, b], dtype=object) > c > -> > array([[0, 1], > [0, 0]], dtype=object) > > On a previous version of numpy (1.02b.dev2975) I get the answer I was expecting: > > array([[0], [0 1]], dtype=object) > This should now be fixed. The code was inappropriately not checking for dimensions when object arrays were being constructed. Now, it raises the appropriate error and then interprets it correctly using the extra object creation code. Users of scipy 0.5.1 will only have to upgrade NumPy to get the fix (the SciPy install won't have to be re-built). -Travis From dalcinl at gmail.com Tue Sep 5 14:33:03 2006 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Sep 2006 15:33:03 -0300 Subject: [Numpy-discussion] order keyword In-Reply-To: <44FDADBC.5030906@ieee.org> References: <44FDADBC.5030906@ieee.org> Message-ID: BTW, in numpy-1.0b1 numpy.zeros((3,3), order='QQQ') pass without generating any error... Is this intended behaviour? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From charlesr.harris at gmail.com Tue Sep 5 14:37:19 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 12:37:19 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: <44FDADBC.5030906@ieee.org> References: <44FDADBC.5030906@ieee.org> Message-ID: On 9/5/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > I am trying to write more document strings for numpy and have a > > question about the order keyword in tostring. The usual allowed values > > of this keyword are "C", "F", or None, but in the tostring method > > there is also the value "Any" which has the same effect as None. I > > wonder if the "Any" shouldn't be removed as None seems to be the > > preferred form in other methods. > I don't think keeping 'Any' as a keyword here is a problem. Yeah, I noticed that the PyArray_OrderConverter just replaces None by "A" and is common to all the methods but the default is set in the calling code. > Also, the default value of order in the tostring method is "C" and it > > seems to me that the principal of least surprise requires None as the > > default so that the order of the array being converted is the default. > I've thought this through several times. There may be a few cases where > 'Any' is appropriate but the user will be expecting 'C' as the default > because that was the only possibility for a long time. It's actually > very problematic to switch to 'Any' as the default. You end up with > lots of surprises as things start behaving very differently then they > used to under Numeric once you transpose the array. > > -Travis Ok, I will add a comment to tofile that the data is written out in "C" order. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 5 14:40:50 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 5 Sep 2006 12:40:50 -0600 Subject: [Numpy-discussion] order keyword In-Reply-To: References: <44FDADBC.5030906@ieee.org> Message-ID: On 9/5/06, Lisandro Dalcin wrote: > > BTW, in numpy-1.0b1 > > numpy.zeros((3,3), order='QQQ') > > pass without generating any error... Is this intended behaviour? This has been fixed: In [3]: zeros((3,3), order='QQQ') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/charris/ TypeError: order not understood Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From torgil.svensson at gmail.com Tue Sep 5 15:49:34 2006 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Tue, 5 Sep 2006 21:49:34 +0200 Subject: [Numpy-discussion] Irregular arrays In-Reply-To: <1157430437.25529.270131342@webmail.messagingengine.com> References: <1156916822.16010.269732980@webmail.messagingengine.com> <1156926065.27232.269739888@webmail.messagingengine.com> <1157405271.21974.270111020@webmail.messagingengine.com> <1157430437.25529.270131342@webmail.messagingengine.com> Message-ID: > Think about gridding physical problems expressed in cylindrical or > spherical coordinates. The natural slices are not rectangles. You can > use rectangular storage but only with O(n^3) waste. I don't get this argument. Are you slicing your spherical coordinates with a cartesian coordinate system? That's surely a waste. Many times bases can be chosen so that you even in an array doesn't waste space. I can't see the point attacking numpy with this "irregular" generalisation. Numpy is a tool very good for it's purpose, we don't really want to trash it with features for solving unspecified general problems. Instead, specify a real world problem, and perhaps someone might have a good tip, otherwise look for other tools. Pytables for example can efficiently store varying length arrays. Maybe start on developing an variable length array yourself that integrates perfectly with Numpy where applicable? //Torgil On 9/5/06, rw679aq02 at sneakemail.com wrote: > Exalted presences and superior intellects aside, the point is not hard > to get: Motivational examples are everywhere. > > Think about gridding physical problems expressed in cylindrical or > spherical coordinates. The natural slices are not rectangles. You can > use rectangular storage but only with O(n^3) waste. > > More abstract solution spaces of math and physics do not usually lend > themselves to rectangular treatments. (I understand finite element > techniques and am not referring to those.) Again, rectangular storage > is possible only with O(n^d) waste, where commonly d>3. > > Granted one may overcome these issues with software development effort; > that insight begs the question. I am looking for teaching software that > already does so. > > I agree that rectangular storage is easiest for software programmers and > hence common. It is not easiest for solving all problems. Students > should explore solutiuon spaces in a proper setting. So I just asked > what numpy could do in this regard. Now I have the plain answer, and am > grateful for it. > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From scipy at mspacek.mm.st Tue Sep 5 19:46:20 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Tue, 05 Sep 2006 16:46:20 -0700 Subject: [Numpy-discussion] Keyword added to searchsorted. In-Reply-To: References: Message-ID: <44FE0C4C.80402@mspacek.mm.st> I agree. This'll allow me to delete some messy code I have to get the same behaviour. I'm amazed by how often I use searchsorted. 'side' sounds like a good keyword name to me. Martin Robert Kern wrote: > Charles R Harris wrote: >> Hi all, >> >> I added the keyword side to the searchsorted method and functions. > > Thank you! Just the other day, I was wishing that we had such a thing. > From pgmdevlist at gmail.com Wed Sep 6 03:57:38 2006 From: pgmdevlist at gmail.com (PGM) Date: Wed, 6 Sep 2006 03:57:38 -0400 Subject: [Numpy-discussion] convention in numpy.correlate? Message-ID: <200609060357.38906.pgmdevlist@gmail.com> Folks, I need to compute some cross-correlations between time series, and I naturally started to use numpy.correlate. Howver, I'm not sure about the convention being used: The crosscorrelation is usually defined as $\gamma_{xy}[k] = \sum_{i}{x[i] y[i+k]}$ So, when I compute >>> numpy.dot(x[:-1],y[1:]) on two 1D series of same size $n$, I should have $\gamma[1]$. With numpy.correlate, I have to use >>>numpy.correlate(x,y,'full')[(n-1)-1] or reverse x and y to get $\gamma[1]$ Is that correct ? P. From seguridad at santander.com.mx Wed Sep 6 05:07:50 2006 From: seguridad at santander.com.mx (Santander Serfin) Date: Wed, 6 Sep 2006 04:07:50 -0500 Subject: [Numpy-discussion] Seguridad Santander (Actualizacion) Message-ID: An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Sep 6 05:52:11 2006 From: pgmdevlist at gmail.com (PGM) Date: Wed, 6 Sep 2006 05:52:11 -0400 Subject: [Numpy-discussion] numpy.fftpack: axis=None ? Message-ID: <200609060552.11394.pgmdevlist@gmail.com> Folks, I was playing around the numpy.fftpack when I ran into the problem below: it seems that axis=None is not valid with fft. Is there a reason for that ? I was assuming a behavior similar to other functions, where axis=None translates to "use a flat array". ------------------------------------------------------ >>> N.fft.fft(N.arange(100).reshape(10,10),128,None) /usr/lib64/python2.4/site-packages/numpy/fft/fftpack.py in fft(a, n, axis) 85 different n's.""" 86 ---> 87 return _raw_fft(a, n, axis, fftpack.cffti, fftpack.cfftf, _fft_cache) 88 89 /usr/lib64/python2.4/site-packages/numpy/fft/fftpack.py in _raw_fft(a, n, axis, init_function, work_function, fft_cache) 44 fft_cache[n] = wsave 45 ---> 46 if a.shape[axis] != n: 47 s = list(a.shape) 48 if s[axis] > n: TypeError: tuple indices must be integers ------------------------------------------------- From ryanlists at gmail.com Wed Sep 6 13:18:03 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 12:18:03 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version Message-ID: I am a Linux user trying to install Numpy/Scipy on a Windows machine in my office. I went to the website and grabbed the two latest versions: scipy = scipy-0.5.0.win32-py2.4.exe numpy = numpy-1.0b5.win32-py2.4.exe Trying to import scipy I get: Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy Overwriting info= from scipy.misc.helpmod (was from numpy.lib.utils) Overwriting who= from scipy.misc.common (was from numpy.lib.utils) Overwriting source= from scipy.misc.helpmod (was from numpy.lib.utils) RuntimeError: module compiled against version 1000000 of C-API but this version of numpy is 1000002 Fatal Python error: numpy.core.multiarray failed to import... exiting. I read about this problem on the list and there was mention of needing a different Numpy version. With a slightly older Numpy I got (numpy-0.9.8.win32-py2.4.exe): Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\scipy\__init__.py", line 32, in ? from numpy import oldnumeric ImportError: cannot import name oldnumeric Are there windows installers available for compatible versions of Scipy and Numpy? Thanks, Ryan From scipy at mspacek.mm.st Wed Sep 6 13:23:01 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Wed, 06 Sep 2006 10:23:01 -0700 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: Message-ID: <44FF03F5.60208@mspacek.mm.st> Ryan, Try installing the latest scipy version 0.51. There's a windows binary for it. Worked fine for me. Martin Ryan Krauss wrote: > I am a Linux user trying to install Numpy/Scipy on a Windows machine > in my office. > > I went to the website and grabbed the two latest versions: > scipy = scipy-0.5.0.win32-py2.4.exe > numpy = numpy-1.0b5.win32-py2.4.exe > > From ryanlists at gmail.com Wed Sep 6 13:30:17 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 12:30:17 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF03F5.60208@mspacek.mm.st> References: <44FF03F5.60208@mspacek.mm.st> Message-ID: I would be glad to try that. Where do I get the installer for scipy 0.51? Thanks, Ryan On 9/6/06, Martin Spacek wrote: > Ryan, > > Try installing the latest scipy version 0.51. There's a windows binary > for it. Worked fine for me. > > Martin > > Ryan Krauss wrote: > > I am a Linux user trying to install Numpy/Scipy on a Windows machine > > in my office. > > > > I went to the website and grabbed the two latest versions: > > scipy = scipy-0.5.0.win32-py2.4.exe > > numpy = numpy-1.0b5.win32-py2.4.exe > > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jg307 at cam.ac.uk Wed Sep 6 13:37:17 2006 From: jg307 at cam.ac.uk (James Graham) Date: Wed, 06 Sep 2006 18:37:17 +0100 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> Message-ID: <44FF074D.5070204@cam.ac.uk> Ryan Krauss wrote: > I would be glad to try that. Where do I get the installer for scipy 0.51? http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download The SciPy website needs to be updated. -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead From ryanlists at gmail.com Wed Sep 6 14:17:57 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 13:17:57 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF074D.5070204@cam.ac.uk> References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: O.K. I think I am up and running. I actually installed Entought and Numpy and Scipy worked together there. I then upgraded the scipy and numpy packages. Is there a compatible matplotlib as well? I was o.k. with mpl from enthought until I switched numerix to numpy. That made mpl unhappy. I downloaded 0.87.5 but I broke something in the process because now even switching back to Numeric doesn't make mpl happy. I may switch to the mpl list if this keeps giving me problems. Ryan On 9/6/06, James Graham wrote: > Ryan Krauss wrote: > > I would be glad to try that. Where do I get the installer for scipy 0.51? > > http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download > > The SciPy website needs to be updated. > > -- > "Eternity's a terrible thought. I mean, where's it all going to end?" > -- Tom Stoppard, Rosencrantz and Guildenstern are Dead > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From bhendrix at enthought.com Wed Sep 6 14:33:54 2006 From: bhendrix at enthought.com (bryce hendrix) Date: Wed, 06 Sep 2006 13:33:54 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: <44FF1492.9060604@enthought.com> Ryan, The Enthought 1.0.0 release shipped a pre-release of matplotlib 0.87.3, svn revision 2478. If you update numpy, you have to update to matching versions of scipy and matplotlib. If you are using binary releases, guessing which version of each is needed is difficult unless builders of the binaries specify exactly which version of numpy was used. If you're comfortable building these packages yourself, the simple solution is to download the latest numpy, install it over the existing numpy, get the latest scipy & matplot lib sources, build them and install them over the existing packages. Bryce Ryan Krauss wrote: > O.K. I think I am up and running. I actually installed Entought and > Numpy and Scipy worked together there. I then upgraded the scipy and > numpy packages. > > Is there a compatible matplotlib as well? I was o.k. with mpl from > enthought until I switched numerix to numpy. That made mpl unhappy. > I downloaded 0.87.5 but I broke something in the process because now > even switching back to Numeric doesn't make mpl happy. I may switch > to the mpl list if this keeps giving me problems. > > Ryan > > On 9/6/06, James Graham wrote: > >> Ryan Krauss wrote: >> >>> I would be glad to try that. Where do I get the installer for scipy 0.51? >>> >> http://prdownloads.sourceforge.net/scipy/scipy-0.5.1.win32-py2.4.exe?download >> >> The SciPy website needs to be updated. >> >> -- >> "Eternity's a terrible thought. I mean, where's it all going to end?" >> -- Tom Stoppard, Rosencrantz and Guildenstern are Dead >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Sep 6 16:50:12 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 06 Sep 2006 13:50:12 -0700 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> Message-ID: <44FF3484.2040700@noaa.gov> Ryan Krauss wrote: > Is there a compatible matplotlib as well? I was o.k. with mpl from > enthought until I switched numerix to numpy. That made mpl unhappy. > I downloaded 0.87.5 but I broke something in the process because now > even switching back to Numeric doesn't make mpl happy. There was an error with the first MPL 0.87.5 builds uploaded. See the MPL list for more info, but these builds should work: http://euclid.uits.iupui.edu/mplfiles/ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Sep 6 17:08:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 15:08:08 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FDAE3C.7000901@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> Message-ID: On 9/5/06, Travis Oliphant wrote: > > Matthew Brett wrote: > > Hi, > > > > > >> This is a result of PyArray_FromAny changing when object arrays are > >> explicitly requested (which they are in this case --- although behind > >> the scenes). > >> > > > > Hmm - I think I am hitting a related bug/feature/surprising change in > > behavior, which is showing up rather obscurely in a failure of the > > scipy.io matlab loading tests: > > > > http://projects.scipy.org/scipy/scipy/ticket/258 > > > > Here's the change I wasn't expecting, present with current SVN: > > > > a = arange(2) > > b = arange(1) > > c = array([a, b], dtype=object) > > c > > -> > > array([[0, 1], > > [0, 0]], dtype=object) > > > > On a previous version of numpy (1.02b.dev2975) I get the answer I was > expecting: > > > > array([[0], [0 1]], dtype=object) > > > > This should now be fixed. The code was inappropriately not checking for > dimensions when object arrays were being constructed. Now, it raises > the appropriate error and then interprets it correctly using the extra > object creation code. > > Users of scipy 0.5.1 will only have to upgrade NumPy to get the fix (the > SciPy install won't have to be re-built). > > -Travis Where is array at this point? I would like to review the documented behaviour and make modifications to the document string if required. What about Robert's idea of a separate constructor for object arrays? Is it something we could introduce on top of the current array constructor? I realize that if we restrict the current array constructor there might be compatibility problems with Numeric code, but introducing something like oarray as a shorthand for object arrays might incourage it's use. Robert also said that Numarray dealt with object arrays as a separate issue and I wonder what they did that we should think about. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Sep 6 19:33:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 06 Sep 2006 17:33:42 -0600 Subject: [Numpy-discussion] numpy.fftpack: axis=None ? In-Reply-To: <200609060552.11394.pgmdevlist@gmail.com> References: <200609060552.11394.pgmdevlist@gmail.com> Message-ID: <44FF5AD6.6050800@ieee.org> PGM wrote: > Folks, > I was playing around the numpy.fftpack when I ran into the problem below: > it seems that axis=None is not valid with fft. Is there a reason for that ? Not really, other than history and the fact that sub-packages of NumPy will often define their own behavior. Perhaps we could implement it, but would this create more confusion as usually when axis=None is accepted then it is the default. -Travis From oliphant.travis at ieee.org Wed Sep 6 19:39:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 06 Sep 2006 17:39:35 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> Message-ID: <44FF5C37.4090909@ieee.org> Charles R Harris wrote: > > Where is array at this point? Basically it supports the old Numeric behavior wherein object array's are treated as before *except* for when an error would have occurred previously when the "new behavior" kicks in. Anything that violates that is a bug needing to be fixed. This leaves the new object-array constructor used less often. It could be exported explicitly into an oarray constructor, but I'm not sure about the advantages of that approach. There are benefits to having object arrays constructed in the same way as other arrays. It turns out many people actually like that feature of Numeric, which is the reason I didn't go the route of numarray which pulled object arrays out. At this point, however, object arrays can even be part of records and so need to be an integral part of the data-type description. Pulling that out is not going to happen. A more intelligent object-array constructor, however, may be a useful tool. -Travis From ryanlists at gmail.com Wed Sep 6 20:44:38 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed, 6 Sep 2006 19:44:38 -0500 Subject: [Numpy-discussion] Numpy that is compatible with Scipy windows installer version In-Reply-To: <44FF3484.2040700@noaa.gov> References: <44FF03F5.60208@mspacek.mm.st> <44FF074D.5070204@cam.ac.uk> <44FF3484.2040700@noaa.gov> Message-ID: Thanks Chris, that worked great. Ryan On 9/6/06, Christopher Barker wrote: > Ryan Krauss wrote: > > Is there a compatible matplotlib as well? I was o.k. with mpl from > > enthought until I switched numerix to numpy. That made mpl unhappy. > > I downloaded 0.87.5 but I broke something in the process because now > > even switching back to Numeric doesn't make mpl happy. > > There was an error with the first MPL 0.87.5 builds uploaded. See the > MPL list for more info, but these builds should work: > > http://euclid.uits.iupui.edu/mplfiles/ > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From fccoelho at gmail.com Wed Sep 6 21:05:51 2006 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed, 6 Sep 2006 22:05:51 -0300 Subject: [Numpy-discussion] f2py with xmingw Message-ID: Hi, I have a module that uses a Fortran extension which I would like to compile for windows with f2py. I wonder If I could do this from Linux using xmingw. Has anyone tried this? thanks, -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 6 21:18:33 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 19:18:33 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FF5C37.4090909@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 9/6/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > Where is array at this point? > Basically it supports the old Numeric behavior wherein object array's > are treated as before *except* for when an error would have occurred > previously when the "new behavior" kicks in. Anything that violates > that is a bug needing to be fixed. > > This leaves the new object-array constructor used less often. It could > be exported explicitly into an oarray constructor, but I'm not sure > about the advantages of that approach. There are benefits to having > object arrays constructed in the same way as other arrays. It turns out > many people actually like that feature of Numeric, which is the reason I > didn't go the route of numarray which pulled object arrays out. > > At this point, however, object arrays can even be part of records and so > need to be an integral part of the data-type description. Pulling that > out is not going to happen. A more intelligent object-array > constructor, however, may be a useful tool. OK. I do have a couple of questions. Let me insert the docs for array and asarray : """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) Return an array from object with the specified date-type. Inputs: object - an array, any object exposing the array interface, any object whose __array__ method returns an array, or any (nested) sequence. dtype - The desired data-type for the array. If not given, then the type will be determined as the minimum type required to hold the objects in the sequence. This argument can only be used to 'upcast' the array. For downcasting, use the .astype(t) method. copy - If true, then force a copy. Otherwise a copy will only occur if __array__ returns a copy, obj is a nested sequence, or a copy is needed to satisfy any of the other requirements order - Specify the order of the array. If order is 'C', then the array will be in C-contiguous order (last-index varies the fastest). If order is 'FORTRAN', then the returned array will be in Fortran-contiguous order (first-index varies the fastest). If order is None, then the returned array may be in either C-, or Fortran-contiguous order or even discontiguous. subok - If True, then sub-classes will be passed-through, otherwise the returned array will be forced to be a base-class array ndmin - Specifies the minimum number of dimensions that the resulting array should have. 1's will be pre-pended to the shape as needed to meet this requirement. """) asarray(a, dtype=None, order=None) Returns a as an array. Unlike array(), no copy is performed if a is already an array. Subclasses are converted to base class ndarray. 1) Is it true that array doesn't always return a copy except by default? asarray says it contrasts with array in this regard. Maybe copy=0 should be deprecated. 2) Is asarray is basically array with copy=0? 3) Is asanyarray basically array with copy=0 and subok=1? 4) Is there some sort of precedence table for conversions? To me it looks like the most deeply nested lists are converted to arrays first, numeric if they contain all numeric types, object otherwise. I assume the algorithm then ascends up through the hierarchy like traversing a binary tree in postorder? 5) All nesting must be to the same depth and the deepest nested items must have the same length. 6) How is the difference between lists and "lists" determined, i.e., In [3]: array([list([1,2,3]),list([1,2])], dtype = object) Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) In [8]: array([array([1,2,3]),array([1,2])], dtype = object) Out[8]: array([[1 2 3], [1 2]], dtype=object) In [9]: array([1,2,3],[1,2]], dtype = object) ------------------------------------------------------------ File "", line 1 array([1,2,3],[1,2]], dtype = object) ^ SyntaxError: invalid syntax Is the difference that list(...) and array(...) are passed as functions (lazy evaluation), but a list is just a list? Sorry to be asking all these questions, but I would like to try making the documentation be a bit of a reference. I am sure I will have more questions ;) -Travis Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 6 21:52:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 6 Sep 2006 19:52:42 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 9/6/06, Charles R Harris wrote: > > > > On 9/6/06, Travis Oliphant wrote: > > > > Charles R Harris wrote: > > > > > > Where is array at this point? > > Basically it supports the old Numeric behavior wherein object array's > > are treated as before *except* for when an error would have occurred > > previously when the "new behavior" kicks in. Anything that violates > > that is a bug needing to be fixed. > > > > This leaves the new object-array constructor used less often. It could > > be exported explicitly into an oarray constructor, but I'm not sure > > about the advantages of that approach. There are benefits to having > > object arrays constructed in the same way as other arrays. It turns out > > many people actually like that feature of Numeric, which is the reason I > > didn't go the route of numarray which pulled object arrays out. > > > > At this point, however, object arrays can even be part of records and so > > need to be an integral part of the data-type description. Pulling that > > out is not going to happen. A more intelligent object-array > > constructor, however, may be a useful tool. > > > OK. I do have a couple of questions. Let me insert the docs for array and > asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > Inputs: > object - an array, any object exposing the array interface, any > object whose __array__ method returns an array, or any > (nested) sequence. > dtype - The desired data-type for the array. If not given, then > the type will be determined as the minimum type required > to hold the objects in the sequence. This argument can > only > be used to 'upcast' the array. For downcasting, use the > .astype(t) method. > copy - If true, then force a copy. Otherwise a copy will only > occur > if __array__ returns a copy, obj is a nested sequence, or > a copy is needed to satisfy any of the other requirements > order - Specify the order of the array. If order is 'C', then the > array will be in C-contiguous order (last-index varies the > fastest). If order is 'FORTRAN', then the returned array > will be in Fortran-contiguous order (first-index varies > the > fastest). If order is None, then the returned array may > be in either C-, or Fortran-contiguous order or even > discontiguous. > subok - If True, then sub-classes will be passed-through, otherwise > the returned array will be forced to be a base-class array > ndmin - Specifies the minimum number of dimensions that the > resulting > array should have. 1's will be pre-pended to the shape as > needed to meet this requirement. > > """) > > asarray(a, dtype=None, order=None) > Returns a as an array. > > Unlike array(), no copy is performed if a is already an array. > Subclasses > are converted to base class ndarray. > > 1) Is it true that array doesn't always return a copy except by default? > asarray says it contrasts with array in this regard. Maybe copy=0 should be > deprecated. > > 2) Is asarray is basically array with copy=0? > > 3) Is asanyarray basically array with copy=0 and subok=1? > > 4) Is there some sort of precedence table for conversions? To me it looks > like the most deeply nested lists are converted to arrays first, numeric if > they contain all numeric types, object otherwise. I assume the algorithm > then ascends up through the hierarchy like traversing a binary tree in > postorder? > > 5) All nesting must be to the same depth and the deepest nested items must > have the same length. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax > > Is the difference that list(...) and array(...) are passed as functions > (lazy evaluation), but a list is just a list? > > Sorry to be asking all these questions, but I would like to try making the > documentation be a bit of a reference. I am sure I will have more questions > ;) > > -Travis > > And, voila, ragged arrays: In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) In [10]: a*2 Out[10]: array([[2 4 6], [2 4]], dtype=object) In [11]: a + a Out[11]: array([[2 4 6], [2 4]], dtype=object) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Sep 6 22:18:39 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Wed, 6 Sep 2006 22:18:39 -0400 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: On 06/09/06, Charles R Harris wrote: > On 9/6/06, Charles R Harris wrote: > > > order - Specify the order of the array. If order is 'C', then the > > array will be in C-contiguous order (last-index varies the > > fastest). If order is 'FORTRAN', then the returned array > > will be in Fortran-contiguous order (first-index varies > the > > fastest). If order is None, then the returned array may > > be in either C-, or Fortran-contiguous order or even > > discontiguous. This one's a bit complicated. If array() is passed a list of lists, there are two different orders that are relevant - the output order of the array, and the order used to interpret the input. I suppose that if L is a lost of lists, array(L)[2,3]==L[2][3], that is, in some sense the arrays are always logically C-ordered even if the underlying representation is different. Does it make sense to specify this somewhere in the docstring? At least it would be good to make it clear that the order parameter affects only the underlying storage format, and not the indexing of the array. A. M. Archibald From oliphant.travis at ieee.org Thu Sep 7 02:54:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 07 Sep 2006 00:54:41 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: <44FFC231.4060507@ieee.org> Charles R Harris wrote: > On 9/6/06, *Charles R Harris* > wrote: > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > wrote: > > Charles R Harris wrote: > > > > Where is array at this point? > Basically it supports the old Numeric behavior wherein object > array's > are treated as before *except* for when an error would have > occurred > previously when the "new behavior" kicks in. Anything that > violates > that is a bug needing to be fixed. > > This leaves the new object-array constructor used less > often. It could > be exported explicitly into an oarray constructor, but I'm not > sure > about the advantages of that approach. There are benefits to > having > object arrays constructed in the same way as other arrays. It > turns out > many people actually like that feature of Numeric, which is > the reason I > didn't go the route of numarray which pulled object arrays out. > > At this point, however, object arrays can even be part of > records and so > need to be an integral part of the data-type description. > Pulling that > out is not going to happen. A more intelligent object-array > constructor, however, may be a useful tool. > > > OK. I do have a couple of questions. Let me insert the docs for > array and asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > Inputs: > object - an array, any object exposing the array interface, any > object whose __array__ method returns an array, or any > (nested) sequence. > dtype - The desired data-type for the array. If not given, > then > the type will be determined as the minimum type > required > to hold the objects in the sequence. This > argument can only > be used to 'upcast' the array. For downcasting, > use the > .astype(t) method. > copy - If true, then force a copy. Otherwise a copy will > only occur > if __array__ returns a copy, obj is a nested > sequence, or > a copy is needed to satisfy any of the other > requirements > order - Specify the order of the array. If order is 'C', > then the > array will be in C-contiguous order (last-index > varies the > fastest). If order is 'FORTRAN', then the > returned array > will be in Fortran-contiguous order (first-index > varies the > fastest). If order is None, then the returned > array may > be in either C-, or Fortran-contiguous order or even > discontiguous. > subok - If True, then sub-classes will be passed-through, > otherwise > the returned array will be forced to be a > base-class array > ndmin - Specifies the minimum number of dimensions that the > resulting > array should have. 1's will be pre-pended to the > shape as > needed to meet this requirement. > > """) > > asarray(a, dtype=None, order=None) > Returns a as an array. > > Unlike array(), no copy is performed if a is already an array. > Subclasses > are converted to base class ndarray. > > 1) Is it true that array doesn't always return a copy except by > default? asarray says it contrasts with array in this regard. > Maybe copy=0 should be deprecated. > > 2) Is asarray is basically array with copy=0? > > 3) Is asanyarray basically array with copy=0 and subok=1? > > 4) Is there some sort of precedence table for conversions? To me > it looks like the most deeply nested lists are converted to arrays > first, numeric if they contain all numeric types, object > otherwise. I assume the algorithm then ascends up through the > hierarchy like traversing a binary tree in postorder? > > 5) All nesting must be to the same depth and the deepest nested > items must have the same length. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax > > Is the difference that list(...) and array(...) are passed as > functions (lazy evaluation), but a list is just a list? > > Sorry to be asking all these questions, but I would like to try > making the documentation be a bit of a reference. I am sure I will > have more questions ;) > > -Travis > > > And, voila, ragged arrays: > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > In [10]: a*2 > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > In [11]: a + a > Out[11]: array([[2 4 6], [2 4]], dtype=object) Now I remember that this was my original motivation for futzing with the object-array constructor in the first place. So, now you get there only after an attempt to make a "rectangular" array first. -Travis From oliphant.travis at ieee.org Thu Sep 7 03:03:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 07 Sep 2006 01:03:03 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> Message-ID: <44FFC427.2030101@ieee.org> Charles R Harris wrote: > OK. I do have a couple of questions. Let me insert the docs for array > and asarray : > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > Return an array from object with the specified date-type. > > 1) Is it true that array doesn't always return a copy except by > default? asarray says it contrasts with array in this regard. Maybe > copy=0 should be deprecated. array is the main creation function. It is a loose wrapper around PyArray_fromAny. copy=0 means don't copy unless you have to. > 2) Is asarray is basically array with copy=0? Yes. > > 3) Is asanyarray basically array with copy=0 and subok=1? Yes. > > 4) Is there some sort of precedence table for conversions? To me it > looks like the most deeply nested lists are converted to arrays first, > numeric if they contain all numeric types, object otherwise. I assume > the algorithm then ascends up through the hierarchy like traversing a > binary tree in postorder? I'm not sure I understand what you mean. The discover-depth and discover-dimensions algorithm figures out what the shape should be and then recursive PySequence_GetItem and PySequence_SetItem is used to copy the information over to the ndarray from the nested sequence. > > 5) All nesting must be to the same depth and the deepest nested items > must have the same length. Yes, there are routines discover_depth and discover_dimensions that are the actual algorithm used. These are adapted from Numeric. > > 6) How is the difference between lists and "lists" determined, i.e., > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > In [9]: array([1,2,3],[1,2]], dtype = object) > ------------------------------------------------------------ > File "", line 1 > array([1,2,3],[1,2]], dtype = object) > ^ > SyntaxError: invalid syntax I think this is just due to a missing [ in In [9]. There is no semantic difference between list([1,2,3]) and [1,2,3] (NumPy will see those things as exactly the same). > > Is the difference that list(...) and array(...) are passed as > functions (lazy evaluation), but a list is just a list? There is nothing like "lazy evaluation" going on. array([1,2,3]) is evaluated returning an object and array([1,2]) is evaluated returning an object and then the two are put into another object array. Equivalent code a = array([1,2,3]) b = array([1,2]) c = array([a,b],dtype=object) Thanks for all your help with documentation. It is very-much appreciated. -Travis From zdm105 at tom.com Sun Sep 10 03:09:07 2006 From: zdm105 at tom.com (=?GB2312?B?IjnUwjIxLTIyyNUvyc+6oyI=?=) Date: Sun, 10 Sep 2006 15:09:07 +0800 Subject: [Numpy-discussion] =?GB2312?B?RE9FzO+/2sq9xrfWyrmks8w=?= Message-ID: An HTML attachment was scrubbed... URL: From ivilata at carabos.com Thu Sep 7 08:12:10 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Thu, 07 Sep 2006 14:12:10 +0200 Subject: [Numpy-discussion] Fix to Int64/string numexpr support Message-ID: <45000C9A.5050902@carabos.com> Hi all, I've detected some small errors in the patches I sent some time ago for adding Int64 and string support to numexpr (see http://www.mail-archive.com/numpy-discussion%40lists.sourceforge.net/msg01551.html). Basically: * ``numpy.string`` was accessed instead of ``numpy.string_`` (looks like one of those not-so-just-aesthetical last-time changes). * The ``copy_args`` argument to ``evaluate()`` and others was no longer needed since discontiguous/unaligned arrays are no longer a special case. I have attached the necessary patch. Bye! :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr_no_copy_args.diff Type: text/x-patch Size: 2712 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: OpenPGP digital signature URL: From svetosch at gmx.net Thu Sep 7 09:29:24 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 07 Sep 2006 15:29:24 +0200 Subject: [Numpy-discussion] reshape: missing arg check? Message-ID: <45001EB4.7080001@gmx.net> Hi, never mind that the following syntax is wrong, but is it supposed to yield that SystemError instead of something more informative? (This is with b5 on win32 and python 2.4.3) >>> b.reshape(3,3,axis = 1) Traceback (most recent call last): File "", line 1, in ? SystemError: NULL result without error in PyObject_Call -sven From charlesr.harris at gmail.com Thu Sep 7 15:22:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 13:22:01 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <44FFC231.4060507@ieee.org> References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: On 9/7/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > On 9/6/06, *Charles R Harris* > > wrote: > > > > > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > > wrote: > > > > Charles R Harris wrote: > > > > > > Where is array at this point? > > Basically it supports the old Numeric behavior wherein object > > array's > > are treated as before *except* for when an error would have > > occurred > > previously when the "new behavior" kicks in. Anything that > > violates > > that is a bug needing to be fixed. > > > > This leaves the new object-array constructor used less > > often. It could > > be exported explicitly into an oarray constructor, but I'm not > > sure > > about the advantages of that approach. There are benefits to > > having > > object arrays constructed in the same way as other arrays. It > > turns out > > many people actually like that feature of Numeric, which is > > the reason I > > didn't go the route of numarray which pulled object arrays out. > > > > At this point, however, object arrays can even be part of > > records and so > > need to be an integral part of the data-type description. > > Pulling that > > out is not going to happen. A more intelligent object-array > > constructor, however, may be a useful tool. > > > > > > OK. I do have a couple of questions. Let me insert the docs for > > array and asarray : > > > > """array(object, dtype=None, copy=1,order=None, subok=0,ndmin=0) > > > > Return an array from object with the specified date-type. > > > > Inputs: > > object - an array, any object exposing the array interface, > any > > object whose __array__ method returns an array, or > any > > (nested) sequence. > > dtype - The desired data-type for the array. If not given, > > then > > the type will be determined as the minimum type > > required > > to hold the objects in the sequence. This > > argument can only > > be used to 'upcast' the array. For downcasting, > > use the > > .astype(t) method. > > copy - If true, then force a copy. Otherwise a copy will > > only occur > > if __array__ returns a copy, obj is a nested > > sequence, or > > a copy is needed to satisfy any of the other > > requirements > > order - Specify the order of the array. If order is 'C', > > then the > > array will be in C-contiguous order (last-index > > varies the > > fastest). If order is 'FORTRAN', then the > > returned array > > will be in Fortran-contiguous order (first-index > > varies the > > fastest). If order is None, then the returned > > array may > > be in either C-, or Fortran-contiguous order or even > > discontiguous. > > subok - If True, then sub-classes will be passed-through, > > otherwise > > the returned array will be forced to be a > > base-class array > > ndmin - Specifies the minimum number of dimensions that the > > resulting > > array should have. 1's will be pre-pended to the > > shape as > > needed to meet this requirement. > > > > """) > > > > asarray(a, dtype=None, order=None) > > Returns a as an array. > > > > Unlike array(), no copy is performed if a is already an array. > > Subclasses > > are converted to base class ndarray. > > > > 1) Is it true that array doesn't always return a copy except by > > default? asarray says it contrasts with array in this regard. > > Maybe copy=0 should be deprecated. > > > > 2) Is asarray is basically array with copy=0? > > > > 3) Is asanyarray basically array with copy=0 and subok=1? > > > > 4) Is there some sort of precedence table for conversions? To me > > it looks like the most deeply nested lists are converted to arrays > > first, numeric if they contain all numeric types, object > > otherwise. I assume the algorithm then ascends up through the > > hierarchy like traversing a binary tree in postorder? > > > > 5) All nesting must be to the same depth and the deepest nested > > items must have the same length. > > > > 6) How is the difference between lists and "lists" determined, i.e., > > > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > > > > In [9]: array([1,2,3],[1,2]], dtype = object) > > ------------------------------------------------------------ > > File "", line 1 > > array([1,2,3],[1,2]], dtype = object) > > ^ > > SyntaxError: invalid syntax > > > > Is the difference that list(...) and array(...) are passed as > > functions (lazy evaluation), but a list is just a list? > > > > Sorry to be asking all these questions, but I would like to try > > making the documentation be a bit of a reference. I am sure I will > > have more questions ;) > > > > -Travis > > > > > > And, voila, ragged arrays: > > > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > > > In [10]: a*2 > > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > > > In [11]: a + a > > Out[11]: array([[2 4 6], [2 4]], dtype=object) > > Now I remember that this was my original motivation for futzing with the > object-array constructor in the first place. So, now you get there only > after an attempt to make a "rectangular" array first. > > -Travis So is this intentional? In [24]: a = array([[],[],[]], dtype=object) In [25]: a.shape Out[25]: (3, 0) In [26]: a = array([], dtype=object) In [27]: a.shape Out[27]: (0,) One could argue that the first array should have shape (3,) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 7 15:29:45 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 13:29:45 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: On 9/7/06, Charles R Harris wrote: > > > > On 9/7/06, Travis Oliphant wrote: > > > > Charles R Harris wrote: > > > On 9/6/06, *Charles R Harris* > > > wrote: > > > > > > > > > > > > On 9/6/06, *Travis Oliphant* < oliphant.travis at ieee.org > > > > wrote: > > > > > > Charles R Harris wrote: > > > > > > > > Where is array at this point? > > > Basically it supports the old Numeric behavior wherein object > > > array's > > > are treated as before *except* for when an error would have > > > occurred > > > previously when the "new behavior" kicks in. Anything that > > > violates > > > that is a bug needing to be fixed. > > > > > > This leaves the new object-array constructor used less > > > often. It could > > > be exported explicitly into an oarray constructor, but I'm not > > > > > sure > > > about the advantages of that approach. There are benefits to > > > having > > > object arrays constructed in the same way as other arrays. It > > > turns out > > > many people actually like that feature of Numeric, which is > > > the reason I > > > didn't go the route of numarray which pulled object arrays > > out. > > > > > > At this point, however, object arrays can even be part of > > > records and so > > > need to be an integral part of the data-type description. > > > Pulling that > > > out is not going to happen. A more intelligent object-array > > > constructor, however, may be a useful tool. > > > > > > > > > OK. I do have a couple of questions. Let me insert the docs for > > > array and asarray : > > > > > > """array(object, dtype=None, copy=1,order=None, > > subok=0,ndmin=0) > > > > > > Return an array from object with the specified date-type. > > > > > > Inputs: > > > object - an array, any object exposing the array interface, > > any > > > object whose __array__ method returns an array, or > > any > > > (nested) sequence. > > > dtype - The desired data-type for the array. If not given, > > > then > > > the type will be determined as the minimum type > > > required > > > to hold the objects in the sequence. This > > > argument can only > > > be used to 'upcast' the array. For downcasting, > > > use the > > > .astype(t) method. > > > copy - If true, then force a copy. Otherwise a copy will > > > only occur > > > if __array__ returns a copy, obj is a nested > > > sequence, or > > > a copy is needed to satisfy any of the other > > > requirements > > > order - Specify the order of the array. If order is 'C', > > > then the > > > array will be in C-contiguous order (last-index > > > varies the > > > fastest). If order is 'FORTRAN', then the > > > returned array > > > will be in Fortran-contiguous order (first-index > > > varies the > > > fastest). If order is None, then the returned > > > array may > > > be in either C-, or Fortran-contiguous order or > > even > > > discontiguous. > > > subok - If True, then sub-classes will be passed-through, > > > otherwise > > > the returned array will be forced to be a > > > base-class array > > > ndmin - Specifies the minimum number of dimensions that the > > > resulting > > > array should have. 1's will be pre-pended to the > > > shape as > > > needed to meet this requirement. > > > > > > """) > > > > > > asarray(a, dtype=None, order=None) > > > Returns a as an array. > > > > > > Unlike array(), no copy is performed if a is already an array. > > > Subclasses > > > are converted to base class ndarray. > > > > > > 1) Is it true that array doesn't always return a copy except by > > > default? asarray says it contrasts with array in this regard. > > > Maybe copy=0 should be deprecated. > > > > > > 2) Is asarray is basically array with copy=0? > > > > > > 3) Is asanyarray basically array with copy=0 and subok=1? > > > > > > 4) Is there some sort of precedence table for conversions? To me > > > it looks like the most deeply nested lists are converted to arrays > > > first, numeric if they contain all numeric types, object > > > otherwise. I assume the algorithm then ascends up through the > > > hierarchy like traversing a binary tree in postorder? > > > > > > 5) All nesting must be to the same depth and the deepest nested > > > items must have the same length. > > > > > > 6) How is the difference between lists and "lists" determined, i.e > > ., > > > > > > In [3]: array([list([1,2,3]),list([1,2])], dtype = object) > > > Out[3]: array([[1, 2, 3], [1, 2]], dtype=object) > > > > > > In [8]: array([array([1,2,3]),array([1,2])], dtype = object) > > > Out[8]: array([[1 2 3], [1 2]], dtype=object) > > > > > > > > > In [9]: array([1,2,3],[1,2]], dtype = object) > > > ------------------------------------------------------------ > > > File "", line 1 > > > array([1,2,3],[1,2]], dtype = object) > > > ^ > > > SyntaxError: invalid syntax > > > > > > Is the difference that list(...) and array(...) are passed as > > > functions (lazy evaluation), but a list is just a list? > > > > > > Sorry to be asking all these questions, but I would like to try > > > making the documentation be a bit of a reference. I am sure I will > > > have more questions ;) > > > > > > -Travis > > > > > > > > > And, voila, ragged arrays: > > > > > > In [9]: a = array([array([1,2,3]),array([1,2])], dtype = object) > > > > > > In [10]: a*2 > > > Out[10]: array([[2 4 6], [2 4]], dtype=object) > > > > > > In [11]: a + a > > > Out[11]: array([[2 4 6], [2 4]], dtype=object) > > > > Now I remember that this was my original motivation for futzing with the > > > > object-array constructor in the first place. So, now you get there only > > after an attempt to make a "rectangular" array first. > > > > -Travis > > > So is this intentional? > > In [24]: a = array([[],[],[]], dtype=object) > > In [25]: a.shape > Out[25]: (3, 0) > > In [26]: a = array([], dtype=object) > > In [27]: a.shape > Out[27]: (0,) > > One could argue that the first array should have shape (3,) > And this doesn't look quite right: In [38]: a = array([[1],[2],[3]], dtype=object) In [39]: a.shape Out[39]: (3, 1) In [40]: a = array([[1],[2,3],[4,5]], dtype=object) In [41]: a.shape Out[41]: (3,) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Glen.Mabey at swri.org Thu Sep 7 15:34:14 2006 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Thu, 7 Sep 2006 14:34:14 -0500 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files Message-ID: <20060907193414.GA7225@swri16wm.electro.swri.edu> A long time ago, Travis wrote: > On a related, but orthogonal note: > > My understanding is that using memory-mapped files for *very* large > files will require modification to the mmap module in Python --- > something I think we should push. One part of that process would be > to add the C-struct array interface to the mmap module and the buffer > object -- perhaps this is how we get the array interface into Python > quickly. Then, if we could make a base-type mmap that did not use > the buffer interface or the sequence interface (similar to the > bigndarray in scipy_core) and therefore by-passed the problems with > Python in those areas, then the current mmap object could inherit from > the base class and provide current functionality while still exposing > the array interface for access to >2GB files on 64-bit systems. > > Who would like to take up the ball for modifying mmap in Python in > this fashion? > > -Travis Did anyone ever "pick up the ball" on this issue? Glen From scipy at mspacek.mm.st Thu Sep 7 16:17:58 2006 From: scipy at mspacek.mm.st (Martin Spacek) Date: Thu, 07 Sep 2006 13:17:58 -0700 Subject: [Numpy-discussion] ndarray.count() ? Message-ID: <45007E76.8030102@mspacek.mm.st> What's the most straightforward way to count, say, the number of 1s or Trues in the array? Or the number of any integer? I was surprised to discover recently that there isn't a count() method as there is for Python lists. Sorry if this has been discussed already, but I'm wondering if there's a reason for its absence. I came across a thread in March: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3066460 that talked a bit about this in terms of speed, but what about just the convenience of having a count() method? Looks like masked arrays have a count method, don't know much about them though. Also, I understand the inaccuracies when converting between binary and decimal floating point representations, and therefore making counting of a specific float value in an array somewhat undefined, yet it seems to work in Python lists: >>> 1.1 1.1000000000000001 >>> a=[1.1, 1.1, 1.2] >>> a [1.1000000000000001, 1.1000000000000001, 1.2] >>> a.count(1.1) 2 >>> a.count(1.1000000000000001) 2 >>> a.count(1.2) 1 Comments? Martin From charlesr.harris at gmail.com Thu Sep 7 16:37:07 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 14:37:07 -0600 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. I don't know about count, but you can gin up something like this In [78]: a = ran.randint(0,2, size=(10,)) In [79]: a Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) In [80]: b = sort(a) In [81]: b.searchsorted(1, side='right') - b.searchsorted(1, side='left') Out[81]: 6 Which counts the number of ones in a. I came across a thread in March: > > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3066460 > > that talked a bit about this in terms of speed, but what about just the > convenience of having a count() method? > > Looks like masked arrays have a count method, don't know much about them > though. > > Also, I understand the inaccuracies when converting between binary and > decimal floating point representations, and therefore making counting of > a specific float value in an array somewhat undefined, yet it seems to > work in Python lists: > > >>> 1.1 > 1.1000000000000001 > >>> a=[1.1, 1.1, 1.2] > >>> a > [1.1000000000000001, 1.1000000000000001, 1.2] > >>> a.count(1.1) > 2 > >>> a.count(1.1000000000000001) > 2 > >>> a.count(1.2) > 1 Well, 1.1 == 1.1000000000000001 and that doesn't change. You probably need to use different precisions to run into problems. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 16:40:24 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 15:40:24 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. Mostly, it's simply easy enough to implement yourself. Not all one-liners should be methods on the array object. (a == value).sum() Of course, there are several different things you might do. You might want to have multiple values broadcast across the array. You might want to reduce the count along a given axis. You might want to use floating point comparison with a tolerance. Putting all of those options into one method reduces the convenience of that method. Putting a crippled implementation (i.e. just that one-liner) expands the already-enormous API of the ndarray object without much benefit. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Thu Sep 7 16:25:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 7 Sep 2006 13:25:04 -0700 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files In-Reply-To: <20060907193414.GA7225@swri16wm.electro.swri.edu> References: <20060907193414.GA7225@swri16wm.electro.swri.edu> Message-ID: <200609071325.04777.haase@msg.ucsf.edu> Hi Glen ! How is that quote really !? The new Python2.5 *is* implementing the needed changes - so go ahead install Python2.5 (rc1 is the latest I think) and report how it works. I would also be very intersted to hear ;-) -Sebastian Haase On Thursday 07 September 2006 12:34, Glen W. Mabey wrote: > A long time ago, Travis wrote: > > On a related, but orthogonal note: > > > > My understanding is that using memory-mapped files for *very* large > > files will require modification to the mmap module in Python --- > > something I think we should push. One part of that process would be > > to add the C-struct array interface to the mmap module and the buffer > > object -- perhaps this is how we get the array interface into Python > > quickly. Then, if we could make a base-type mmap that did not use > > the buffer interface or the sequence interface (similar to the > > bigndarray in scipy_core) and therefore by-passed the problems with > > Python in those areas, then the current mmap object could inherit from > > the base class and provide current functionality while still exposing > > the array interface for access to >2GB files on 64-bit systems. > > > > Who would like to take up the ball for modifying mmap in Python in > > this fashion? > > > > -Travis > > Did anyone ever "pick up the ball" on this issue? > > Glen > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From alexander.belopolsky at gmail.com Thu Sep 7 16:47:07 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 7 Sep 2006 16:47:07 -0400 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <45007E76.8030102@mspacek.mm.st> References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. > You don't really need count with ndarrays: >>> from numpy import * >>> a = array([1,2,3,1,2,3,1,2]) >>> (a==3).sum() 2 From ndarray at mac.com Thu Sep 7 16:48:07 2006 From: ndarray at mac.com (Sasha) Date: Thu, 7 Sep 2006 16:48:07 -0400 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: On 9/7/06, Martin Spacek wrote: > What's the most straightforward way to count, say, the number of 1s or > Trues in the array? Or the number of any integer? > > I was surprised to discover recently that there isn't a count() method > as there is for Python lists. Sorry if this has been discussed already, > but I'm wondering if there's a reason for its absence. > You don't really need count with ndarrays: >>> from numpy import * >>> a = array([1,2,3,1,2,3,1,2]) >>> (a==3).sum() 2 From mspacek at mm.st Thu Sep 7 17:21:10 2006 From: mspacek at mm.st (Martin Spacek) Date: Thu, 07 Sep 2006 14:21:10 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: <45008D46.5050008@mm.st> Great! That's exactly what I wanted. Works with floats too. Thanks, Martin Robert Kern wrote: > Mostly, it's simply easy enough to implement yourself. Not all one-liners should > be methods on the array object. > > (a == value).sum() > From mike.ressler at gmail.com Thu Sep 7 16:18:19 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Thu, 7 Sep 2006 13:18:19 -0700 Subject: [Numpy-discussion] 2 GB limit for memmap'ed files In-Reply-To: <20060907193414.GA7225@swri16wm.electro.swri.edu> References: <20060907193414.GA7225@swri16wm.electro.swri.edu> Message-ID: <268febdf0609071318la00d3evaac8d6640b967acb@mail.gmail.com> On 9/7/06, Glen W. Mabey wrote: > > A long time ago, Travis wrote: > > > > My understanding is that using memory-mapped files for *very* large > > files will require modification to the mmap module in Python --- > > Did anyone ever "pick up the ball" on this issue? This works with python-2.5 betas and above, and numpy 1.0 betas and above. I frequently do 10+ GB mmaps. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Sep 7 18:15:33 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 07 Sep 2006 16:15:33 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FCA9E0.4040908@ieee.org> <1e2af89e0609050634k781c543fhec47481d6a20b8e9@mail.gmail.com> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> Message-ID: <45009A05.5020007@ee.byu.edu> Charles R Harris wrote: > > So is this intentional? > > In [24]: a = array([[],[],[]], dtype=object) > > In [25]: a.shape > Out[25]: (3, 0) > > In [26]: a = array([], dtype=object) > > In [27]: a.shape > Out[27]: (0,) > > One could argue that the first array should have shape (3,) > Yes, it's intentional because it's the old behavior of Numeric. And it follows the rule that object arrays don't do anything special unless the old technique of using [] as 'dimension delimiters' breaks down. > > And this doesn't look quite right: > > In [38]: a = array([[1],[2],[3]], dtype=object) > > In [39]: a.shape > Out[39]: (3, 1) > > In [40]: a = array([[1],[2,3],[4,5]], dtype=object) > > In [41]: a.shape > Out[41]: (3,) > Again, same reason as before. The first example works fine to construct a rectangular array of object arrays of dimension 2. The second only does if we limit the number of dimensions to 1. The rule is that array needs nested lists with the same number of dimensions unless you have object arrays. Then, the dimensionality will be determined by finding the largest number of dimensions possible for consistency of shape. -Travis From rex at nosyntax.com Thu Sep 7 18:33:36 2006 From: rex at nosyntax.com (rex) Date: Thu, 7 Sep 2006 15:33:36 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> Message-ID: <20060907223336.GH5025@x2.nosyntax.com> Charles R Harris [2006-09-07 15:04]: > I don't know about count, but you can gin up something like this > > In [78]: a = ran.randint(0,2, size=(10,)) > > In [79]: a > Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) This exposed inconsistent randint() behavior between SciPy and the Python random module. The Python randint includes the upper endpoint. The SciPy version excludes it. >>> from random import randint >>> for i in range(50): ... print randint(0,2), ... 0 1 1 1 1 1 0 0 2 1 1 0 2 2 1 2 0 2 0 0 0 2 2 2 2 2 2 2 1 2 2 0 0 1 2 2 0 1 1 0 2 0 1 2 1 2 2 2 1 1 >>> from scipy import * >>> print random.randint(0,2, size=(100,)) [0 1 1 1 1 0 1 1 0 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 1 1 0 1 1 1 0 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 1 0 1 1 1 1 1 1 1 0 1 1 0 0 1 0 1 1 0 0 0 0 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 0 0] -rex From a.h.jaffe at gmail.com Thu Sep 7 18:46:15 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 07 Sep 2006 23:46:15 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy Message-ID: Hi all, It seems that scipy and numpy define rfft differently. numpy returns n/2+1 complex numbers (so the first and last numbers are actually real) with the frequencies equivalent to the positive part of the fftfreq, whereas scipy returns n real numbers with the frequencies as in rfftfreq (i.e., two real numbers at the same frequency, except for the highest and lowest) [All of the above for even n; but the difference between numpy and scipy remains for odd n.] I think the numpy behavior makes more sense, as it doesn't require any unpacking after the fact, at the expense of a tiny amount of wasted space. But would this in fact require scipy doing extra work from whatever the 'native' real_fft (fftw, I assume) produces? Anyone else have an opinion? Andrew From charlesr.harris at gmail.com Thu Sep 7 18:48:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 16:48:51 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <45009A05.5020007@ee.byu.edu> References: <44FB29CA.9070808@colorado.edu> <44FDAE3C.7000901@ieee.org> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: On 9/7/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > So is this intentional? > > > > In [24]: a = array([[],[],[]], dtype=object) > > > > In [25]: a.shape > > Out[25]: (3, 0) > > > > In [26]: a = array([], dtype=object) > > > > In [27]: a.shape > > Out[27]: (0,) > > > > One could argue that the first array should have shape (3,) > > > Yes, it's intentional because it's the old behavior of Numeric. And it > follows the rule that object arrays don't do anything special unless the > old technique of using [] as 'dimension delimiters' breaks down. > > > > > And this doesn't look quite right: > > > > In [38]: a = array([[1],[2],[3]], dtype=object) > > > > In [39]: a.shape > > Out[39]: (3, 1) > > > > In [40]: a = array([[1],[2,3],[4,5]], dtype=object) > > > > In [41]: a.shape > > Out[41]: (3,) > > > > Again, same reason as before. The first example works fine to construct > a rectangular array of object arrays of dimension 2. The second only > does if we limit the number of dimensions to 1. > > The rule is that array needs nested lists with the same number of > dimensions unless you have object arrays. Then, the dimensionality will > be determined by finding the largest number of dimensions possible for > consistency of shape. So there is a 'None' trick: In [93]: a = array([[[2]], None], dtype=object) In [94]: a[0] Out[94]: [[2]] I wonder if it wouldn't be useful to have a 'depth' keyword. Thus depth=None is current behavior, but array([], depth=0) would produce a zero dimensional array containing an empty list. Although I notice from playing with dictionaries that a zero dimensional array containing a dictionary isn't very useful. array([[],[]], depth=1) would produce a one dimensional array containing two empty lists, etc. I can see it is difficult to get something truely general with the current syntax without a little bit of extra information. Another question, what property must an object possess to be a container type argument in array? There are sequence type objects, and array type objects. Are there more or is everything else treated as an object? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 18:57:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 17:57:00 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <20060907223336.GH5025@x2.nosyntax.com> References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> Message-ID: rex wrote: > Charles R Harris [2006-09-07 15:04]: >> I don't know about count, but you can gin up something like this >> >> In [78]: a = ran.randint(0,2, size=(10,)) >> >> In [79]: a >> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) > > This exposed inconsistent randint() behavior between SciPy and the Python > random module. The Python randint includes the upper endpoint. The SciPy > version excludes it. numpy.random.random_integers() includes the upper bound, if you like. numpy.random does not try to emulate the standard library's random module. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Sep 7 19:03:59 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 17:03:59 -0600 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: On 9/7/06, Andrew Jaffe wrote: > > Hi all, > > It seems that scipy and numpy define rfft differently. > > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? > > Anyone else have an opinion? Yes, I prefer the scipy version because the result is actually a complex array and can immediately be use as the coefficients of the fft for frequencies <= Nyquist. I suspect, without checking, that what you get in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as the coefficient of the second frequency, etc. This makes it difficult to multiply by a complex transfer function or phase shift the result to rotate the original points by some fractional amount. As to unpacking, for some algorithms the two real coefficients are packed into the real and complex parts of the zero frequency so all that is needed is an extra complex slot at the end. Other algorithms produce what you describe. I just think it is more convenient to think of the real fft as an efficient complex fft that only computes the coefficients <= Nyquist because Hermitean symmetry determines the rest. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstrunk at enthought.com Thu Sep 7 19:19:01 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Thu, 7 Sep 2006 18:19:01 -0500 Subject: [Numpy-discussion] ISP changeover Tuesday 9/12 7:00 PM Central Message-ID: <200609071819.01770.jstrunk@enthought.com> Good afternoon, Unfortunately, our recent change in internet service providers is not working out. We will be switching to a more reliable provider on Tuesday 9/12 at 7:00 PM Central. Please allow for up to two hours of downtime. I will send an email announcing the start and completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. The problem was that our in-building internet service provider neglected to mention that our internet connection was over a wireless link to a different building. This link had very high latency. They have fixed this problem, but we feel that wireless is not stable enough for our needs. In order to provide higher quality service, we will be using 3 bonded T1s from AT&T after this switchover. Please pass this message along to people that I have missed. If you have any questions, please direct them to me. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From peridot.faceted at gmail.com Thu Sep 7 19:49:08 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Thu, 7 Sep 2006 19:49:08 -0400 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: Maybe I should stay out of this, but it seems like constructing object arrays is complicated and involves a certain amount of guesswork on the part of Numeric. For example, if you do array([a,b,c]).shape(), the answer is normally (3,) unless a b and c happen to all be lists of the same length, at which point your array could have a much more complicated shape... but as the person who wrote "array([a,b,c])" it's tempting to assume that the result has shape (3,), only to discover subtle bugs much later. If we were writing an array-creation function from scratch, would there be any reason to include object-array creation in the same function as uniform array creation? It seems like a bad idea to me. If not, the problem is just compatibility with Numeric. Why not simply write a wrapper function in python that does Numeric-style guesswork, and put it in the compatibility modules? How much code will actually break? A. M. Archibald From a.h.jaffe at gmail.com Thu Sep 7 20:04:52 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 08 Sep 2006 01:04:52 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: <4500B3A4.7040704@gmail.com> Hi Charles, Charles R Harris wrote: > On 9/7/06, *Andrew Jaffe* > wrote: > > Hi all, > > It seems that scipy and numpy define rfft differently. > > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the > difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? > > Anyone else have an opinion? > > Yes, I prefer the scipy version because the result is actually a complex > array and can immediately be use as the coefficients of the fft for > frequencies <= Nyquist. I suspect, without checking, that what you get > in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as > the coefficient of the second frequency, etc. This makes it difficult to > multiply by a complex transfer function or phase shift the result to > rotate the original points by some fractional amount. > > As to unpacking, for some algorithms the two real coefficients are > packed into the real and complex parts of the zero frequency so all that > is needed is an extra complex slot at the end. Other algorithms produce > what you describe. I just think it is more convenient to think of the > real fft as an efficient complex fft that only computes the coefficients > <= Nyquist because Hermitean symmetry determines the rest. Unless I misunderstand, I think you've got it backwards: - numpy.fft.rfft produces the correct complex array (with no strange packing), with frequencies (0, 1, 2, ... n/2) - scipy.fftpack.rfft produces a single real array, in the correct order, but with frequencies (0, 1, 1, 2, 2, ..., n/2) -- as given by scipy.fftpack's rfftfreq function. So I think you prefer numpy, not scipy. This is complicated by the fact that (I think) numpy.fft shows up as scipy.fft, so functions with the same name in scipy.fft and scipy.fftpack aren't actually the same! Andrew From rex at nosyntax.com Thu Sep 7 20:14:22 2006 From: rex at nosyntax.com (rex) Date: Thu, 7 Sep 2006 17:14:22 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> Message-ID: <20060908001422.GJ5025@x2.nosyntax.com> Robert Kern [2006-09-07 16:35]: > rex wrote: > > Charles R Harris [2006-09-07 15:04]: > >> I don't know about count, but you can gin up something like this > >> > >> In [78]: a = ran.randint(0,2, size=(10,)) > >> > >> In [79]: a > >> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) > > > > This exposed inconsistent randint() behavior between SciPy and the Python > > random module. The Python randint includes the upper endpoint. The SciPy > > version excludes it. > > numpy.random.random_integers() includes the upper bound, if you like. > numpy.random does not try to emulate the standard library's random module. I'm not in a position to argue the merits, but IMHO, when code that previously worked silently starts returning subtly bad results after importing numpy, there is a problem. What possible upside is there in having randint() behave one way in the random module and silently behave differently in numpy? More generally, since numpy.random does not try to emulate the random module, how does one convert from code that uses the random module to numpy? Is randint() the only silent problem, or are there others? If so, how does one discover them? Are they documented anywhere? I deeply appreciate the countless hours the core developers have contributed to numpy/scipy, but sometimes I think you are too close to the problems to fully appreciate the barriers to widespread adoption such silent "gotchas" present. If the code breaks, fine, you know there's a problem. When it runs, but returns wrong -- but not obviously wrong -- results, there's a serious problem that will deter a significant number of people from ever trying the product again. Again, what is the upside of changing the behavior of the standard library's randint() without also changing the name? -rex From charlesr.harris at gmail.com Thu Sep 7 20:42:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Sep 2006 18:42:08 -0600 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: <4500B3A4.7040704@gmail.com> References: <4500B3A4.7040704@gmail.com> Message-ID: On 9/7/06, Andrew Jaffe wrote: > > Hi Charles, > > Charles R Harris wrote: > > On 9/7/06, *Andrew Jaffe* > > wrote: > > > > Hi all, > > > > It seems that scipy and numpy define rfft differently. > > > > numpy returns n/2+1 complex numbers (so the first and last numbers > are > > actually real) with the frequencies equivalent to the positive part > of > > the fftfreq, whereas scipy returns n real numbers with the > frequencies > > as in rfftfreq (i.e., two real numbers at the same frequency, except > for > > the highest and lowest) [All of the above for even n; but the > > difference > > between numpy and scipy remains for odd n.] > > > > I think the numpy behavior makes more sense, as it doesn't require > any > > unpacking after the fact, at the expense of a tiny amount of wasted > > space. But would this in fact require scipy doing extra work from > > whatever the 'native' real_fft (fftw, I assume) produces? > > > > Anyone else have an opinion? > > > > Yes, I prefer the scipy version because the result is actually a complex > > array and can immediately be use as the coefficients of the fft for > > frequencies <= Nyquist. I suspect, without checking, that what you get > > in numpy is a real array with f[0] == zero frequency, f[1] + 1j* f[2] as > > the coefficient of the second frequency, etc. This makes it difficult to > > multiply by a complex transfer function or phase shift the result to > > rotate the original points by some fractional amount. > > > > As to unpacking, for some algorithms the two real coefficients are > > packed into the real and complex parts of the zero frequency so all that > > is needed is an extra complex slot at the end. Other algorithms produce > > what you describe. I just think it is more convenient to think of the > > real fft as an efficient complex fft that only computes the coefficients > > <= Nyquist because Hermitean symmetry determines the rest. > > Unless I misunderstand, I think you've got it backwards: > - numpy.fft.rfft produces the correct complex array (with no strange > packing), with frequencies (0, 1, 2, ... n/2) > > - scipy.fftpack.rfft produces a single real array, in the correct > order, but with frequencies (0, 1, 1, 2, 2, ..., n/2) -- as given by > scipy.fftpack's rfftfreq function. > > So I think you prefer numpy, not scipy. Ah, well then, yes. I prefer Numpy. IIRC, one way to get the Scipy ordering is to use the Hartley transform as the front to the real transform. And, now that you mention it, there was a big discussion about it in the scipy list way back when with yours truly pushing the complex form. I don't recall that anything was settled, rather the natural output of the algorithm they were using prevailed by default. Maybe you could write a front end that did the right thing? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fccoelho at gmail.com Thu Sep 7 21:18:46 2006 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu, 7 Sep 2006 22:18:46 -0300 Subject: [Numpy-discussion] Fwd: f2py with xmingw In-Reply-To: References: Message-ID: Hi, I have a module that uses a Fortran extension which I would like to compile for windows with f2py. I wonder If I could do this from Linux using xmingw. Has anyone tried this? thanks, -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- "Laws are like sausages. It's better not to see them being made." Otto von Bismark -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 7 23:35:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 07 Sep 2006 22:35:06 -0500 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: <20060908001422.GJ5025@x2.nosyntax.com> References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> <20060908001422.GJ5025@x2.nosyntax.com> Message-ID: rex wrote: > Robert Kern [2006-09-07 16:35]: >> rex wrote: >>> Charles R Harris [2006-09-07 15:04]: >>>> I don't know about count, but you can gin up something like this >>>> >>>> In [78]: a = ran.randint(0,2, size=(10,)) >>>> >>>> In [79]: a >>>> Out[79]: array([0, 1, 0, 1, 1, 0, 0, 1, 1, 1]) >>> This exposed inconsistent randint() behavior between SciPy and the Python >>> random module. The Python randint includes the upper endpoint. The SciPy >>> version excludes it. >> numpy.random.random_integers() includes the upper bound, if you like. >> numpy.random does not try to emulate the standard library's random module. > > I'm not in a position to argue the merits, but IMHO, when code that > previously worked silently starts returning subtly bad results after > importing numpy, there is a problem. What possible upside is there in > having randint() behave one way in the random module and silently behave > differently in numpy? I don't understand you. Importing numpy does not change the standard library's random module in any way. There is no silent difference in behavior. If you use numpy.random you get one set of behavior. If you use random, you get another. Pick the one you want. They're not interchangeable, and nothing suggests that they ought to be. > More generally, since numpy.random does not try to emulate the random > module, how does one convert from code that uses the random module to > numpy? Is randint() the only silent problem, or are there others? If so, > how does one discover them? Are they documented anywhere? The docstrings in that module are complete. > I deeply appreciate the countless hours the core developers have > contributed to numpy/scipy, but sometimes I think you are too close to > the problems to fully appreciate the barriers to widespread adoption such > silent "gotchas" present. If the code breaks, fine, you know there's a > problem. When it runs, but returns wrong -- but not obviously wrong -- > results, there's a serious problem that will deter a significant number > of people from ever trying the product again. > > Again, what is the upside of changing the behavior of the standard > library's randint() without also changing the name? Again, numpy.random has nothing to do with the standard library module random. The names of the functions match those in the PRNG facilities that used to be in Numeric and scipy which numpy.random is replacing. Specifically, numpy.random.randint() derives its behavior from Numeric's RandomArray.randint(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rahn at avg-joe.com Fri Sep 8 06:35:35 2006 From: rahn at avg-joe.com (Mabelle Reach) Date: Fri, 8 Sep 2006 03:35:35 -0700 Subject: [Numpy-discussion] PHAruRMACY Message-ID: <000001c6d332$84d20620$cc58a8c0@nyaz> Hi All yo k ur P n HAR c MAC x Y di f rect i ly from the man a ufa d cture o r, Your c w hanc f e to eco q no p mize w f ith us http://www.pumationdesun.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From listservs at mac.com Fri Sep 8 09:20:27 2006 From: listservs at mac.com (listservs at mac.com) Date: Fri, 8 Sep 2006 09:20:27 -0400 Subject: [Numpy-discussion] numpy test failure: "ctypes is not available" Message-ID: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> I have built and installed numpy from svn on an Intel mac, and am having test failures that do not occur on PPC: In [8]: numpy.test() Found 13 tests for numpy.core.umath Found 8 tests for numpy.lib.arraysetops Found 3 tests for numpy.fft.helper Found 1 tests for numpy.lib.ufunclike Found 4 tests for numpy.ctypeslib Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 3 tests for numpy.lib.getlimits Found 31 tests for numpy.core.numerictypes Found 4 tests for numpy.core.scalarmath Found 10 tests for numpy.lib.twodim_base Found 46 tests for numpy.lib.shape_base Found 4 tests for numpy.lib.index_tricks Found 32 tests for numpy.linalg.linalg Found 5 tests for numpy.distutils.misc_util Found 42 tests for numpy.lib.type_check Found 163 tests for numpy.core.multiarray Found 36 tests for numpy.core.ma Found 10 tests for numpy.core.defmatrix Found 39 tests for numpy.lib.function_base Found 0 tests for __main__ .........................EEEE........................................... ........................................................................ ........................................................................ ........................................................................ ........................................................................ ........................................................................ ......................................................... ====================================================================== ERROR: check_dtype (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 10, in check_dtype p = ndpointer(dtype=dt) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_flags (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 54, in check_flags p = ndpointer(flags='FORTRAN') File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_ndim (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 36, in check_ndim p = ndpointer(ndim=0) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ====================================================================== ERROR: check_shape (numpy.tests.test_ctypeslib.test_ndpointer) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 46, in check_shape p = ndpointer(shape=(1,2)) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy raise ImportError, "ctypes is not available." ImportError: ctypes is not available. ---------------------------------------------------------------------- Ran 489 tests in 0.528s FAILED (errors=4) Out[8]: Any clues? -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From faltet at carabos.com Fri Sep 8 10:41:21 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 8 Sep 2006 16:41:21 +0200 Subject: [Numpy-discussion] numpy test failure: "ctypes is not available" In-Reply-To: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> References: <7EE9A3E7-5F4C-45B6-9E2D-C80553FEF8BC@mac.com> Message-ID: <200609081641.22048.faltet@carabos.com> A Divendres 08 Setembre 2006 15:20, listservs at mac.com va escriure: > I have built and installed numpy from svn on an Intel mac, and am > having test failures that do not occur on PPC: > > In [8]: numpy.test() > Found 13 tests for numpy.core.umath > Found 8 tests for numpy.lib.arraysetops > Found 3 tests for numpy.fft.helper > Found 1 tests for numpy.lib.ufunclike > Found 4 tests for numpy.ctypeslib > Found 1 tests for numpy.lib.polynomial > Found 8 tests for numpy.core.records > Found 26 tests for numpy.core.numeric > Found 3 tests for numpy.lib.getlimits > Found 31 tests for numpy.core.numerictypes > Found 4 tests for numpy.core.scalarmath > Found 10 tests for numpy.lib.twodim_base > Found 46 tests for numpy.lib.shape_base > Found 4 tests for numpy.lib.index_tricks > Found 32 tests for numpy.linalg.linalg > Found 5 tests for numpy.distutils.misc_util > Found 42 tests for numpy.lib.type_check > Found 163 tests for numpy.core.multiarray > Found 36 tests for numpy.core.ma > Found 10 tests for numpy.core.defmatrix > Found 39 tests for numpy.lib.function_base > Found 0 tests for __main__ > .........................EEEE........................................... > ........................................................................ > ........................................................................ > ........................................................................ > ........................................................................ > ........................................................................ > ......................................................... > ====================================================================== > ERROR: check_dtype (numpy.tests.test_ctypeslib.test_ndpointer) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/numpy/tests/test_ctypeslib.py", line 10, in > check_dtype > p = ndpointer(dtype=dt) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/numpy/ctypeslib.py", line 15, in _dummy > raise ImportError, "ctypes is not available." > ImportError: ctypes is not available. This may be due to the fact that you are using Python 2.4 here and ctypes comes with Python2.5. Switch to 2.5, install ctypes separately or feel free to ignore this. I suppose that a check has to be set up in the tests to avoid ctypes ones to be checked in case ctypes is not available. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From dd55 at cornell.edu Fri Sep 8 11:03:02 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri, 8 Sep 2006 11:03:02 -0400 Subject: [Numpy-discussion] missing import in numpy.oldnumeric.mlab Message-ID: <200609081103.02628.dd55@cornell.edu> This was just reported on mpl-dev: In [1]: from numpy.oldnumeric import mlab In [2]: mlab.eye(3) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /home/darren/ /usr/lib64/python2.4/site-packages/numpy/oldnumeric/mlab.py in eye(N, M, k, typecode, dtype) 22 dtype = convtypecode(typecode, dtype) 23 if M is None: M = N ---> 24 m = nn.equal(nn.subtract.outer(nn.arange(N), nn.arange(M)),-k) 25 if m.dtype != dtype: 26 return m.astype(dtype) NameError: global name 'nn' is not defined From listservs at mac.com Fri Sep 8 11:31:01 2006 From: listservs at mac.com (listservs at mac.com) Date: Fri, 8 Sep 2006 11:31:01 -0400 Subject: [Numpy-discussion] problems with numpy/f2py module on Intel Mac Message-ID: <9A3362CE-CA49-4241-A5C6-9FA478BAD0F9@mac.com> After building and installing numpy from svn on an Intel Mac, I can successfully build f2py modules, but I get the following linking error: ImportError: Failure linking new module: /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/ flib.so: Symbol not found: ___dso_handle Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/PyMC/flib.so Expected in: dynamic lookup I'm using gfortran. Has anyone seen these types of errors before? I do not get them on PPC. Thanks, -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From zdm105 at tom.com Mon Sep 11 14:30:49 2006 From: zdm105 at tom.com (=?GB2312?B?IjnUwjIxLTIyyNUvyc+6oyI=?=) Date: Tue, 12 Sep 2006 02:30:49 +0800 Subject: [Numpy-discussion] =?GB2312?B?RE9FzO+/2sq9xrfWyrmks8w=?= Message-ID: An HTML attachment was scrubbed... URL: From jsz at kalliness.fsnet.co.uk Fri Sep 8 16:16:53 2006 From: jsz at kalliness.fsnet.co.uk (Sadie Craft) Date: Fri, 8 Sep 2006 22:16:53 +0200 Subject: [Numpy-discussion] regurgitation Message-ID: <001a01c6d384$3131d142$d8d71853@ybxm> Is it really true that no influence is ever brought to bear onanybody about anything? If Diana did want anything youd be on the spot. Andchanging her tone, she said coldly: After all, Captain Cherrellis a D. A good many, said Dinny, seem to drop first, and then do therunning. She did not talk, however,leaving him to begin if he wished. Personally, I dontbelieve Aunt May can put us up. Especially when theyve been to the same school. Not more than I am, dont worry about that. Its as good a place, I suppose, asyou could find. Arrived at Shropshire House Sir Lawrence said:Can we see the Marquess, Pommett? Her knowledge of London was small, and she hailed the first cab. She passed Dinny andAdrian as they were coming in. The man came in again and drewthe curtains. Itrepresented with a moderate degree of certainty a young womanwithout clothes. The man came in again and drewthe curtains. Is your extradition really likely,Hubert? The hat ropes did it, Dinny, and let me tell you that the grapeshave been sour ever since. Professor Hallorsen was expected in at five and should at once begiven the message. Lord Saxenden knows me, and raisedher eyes. I shall sleep with my emergency suit-case; one can always get a taxi here at a moments notice. Shake hands, Millie, and remember what I said. Then, swiftly, looking neither to left nor right, she returned toAdrian. Go off your chump, Cherrell,then youll know what it means to be lonely for the rest of yourdays. But privilege is only justified nowadays byrunning till you drop. Your Auntnever ceases to throw them in my teeth. Puzzlin, theyheard her say as they entered the library. Either Im over-sensitive, Dinny, or this particular afflictiondoes seem to me too dreadful. She bent her head, and still didnt speak. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sister.gif Type: image/gif Size: 15866 bytes Desc: not available URL: From dpinte at itae.be Fri Sep 8 20:32:20 2006 From: dpinte at itae.be (Didrik Pinte) Date: Sat, 09 Sep 2006 02:32:20 +0200 Subject: [Numpy-discussion] gdal numpy/numeric/matplotlib problem Message-ID: <1157761940.23138.27.camel@geru-itae> Hi, I am running a Debian/Sid system and face a problem when using the gdal python bindinds. The gdal python bindings are linked with Numeric. In one of my application i'm using some numpy methods. This is working fine BUT when I add the use of the matploblib library to my application all the calls to gdal specific methods are broken. I have attached a basic example. The first test fails if I import the pylab module. The second one that can be run with any shapefile shows that when pylab is loaded, some GDAL methods raises GEOS exceptions. Commenting the "import pylab" line shows that without pylab there is no exceptions, nor problems. They are two workarounds : [1] a real one : using matplotlib with the Numeric lib [2] a fake one : renaming /usr/lib/python2.4/site-packages/numpy/core/multiarray.so to another name, the tests does not fail. Does anybody have a suggestion to correct this problem ? I can provide more details if needed. Best regards, -- Didrik -------------- next part -------------- A non-text attachment was scrubbed... Name: test_gdalnumpy.py Type: text/x-python Size: 1469 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From mmb at pwdfoundation.org Sat Sep 9 09:20:20 2006 From: mmb at pwdfoundation.org (Roger Mclaughlin) Date: Sat, 9 Sep 2006 15:20:20 +0200 Subject: [Numpy-discussion] cobblestone Message-ID: <001a01c6d413$1cecdd3c$7c824b52@shc.rp> We are taught that in the Bible, youremember. Possibly he meant to melt away in the darkness, leaving us in thewilderness to our fate. He did not seem in the least surprised, indeed he said he knew thatthis was so. Why must he be our companion, Macumazahn? In the end I did so, very unwillingly,to please my relations. Well, hereplied, with a twinkle in his eye, if I were you I should go andlook for the lady there. I realizedthis when I made up my mind to continue the journey to Lake Monealone. Also she was rude enoughto believe and to tell me she believed that I was mad. Odd to have met you in this fashion,but so is everything in this place. Do you mean to say that you tried to do that, Mr. Oh, only that I seemed different in every way. No, always when I was awake and looking at the stars, andgenerally when I was in the open air. I willfollow with the wanderer as soon as he can walk. Then he lay down in the tentand fell at once into a profound sleep. My father was cracked on astrology andcast mine when I was born. I suppose I inherited it frommy father, at any rate there it is. Then he lay down in the tentand fell at once into a profound sleep. This pistol is much lighter triggered than I thought or perhaps theheat has affected the spring. From that moment, for example, I hated thedissipations which had attracted me. Go with them to the borders of the lake, where perchanceone will meet you. From that moment, for example, I hated thedissipations which had attracted me. Indeed, you would be wise to turnback, for I think that death is very fond of Lake Mone. There you will find those who will guideyou. He did not seem in the least surprised, indeed he said he knew thatthis was so. Lastly it was necessary to provide him withone of the spare Winchester rifles and some cartridges. Quatermain, I am adreamer and what is called a mystic. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: obediently.gif Type: image/gif Size: 12923 bytes Desc: not available URL: From ndbecker2 at gmail.com Sat Sep 9 10:18:24 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 9 Sep 2006 10:18:24 -0400 Subject: [Numpy-discussion] 1.0b5 build failure Message-ID: <200609091018.24835.ndbecker2@gmail.com> Fedora FC5 linux, with BLAS, LAPACK, ATLAS, FFTW compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c numpy/core/src/multiarraymodule.c: In function ?initmultiarray?: numpy/core/src/multiarraymodule.c:6860: error: ?NPY_ALLOW_THREADS? undeclared (first use in this function) numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier is reported only once numpy/core/src/multiarraymodule.c:6860: error: for each function it appears in.) numpy/core/src/multiarraymodule.c: In function ?initmultiarray?: numpy/core/src/multiarraymodule.c:6860: error: ?NPY_ALLOW_THREADS? undeclared (first use in this function) numpy/core/src/multiarraymodule.c:6860: error: (Each undeclared identifier is reported only once numpy/core/src/multiarraymodule.c:6860: error: for each function it appears in.) error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC -Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-x86_64-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 From ndbecker2 at gmail.com Sat Sep 9 10:50:24 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 09 Sep 2006 10:50:24 -0400 Subject: [Numpy-discussion] 1.0b5 build failure References: <200609091018.24835.ndbecker2@gmail.com> Message-ID: Sorry, actually that's today's numpy svn - not 1.0b5. From fperez.net at gmail.com Sat Sep 9 11:10:37 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 9 Sep 2006 09:10:37 -0600 Subject: [Numpy-discussion] 1.0b5 build failure In-Reply-To: References: <200609091018.24835.ndbecker2@gmail.com> Message-ID: On 9/9/06, Neal Becker wrote: > Sorry, actually that's today's numpy svn - not 1.0b5. Make sure you have a clean build environment, the current SVN builds just fine on my Dapper 6.06 box; I just rebuilt it a second ago. Cheers, f ps. Here's my little build script, which I use to rebuild numpy/scipy/matplotlib in one shot: longs[scipy]> cat makeall #!/bin/sh ./makepkg numpy ./makepkg scipy ./makepkg matplotlib # EOF which uses the simple makepkg (just edit the PYTHON and PYPREFIX vars to suit your config): longs[scipy]> cat makepkg #!/bin/sh PACKAGE=$1 PYTHON=python2.4 PYPREFIX=$HOME/tmp/local #REVISION="HEAD" #REVISION=2772 # for numpy #REVISION=2054 # for scipy #REVISION="{2006-07-06}" #svn update -r $REVISION ${PACKAGE} svn update ${PACKAGE} export PYTHONPATH=${PYPREFIX}/lib/${PYTHON}/site-packages:${PYTHONPATH}: # remove existing ${PACKAGE} to make sure the build doesn't pick up spurious things rm -rf $PYPREFIX/lib/${PYTHON}/site-packages/${PACKAGE} # make/install cd ${PACKAGE} rm -rf build $PYTHON setup.py install --prefix=$PYPREFIX From stevenj at alum.mit.edu Sat Sep 9 12:50:58 2006 From: stevenj at alum.mit.edu (Steven G. Johnson) Date: Sat, 09 Sep 2006 09:50:58 -0700 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: References: Message-ID: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> Andrew Jaffe wrote: > numpy returns n/2+1 complex numbers (so the first and last numbers are > actually real) with the frequencies equivalent to the positive part of > the fftfreq, whereas scipy returns n real numbers with the frequencies > as in rfftfreq (i.e., two real numbers at the same frequency, except for > the highest and lowest) [All of the above for even n; but the difference > between numpy and scipy remains for odd n.] > > I think the numpy behavior makes more sense, as it doesn't require any > unpacking after the fact, at the expense of a tiny amount of wasted > space. But would this in fact require scipy doing extra work from > whatever the 'native' real_fft (fftw, I assume) produces? As an author of FFTW, let me interject a couple of points into this discussion. First, if you are using FFTW, then its real-input r2c routines "natively" produce output in the "unpacked" numpy format as described above: an array of n/2+1 complex numbers. Any "packed" format would require some data permutations. Other FFT implementations use a variety of formats. Second, the *reason* why FFTW's r2c routines produce unpacked output is largely because "packed" formats do not generalize well to multi-dimensional FFTs, while the "unpacked" format does. (Packed formats are *possible* for multidimensional transforms, but become increasingly intricate as you add more dimensions.) Additionally, I personally find the unpacked format more convenient in most applications. (FFTW does actually support a different, "packed" format in its r2r interface, but only for 1d FFTs. The reason for this has do to with advantages of that format for odd sizes. We recommend that most users employ the r2c interface, however; our r2c interface is generally faster for even sizes.) I hope this is helpful. Cordially, Steven G. Johnson From tzdorsgmsb at esmas.com Sat Sep 9 18:11:21 2006 From: tzdorsgmsb at esmas.com (whieofvv llolkgcls) Date: Sat, 09 Sep 2006 23:11:21 +0100 Subject: [Numpy-discussion] Your PR man AETR Message-ID: <0429640572.fNGGRG-46475387-3458@esmas.com> W A T C H O U T! Here comes the big one! All signs show that AETR is going to Explode! ALLIANCE ENTERPRISE (AETR) Current Price 0.50 Add this gem to your watch list, and watch it trad closely! NEWS RELEASE TAECORP ANNOUNCES BREAKTHROUGH IN REMOVING DEADLY LANDMINES. TaeCorp Announces That NASA Technology Will Be a Key Contributor to Resolving World's Landmine Dilemma Sept. 6, 2006--The Alliance Enterprise Corporation ("TaeCorp") is pleased to announce that NASA technology will be the key component in keeping our landmine locator (LML) system operational year round, especially in inclement weather. TaeCorp and Ice Management have agreed that Ice Management will customize their de-icer system exclusively for TaeCorp's LML air vehicles that are to be deployed for landmine removal and home security MILL VALLEY, CALIFORNIA August 25, 2006 - The Alliance Enterprise Corporation announced today a BREAKTHROUGH in developing an Aerial Landmine System aimed at locating, detecting and mapping deadly landmines. More than 100 million landmines in 83 countries are holding international communities and industries hostage, preventing the investment in and development of productive lands and the re-building of infrastructure. A broad variety of landmines have been scattered over productive areas effectively crippling the economy and disabling thousands of children and adults. There are no reliable records that accurately show where these devastating landmines lie in wait for their victims. With the present day costs to clear a single land mine ranging between ,000 to ,500, solving the problem of de-mining lands will reach billions of dollars. TaeCorp has developed a technology based, cost effective solution to this problem using its three tiered approach to scanning, mapping and removing landmines. TaeCorp's System will provide many social and economic benefits to countries and their industries including oil and gas, mining, agriculture, roads and infrastructure development. About TaeCorp. TaeCorp's vision is to be the recognized leader in providing Aerial Detection Systems including global de-mining, clearing a path to a safer planet for all humankind. TaeCorp's mission is to reclaim lands around the globe embedded with landmines that victimize countries and their stakeholders. Watch AETR like a HAWK come Monday You will be convinced if you see ----------------------- What goes up must come down. Stop and smell the roses. Your barking up the wrong tree. Season of mists and mellow fruitfulness. You can't squeeze blood out of a turnip. A rolling stone gathers no moss. Through the grapevine. Say it with flowers. What goes up must come down. Plant kindness and gather love. You never miss the water till the well runs dry. When the cows come home. Water under the bridge. Tools of the trade. Two peas in a pod. Sitting on the fence. Put to bed with a shovel. We hung them out to dry. When you get lemons, make lemonade.(When life gives you scraps make quilts.) Rare as walking on water. A tree does not move unless there is wind. Slow as molasses in January. We'll cross that bridge when we come to it. Sweet as apple pie. There is always next year. Stone cold sober. Raking in the dough. Waking up with the chickens. A weed is no more than a flower in disguise. That's water under the bridge. Stop, look and listen. Spring to mind. Tastes like chicken. Tools of the trade. A snail's pace. Plant kindness and gather love. Still water runs dirty and deep. What goes down usually comes up. You have to separate the chaff from the wheat. Thick as a brick. When it rains it pours. Put off the scent. Walking on water. Take time to smell the roses. Schools out for summer. When pigs fly. From a.h.jaffe at bakerjaffe.plus.com Mon Sep 11 11:02:08 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 11 Sep 2006 16:02:08 +0100 Subject: [Numpy-discussion] rfft different in numpy vs scipy In-Reply-To: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> References: <1157820658.981272.299330@m79g2000cwm.googlegroups.com> Message-ID: <45057A70.2040709@bakerjaffe.plus.com> Steven G. Johnson wrote: > Andrew Jaffe wrote: >> numpy returns n/2+1 complex numbers (so the first and last numbers are >> actually real) with the frequencies equivalent to the positive part of >> the fftfreq, whereas scipy returns n real numbers with the frequencies >> as in rfftfreq (i.e., two real numbers at the same frequency, except for >> the highest and lowest) [All of the above for even n; but the difference >> between numpy and scipy remains for odd n.] >> >> I think the numpy behavior makes more sense, as it doesn't require any >> unpacking after the fact, at the expense of a tiny amount of wasted >> space. But would this in fact require scipy doing extra work from >> whatever the 'native' real_fft (fftw, I assume) produces? > > As an author of FFTW, let me interject a couple of points into this > discussion. > > First, if you are using FFTW, then its real-input r2c routines > "natively" produce output in the "unpacked" numpy format as described > above: an array of n/2+1 complex numbers. Any "packed" format would > require some data permutations. Other FFT implementations use a > variety of formats. > > Second, the *reason* why FFTW's r2c routines produce unpacked output is > largely because "packed" formats do not generalize well to > multi-dimensional FFTs, while the "unpacked" format does. (Packed > formats are *possible* for multidimensional transforms, but become > increasingly intricate as you add more dimensions.) Additionally, I > personally find the unpacked format more convenient in most > applications. > > I hope this is helpful. OK -- so it appears that all (three) of the votes so far are in support of the numpy convention -- a complex result. If this is something that can be done in pure python, I'm willing to give it a stab, but I'm probably not capable of handling any python/C issues. Does anyone out there understand the interaction between fftpack (C & python?), fftw (C), scipy and numpy in this context well enough to give some advice? Yours, Andrew From listservs at mac.com Mon Sep 11 11:45:08 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 11 Sep 2006 11:45:08 -0400 Subject: [Numpy-discussion] sum returns grand total? Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I must have missed something -- since when does summing a multidimensional array return the grand total? I thought that it should sum over axis zero by default (at least, thats what the docs say). Here's what I am getting: (Pdb) pre Out[6]: array([[[ 13, 3, 1, 33, 52], [ 1, 0, 0, 0, 2], [ 4, 0, 0, 2, 6], [ 15, 1, 0, 20, 54]], [[380, 101, 9, 267, 321], [ 20, 4, 0, 7, 16], [ 31, 3, 1, 31, 22], [314, 27, 4, 276, 391]]], dtype=int64) (Pdb) sum(pre) Out[6]: 2432L Is this correct behaviour? - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFBYSJGKaDpEGshJ0RAiPhAJ0d/MCGMciLgB2ZOwXf1FVj26NpTgCdG7h8 M83+mZsYGP2svVBGn8RtbNw= =mby0 -----END PGP SIGNATURE----- From oliphant at ee.byu.edu Mon Sep 11 11:47:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 11 Sep 2006 09:47:18 -0600 Subject: [Numpy-discussion] sum returns grand total? In-Reply-To: References: Message-ID: <45058506.3060403@ee.byu.edu> listservs at mac.com wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >I must have missed something -- since when does summing a >multidimensional array return the grand total? I thought that it >should sum over axis zero by default (at least, thats what the docs >say). Here's what I am getting: > >(Pdb) pre >Out[6]: >array([[[ 13, 3, 1, 33, 52], > [ 1, 0, 0, 0, 2], > [ 4, 0, 0, 2, 6], > [ 15, 1, 0, 20, 54]], > > [[380, 101, 9, 267, 321], > [ 20, 4, 0, 7, 16], > [ 31, 3, 1, 31, 22], > [314, 27, 4, 276, 391]]], dtype=int64) >(Pdb) sum(pre) >Out[6]: 2432L > >Is this correct behaviour? > > Yes, you need to specify which axis you want to sum over if you have a multi-dimensional array. This is the "basic" behavior of most functions in numpy unless you import them from oldnumeric. See the fix_default_axis.py file for help on converting code that worked for numpy 0.9.8 to 1.0 where this switch was made. -Travis From aisaac at american.edu Mon Sep 11 12:21:29 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 11 Sep 2006 12:21:29 -0400 Subject: [Numpy-discussion] sum returns grand total? In-Reply-To: <45058506.3060403@ee.byu.edu> References: <45058506.3060403@ee.byu.edu> Message-ID: On Mon, 11 Sep 2006, Travis Oliphant apparently wrote: > Yes, you need to specify which axis you want to sum over > if you have a multi-dimensional array. This is the > "basic" behavior of most functions And a good behavior indeed! Cheers, Alan Isaac From rex at nosyntax.com Mon Sep 11 16:38:33 2006 From: rex at nosyntax.com (rex) Date: Mon, 11 Sep 2006 13:38:33 -0700 Subject: [Numpy-discussion] ndarray.count() ? In-Reply-To: References: <45007E76.8030102@mspacek.mm.st> <20060907223336.GH5025@x2.nosyntax.com> <20060908001422.GJ5025@x2.nosyntax.com> Message-ID: <20060911203833.GE5064@x2.nosyntax.com> Robert Kern [2006-09-08 06:51]: > rex wrote: > > Robert Kern [2006-09-07 16:35]: > >> rex wrote: > >>> This exposed inconsistent randint() behavior between SciPy and the Python > >>> random module. The Python randint includes the upper endpoint. The SciPy > >>> version excludes it. > > > > I'm not in a position to argue the merits, but IMHO, when code that > > previously worked silently starts returning subtly bad results after > > importing numpy, there is a problem. What possible upside is there in > > having randint() behave one way in the random module and silently behave > > differently in numpy? > > I don't understand you. That's because I wasn't making any sense. :( > Importing numpy does not change the standard library's > random module in any way. There is no silent difference in behavior. If you use > numpy.random you get one set of behavior. If you use random, you get another. > Pick the one you want. They're not interchangeable, and nothing suggests that > they ought to be. Of course you're right. I thought the name would be overwritten, and it isn't. Sorry for wasting your time. :( Thanks, -rex From dgssgsd at tom.com Mon Sep 11 16:58:31 2006 From: dgssgsd at tom.com (=?GB2312?B?1cXPyMn6?=) Date: Tue, 12 Sep 2006 04:58:31 +0800 Subject: [Numpy-discussion] =?GB2312?B?tPq/qreixrE=?= Message-ID: ????????(??/??)??? ????????????????????????????????.???? ?????????????????6%?????????1-1.5%????????? ???????????????? ???????????????????????????????????? ??????????????????? ? ?????13590116835 ??????? E- MAIL ?szhw1359011 at 126.com ? ??????????????? ????????????????????????? ? ?? From michael.sorich at gmail.com Tue Sep 12 03:38:55 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue, 12 Sep 2006 17:08:55 +0930 Subject: [Numpy-discussion] creating a masked array with a sequence containing a masked scalar array Message-ID: <16761e100609120038m2aa21600qbea2c71d571cc9e9@mail.gmail.com> It seems that ma.array does not accept a sequence which contains a masked scalar array (not sure if this is the correct term). Is this deliberate? #numpy 1.0b5, python 2.3.x, winXP import numpy as N print N.ma.array([1,2,3, N.ma.array(1, mask=True)]) Traceback (most recent call last): File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_comm.py", line 529, in doIt result = pydevd_vars.evaluateExpression( self.thread_id, self.frame_id, self.expression, self.doExec ) File "C:\eclipse\plugins\org.python.pydev.debug_1.2.2\pysrc\pydevd_vars.py", line 258, in evaluateExpression exec expression in frame.f_globals, frame.f_locals File "", line 1, in ? File "C:\Python23\Lib\site-packages\numpy\core\ma.py", line 562, in __init__ c = numeric.array(data, dtype=tc, copy=True, order=order) TypeError: an integer is required #works fine if scalar masked array is not masked print N.ma.array([1,2,3, N.ma.array(1)]) [1 2 3 1] From thibault at physics.cornell.edu Tue Sep 12 09:52:03 2006 From: thibault at physics.cornell.edu (Pierre Thibault) Date: Tue, 12 Sep 2006 09:52:03 -0400 Subject: [Numpy-discussion] In-place operations Message-ID: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Hi, I would like to have information on the best techniques to do in-place calculations and to minimize temporary array creations. To me this seems to be very important whenever the arrays become very large. I already know about the ufunc in-place functionality (which is great). More specifically, here are examples that occured in my code 1) FFTs: Let A and B be two large arrays, already allocated. I want the fft of A to be stored in B. If I just type B = fft(A), there is a temprary array creation, right? Is it possible to avoid that? 2) Function output: In general, I think the same thing happens with functions like def f1(array_in): array_out = # something using array_in return array_out Then, if B is already allocated, writing B = f1(A) involves again a temporary array creation I thought instead of doing something like def f2(array_in, array_out): array_out[:] = # something # Is this good practice? and call f2(A,B). If I understand well, this still requires a temporary array creation. Is there another way of doing that (appart from actually looping through the indices of A and B)? I know that the use of f2 has one clear advantage: it makes sure that whatever was in B is discarded. With f1, the following could happen: A # contains something B # contains something C = B B = f1(A) C still contains whatever was in B. This could be what you wanted, but if you consider C just as a reference to B, this is not good. I guess these considerations are not standard python problems because you expect python to take care of memory issues. With big arrays in scientific computations, I feel the question is more relevant. I might be wrong... Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From jstrunk at enthought.com Tue Sep 12 10:26:51 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 09:26:51 -0500 Subject: [Numpy-discussion] REMINDER: ISP changeover today 9/12 7:00 PM Central Message-ID: <200609120926.51762.jstrunk@enthought.com> Good morning, This is a reminder for this evening's network maintenance. Unfortunately, our recent change in internet service providers is not working out. We will be switching to a more reliable provider tonight at 7:00 PM Central. Please allow for up to two hours of downtime. I will send an email announcing the start and completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. Please pass this message along to people that I have missed. If you have any questions, please direct them to me. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From faltet at carabos.com Tue Sep 12 10:51:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 12 Sep 2006 16:51:03 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: <1158072663.3973.21.camel@localhost.localdomain> Hello Pierre, El dt 12 de 09 del 2006 a les 09:52 -0400, en/na Pierre Thibault va escriure: > Hi, > > I would like to have information on the best techniques to do in-place > calculations and to minimize temporary array creations. To me this > seems to be very important whenever the arrays become very large. > > I already know about the ufunc in-place functionality (which is great). > > More specifically, here are examples that occured in my code > > 1) FFTs: Let A and B be two large arrays, already allocated. I want > the fft of A to be stored in B. If I just type B = fft(A), there is a > temprary array creation, right? Is it possible to avoid that? Well, in some way, there is a temporary array creation that is immediately bound to B, so in the end, the temporary is not so temporary, but a new (bounded) object. Obviously, the object that was referencing B is freed (unless there is another reference to it). > > 2) Function output: In general, I think the same thing happens with > functions like > > def f1(array_in): > array_out = # something using array_in > return array_out > > Then, if B is already allocated, writing B = f1(A) involves again a > temporary array creation Again, it depends what do you understand by 'temporary'. > > I thought instead of doing something like > > def f2(array_in, array_out): > array_out[:] = # something > # Is this good practice? > > and call f2(A,B). > > If I understand well, this still requires a temporary array creation. > Is there another way of doing that (appart from actually looping > through the indices of A and B)? Here I'd say that yes, you are creating a truly temporary object and then assign element by element to B. > > I know that the use of f2 has one clear advantage: it makes sure that > whatever was in B is discarded. With f1, the following could happen: > > A # contains something > B # contains something > C = B > B = f1(A) > > C still contains whatever was in B. This could be what you wanted, but > if you consider C just as a reference to B, this is not good. This is not good if you want to get rid of the object pointed by B, but in general, this is considered a nice feature of python. > > I guess these considerations are not standard python problems because > you expect python to take care of memory issues. With big arrays in > scientific computations, I feel the question is more relevant. I might > be wrong... If you are worried about wasting memory, just get familiar with this rule: an object only exists in memory when it is referenced (bounded) by a variable. When the object is no longer referenced, its memory becomes freed and is available to the system for later reuse. With this, B = fft(A) will create a new object (the FFT of A), the object pointed by B will be freed (if there is not any other reference to it) and the new object will be bound to B. If what you want is to avoid having in memory the three objects (namely A, old B and new B) at the same time, you can do something like: del B # deletes reference to object pointed by B B = fft(A) # B gets bounded to new FFT object HTH, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From dd55 at cornell.edu Tue Sep 12 12:26:10 2006 From: dd55 at cornell.edu (Darren Dale) Date: Tue, 12 Sep 2006 12:26:10 -0400 Subject: [Numpy-discussion] numpy.distutils compatibility with subversion-1.4 Message-ID: <200609121226.10259.dd55@cornell.edu> This morning I updated to subversion-1.4. There have been several significant enhancements in this release (see http://subversion.tigris.org/svn_1.4_releasenotes.html), and it appears that numpy.distutils is not compatible with working copies created with the new svn. I submitted a ticket (#276), but I figured it was worth a post here, just a "heads up" for anyone not watching the trac website. $ python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' No module named __svn_version__ Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/home/darren/src/numpy/numpy/distutils/core.py", line 144, in setup config = configuration() File "setup.py", line 48, in configuration config.add_subpackage('numpy') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in add_subpackage caller_level = 2) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in get_subpackage caller_level = caller_level + 1) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "./numpy/setup.py", line 8, in configuration config.add_subpackage('f2py') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 753, in add_subpackage caller_level = 2) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 736, in get_subpackage caller_level = caller_level + 1) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 683, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/f2py/setup.py", line 39, in configuration config.make_svn_version_py() File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1298, in make_svn_version_py self.add_data_files(('', generate_svn_version_py())) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 1281, in generate_svn_version_py assert revision is not None,'hmm, why I am not inside SVN tree???' AssertionError: hmm, why I am not inside SVN tree??? Darren From thibault at physics.cornell.edu Tue Sep 12 13:17:22 2006 From: thibault at physics.cornell.edu (Pierre Thibault) Date: Tue, 12 Sep 2006 13:17:22 -0400 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158072663.3973.21.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> Message-ID: <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> Hello again, On 9/12/06, Francesc Altet wrote: > Hello Pierre, > [...] > > Well, in some way, there is a temporary array creation that is > immediately bound to B, so in the end, the temporary is not so > temporary, but a new (bounded) object. Obviously, the object that was > referencing B is freed (unless there is another reference to it). ok, I guess I was aware of all that. My worries are related to two cases: 1) When the mere creation of a new array is prohibitive. 2) When what I really want to change is the _content_ of an array. Using assignment (=) disconnects B from the array it refers to, so that what used to be true (C=B) is not anymore. I understant that B[:] = ... solves the problem 2), though I don't know if this notation is recommended; but I would like to know if there is anyway to solve 1), in the way ufuncs can do. I had fun using kde's "performance monitor" (ksysguard) to see the difference between a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) a = numpy.exp(a) and a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) numpy.exp(a,out=a) Not only is the latter faster, but I could see a large glitch in the memory usage for the former. Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From peridot.faceted at gmail.com Tue Sep 12 13:19:05 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 12 Sep 2006 13:19:05 -0400 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: On 12/09/06, Pierre Thibault wrote: > I would like to have information on the best techniques to do in-place > calculations and to minimize temporary array creations. To me this > seems to be very important whenever the arrays become very large. The first rule of optimization: don't do it yet. You can usually go through and banish temporary arrays (using ufuncs and so on) at the cost of readability, code encapsulation, and thread-safe-ness. But it may not do what you want. I had an image-processing code that was taking longer than I thought it should and using two hundred megabytes or so of RAM. So I rewrote it, with a certain amount of pain, in a way that it used the fewest possible temporary arrays. It didn't run any faster, and it then took five hundred megabytes. Because all the arrays ended up being in memory at once, the memory footprint increased drastically. malloc() is fast, typically just a handful of instructions; if you're allocating a giant array, it's almost certainly being allocated using mmap(), and it can be released back to the OS on deallocation. But you probably still want to avoid temporary arrays. So: > More specifically, here are examples that occured in my code > > 1) FFTs: Let A and B be two large arrays, already allocated. I want > the fft of A to be stored in B. If I just type B = fft(A), there is a > temprary array creation, right? Is it possible to avoid that? Doing an FFT in-place is a major challenge, and involves its own slowdowns, so generally high-level toolkits don't bother. But fft seems to be like many functions (those generated by interp1d, for example) that insist on malloc()ing their own arrays to return. Short of rooting around in the numpy/scipy code, there's no real way around this for such functions. The best you can do is make actual use of the allocated array (rather than copy its contents to *another* array and discard it). > 2) Function output: In general, I think the same thing happens with > functions like > > def f1(array_in): > array_out = # something using array_in > return array_out > > Then, if B is already allocated, writing B = f1(A) involves again a > temporary array creation Uh, no, not really. The way you have written f1, it probably malloc()s space for array_out. the address of that space (roughly) is saved in the array_out variable. If you write B=f1(A), you are just storing the address in B. The memory is not copied. Even if you do B=f1(A)[::10] you don't copy the memory. > I thought instead of doing something like > > def f2(array_in, array_out): > array_out[:] = # something > # Is this good practice? > > and call f2(A,B). This is a ufunc-like solution; you could even make array_out an optional argument, and return it. > If I understand well, this still requires a temporary array creation. > Is there another way of doing that (appart from actually looping > through the indices of A and B)? It depends what #something is. If, say, it is 2*array_in, you can simply do multiply(array_in,2,array_out) to avoid any dynamic allocation. > I guess these considerations are not standard python problems because > you expect python to take care of memory issues. With big arrays in > scientific computations, I feel the question is more relevant. I might > be wrong... Some of these issues come up when dealing with mutable objects (lists, dictionaries, and so on). Some of them (the fact that python variables contain simply references) are discussed in various python FAQs. A. M. Archibald From faltet at carabos.com Tue Sep 12 13:49:28 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 12 Sep 2006 19:49:28 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> Message-ID: <1158083368.3973.34.camel@localhost.localdomain> El dt 12 de 09 del 2006 a les 13:17 -0400, en/na Pierre Thibault va escriure: > Hello again, > > On 9/12/06, Francesc Altet wrote: > > Hello Pierre, > > [...] > > > > Well, in some way, there is a temporary array creation that is > > immediately bound to B, so in the end, the temporary is not so > > temporary, but a new (bounded) object. Obviously, the object that was > > referencing B is freed (unless there is another reference to it). > > ok, I guess I was aware of all that. My worries are related to two cases: > 1) When the mere creation of a new array is prohibitive. As Archibald said in other message, creation of a big array is not an issue (malloc is very fast) and indepent of the size: >>> Timer("a=numpy.array(100,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.19819307327270508, 0.14915895462036133, 0.14999985694885254] >>> Timer("a=numpy.array(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.15171599388122559, 0.14998698234558105, 0.14901280403137207] that is 15 us (in my old machine) irregardingly of the size. [BTW, numpy.empty seems twice as slower in my machine. Why? >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.37033700942993164, 0.31780219078063965, 0.31607294082641602] ] > 2) When what I really want to change is the _content_ of an array. > Using assignment (=) disconnects B from the array it refers to, so > that what used to be true (C=B) is not anymore. > > I understant that B[:] = ... solves the problem 2), though I don't > know if this notation is recommended; but I would like to know if > there is anyway to solve 1), in the way ufuncs can do. > > I had fun using kde's "performance monitor" (ksysguard) to see the > difference between > > a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) > a = numpy.exp(a) > > and > > a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048) > numpy.exp(a,out=a) > > Not only is the latter faster, but I could see a large glitch in the > memory usage for the former. Yes, it seems that some ufuncs have an additional "out" parameter that I was not aware of. Well, it that case, this parameter in the fft function would solve your needs, although I don't know how complicated would be this. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Sep 12 15:28:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 12 Sep 2006 13:28:01 -0600 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158083368.3973.34.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> Message-ID: <45070A41.90001@ee.byu.edu> Francesc Altet wrote: >>>>Timer("a=numpy.array(100,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.19819307327270508, 0.14915895462036133, 0.14999985694885254] > > >>>>Timer("a=numpy.array(10000,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.15171599388122559, 0.14998698234558105, 0.14901280403137207] > >that is 15 us (in my old machine) irregardingly of the size. > > Ummm.. You are not creating empty arrays here. You are creating a 0-d array with a single entry. >[BTW, numpy.empty seems twice as slower in my machine. Why? > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import >>>> >>>> >numpy").repeat(3,10000) >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] >] > > Now, you are creating an empty array with 10000 elements in it. Best, -Travis From haase at msg.ucsf.edu Tue Sep 12 16:33:21 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 12 Sep 2006 13:33:21 -0700 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? Message-ID: <200609121333.21254.haase@msg.ucsf.edu> Hi ! I have a nice ndarray image viewer built on OpenGL - one annoyance though is that is crashes if the array contains any NaN, or inf, ... So, I found N.nan_to_num - but OpenGL (read: video cards) supports only single precision float (N.float32) So I get this: >>> N.nan_to_num([N.inf]) [ 1.79769313e+308] >>> N.nan_to_num([N.inf]).astype(N.float32) [ inf] Could nan_to_num() get an optional dtype argument ? Thanks, Sebastian Haase From haase at msg.ucsf.edu Tue Sep 12 16:39:54 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 12 Sep 2006 13:39:54 -0700 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? In-Reply-To: <200609121333.21254.haase@msg.ucsf.edu> References: <200609121333.21254.haase@msg.ucsf.edu> Message-ID: <200609121339.54126.haase@msg.ucsf.edu> On Tuesday 12 September 2006 13:33, Sebastian Haase wrote: > Hi ! > I have a nice ndarray image viewer built on OpenGL - one annoyance though > is that is crashes if the array contains any NaN, or inf, ... > > So, I found N.nan_to_num - but OpenGL (read: video cards) supports only > single precision float (N.float32) > > So I get this: > >>> N.nan_to_num([N.inf]) > > [ 1.79769313e+308] > > >>> N.nan_to_num([N.inf]).astype(N.float32) > > [ inf] > > Could nan_to_num() get an optional dtype argument ? > > Thanks, > Sebastian Haase OK - sorry for the noise... this is how to do it: >>> N.nan_to_num( N.array([N.nan,N.inf,-N.inf],N.float32) ) [ 0.00000000e+00 3.40282347e+38 -3.40282347e+38] In the meantime I noticed that there is a N.asarray_chkfinite() function but no equivalent for N.asanyarray() ! Should N.asanyarray_chkfinite() be added !? -Sebastian From jstrunk at enthought.com Tue Sep 12 19:30:03 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 18:30:03 -0500 Subject: [Numpy-discussion] ISP changeover beginning in 30 minutes (7:00 PM Central) Message-ID: <200609121830.03566.jstrunk@enthought.com> Good evening, We will begin switching our internet service over in about 30 minutes. Please allow for up to two hours of downtime. I will send an email announcing the completion of this maintenance. This will effect all Enthought servers as well as the SciPy server which hosts many Open Source projects. We apologize for the inconvenience. Jeff Strunk Enthought, Inc. From aisaac at american.edu Tue Sep 12 20:51:19 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 12 Sep 2006 20:51:19 -0400 Subject: [Numpy-discussion] what is nan_to_num() for float32 ? In-Reply-To: <200609121333.21254.haase@msg.ucsf.edu> References: <200609121333.21254.haase@msg.ucsf.edu> Message-ID: On Tue, 12 Sep 2006, Sebastian Haase apparently wrote: > I have a nice ndarray image viewer built on OpenGL Please post (with license). Thanks! Alan Isaac From jstrunk at enthought.com Tue Sep 12 21:11:58 2006 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 12 Sep 2006 20:11:58 -0500 Subject: [Numpy-discussion] ISP Changeover complete Message-ID: <200609122011.58960.jstrunk@enthought.com> Good evening, We have completed our internet service switchover. Thank you for your patience. If you have any trouble, please let me know. Thank you, Jeff Strunk IT Administrator Enthought, Inc. From faltet at carabos.com Wed Sep 13 03:15:49 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 13 Sep 2006 09:15:49 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <45070A41.90001@ee.byu.edu> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> Message-ID: <1158131750.3968.36.camel@localhost.localdomain> El dt 12 de 09 del 2006 a les 13:28 -0600, en/na Travis Oliphant va escriure: > >[BTW, numpy.empty seems twice as slower in my machine. Why? > > > > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > >>>> > >>>> > >numpy").repeat(3,10000) > >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] > >] > > > > > Now, you are creating an empty array with 10000 elements in it. Ups, my bad. So, here are the correct times for array creation: >>> Timer("a=numpy.empty(10,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.083303928375244141, 0.080381870269775391, 0.077172040939331055] >>> Timer("a=numpy.empty(100,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.086454868316650391, 0.084085941314697266, 0.083555936813354492] >>> Timer("a=numpy.empty(1000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.084996223449707031, 0.082299947738647461, 0.081347942352294922] >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.31068897247314453, 0.30376386642456055, 0.30176281929016113] >>> Timer("a=numpy.empty(100000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.42552995681762695, 0.36864185333251953, 0.36870002746582031] >>> Timer("a=numpy.empty(1000000,dtype=numpy.complex128)", "import numpy").repeat(3,10000) [0.48045611381530762, 0.41251182556152344, 0.40645909309387207] So, it seems that there are a certain time dependency with size array of 10 elements --> 7.7 us array of 100 elements --> 8.4 us array of 1000 elements --> 8.1 us array of 10000 elements --> 30.2 us array of 100000 elements --> 36.9 us array of 1000000 elements --> 40.6 us Well, it seems that malloc actually takes more time when asking for more space. However, this can't be the reason why Pierre is seeing that: a = numpy.exp(a) [1] is slower than numpy.exp(a,out=a) [2] as I'd say that this increment in time is negligible compared with processing times of those big arrays. In fact, here are my times: >>> Timer("a = numpy.exp(a)", "import numpy;a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)").repeat(3,1) [2.5527338981628418, 2.5427830219268799, 2.5074479579925537] >>> Timer("numpy.exp(a,out=a)", "import numpy;a = numpy.random.rand(2048,2048) + 1j * numpy.random.rand(2048,2048)").repeat(3,1) [2.5298278331756592, 2.5082788467407227, 2.5222280025482178] So, both times are comparable. Perhaps what Pierre is seeing is that he is approaching the limits of memory in his system and because [1] takes more memory than [2] (two objects in memory instead of one) perhaps the former is causing the OS to start swapping. However a quick look with top at the processes, says that both [1] and [2] takes similar amounts of memory (~ 170 MB peak) and, as arrays take 64 MB each, in both cases the used memory seems higher than the required at first sight. Mmmm, the only explanation is that the exp() ufunc does require temporaries, although this is a bit strange as exp() works element wise. I recognize that I'm a bit lost here... -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From a.u.r.e.l.i.a.n at gmx.net Wed Sep 13 04:23:50 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 13 Sep 2006 10:23:50 +0200 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> Message-ID: <200609131023.50534.a.u.r.e.l.i.a.n@gmx.net> Hi, one word in advance, instead of optimizing it is advisable to seek for a way to refactorize the algorithm using smaller arrays, since this kind of optimization almost certainly reduces readability. If you do it, comment well. ;-) If you have very large arrays and want to do some arithmetics on it, say B = 2*B + C you can use inplace operators to avoid memory overhead: B *= 2 B += C Another trick which works in most situations is to do the outermost loop in python: for i in xrange(len(B)): B[i] = 2*B[i] + C[i] This reduces the temporary array size to 1/len(B) while still being fast (if the other dimensions are large enough). For very large 1d arrays, you could split them into chunks of a certain size. However, you have to be careful that your calculation does not access already-calculated elements of B. Consider the following example: In [2]: B=arange(10) In [3]: B+B[::-1] Out[3]: array([9, 9, 9, 9, 9, 9, 9, 9, 9, 9]) In [4]: B += B[::-1] In [5]: B Out[5]: array([ 9, 9, 9, 9, 9, 14, 15, 16, 17, 18]) Johannes From zeldey at i-enternet.com Wed Sep 13 07:08:55 2006 From: zeldey at i-enternet.com (Paxton Pereda) Date: Wed, 13 Sep 2006 04:08:55 -0700 Subject: [Numpy-discussion] PHAlxcRMA Message-ID: <000001c6d725$00f3d760$790ea8c0@fnue> Hi QUIT OVE t RPA e YIN a G FOR YOU e R P o HAR z MAC j Y S u AV k E u p p to 50 w % wi v th http://www.prlawoec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From klemm at phys.ethz.ch Wed Sep 13 11:09:40 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Wed, 13 Sep 2006 17:09:40 +0200 Subject: [Numpy-discussion] numpy-1.0b5 can't find ATLAS Message-ID: Hello, when I try to build numpy-1.0b5 it doesn't find my ATLAS libraries. I had to build hem from scratch (version 3.7.17) and the compile went well. I have installed them under /scratch/python2.4/atlas furthermore, I have blas and lapack installed under /scatch/python2.4/lib I adjusted the environment variables as: BLAS=/scratch/python2.4/lib/libfblas.a LAPACK=/scratch/python2.4/lib/libflapack.a ATLAS=/scratch/python2.4/atlas and my site.cfg looks like [atlas] library_dirs = /scratch/python2.4/atlas/lib atlas_libs = lapack, blas, cblas, atlas include_dirs = /scratch/python2.4/atlas/include python setup.py config then finds the blas and lapack libraries under /scratch/python2.4/lib but does not find the atlas libraries. What am I doing wrong here? Hanno P.S.: The output of python setup.py config reads: atlas_threads_info: Setting PTATLAS=ATLAS libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include libraries lapack_atlas not found in /scratch/python2.4/atlas/include libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/lib libraries lapack_atlas not found in /scratch/python2.4/atlas/lib libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/lib libraries lapack_atlas not found in /scratch/python2.4/lib libraries lapack,blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack,blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack_atlas not found in /scratch/python2.4/atlas/include/atlas libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/include libraries lapack_atlas not found in /scratch/python2.4/atlas/include libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/atlas/lib libraries lapack_atlas not found in /scratch/python2.4/atlas/lib libraries lapack,blas,cblas,atlas not found in /scratch/python2.4/lib libraries lapack_atlas not found in /scratch/python2.4/lib libraries lapack,blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack,blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /scratch/src/numpy-1.0b5/numpy/distutils/system_info.py:1205: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/scratch/python2.4/lib'] language = f77 FOUND: libraries = ['lapack', 'fblas'] library_dirs = ['/scratch/python2.4/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running config -- Hanno Klemm klemm at phys.ethz.ch From rng7 at cornell.edu Wed Sep 13 14:32:29 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Wed, 13 Sep 2006 14:32:29 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? Message-ID: <45084EBD.5050205@cornell.edu> Hi all, I'm migrating an application from Numeric to numpy, and I've run into a significant application slowdown related to arithmetic on array-scalars. The inner loop of the application is integrating a nonlinear set of differential equations using odeint, with the rhs a dynamically-generated (only once) python function. In that function I copy the entries of the current x array to a bunch of local variables, do a bunch of arithmetic, and assign the results to a dx_dt array. The arithmetic is approximately 3x slower using numpy than Numeric, because numpy returns array-scalars while Numeric returns normal scalars. (Simple example below.) I can wrap all my arrays accesses with float() casts, but that introduces a noticable overhead (~50% for problems of interest). I'm guessing speeding up the scalar-array math would be difficult, if not impossible. (Maybe I'm wrong?) I notice that numpy_array.item() will give me the first element as a normal scalar. Would it be possible for numpy_array.item(N) to return the Nth element of the array as a normal scalar? Thanks a bunch, Ryan The effect can be isolated as (running in python 2.4 on a 32-bit Athlon): In [1]: import Numeric, numpy In [2]: a_old, a_new = Numeric.array([1.0, 2.0]), numpy.array([1.0, 2.0]) In [3]: b_old, b_new = a_old[0], a_new[0] In [4]: %time for ii in xrange(1000000):c = b_old + 1.0 CPU times: user 0.40 s, sys: 0.00 s, total: 0.40 s Wall time: 0.40 In [5]: %time for ii in xrange(1000000):c = b_new + 1.0 CPU times: user 1.20 s, sys: 0.00 s, total: 1.20 s Wall time: 1.22 In [6]: Numeric.__version__, numpy.__version__ Out[6]: ('24.2', '1.0b5') -- Ryan Gutenkunst | Cornell LASSP | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)227-7914 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From oliphant.travis at ieee.org Wed Sep 13 15:07:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 13:07:03 -0600 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <45084EBD.5050205@cornell.edu> References: <45084EBD.5050205@cornell.edu> Message-ID: <450856D7.6040601@ieee.org> Ryan Gutenkunst wrote: > Hi all, > > I'm migrating an application from Numeric to numpy, and I've run into a > significant application slowdown related to arithmetic on array-scalars. > > Yeah, that is a common thing. There are two factors: 1) array indexing and 2) array scalar math. I don't think array scalar math has to be slower in principle then Python code. I think it's ability to handle interaction with multiple scalars that is operating more slowly right now. It could be sped up. The array indexing code is slower (in fact Numeric's indexing code is slower then just using lists also). > I'm guessing speeding up the scalar-array math would be difficult, if > not impossible. (Maybe I'm wrong?) > I think scalar-array math could be sped up quite a bit. I haven't done much in that area at all. Right now a lot of setup code is handled generically instead of type-specifically like it could be. > I notice that numpy_array.item() will give me the first element as a > normal scalar. Would it be possible for numpy_array.item(N) to return > the Nth element of the array as a normal scalar? > Now this is an interesting idea. It would allow you to by-pass the slow-indexing as well as the array scalar computation should you desire it. I like it and am going to add it unless there are convincing objections. -Travis From oliphant.travis at ieee.org Wed Sep 13 16:18:28 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 14:18:28 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend Message-ID: <45086794.8020704@ieee.org> I'd like to make the first release-candidate of NumPy 1.0 this weekend. Any additions wanting to make the first official release candidate should be checked in by then. -Travis From fullung at gmail.com Wed Sep 13 18:41:34 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 14 Sep 2006 00:41:34 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> Message-ID: Hello all I just ran NumPy and SciPy through Valgrind, and everything looks clean on that the NumPy side. Some other things that could be fixed for RC1: - GCC 4.1.1 warning in ufuncobject.c: numpy/core/src/ufuncobject.c: In function ?PyUFunc_RegisterLoopForType?: numpy/core/src/ufuncobject.c:3215: warning: "cmp" may be used uninitialized in this function - Time to kill the dft package? /usr/lib/python2.4/site-packages/numpy/dft/__init__.py:2: UserWarning: The dft subpackage will be removed by 1.0 final -- it is now called fft warnings.warn("The dft subpackage will be removed by 1.0 final -- it is now called fft") - Test failure ====================================================================== ERROR: check_instance_methods (numpy.core.tests.test_defmatrix.test_matrix_return) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/numpy/core/tests/test_defmatrix.py", line 166, in check_instance_methods b = f(*args) ValueError: setitem must have at least one argument Although not strictly NumPy issues, the following crops up when you run the SciPy test suite through Valgrind: Valgrind warnings when running scipy.integrate.tests.test_integrate.test_odeint http://projects.scipy.org/scipy/scipy/ticket/246 Valgrind warnings when running Cephes tests http://projects.scipy.org/scipy/scipy/ticket/247 Memory leak in fitpack http://projects.scipy.org/scipy/scipy/ticket/248 I think I've figured out #248. #246 should be relatively easy to fix. #247 is... interesting. Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Travis Oliphant > Sent: 13 September 2006 22:18 > To: numpy-discussion > Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend > > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis From haase at msg.ucsf.edu Wed Sep 13 18:39:38 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 13 Sep 2006 15:39:38 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: <200609131539.39147.haase@msg.ucsf.edu> Hi! I would like to hear about three tickets I submitted some time ago: Ticket #230 a**2 not executed as a*a if a.dtype = int32 is this easy to fix ? Ticket #229 numpy.random.poisson(0) should return 0 I hope there is agreement that the edge-case of 0 should/could be handled without raising an exception. I submitted a patch (please test first!) any comments on this one. Ticket #188 dtype should have nicer str representation Is this one now officially dead ? " I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Wed Sep 13 20:08:43 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 18:08:43 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <200609131539.39147.haase@msg.ucsf.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> Message-ID: <45089D8B.3070602@ieee.org> Sebastian Haase wrote: > Hi! > I would like to hear about three tickets I submitted some time ago: > > Ticket #230 a**2 not executed as a*a if a.dtype = int32 > is this easy to fix ? > > Fixed. Now, all arrays with a**2 are executed as a*a (float arrays are still executed as square(a) (is this needed)? > Ticket #229 numpy.random.poisson(0) should return 0 > I hope there is agreement that the edge-case of 0 should/could be handled > without raising an exception. I submitted a patch (please test first!) > any comments on this one. > Fixed. This seems reasonable to me. > Ticket #188 dtype should have nicer str representation > Is this one now officially dead ? > " but str() should rather return 'int32 (little endian)' > It's not necessarily dead, the problem is complexity of implementation and more clarity about how all dtypes are supposed to be printed, not just this particular example. A patch would be very helpful here. If desired it could be implemented in _internal.py and called from there in arrayobject.c But, to get you thinking... How should the following be printed dtype('c4') dtype('a4,i8,3f4') dtype('3f4') -Travis From matthew.brett at gmail.com Wed Sep 13 20:09:50 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 14 Sep 2006 01:09:50 +0100 Subject: [Numpy-discussion] iscomplex on strings Message-ID: <1e2af89e0609131709v3b57a005j9056112f2021858b@mail.gmail.com> Hi, I was surprised by this - but maybe I shouldn't have been: In [7]:iscomplex('a') Out[7]:True In [8]:iscomplex(u'a') Out[8]:True Best, Matthew From matthew.brett at gmail.com Wed Sep 13 20:13:52 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 14 Sep 2006 01:13:52 +0100 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FF5C37.4090909@ieee.org> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> Message-ID: <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Hi, > For example, if you do array([a,b,c]).shape(), the answer is normally > (3,) unless a b and c happen to all be lists of the same length, at > which point your array could have a much more complicated shape... but > as the person who wrote "array([a,b,c])" it's tempting to assume that > the result has shape (3,), only to discover subtle bugs much later. Very much agree with this. > If we were writing an array-creation function from scratch, would > there be any reason to include object-array creation in the same > function as uniform array creation? It seems like a bad idea to me. > > If not, the problem is just compatibility with Numeric. Why not simply > write a wrapper function in python that does Numeric-style guesswork, > and put it in the compatibility modules? How much code will actually > break? Can I encourage any more comments? This suggestion seems very sensible to me, and I guess this is our very last chance to change this. The current behavior does seem to violate least surprise - at least to my eye. Best, Matthew From charlesr.harris at gmail.com Wed Sep 13 21:07:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:07:39 -0600 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> References: <44FB29CA.9070808@colorado.edu> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Message-ID: On 9/13/06, Matthew Brett wrote: > > Hi, > > > For example, if you do array([a,b,c]).shape(), the answer is normally > > (3,) unless a b and c happen to all be lists of the same length, at > > which point your array could have a much more complicated shape... but > > as the person who wrote "array([a,b,c])" it's tempting to assume that > > the result has shape (3,), only to discover subtle bugs much later. > > Very much agree with this. > > > If we were writing an array-creation function from scratch, would > > there be any reason to include object-array creation in the same > > function as uniform array creation? It seems like a bad idea to me. > > > > If not, the problem is just compatibility with Numeric. Why not simply > > write a wrapper function in python that does Numeric-style guesswork, > > and put it in the compatibility modules? How much code will actually > > break? > > Can I encourage any more comments? This suggestion seems very > sensible to me, and I guess this is our very last chance to change > this. The current behavior does seem to violate least surprise - at > least to my eye. I've been thinking about how to write a new constructor for objects. Because array has been at the base of numpy for many years I think it is too late to change it now, but perhaps a new and more predictable constructor for objects may eventually displace it. The main problem in constructing arrays of objects is more information needs to be supplied because the user's intention can't be reliably deduced from the current syntax. That said, I have no idea how widespread the use of object arrays is and so don't know how much it really matters. I don't use them much myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 13 21:18:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:18:39 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: On 9/13/06, Travis Oliphant wrote: > > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. There are a few cleanups and added functionality I have in mind but nothing that would affect the release. Do you plan to keep the 1.0 release as is with only added fixes and then make a 1.1 release not too long after that contains additions, or are you thinking that modifications that don't affect the API should all go into 1.0.x or some such? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 13 21:24:28 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 13 Sep 2006 19:24:28 -0600 Subject: [Numpy-discussion] In-place operations In-Reply-To: <1158131750.3968.36.camel@localhost.localdomain> References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> <1158131750.3968.36.camel@localhost.localdomain> Message-ID: On 9/13/06, Francesc Altet wrote: > > El dt 12 de 09 del 2006 a les 13:28 -0600, en/na Travis Oliphant va > escriure: > > >[BTW, numpy.empty seems twice as slower in my machine. Why? > > > > > > > > >>>>Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > > >>>> > > >>>> > > >numpy").repeat(3,10000) > > >[0.37033700942993164, 0.31780219078063965, 0.31607294082641602] > > >] > > > > > > > > Now, you are creating an empty array with 10000 elements in it. > > Ups, my bad. So, here are the correct times for array creation: > > >>> Timer("a=numpy.empty(10,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.083303928375244141, 0.080381870269775391, 0.077172040939331055] > >>> Timer("a=numpy.empty(100,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.086454868316650391, 0.084085941314697266, 0.083555936813354492] > >>> Timer("a=numpy.empty(1000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.084996223449707031, 0.082299947738647461, 0.081347942352294922] > >>> Timer("a=numpy.empty(10000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.31068897247314453, 0.30376386642456055, 0.30176281929016113] > >>> Timer("a=numpy.empty(100000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.42552995681762695, 0.36864185333251953, 0.36870002746582031] > >>> Timer("a=numpy.empty(1000000,dtype=numpy.complex128)", "import > numpy").repeat(3,10000) > [0.48045611381530762, 0.41251182556152344, 0.40645909309387207] > > So, it seems that there are a certain time dependency with size > > array of 10 elements --> 7.7 us > array of 100 elements --> 8.4 us > array of 1000 elements --> 8.1 us > array of 10000 elements --> 30.2 us > array of 100000 elements --> 36.9 us > array of 1000000 elements --> 40.6 us The transition looks a bit like a cache effect, although I don't see why the cache should enter in. But all the allocations look pretty fast to me. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Sep 13 23:25:39 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 13 Sep 2006 20:25:39 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45089D8B.3070602@ieee.org> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> Message-ID: <4508CBB3.1070104@msg.ucsf.edu> Travis Oliphant wrote: >> Ticket #188 dtype should have nicer str representation >> Is this one now officially dead ? >> "> but str() should rather return 'int32 (little endian)' >> > It's not necessarily dead, the problem is complexity of implementation > and more clarity about how all dtypes are supposed to be printed, not > just this particular example. A patch would be very helpful here. If > desired it could be implemented in _internal.py and called from there in > arrayobject.c > > But, to get you thinking... How should the following be printed > > dtype('c4') > > dtype('a4,i8,3f4') > > dtype('3f4') > > > -Travis I would argue that if the simple cases were addressed first those would cover 90% (if not 99% for most people) of the cases occurring in people's daily use. For complex types (like 'a4,i8,3f4') I actually think the current text is compact and good. (Lateron one could think about 'c4' --> '4 chars' '3f4' --> '3 float32s' but already I don't know: is there any difference between 'c4' and '4c1'? What is the difference between 'c4' and 'a4' !? ) My main focus is on the fact that you might read ' References: <072.99fbb56cad8c4060e4cd98531f18384d@scipy.net> <081.27063a6fc6f370624d8d3a3c75880a1e@scipy.net> Message-ID: <4508CDF9.4010706@msg.ucsf.edu> Travis, what is the "new string directives as the first element of the item tuple" !? I always liked the idea of having a "shortest possible" way for creating (or concatenating) rows with "r_" *and* columns with "c_" ! Why did the "c_" have to be removed !? Thanks, Sebastan NumPy wrote: > #235: r_, c_, hstack, vstack, column_stack should be made more consistent > -------------------------+-------------------------------------------------- > Reporter: baxissimo | Owner: somebody > Type: enhancement | Status: closed > Priority: normal | Milestone: > Component: numpy.lib | Version: devel > Severity: normal | Resolution: fixed > Keywords: | > -------------------------+-------------------------------------------------- > Changes (by oliphant): > > * status: new => closed > * resolution: => fixed > > Comment: > > r_ is the only current quick-creator. You can now get the functionality > of all others using string directives as the first element of the item > tuple. > > Column stack was fixed. > From oliphant.travis at ieee.org Wed Sep 13 23:40:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:40:59 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: References: <45086794.8020704@ieee.org> Message-ID: <4508CF4B.3010904@ieee.org> Charles R Harris wrote: > > > On 9/13/06, *Travis Oliphant* > wrote: > > I'd like to make the first release-candidate of NumPy 1.0 this > weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > > There are a few cleanups and added functionality I have in mind but > nothing that would affect the release. Do you plan to keep the 1.0 > release as is with only added fixes and then make a 1.1 release not > too long after that contains additions, or are you thinking that > modifications that don't affect the API should all go into 1.0.x or > some such? The plan is for 1.0.x to contain modifications that don't affect the API (good additions should be O.K.). We want extensions compiled against 1.0.x to work for a long time. The 1.1 release won't be for at least a year and probably longer. 1.0.1 would be a maintenance release of the 1.0 release. -Travis From oliphant.travis at ieee.org Wed Sep 13 23:44:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:44:08 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <4508CBB3.1070104@msg.ucsf.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> Message-ID: <4508D008.8010305@ieee.org> Sebastian Haase wrote: > Travis Oliphant wrote: > > >> It's not necessarily dead, the problem is complexity of implementation >> and more clarity about how all dtypes are supposed to be printed, not >> just this particular example. A patch would be very helpful here. If >> desired it could be implemented in _internal.py and called from there in >> arrayobject.c >> >> But, to get you thinking... How should the following be printed >> >> dtype('c4') >> >> dtype('a4,i8,3f4') >> >> dtype('3f4') >> >> >> -Travis >> > > > I would argue that if the simple cases were addressed first those would > cover 90% (if not 99% for most people) of the cases occurring in > people's daily use. > For complex types (like 'a4,i8,3f4') I actually think the current text > is compact and good. > (Lateron one could think about > 'c4' --> '4 chars' > '3f4' --> '3 float32s' > > but already I don't know: is there any difference between 'c4' and > '4c1'? What is the difference between 'c4' and 'a4' !? > ) > > > My main focus is on the fact that you might read ' "less" than 4-bytes int, which is very confusing ! > I can agree it's confusing at first, but it's the same syntax the struct module uses which is the Python precedent for this. > As far as a patch is concerned: is _internal.py already being called now > from arrayobject.c for the str() and repr() methods ? And is there so > Yes, you can easily make a call to _internal.py from arrayobject.c (it's how some things are actually implemented). If you just provide a Python function to call for dtype.__str__ then that would suffice. > far any difference in str() and repr() ? > I assume that repr() has to stay exactly the way it is right now - right !? > > Yeah, the repr() probably needs to stay the same -Travis From oliphant.travis at ieee.org Wed Sep 13 23:56:22 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 13 Sep 2006 21:56:22 -0600 Subject: [Numpy-discussion] [Numpy-tickets] [NumPy] #235: r_, c_, hstack, vstack, column_stack should be made more consistent In-Reply-To: <4508CDF9.4010706@msg.ucsf.edu> References: <072.99fbb56cad8c4060e4cd98531f18384d@scipy.net> <081.27063a6fc6f370624d8d3a3c75880a1e@scipy.net> <4508CDF9.4010706@msg.ucsf.edu> Message-ID: <4508D2E6.1020109@ieee.org> Sebastian Haase wrote: > Travis, > what is the "new string directives as the first element of the item > tuple" !? > These have been there for a while, but I recently added a couple of capabilities. > I always liked the idea of having a "shortest possible" way for creating > (or concatenating) > rows with "r_" > *and* > columns with "c_" > ! > > Why did the "c_" have to be removed !? > It wasn't removed, I thought to deprecate it. Owing to your response and the fact that others seem to use c_ quite a bit, I've kept it as a short hand for r_['1,2,0', ...] This means that arrays will be concatenated along the 1st axis after being up-graded to (at-least) 2-dimensional arrays with 1's placed at the end of the new shape. Thus, c_[[1,2,3],[4,5,6] produces array([[1, 4], [2, 5], [3, 6]]) This is a bit different if you were using c_ when you should have been using r_. -Travis From hfgi at grover5352.freeserve.co.uk Thu Sep 14 01:41:07 2006 From: hfgi at grover5352.freeserve.co.uk (Herbert Mcdermott) Date: Thu, 14 Sep 2006 08:41:07 +0300 Subject: [Numpy-discussion] learning Message-ID: <001b01c6d7c1$2292490a$a64d6055@anorwq.vfvdf> A slow strangling gurgle escaped from Eugenes screwed lips. Then, after a moments silence, he went on:Would you boys care to see the body? And Old Bens got the nicest place of all. A slow strangling gurgle escaped from Eugenes screwed lips. The air was a chill dusky pearl: above him a few palestars were out. He watcheduntil the first shovel of dirt had been thrown into the grave. Where is the can in which they throw the parts? Luke, said Harry Tugman, looking up from his paper, I wascertainly sorry to hear about Ben. He stood awkwardly, chilled, not knowinghow to continue. Bens grave had been that day freshlysodded: there was a sharp cold smell of earth there. He stood awkwardly, chilled, not knowinghow to continue. When he came to the gate of the cemetery he found it open. He was a man past fifty, with apleasant red face, brown mustaches, and a gentle placid manner. Well, then, said Horse Hines decisively, I was going to suggestto you boys that you take this one. And it will not lookso bad, if only I can save the front ones. Boys, he said, coming up to them sorrowfully,I was mighty sorry to hear of your trouble. Now, said Horse Hines quietly, I know the family doesnt wantanything cheap. The laurel, the lizard, and the stone will come no more. He put his hand affectionatelyupon a handsome casket at his side. The long barbarism of burial was at an end. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ice skate.gif Type: image/gif Size: 13290 bytes Desc: not available URL: From nwagner at iam.uni-stuttgart.de Thu Sep 14 02:47:32 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 14 Sep 2006 08:47:32 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45086794.8020704@ieee.org> References: <45086794.8020704@ieee.org> Message-ID: <4508FB04.1060709@iam.uni-stuttgart.de> Travis Oliphant wrote: > I'd like to make the first release-candidate of NumPy 1.0 this weekend. > > Any additions wanting to make the first official release candidate > should be checked in by then. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Is it possible to circumvent the error messages if one uses Python2.4 ? ImportError: ctypes is not available. Nils From strawman at astraw.com Thu Sep 14 05:11:05 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 14 Sep 2006 02:11:05 -0700 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <4508D008.8010305@ieee.org> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> Message-ID: <45091CA9.60702@astraw.com> Travis Oliphant wrote: > Sebastian Haase wrote: > >> Travis Oliphant wrote: >> >> >> >>> It's not necessarily dead, the problem is complexity of implementation >>> and more clarity about how all dtypes are supposed to be printed, not >>> just this particular example. A patch would be very helpful here. If >>> desired it could be implemented in _internal.py and called from there in >>> arrayobject.c >>> >>> But, to get you thinking... How should the following be printed >>> >>> dtype('c4') >>> >>> dtype('a4,i8,3f4') >>> >>> dtype('3f4') >>> >>> >>> -Travis >>> >>> >> I would argue that if the simple cases were addressed first those would >> cover 90% (if not 99% for most people) of the cases occurring in >> people's daily use. >> For complex types (like 'a4,i8,3f4') I actually think the current text >> is compact and good. >> (Lateron one could think about >> 'c4' --> '4 chars' >> '3f4' --> '3 float32s' >> >> but already I don't know: is there any difference between 'c4' and >> '4c1'? What is the difference between 'c4' and 'a4' !? >> ) >> >> >> My main focus is on the fact that you might read '> "less" than 4-bytes int, which is very confusing ! >> >> > I can agree it's confusing at first, but it's the same syntax the struct > module uses which is the Python precedent for this. > I'm happy with seeing the repr() value since I know what it means, but I can see Sebastian's point. Perhaps there's a middle ground -- the str() representation for simple dtypes could contain both the repr() value and an English description. For example, something along the lines of "dtype(' References: <1b1c766f0609120652n72141761p8875ce81458260ff@mail.gmail.com> <1158072663.3973.21.camel@localhost.localdomain> <1b1c766f0609121017t2acbf925x7acc7d54d92efd86@mail.gmail.com> <1158083368.3973.34.camel@localhost.localdomain> <45070A41.90001@ee.byu.edu> <1158131750.3968.36.camel@localhost.localdomain> Message-ID: <1b1c766f0609140601h294cb1beq3dfd4e83d2ff3cba@mail.gmail.com> On 9/13/06, Francesc Altet wrote: > Well, it seems that malloc actually takes more time when asking for more > space. However, this can't be the reason why Pierre is seeing that: > > a = numpy.exp(a) [1] > > is slower than > > numpy.exp(a,out=a) [2] > > as I'd say that this increment in time is negligible compared with > processing times of those big arrays. In fact, here are my times: > > >>> Timer("a = numpy.exp(a)", "import numpy;a = > numpy.random.rand(2048,2048) + 1j * > numpy.random.rand(2048,2048)").repeat(3,1) > [2.5527338981628418, 2.5427830219268799, 2.5074479579925537] > >>> Timer("numpy.exp(a,out=a)", "import numpy;a = > numpy.random.rand(2048,2048) + 1j * > numpy.random.rand(2048,2048)").repeat(3,1) > [2.5298278331756592, 2.5082788467407227, 2.5222280025482178] > > So, both times are comparable. Yeah, sorry about that: I had not checked carefully the timing. It seemed slower to me, but you're right, this is not a problem as long as there is enough free RAM. Ok, I'll go back to my coding and do like I should always do: care about optimization later. Thanks for all the comments and explanations! Pierre -- Pierre Thibault 616 Clark Hall, Cornell University (607) 255-5522 From martin.wiechert at gmx.de Thu Sep 14 10:16:26 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Thu, 14 Sep 2006 16:16:26 +0200 Subject: [Numpy-discussion] `_import_array' defined but not used Message-ID: <200609141616.26963.martin.wiechert@gmx.de> Hi gurus, I'm debugging a C-extension module that uses numpy. My version is 1.0b1. Can I safely ignore the following compiler warning? .../lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:903: warning: `_import_array' defined but not used Any help would be appreciated. Regards, Martin Wiechert From faltet at carabos.com Thu Sep 14 11:10:19 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 14 Sep 2006 17:10:19 +0200 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <45091CA9.60702@astraw.com> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> Message-ID: <1158246619.3963.5.camel@localhost.localdomain> El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va escriure: > >> My main focus is on the fact that you might read ' >> "less" than 4-bytes int, which is very confusing ! > >> > >> > > I can agree it's confusing at first, but it's the same syntax the struct > > module uses which is the Python precedent for this. > > > I'm happy with seeing the repr() value since I know what it means, but I > can see Sebastian's point. Perhaps there's a middle ground -- the str() > representation for simple dtypes could contain both the repr() value and > an English description. For example, something along the lines of > "dtype(' the repr() string could be given without any kind of English translation. +1 I was very used (and happy) to the numarray string representation for types ('Int32', 'Complex64'...) and looking at how NumPy represents it now, I'd say that this is a backwards step in readability. Something like '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From laidler at stsci.edu Thu Sep 14 11:17:53 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Thu, 14 Sep 2006 11:17:53 -0400 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <1158246619.3963.5.camel@localhost.localdomain> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> <1158246619.3963.5.camel@localhost.localdomain> Message-ID: <450972A1.5080802@stsci.edu> Francesc Altet wrote: >El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va >escriure: > > >>>>My main focus is on the fact that you might read '>>>"less" than 4-bytes int, which is very confusing ! >>>> >>>> >>>> >>>> >>>I can agree it's confusing at first, but it's the same syntax the struct >>>module uses which is the Python precedent for this. >>> >>> >>> >>I'm happy with seeing the repr() value since I know what it means, but I >>can see Sebastian's point. Perhaps there's a middle ground -- the str() >>representation for simple dtypes could contain both the repr() value and >>an English description. For example, something along the lines of >>"dtype('>the repr() string could be given without any kind of English translation. >> >> > >+1 > >I was very used (and happy) to the numarray string representation for >types ('Int32', 'Complex64'...) and looking at how NumPy represents it >now, I'd say that this is a backwards step in readability. Something >like 'high-level one like NumPy, IMO. > > I agree entirely. The first type I got ' She explained the route and the exact situation of the house. Theres plenty of time, answered the dutiful nephew coolly. Somebody turned the handle of the door and drew back the bolts. Getting down, he tried the gates, and found one was fastened by a slip catch. No blood on the floor, mused Sneed, and stared up at the open skylight. She tried to pull the bed from the wall, but it was a heavy oaken affair and beyond her strength. In a few words he gave the gist of the terrible message which had reached him. She listened at the door, her senses tense, and heard a faint, deep sobbing, then heard no more. I do not expect you to do, that, said Mr Cody hastily. My idea, the man went on, calmness itself, was to meet her halfway. Your daughters friend is a fairly rich young lady? You dare accuse your aunt of being - -Ive got a great respect for my aunt. Ive got a car down here - - Mine is faster. The two men looked at one another without a word. She has a girl friend and often dines with her - probably she will go on to a theatre afterwards. She explained the route and the exact situation of the house. You dare accuse your aunt of being - -Ive got a great respect for my aunt. The hand gripped the bottom of the door and strove to lift it. Ill be under the railway arch in Brixton Road. There came no sound but the slow, solemn ticking of a dock on a landing above. Presently Mrs Cody came out alone, and, going downstairs, unlocked the library and went in. In the hall the chauffeur was lighting a cigarette. I shall not sign any document that I havent read, she replied, and laid down the pen. You ask too many questions; hes been complainin about you. The excuse was so flimsy, she told herself. We do not support such a luxury, Mrs Cody, she said. She knew it was covered with oilcloth, and it was on this that the feet were moving. Whilst the stout man watched admiringly, he removed the whole pane and drew it out. Bare feet must have wandered aimlessly here - the footmarks are on every rug. Ive sent for the Scotland Yard photographer and the local police, he said. He was on a level ledge of roof about three feet wide. As the machine sped southward he told of Sybils disappearance. A spring lock, explained Sneed; fastens automatically when its closed. She listened, and for a long time there was nothing to break the silence. Until then he had not made any reference to the story the librarian had told him. I dont know what your dirty business is - -Mr Cody exploded in anger. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: foreseeable.gif Type: image/gif Size: 7943 bytes Desc: not available URL: From charlesr.harris at gmail.com Thu Sep 14 12:24:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 14 Sep 2006 10:24:51 -0600 Subject: [Numpy-discussion] NumPy 1.0 release-candidate 1.0 this weekend In-Reply-To: <450972A1.5080802@stsci.edu> References: <45086794.8020704@ieee.org> <200609131539.39147.haase@msg.ucsf.edu> <45089D8B.3070602@ieee.org> <4508CBB3.1070104@msg.ucsf.edu> <4508D008.8010305@ieee.org> <45091CA9.60702@astraw.com> <1158246619.3963.5.camel@localhost.localdomain> <450972A1.5080802@stsci.edu> Message-ID: On 9/14/06, Victoria G. Laidler wrote: > > Francesc Altet wrote: > > >El dj 14 de 09 del 2006 a les 02:11 -0700, en/na Andrew Straw va > >escriure: > > > > > >>>>My main focus is on the fact that you might read ' >>>>"less" than 4-bytes int, which is very confusing ! > >>>> > >>>> > >>>> > >>>> > >>>I can agree it's confusing at first, but it's the same syntax the > struct > >>>module uses which is the Python precedent for this. > >>> > >>> > >>> > >>I'm happy with seeing the repr() value since I know what it means, but I > >>can see Sebastian's point. Perhaps there's a middle ground -- the str() > >>representation for simple dtypes could contain both the repr() value and > >>an English description. For example, something along the lines of > >>"dtype(' >>the repr() string could be given without any kind of English > translation. > >> > >> > > > >+1 > > > >I was very used (and happy) to the numarray string representation for > >types ('Int32', 'Complex64'...) and looking at how NumPy represents it > >now, I'd say that this is a backwards step in readability. Something > >like ' >high-level one like NumPy, IMO. > > > > > I agree entirely. > The first type I got ' hell is that?" > It looked disturbingly like line-noise corrupted text to me! (Blast from > the past...) > > +1 from me as well. Just to balance the voting, I think things are fine as they are. As Travis says, the '<' is already used in the Python structure module and the rest doesn't take much time getting used to. However, the docstring for the dtype class is a bit lacking. It shouldn't be too much work to fix that up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Sep 14 12:29:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 14 Sep 2006 09:29:02 -0700 Subject: [Numpy-discussion] Problem with concatenate and object arrays In-Reply-To: References: <44FB29CA.9070808@colorado.edu> <44FFC231.4060507@ieee.org> <45009A05.5020007@ee.byu.edu> <1e2af89e0609131713l562b51d8ge89aed4fc0c6942f@mail.gmail.com> Message-ID: <4509834E.4000007@noaa.gov> Charles R Harris wrote: >> > Why not simply >> > write a wrapper function in python that does Numeric-style guesswork, >> > and put it in the compatibility modules? >> Can I encourage any more comments? +1 > The main problem in constructing arrays > of objects is more information needs to be supplied because the user's > intention can't be reliably deduced from the current syntax. I wrote about this a bit early in this conversation, and as I thought about it. I'm not sure it's possible _- you could specify a rank, or a shape, but in general, there wouldn't be a unique way to translate an given hierarchy of sequences into a particular shape: imagine four levels of nested lists, asked to turn into a rank-3 array. This is why it may be best to simply recommend that people create an empty array of the shape they need, then put the objects into it - it's the only way to construct what you need reliably. However, an object array constructor that take a rank as an argument might well work for most cases, as long as there is a clearly documented and consistent way to handle extra levels of sequences: perhaps specify that any extra levels of nesting always go to the last dimension (or the first). That being said, it's still dangerous -- what levels of nesting are allowed would depend on which sequences *happen* to be the same size. Also the code would be a pain to write! I wonder how often people need to use objects arrays when they don't know when writing the code what shape they need? this is making me think that maybe all we really need is a little syntactic sugar for creating empty object arrays: numpy.ObjectArray(shape) Not much different than: numpy.empty(shape, dtype=numpy.object) but a little cleaner an more obvious to new users that are primarily interested in object arrays -- analogous to ones() and zeros() > That said, I > have no idea how widespread the use of object arrays is and so don't know > how much it really matters. If we ever get nd-arrays into the standard lib (or want to see wider use of them in any case), I think that object arrays are critical. Right now, people think they don't have a use for numpy if they aren't doing serious number crunching -- it's seen mostly as a way to speed up computations on lots of numbers. However, I think nd-arrays have LOTS of other applications, for anything where the data fits well in to a "rectangular" data structure. n-d slicing is a wonderful thing! As numpy gets wider use -- object arrays will be a very big draw. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rng7 at cornell.edu Thu Sep 14 18:15:15 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Thu, 14 Sep 2006 18:15:15 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <450856D7.6040601@ieee.org> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> Message-ID: <4509D473.7020905@cornell.edu> Hi Travis, Travis Oliphant wrote: > Ryan Gutenkunst wrote: >> I notice that numpy_array.item() will give me the first element as a >> normal scalar. Would it be possible for numpy_array.item(N) to return >> the Nth element of the array as a normal scalar? >> > Now this is an interesting idea. It would allow you to by-pass the > slow-indexing as well as the array scalar computation should you desire > it. I like it and am going to add it unless there are convincing > objections. > > -Travis Thanks for the quick response, but either there's a bug, or I'm using things wrong. It appears to work to work in the N-D case, but not 1-D. I looked at the change you made, but my grasp of the C-API is too weak to isolate the problem. >>> import numpy >>> numpy.__version__ '1.0rc1.dev3154' >>> a = numpy.array([[1.0, 2.0], [3.0, 4.0]]) >>> a.item(1,1) 4.0 >>> a = numpy.array([1.0, 2.0]) >>> a.item(0) 1.0 >>> a.item(1) 1.7765824089018436e-307 >>> a.item((1,)) 1.7765824089018436e-307 Thanks for your help, Ryan -- Ryan Gutenkunst | Cornell LASSP | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)227-7914 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From oliphant at ee.byu.edu Thu Sep 14 19:40:56 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 17:40:56 -0600 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <4509D473.7020905@cornell.edu> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> <4509D473.7020905@cornell.edu> Message-ID: <4509E888.2050800@ee.byu.edu> Ryan Gutenkunst wrote: >Hi Travis, > >Travis Oliphant wrote: > > >>Ryan Gutenkunst wrote: >> >> >>>I notice that numpy_array.item() will give me the first element as a >>>normal scalar. Would it be possible for numpy_array.item(N) to return >>>the Nth element of the array as a normal scalar? >>> >>> >>> >>Now this is an interesting idea. It would allow you to by-pass the >>slow-indexing as well as the array scalar computation should you desire >>it. I like it and am going to add it unless there are convincing >>objections. >> >>-Travis >> >> > >Thanks for the quick response, but either there's a bug, or I'm using >things wrong. It appears to work to work in the N-D case, but not 1-D. I >looked at the change you made, but my grasp of the C-API is too weak to >isolate the problem. > > It's a silly bug. Please check out the latest SVN. -Travis From oliphant at ee.byu.edu Thu Sep 14 19:58:17 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 17:58:17 -0600 Subject: [Numpy-discussion] Experience with Visit? Message-ID: <4509EC99.2060002@ee.byu.edu> Has anybody had any experience with the 3-D visualization software VISIT? It has Python bindings and seems to be pretty sophisticated. I'm wondering why I haven't heard more about it. http://www.llnl.gov/visit/ -Travis From brendansimons at yahoo.ca Thu Sep 14 20:20:28 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Thu, 14 Sep 2006 20:20:28 -0400 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: Message-ID: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Hi all, Just wondering if there was an arbitrary axis iterator in numpy, or if not, if there's demand for one. What I'm looking for is something which would allow me to do something like (vectorFunc(column) for column in array.axisIter(1) ) without a bunch of for loops and slicing. Thoughts? Brendan __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From oliphant at ee.byu.edu Thu Sep 14 20:31:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 14 Sep 2006 18:31:43 -0600 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: <4509F46F.6090000@ee.byu.edu> Brendan Simons wrote: >Hi all, > >Just wondering if there was an arbitrary axis iterator in numpy, or >if not, if there's demand for one. What I'm looking for is something >which would allow me to do something like (vectorFunc(column) for >column in array.axisIter(1) ) without a bunch of for loops and slicing. > > Hmm... I can't think of something directly in Python, but it would be pretty easy to add. You could probably also do something clever by creating your own ufunc with frompyfunc and object arrays. In C, this would be easy. In the C-API there is a function PyArray_IterAllButAxis which provides an iterator that iterates over all axes but one. Then, you would call the vectorFunc for each element in the loop. The ufuncs use this functionality to call the underlying 1-d loops. -Travis From wbaxter at gmail.com Thu Sep 14 20:53:00 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 15 Sep 2006 09:53:00 +0900 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: Iteration over axis 0 is built-in, so you can already do (vectorFunc(row) for row in array) And you can use transpose() to make it so the axis you want to iterate over is axis 0. (vectorFunc(col) for col in array.transpose(1,0)) Or just use the .T attribute (vectorFunc(col) for col in array.T) So it seems kind of a toss-up whether it's worth adding a specific API to do that. The implementation would probably just return the transpose with the given axis in the zero slot. Something like: def axisiter(arr, i): ax = [i] + range(arr.ndim) del ax[i+1] return arr.transpose(ax) --bb On 9/15/06, Brendan Simons wrote: > Hi all, > > Just wondering if there was an arbitrary axis iterator in numpy, or > if not, if there's demand for one. What I'm looking for is something > which would allow me to do something like (vectorFunc(column) for > column in array.axisIter(1) ) without a bunch of for loops and slicing. > > Thoughts? > Brendan From rng7 at cornell.edu Thu Sep 14 20:59:59 2006 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Thu, 14 Sep 2006 20:59:59 -0400 Subject: [Numpy-discussion] Avoiding array-scalar arithmetic? In-Reply-To: <4509E888.2050800@ee.byu.edu> References: <45084EBD.5050205@cornell.edu> <450856D7.6040601@ieee.org> <4509D473.7020905@cornell.edu> <4509E888.2050800@ee.byu.edu> Message-ID: Travis, Thanks for the quick response. My application is back up to its old speed. Thanks also for spearheading the numpy/scipy projects. It's certainly made my work much, much more productive. Cheers, Ryan On Sep 14, 2006, at 7:40 PM, Travis Oliphant wrote: > Ryan Gutenkunst wrote: >> Thanks for the quick response, but either there's a bug, or I'm using >> things wrong. It appears to work to work in the N-D case, but not >> 1-D. I >> looked at the change you made, but my grasp of the C-API is too weak >> to >> isolate the problem. >> >> > It's a silly bug. Please check out the latest SVN. > > -Travis -- Ryan Gutenkunst | Cornell Dept. of Physics | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)255-6068 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From tim.hochberg at ieee.org Thu Sep 14 21:20:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 14 Sep 2006 18:20:08 -0700 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> Message-ID: <4509FFC8.5000309@ieee.org> Bill Baxter wrote: > Iteration over axis 0 is built-in, so you can already do > (vectorFunc(row) for row in array) > And you can use transpose() to make it so the axis you want to iterate > over is axis 0. > (vectorFunc(col) for col in array.transpose(1,0)) > Or just use the .T attribute > (vectorFunc(col) for col in array.T) > > So it seems kind of a toss-up whether it's worth adding a specific API > to do that. The implementation would probably just return the > transpose with the given axis in the zero slot. Something like: > > def axisiter(arr, i): > ax = [i] + range(arr.ndim) > del ax[i+1] > return arr.transpose(ax) > > --bb > Isn't swapaxis appropriate for this? In other words: for x in arr.swapaxis(0, i): #... should be more or less the same as: for x in axister(arr, i): #.... and it already exists. I'm in a hurry, so I haven't checked the details here, but the basic idea seems sound. -tim > On 9/15/06, Brendan Simons wrote: > >> Hi all, >> >> Just wondering if there was an arbitrary axis iterator in numpy, or >> if not, if there's demand for one. What I'm looking for is something >> which would allow me to do something like (vectorFunc(column) for >> column in array.axisIter(1) ) without a bunch of for loops and slicing. >> >> Thoughts? >> Brendan >> > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From haase at msg.ucsf.edu Thu Sep 14 21:20:53 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 14 Sep 2006 18:20:53 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? Message-ID: <200609141820.53325.haase@msg.ucsf.edu> Hi, what I'm asking is if numpy has an equivalent to numarray's info() function: >>> na.arange(10).info() class: shape: (10,) strides: (4,) byteoffset: 0 bytestride: 4 itemsize: 4 aligned: 1 contiguous: 1 buffer: data pointer: 0x085b7ec8 (DEBUG ONLY) byteorder: 'little' byteswap: 0 type: Int32 This was always helpful to me when debugging C binding code. Especially I'm asking if there is any way to get the memory address of an array - for debugging purposes only - of course ;-) Thanks, Sebastian Haase From wbaxter at gmail.com Thu Sep 14 22:05:13 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 15 Sep 2006 11:05:13 +0900 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: <4509FFC8.5000309@ieee.org> References: <940902B0-B4D2-4749-A68C-53C806004934@yahoo.ca> <4509FFC8.5000309@ieee.org> Message-ID: On 9/15/06, Tim Hochberg wrote: > Isn't swapaxis appropriate for this? In other words: > You're right. Just didn't think of that. Never used swapaxes before. def axisiter(arr, i): return arr.swapaxes(0,i) --bb From brendansimons at yahoo.ca Thu Sep 14 23:27:59 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Thu, 14 Sep 2006 23:27:59 -0400 Subject: [Numpy-discussion] Axis Iterator? In-Reply-To: References: Message-ID: <9E3412CA-464A-409D-8AAE-A3CE8D28AA2F@yahoo.ca> Oh that's cool. For some reason I thought that the built in iterator (for i in array) iterated over cells, not the first axis. I also didn't think about swapaxes. Is there any desire to add a convenience function or method as follows? def axisIter(selfOrArr, i): return iter(selfOrArr.swapAxes(0,i)) Thanks for everyone who helped out. I've got something that works now. Cheers. Brendan On 14-Sep-06, at 10:05 PM, numpy-discussion- request at lists.sourceforge.net wrote: > Date: Fri, 15 Sep 2006 11:05:13 +0900 > From: "Bill Baxter" > Subject: Re: [Numpy-discussion] Axis Iterator? > To: "Discussion of Numerical Python" > > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 9/15/06, Tim Hochberg wrote: >> Isn't swapaxis appropriate for this? In other words: >> > > You're right. Just didn't think of that. Never used swapaxes before. > > def axisiter(arr, i): > return arr.swapaxes(0,i) > > --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Sep 15 02:10:29 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 08:10:29 +0200 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: <1158300630.3964.9.camel@localhost.localdomain> El dj 14 de 09 del 2006 a les 18:20 -0700, en/na Sebastian Haase va escriure: > Especially I'm asking if there is any way to get the memory address of an > array - for debugging purposes only - of course ;-) For this, you can print the data buffer: In [1]:import numpy In [2]:a=numpy.array([1]) In [3]:a.data Out[3]: although I'm not sure which number is the memory address I'd say it's the last one. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From fullung at gmail.com Fri Sep 15 05:51:31 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 11:51:31 +0200 Subject: [Numpy-discussion] how to get info about internals of an arrayobject ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase > Sent: 15 September 2006 03:21 > To: numpy-discussion > Subject: [Numpy-discussion] how to get info about internals of an > arrayobject ? > > Hi, > what I'm asking is if numpy has an equivalent to numarray's info() > function: > > This was always helpful to me when debugging C binding code. > > Especially I'm asking if there is any way to get the memory address of an > array - for debugging purposes only - of course ;-) numpy.array([]).__array_interface__['data'][0] Cheers, Albert From lroubeyrie at limair.asso.fr Fri Sep 15 08:49:51 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Fri, 15 Sep 2006 14:49:51 +0200 Subject: [Numpy-discussion] recarray Message-ID: <200609151449.51529.lroubeyrie@limair.asso.fr> Hi all, I try to use recarray with rec.fromrecords on time-series, datas come from a file where they are stored in csv format, with after each data colum there is one column meanning the state of the data, and the first column is for dates. Then, is it possible to directly transform column of strings to a integer one (or datetime one), and to remove a not used column? thanks -- Lionel From satyaupadhya at yahoo.co.in Fri Sep 15 09:09:25 2006 From: satyaupadhya at yahoo.co.in (Satya Upadhya) Date: Fri, 15 Sep 2006 14:09:25 +0100 (BST) Subject: [Numpy-discussion] degree matrix construction Message-ID: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Dear Friends, my question is the following: Suppose i have the following code: >>> from LinearAlgebra import * >>> from Numeric import * >>> A = [1,2,1,3,1,3,4,1,2] >>> B = reshape(A,(3,3)) >>> C = sum(B,1) >>> C array([4, 7, 7]) >>> Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix with diagonal elements 4,7,7 (obtained from the elements of C) and all off-diagonal elements equal to 0. Could some kind soul kindly tell me how to do this. I've looked at the help for the diagonal function and i am unable to do what i wish to. Furthermore i dont understand the meaning of axis1 and axis2: >>> help (diagonal) Help on function diagonal in module Numeric: diagonal(a, offset=0, axis1=0, axis2=1) diagonal(a, offset=0, axis1=0, axis2=1) returns all offset diagonals defined by the given dimensions of the array. >>> Thanking you, Satya --------------------------------- Find out what India is talking about on - Yahoo! Answers India Send FREE SMS to your friend's mobile from Yahoo! Messenger Version 8. Get it NOW -------------- next part -------------- An HTML attachment was scrubbed... URL: From joris at ster.kuleuven.be Fri Sep 15 09:18:55 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Fri, 15 Sep 2006 15:18:55 +0200 Subject: [Numpy-discussion] degree matrix construction In-Reply-To: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> References: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Message-ID: <200609151518.55585.joris@ster.kuleuven.be> Hi, [SU]: Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix with diagonal elements 4,7,7 You might have a look at the Numpy Example List, at the function diag(). Ciao, Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From joris at ster.kuleuven.be Fri Sep 15 09:22:11 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Fri, 15 Sep 2006 15:22:11 +0200 Subject: [Numpy-discussion] degree matrix construction Message-ID: <200609151522.11577.joris@ster.kuleuven.be> Forgot the link to the NEL: http://www.scipy.org/Numpy_Example_List J. Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From charlesr.harris at gmail.com Fri Sep 15 09:40:45 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 15 Sep 2006 07:40:45 -0600 Subject: [Numpy-discussion] degree matrix construction In-Reply-To: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> References: <20060915130925.81735.qmail@web8505.mail.in.yahoo.com> Message-ID: On 9/15/06, Satya Upadhya wrote: > > Dear Friends, > my question is the following: > > Suppose i have the following code: > > >>> from LinearAlgebra import * > > >>> from Numeric import * > >>> A = [1,2,1,3,1,3,4,1,2] > >>> B = reshape(A,(3,3)) > >>> C = sum(B,1) > >>> C > array([4, 7, 7]) > >>> > > Now, my problem is to construct a degree matrix D which is a 3 * 3 matrix > with diagonal elements 4,7,7 (obtained from the elements of C) and all > off-diagonal elements equal to 0. > Is this what you want to do? In [2]: a = array([4, 7, 7]) In [3]: diagflat(a) Out[3]: array([[4, 0, 0], [0, 7, 0], [0, 0, 7]]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 15 09:57:28 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Sep 2006 08:57:28 -0500 Subject: [Numpy-discussion] recarray In-Reply-To: <200609151449.51529.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> Message-ID: Lionel Roubeyrie wrote: > Hi all, > I try to use recarray with rec.fromrecords on time-series, datas come from a > file where they are stored in csv format, with after each data colum there is > one column meanning the state of the data, and the first column is for dates. > Then, is it possible to directly transform column of strings to a integer one > (or datetime one), and to remove a not used column? When I import CSV files into record arrays, I usually read in all of the data and transpose the list of rows to get a list of columns. Then I can remove columns and transform them _en masse_, usually with map(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Fri Sep 15 10:05:55 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 16:05:55 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: References: <200609151449.51529.lroubeyrie@limair.asso.fr> Message-ID: <200609151605.56207.faltet@carabos.com> A Divendres 15 Setembre 2006 15:57, Robert Kern va escriure: > Lionel Roubeyrie wrote: > > Hi all, > > I try to use recarray with rec.fromrecords on time-series, datas come > > from a file where they are stored in csv format, with after each data > > colum there is one column meanning the state of the data, and the first > > column is for dates. Then, is it possible to directly transform column of > > strings to a integer one (or datetime one), and to remove a not used > > column? > > When I import CSV files into record arrays, I usually read in all of the > data and transpose the list of rows to get a list of columns. Then I can > remove columns and transform them _en masse_, usually with map(). Another possibility is to play with columns directly from the initial recarray. The next is an example: In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) In [102]: ra Out[102]: recarray([('1111', 825307441, 2.5784852031307537e-09), ('1111', 825307441, 2.5784852031307537e-09), ('1111', 825307441, 2.5784852031307537e-09)], dtype=[('f0', '|S4'), ('f1', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Fri Sep 15 10:13:41 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:13:41 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes Message-ID: <200609151613.42137.martin.wiechert@gmx.de> Hi list, I'm using PyArray_DescrConverter with a dict object to create a "struct-like" dtype from C. As the struct contains different data types I run into "unaligned access" problems. Is there a way to force alignment or to get trailing unused bytes in the dtpye? Thanks, Martin From martin.wiechert at gmx.de Fri Sep 15 10:15:19 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:15:19 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <200609151615.19950.martin.wiechert@gmx.de> On Friday 15 September 2006 16:13, Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a > "struct-like" dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. on IA64 I should have mentioned > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > > Thanks, Martin > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cimrman3 at ntc.zcu.cz Fri Sep 15 10:28:30 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 15 Sep 2006 16:28:30 +0200 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: <4509EC99.2060002@ee.byu.edu> References: <4509EC99.2060002@ee.byu.edu> Message-ID: <450AB88E.7040802@ntc.zcu.cz> Travis Oliphant wrote: > Has anybody had any experience with the 3-D visualization software > VISIT? It has Python bindings and seems to be pretty sophisticated. > I'm wondering why I haven't heard more about it. > > http://www.llnl.gov/visit/ No reaction up to now, so... I have just tried the 'getting started' part and was quite impressed, thanks for posting the link! Up to now I have used ParaView and was very satisfied, but the Python bindings of VisIt are a great lure. r. From faltet at carabos.com Fri Sep 15 10:40:39 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 15 Sep 2006 16:40:39 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <200609151640.40220.faltet@carabos.com> A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a > "struct-like" dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? One possible solution is to declare void ('V' charcode) types for filling the gaps. For example: In [118]: ra=numpy.rec.array("1"*300, dtype=[('sval','0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Fri Sep 15 10:56:31 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 16:56:31 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151640.40220.faltet@carabos.com> References: <200609151613.42137.martin.wiechert@gmx.de> <200609151640.40220.faltet@carabos.com> Message-ID: <200609151656.31753.martin.wiechert@gmx.de> On Friday 15 September 2006 16:40, Francesc Altet wrote: > A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure: > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a > > "struct-like" dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > One possible solution is to declare void ('V' charcode) types for filling > the gaps. For example: > > > In [118]: ra=numpy.rec.array("1"*300, > dtype=[('sval',' > In [119]: ra['sval'] > Out[119]: > recarray([1111, 1111], > dtype='|S4') > > In [120]: ra['dval'] > Out[120]: array([ 9.73041595e-72, 9.73041595e-72]) > Hm, not exactly elegant but should work. Thanks Francesc! Note that all but trailing gaps can be controlled using a dictionary descriptor with an offsets key: >>> ra = zeros ((10,), dtype = {'names': ['sval', 'dval'], 'formats': ['>> ra.dtype dtype([('sval', '|S4'), ('', '|V4'), ('dval', ' You can still access the empty spaces if you want (although it is > nonsense): > > In [121]: ra['unused'] > Out[121]: > recarray([1111, 1111], > dtype='|V4') > > Cheers, From fullung at gmail.com Fri Sep 15 11:53:11 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 17:53:11 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: Hello all In [1]: import numpy as N In [3]: N.dtype({'names' : ['x', 'y'], 'formats' : [N.intc, N.float64]}, align=True) Out[3]: dtype([('x', ' String Form: Namespace: Interactive File: c:\python24\lib\site-packages\numpy\__init__.py Docstring: Cheers, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > Sent: 15 September 2006 16:14 > To: numpy-discussion > Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / > trailingunused bytes > > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a "struct- > like" > dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > > Thanks, Martin From martin.wiechert at gmx.de Fri Sep 15 11:56:37 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 17:56:37 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: References: Message-ID: <200609151756.37869.martin.wiechert@gmx.de> On Friday 15 September 2006 17:53, Albert Strasheim wrote: > Hello all > > In [1]: import numpy as N > > In [3]: N.dtype({'names' : ['x', 'y'], > 'formats' : [N.intc, N.float64]}, > align=True) > Out[3]: dtype([('x', ' > The reason you might not have discovered this: > > In [2]: N.dtype? > Type: type > Base Class: > String Form: > Namespace: Interactive > File: c:\python24\lib\site-packages\numpy\__init__.py > Docstring: > > Thanks Albert! Do you also know the corresponding C-API function? It cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no "align", right? > Cheers, > > Albert > > > -----Original Message----- > > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > > Sent: 15 September 2006 16:14 > > To: numpy-discussion > > Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / > > trailingunused bytes > > > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a "struct- > > like" > > dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > > > Thanks, Martin > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Michael_OKeefe at nrel.gov Fri Sep 15 12:23:24 2006 From: Michael_OKeefe at nrel.gov (O'Keefe, Michael) Date: Fri, 15 Sep 2006 10:23:24 -0600 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: <450AB88E.7040802@ntc.zcu.cz> Message-ID: I haven't tried VisIT before but thanks for the link. I also downloaded and am checking it out. Along this same line of discussion, has anyone tried OOF2 which is an FEA package that also has some strong python connections? http://www.ctcms.nist.gov/oof/oof2.html I'm working on a Windows machine and it the current code-base doesn't seem to support Windows out of the box (if at all). Looks like you can put it together for *nix or Mac OS X, though... --Michael > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net > [mailto:numpy-discussion-bounces at lists.sourceforge.net] On > Behalf Of Robert Cimrman > Sent: Friday, September 15, 2006 8:29 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] Experience with Visit? > > Travis Oliphant wrote: > > Has anybody had any experience with the 3-D visualization software > > VISIT? It has Python bindings and seems to be pretty > sophisticated. > > I'm wondering why I haven't heard more about it. > > > > http://www.llnl.gov/visit/ > > No reaction up to now, so... > > I have just tried the 'getting started' part and was quite impressed, > thanks for posting the link! Up to now I have used ParaView > and was very > satisfied, but the Python bindings of VisIt are a great lure. > > r. > > -------------------------------------------------------------- > ----------- > Using Tomcat but need to do more? Need to support web > services, security? > Get stuff done quickly with pre-integrated technology to make > your job easier > Download IBM WebSphere Application Server v.1.0.1 based on > Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057& > dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Fri Sep 15 12:44:23 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 10:44:23 -0600 Subject: [Numpy-discussion] how to get info about internals of an arrayobject ? In-Reply-To: References: Message-ID: <450AD867.1020301@ieee.org> Albert Strasheim wrote: >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase >> Sent: 15 September 2006 03:21 >> To: numpy-discussion >> Subject: [Numpy-discussion] how to get info about internals of an >> arrayobject ? >> >> Hi, >> what I'm asking is if numpy has an equivalent to numarray's info() >> function: >> > > >> >> > > >> This was always helpful to me when debugging C binding code. >> >> Especially I'm asking if there is any way to get the memory address of an >> array - for debugging purposes only - of course ;-) >> > > numpy.array([]).__array_interface__['data'][0] > Or numpy.ctypes._as_parameter_ (should work even if you don't have ctypes installed) -Travis From emsellem at obs.univ-lyon1.fr Fri Sep 15 12:49:09 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Fri, 15 Sep 2006 18:49:09 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ? Message-ID: <450AD985.3050005@obs.univ-lyon1.fr> Hi, I am facing a rather frustrating problem with numpy/scipy: after upgrading to svn numpy and scipy, and trying to remove most of the now unnecessary casting (floats) the program I wrote does not give me the right answer. It seems that the answer is somewhat scaled down (but not in a simple way). I had this kind of behaviour some time ago when scipy and numpy were not fully compatible and then I had to cast the floats explicitely in my program to make it work. But now it SHOULD be fully compatible as far as I understand the issue. So my question is: is there any remaining flaws in terms of casts/format between scipy and numpy? I am specifically using the scipy functions: - special.erf, special.erfc, orthogonal.ps_roots, integrate.quad... and the numpy functions : sum, sqrt, exp, sin, cos, arctan. I am doing all the calculations using numpy.float32. Did anybody witness weird behaviour recently in this context? (I don't really know what to do now since I already spent hours trying to find the problem: the point is that the final answer is wrong for sure, but the intermediate calculations are not easily checked since I do not know what to really expect there. The program was working before I touched the casting + upgraded numpy/scipy) thanks for any help here, and cheers. Eric From oliphant.travis at ieee.org Fri Sep 15 13:00:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 11:00:14 -0600 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609141820.53325.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> Message-ID: <450ADC1E.2060004@ieee.org> Sebastian Haase wrote: > Hi, > what I'm asking is if numpy has an equivalent to numarray's info() function: > >>>> na.arange(10).info() >>>> numpy.numarray.info(numpy.arange(10)) (get recent SVN as there were some bugs just fixed. -Travis From oliphant.travis at ieee.org Fri Sep 15 13:14:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 11:14:15 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151756.37869.martin.wiechert@gmx.de> References: <200609151756.37869.martin.wiechert@gmx.de> Message-ID: <450ADF67.5000709@ieee.org> Martin Wiechert wrote: > Thanks Albert! Do you also know the corresponding C-API function? It cannot be > PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no > "align", right? > The DescrConverter function is meant for "O&"-style conversions. It can't accept an align function. We could possibly add something to the converter to allow specification of alignment through the object to be converted. Or, you can just call the __new__ method of the PyArrayDescr_Type object res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", "Oi", dict_object, 1)) or call the tp->new method directly: args = Py_BuildValue("Oi", dict_object, 1); PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); Py_DECREF(args); (I think passing in NULL for the keywords is O.K., but I haven't checked it). -Travis From martin.wiechert at gmx.de Fri Sep 15 13:53:14 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 19:53:14 +0200 Subject: [Numpy-discussion] =?iso-8859-1?q?PyArray=5FDescrConverter_-_alig?= =?iso-8859-1?q?nment_/=09trailingunused_bytes?= In-Reply-To: <450ADF67.5000709@ieee.org> References: <200609151756.37869.martin.wiechert@gmx.de> <450ADF67.5000709@ieee.org> Message-ID: <200609151953.14405.martin.wiechert@gmx.de> On Friday 15 September 2006 19:14, Travis Oliphant wrote: > Martin Wiechert wrote: > > Thanks Albert! Do you also know the corresponding C-API function? It > > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > > signature has no "align", right? > > The DescrConverter function is meant for "O&"-style conversions. It > can't accept an align function. We could possibly add something to the > converter to allow specification of alignment through the object to be > converted. > I begin to see the light.... For dictionaries one could maybe just add an optional key "align". Also an optional key "elsize" or "itemsize" to force the total size of the record may sometimes be useful. E.g. one may want to faithfully map a given C struct. (That's why I was asking for trailing unused bytes.) But I understand that other things have higher priority. > Or, you can just call the __new__ method of the PyArrayDescr_Type object > > res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", > "Oi", dict_object, 1)) > > or call the tp->new method directly: > > args = Py_BuildValue("Oi", dict_object, 1); > PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); > Py_DECREF(args); > Thank you! I'll try this. > (I think passing in NULL for the keywords is O.K., but I haven't checked > it). > > -Travis > One final question. To me the repr of a dtype with gaps is a little bit puzzling: >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Fri Sep 15 14:05:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 12:05:50 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <200609151613.42137.martin.wiechert@gmx.de> References: <200609151613.42137.martin.wiechert@gmx.de> Message-ID: <450AEB7E.3050009@ieee.org> Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a "struct-like" > dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > Besides calling the tp_new function (or the method __new__), I added PyArray_DescrAlignConverter to the C-API. It is at the end and so does not require re-compilation of extension modules (unless you want to use it...) -Travis From martin.wiechert at gmx.de Fri Sep 15 14:03:54 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 20:03:54 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailing unused bytes In-Reply-To: <450AEB7E.3050009@ieee.org> References: <200609151613.42137.martin.wiechert@gmx.de> <450AEB7E.3050009@ieee.org> Message-ID: <200609152003.54715.martin.wiechert@gmx.de> On Friday 15 September 2006 20:05, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a > > "struct-like" dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > Besides calling the tp_new function (or the method __new__), I added > PyArray_DescrAlignConverter to the C-API. > > It is at the end and so does not require re-compilation of extension > modules (unless you want to use it...) > Thanks Travis. I appreciate your work. > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From fullung at gmail.com Fri Sep 15 14:13:29 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 20:13:29 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151953.14405.martin.wiechert@gmx.de> Message-ID: Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Martin Wiechert > Sent: 15 September 2006 19:53 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion]PyArray_DescrConverter - alignment / > trailingunused bytes > > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > > Martin Wiechert wrote: > > > Thanks Albert! Do you also know the corresponding C-API function? It > > > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > > > signature has no "align", right? > > > > The DescrConverter function is meant for "O&"-style conversions. It > > can't accept an align function. We could possibly add something to the > > converter to allow specification of alignment through the object to be > > converted. > > > > One final question. To me the repr of a dtype with gaps is a little bit > puzzling: > > >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' 'offsets': [0, 16, 24]}) > dtype([('a', '|S4'), ('', '|V12'), ('b', ' ' > There should be no gap between "b" and "c" but still the repr has ('', > '|V12') > between them. Am I missing something? For performance reasons, compilers will typically align integers (and probably floats) on 4-byte boundaries and apparently, doubles on 16-byte boundaries. Because compilers align like this, so does NumPy. This allows you to: 1. Take any kind of C struct definition 2. Convert it to a dtype 3. Create a NumPy array with this dtype 4. Pass the array's data pointer to C code 5. Cast the data pointer to a pointer to your C struct 6. Operate on the pointer to struct as if it were allocated in C Cheers, Albert From fullung at gmail.com Fri Sep 15 14:16:59 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 15 Sep 2006 20:16:59 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment/ trailingunused bytes In-Reply-To: Message-ID: Argh > > > > > One final question. To me the repr of a dtype with gaps is a little bit > > puzzling: > > > > >>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > 'offsets': [0, 16, 24]}) > > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > ' > > > There should be no gap between "b" and "c" but still the repr has ('', > > '|V12') > > between them. Am I missing something? I see you're not specifying align=True here. Ignore my last message. Cheers, Albert From oliphant.travis at ieee.org Fri Sep 15 14:27:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 15 Sep 2006 12:27:00 -0600 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <200609151953.14405.martin.wiechert@gmx.de> References: <200609151756.37869.martin.wiechert@gmx.de> <450ADF67.5000709@ieee.org> <200609151953.14405.martin.wiechert@gmx.de> Message-ID: <450AF074.1090806@ieee.org> Martin Wiechert wrote: > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > >> Martin Wiechert wrote: >> >>> Thanks Albert! Do you also know the corresponding C-API function? It >>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose >>> signature has no "align", right? >>> >> The DescrConverter function is meant for "O&"-style conversions. It >> can't accept an align function. We could possibly add something to the >> converter to allow specification of alignment through the object to be >> converted. >> >> > > I begin to see the light.... > > For dictionaries one could maybe just add an optional key "align". > Also an optional key "elsize" or "itemsize" to force the total size of the > record may sometimes be useful. E.g. one may want to faithfully map a given C > struct. (That's why I was asking for trailing unused bytes.) > > But I understand that other things have higher priority. > > >> Or, you can just call the __new__ method of the PyArrayDescr_Type object >> >> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", >> "Oi", dict_object, 1)) >> >> or call the tp->new method directly: >> >> args = Py_BuildValue("Oi", dict_object, 1); >> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); >> Py_DECREF(args); >> >> > > Thank you! I'll try this. > > >> (I think passing in NULL for the keywords is O.K., but I haven't checked >> it). >> >> -Travis >> >> > > One final question. To me the repr of a dtype with gaps is a little bit > puzzling: > > >>>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['>>> > 'offsets': [0, 16, 24]}) > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > There should be no gap between "b" and "c" but still the repr has ('', '|V12') > between them. Am I missing something? > There was a bug I just fixed in the representation of these structures with gaps. It should be fixed in SVN, now. -Travis From martin.wiechert at gmx.de Fri Sep 15 14:22:44 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 15 Sep 2006 20:22:44 +0200 Subject: [Numpy-discussion] PyArray_DescrConverter - alignment / trailingunused bytes In-Reply-To: <450AF074.1090806@ieee.org> References: <200609151953.14405.martin.wiechert@gmx.de> <450AF074.1090806@ieee.org> Message-ID: <200609152022.45061.martin.wiechert@gmx.de> On Friday 15 September 2006 20:27, Travis Oliphant wrote: > Martin Wiechert wrote: > > On Friday 15 September 2006 19:14, Travis Oliphant wrote: > >> Martin Wiechert wrote: > >>> Thanks Albert! Do you also know the corresponding C-API function? It > >>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose > >>> signature has no "align", right? > >> > >> The DescrConverter function is meant for "O&"-style conversions. It > >> can't accept an align function. We could possibly add something to the > >> converter to allow specification of alignment through the object to be > >> converted. > > > > I begin to see the light.... > > > > For dictionaries one could maybe just add an optional key "align". > > Also an optional key "elsize" or "itemsize" to force the total size of > > the record may sometimes be useful. E.g. one may want to faithfully map a > > given C struct. (That's why I was asking for trailing unused bytes.) > > > > But I understand that other things have higher priority. > > > >> Or, you can just call the __new__ method of the PyArrayDescr_Type object > >> > >> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__", > >> "Oi", dict_object, 1)) > >> > >> or call the tp->new method directly: > >> > >> args = Py_BuildValue("Oi", dict_object, 1); > >> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL); > >> Py_DECREF(args); > > > > Thank you! I'll try this. > > > >> (I think passing in NULL for the keywords is O.K., but I haven't checked > >> it). > >> > >> -Travis > > > > One final question. To me the repr of a dtype with gaps is a little bit > > > > puzzling: > >>>> dtype ({'names': ['a', 'b', 'c'], 'formats': [' > > > 'offsets': [0, 16, 24]}) > > dtype([('a', '|S4'), ('', '|V12'), ('b', ' > ' > > > There should be no gap between "b" and "c" but still the repr has ('', > > '|V12') between them. Am I missing something? > > There was a bug I just fixed in the representation of these structures > with gaps. It should be fixed in SVN, now. > > > -Travis > > Thanks again. > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Fri Sep 15 15:07:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Sep 2006 14:07:25 -0500 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ? In-Reply-To: <450AD985.3050005@obs.univ-lyon1.fr> References: <450AD985.3050005@obs.univ-lyon1.fr> Message-ID: Eric Emsellem wrote: > Hi, > > I am facing a rather frustrating problem with numpy/scipy: after > upgrading to svn numpy and scipy, and trying to remove most of the now > unnecessary casting (floats) the program I wrote does not give me the > right answer. It seems that the answer is somewhat scaled down (but not > in a simple way). I had this kind of behaviour some time ago when scipy > and numpy were not fully compatible and then I had to cast the floats > explicitely in my program to make it work. But now it SHOULD be fully > compatible as far as I understand the issue. So my question is: > is there any remaining flaws in terms of casts/format between scipy and > numpy? I am specifically using the scipy functions: > - special.erf, special.erfc, orthogonal.ps_roots, integrate.quad... > and the numpy functions : sum, sqrt, exp, sin, cos, arctan. > I am doing all the calculations using numpy.float32. Could you give us a short piece of code that demonstrates the problem? You don't state what flaws you were working around before. Note that some of those functions in scipy are unavoidably implemented in double precision at the C/FORTRAN level. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hetland at tamu.edu Fri Sep 15 15:15:55 2006 From: hetland at tamu.edu (Rob Hetland) Date: Fri, 15 Sep 2006 14:15:55 -0500 Subject: [Numpy-discussion] Experience with Visit? In-Reply-To: References: Message-ID: I have looked at it, and I even know somebody who is using python to put in a data reader for FVCOM (a finite volume ocean model). He has a prototype working. I have no experience with the python bindings personally. I looked into compiling it from source (for an intel mac), but it was daunting so I gave up. I also contacted them to try to put in lower level python bindings, so that we could create 3D graphics from the command line, but they didn't seem too interested in that. Otherwise, it is a very nice workable app. -Rob On Sep 15, 2006, at 11:23 AM, O'Keefe, Michael wrote: > I haven't tried VisIT before but thanks for the link. I also > downloaded and am checking it out. > > Along this same line of discussion, has anyone tried OOF2 which is > an FEA package that also has some strong python connections? > > http://www.ctcms.nist.gov/oof/oof2.html > > I'm working on a Windows machine and it the current code-base > doesn't seem to support Windows out of the box (if at all). Looks > like you can put it together for *nix or Mac OS X, though... > > --Michael > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net >> [mailto:numpy-discussion-bounces at lists.sourceforge.net] On >> Behalf Of Robert Cimrman >> Sent: Friday, September 15, 2006 8:29 >> To: Discussion of Numerical Python >> Subject: Re: [Numpy-discussion] Experience with Visit? >> >> Travis Oliphant wrote: >>> Has anybody had any experience with the 3-D visualization software >>> VISIT? It has Python bindings and seems to be pretty >> sophisticated. >>> I'm wondering why I haven't heard more about it. >>> >>> http://www.llnl.gov/visit/ >> >> No reaction up to now, so... >> >> I have just tried the 'getting started' part and was quite impressed, >> thanks for posting the link! Up to now I have used ParaView >> and was very >> satisfied, but the Python bindings of VisIt are a great lure. >> >> r. >> >> -------------------------------------------------------------- >> ----------- >> Using Tomcat but need to do more? Need to support web >> services, security? >> Get stuff done quickly with pre-integrated technology to make >> your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on >> Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057& >> dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > > ---------------------------------------------------------------------- > --- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion ---- Rob Hetland, Associate Professor Dept. of Oceanography, Texas A&M University http://pong.tamu.edu/~rob phone: 979-458-0096, fax: 979-845-6331 From haase at msg.ucsf.edu Fri Sep 15 15:13:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 15 Sep 2006 12:13:43 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <450ADC1E.2060004@ieee.org> References: <200609141820.53325.haase@msg.ucsf.edu> <450ADC1E.2060004@ieee.org> Message-ID: <200609151213.43655.haase@msg.ucsf.edu> On Friday 15 September 2006 10:00, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > > > what I'm asking is if numpy has an equivalent to numarray's info() function: > >>>> na.arange(10).info() > > numpy.numarray.info(numpy.arange(10)) > > (get recent SVN as there were some bugs just fixed. > > -Travis Thanks, should this maybe also be added somewhere in the numpy module itself !? I guess the question is, what the original intent was for numarray to put it in (only for debugging ?) -- then we could decide if this would also make sense for numpy. I have never used numpy.info(obj) - I don't know what is does (compared to __doc__) but maybe it could be extended like if isinstance( obj , N.ndarray): print N.numarray.info( obj ) right now it prints nothings (and returns None) Thanks, Sebastian Haase From oliphant at ee.byu.edu Fri Sep 15 15:49:56 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 15 Sep 2006 13:49:56 -0600 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <200609151213.43655.haase@msg.ucsf.edu> References: <200609141820.53325.haase@msg.ucsf.edu> <450ADC1E.2060004@ieee.org> <200609151213.43655.haase@msg.ucsf.edu> Message-ID: <450B03E4.6080101@ee.byu.edu> Sebastian Haase wrote: >On Friday 15 September 2006 10:00, Travis Oliphant wrote: > > >>Sebastian Haase wrote: >> >> >>>Hi, >>> >>>what I'm asking is if numpy has an equivalent to numarray's info() >>> >>> >function: > > >>>>>>na.arange(10).info() >>>>>> >>>>>> >>numpy.numarray.info(numpy.arange(10)) >> >>(get recent SVN as there were some bugs just fixed. >> >>-Travis >> >> > >Thanks, > >should this maybe also be added somewhere in the numpy module itself !? >I guess the question is, what the original intent was for numarray to put it >in (only for debugging ?) -- then we could decide if this would also make >sense for numpy. > >I have never used numpy.info(obj) - I don't know what is does (compared to >__doc__) > It prints the doc string of an object searching for it better than help seems to do and without a "pager". Compare numpy.info(numpy.sin) versus help(numpy.sin) I don't know what the info method was for other than debugging. What about having the __doc__ attribute of an ndarray return the info? -Travis From haase at msg.ucsf.edu Fri Sep 15 16:49:41 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 15 Sep 2006 13:49:41 -0700 Subject: [Numpy-discussion] how to get info about internals of an array object ? In-Reply-To: <450B03E4.6080101@ee.byu.edu> References: <200609141820.53325.haase@msg.ucsf.edu> <200609151213.43655.haase@msg.ucsf.edu> <450B03E4.6080101@ee.byu.edu> Message-ID: <200609151349.41809.haase@msg.ucsf.edu> On Friday 15 September 2006 12:49, Travis Oliphant wrote: > Sebastian Haase wrote: > >On Friday 15 September 2006 10:00, Travis Oliphant wrote: > >>Sebastian Haase wrote: > >>>Hi, > >>> > >>>what I'm asking is if numpy has an equivalent to numarray's info() > > > >function: > >>>>>>na.arange(10).info() > >> > >>numpy.numarray.info(numpy.arange(10)) > >> > >>(get recent SVN as there were some bugs just fixed. > >> > >>-Travis > > > >Thanks, > > > >should this maybe also be added somewhere in the numpy module itself !? > >I guess the question is, what the original intent was for numarray to put > > it in (only for debugging ?) -- then we could decide if this would also > > make sense for numpy. > > > >I have never used numpy.info(obj) - I don't know what is does (compared to > >__doc__) > > It prints the doc string of an object searching for it better than help > seems to do and without a "pager". > > Compare numpy.info(numpy.sin) versus help(numpy.sin) In PyShell I would miss the output of numpy.info(numpy.sin) since it goes to the (C) stdout and not to the (sometimes redirected ) python sys.stdout. So it seems it prints N.sin.__doc__ The output from help(...) goes to the (python) sys.stdout (or stderr !?) and is quite long (and mostly just generec info about ufuncs ...) > > I don't know what the info method was for other than debugging. > > What about having the __doc__ attribute of an ndarray return the info? > Maybe, but honestly I would *not* expect any kind of "live inspection" to be done by __doc__ I would expect more like "ndarray is an efficient way to handle large data sets in python - it is the bases for Numerical Python, see www.scipy.org " Maybe info() should just be an array attribute - just like it was it numarray. -Sebastian From yukihana at yahoo.co.jp Sat Sep 16 08:15:41 2006 From: yukihana at yahoo.co.jp (=?iso-2022-jp?B?eXVraWhhbmE=?=) Date: Sat, 16 Sep 2006 12:15:41 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?02 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?02 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ?????????????????????????????????? ???????????? ??????????3-6-4-533 ?????? 139-3668-7892 From philluewan at alliedvaughn.com Sat Sep 16 09:18:38 2006 From: philluewan at alliedvaughn.com (Sib Ridenhour) Date: Sat, 16 Sep 2006 06:18:38 -0700 Subject: [Numpy-discussion] PHxoeARMA Message-ID: <01c6d992$e87da9c0$0700000a@vieja> Hi QUIT OVE k RPA y YIN r G FOR Y c OUR PH l AR g MAC n Y S w AV i E up t h o 50 z % wit p h http://www.criaboedri.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From irbdavid at gmail.com Sat Sep 16 11:46:28 2006 From: irbdavid at gmail.com (David Andrews) Date: Sat, 16 Sep 2006 17:46:28 +0200 Subject: [Numpy-discussion] 1.0b5 Installation on OS X Message-ID: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> Hi, I'm unable to install the 1.0b5 release from the .mpkg on OS X - when it comes to the two checkboxes to select the platlib and the scripts, both are greyed out. The installer then claims 'Cannot continue: Nothing to install'. Any suggestions? Cheers, Dave From Carmen at tal.is Sat Sep 16 13:39:48 2006 From: Carmen at tal.is (Casandra) Date: Sat, 16 Sep 2006 23:39:48 +0600 Subject: [Numpy-discussion] A Job offer Message-ID: Dear Friend, Taken into account the expansion of a field of activity of our firm, which is located in Europe and engaged in modeling and fashion industry in the world markets. We require Project Manager. The job is of a partial employment, which means that you will not spend a lot of energy and time on it. The salary depends on the price of projects executed by you (about 30000-50000 dollars a year). If our offer has interested you and you are a serious and hardworking person, who is not afraid to open a new field of activity for yourself, you are probably the person that we are looking for. Also, we will be glad to see you in our company. To suit this job you have to be at least 21 years of age or older and be a US or Australian citizen. Tahnk you. galina_frolova at gawab.com From martin.wiechert at gmx.de Sat Sep 16 14:49:32 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sat, 16 Sep 2006 20:49:32 +0200 Subject: [Numpy-discussion] concatenation changes dtype Message-ID: <200609162049.33244.martin.wiechert@gmx.de> Hi list, Apparently r_[x,x] does not necessarily have the same dtype as x: >>> from numpy import * >>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>> x = zeros ((1,), dt) >>> x.dtype dtype([('a', '|i1'), ('f4', '>> r_[x,x].dtype dtype('object') >>> >>> import numpy >>> numpy.version.version '1.0rc1.dev3171' Does anybody know how to avoid this change of dtype? Thanks, Martin. From oliphant.travis at ieee.org Sat Sep 16 18:29:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 16 Sep 2006 16:29:01 -0600 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <200609162049.33244.martin.wiechert@gmx.de> References: <200609162049.33244.martin.wiechert@gmx.de> Message-ID: <450C7AAD.6030705@ieee.org> Martin Wiechert wrote: > Hi list, > > Apparently r_[x,x] does not necessarily have the same dtype as x: > > >>>> from numpy import * >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>>> x = zeros ((1,), dt) >>>> x.dtype >>>> > dtype([('a', '|i1'), ('f4', ' >>>> r_[x,x].dtype >>>> > dtype('object') > >>>> import numpy >>>> numpy.version.version >>>> > '1.0rc1.dev3171' > > Does anybody know how to avoid this change of dtype? > Thanks for the check. It should be fixed now in SVN. There was an inappropriate force to object arrays which neglected this case. -Travis From oliphant.travis at ieee.org Sat Sep 16 18:36:39 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 16 Sep 2006 16:36:39 -0600 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <200609162049.33244.martin.wiechert@gmx.de> References: <200609162049.33244.martin.wiechert@gmx.de> Message-ID: <450C7C77.4050006@ieee.org> Martin Wiechert wrote: > Hi list, > > Apparently r_[x,x] does not necessarily have the same dtype as x: > > >>>> from numpy import * >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) >>>> x = zeros ((1,), dt) >>>> x.dtype >>>> > dtype([('a', '|i1'), ('f4', ' Did you mean to make a data-type that was a 1-byte integer followed by a 4-byte integer with field names of 'a' and 'f4'? Perhaps you meant: dtype([('a','f4'),('b','i4')]) which is a 4-byte float followed by a 4-byte integer. -Travis From martin.wiechert at gmx.de Sat Sep 16 20:15:23 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 17 Sep 2006 02:15:23 +0200 Subject: [Numpy-discussion] concatenation changes dtype In-Reply-To: <450C7C77.4050006@ieee.org> References: <200609162049.33244.martin.wiechert@gmx.de> <450C7C77.4050006@ieee.org> Message-ID: <200609170215.23435.martin.wiechert@gmx.de> On Sunday 17 September 2006 00:36, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > Apparently r_[x,x] does not necessarily have the same dtype as x: > >>>> from numpy import * > >>>> dt = dtype ([('a', 'b'), ('f4', 'i4')]) > >>>> x = zeros ((1,), dt) > >>>> x.dtype > > > > dtype([('a', '|i1'), ('f4', ' > Did you mean to make a data-type that was a 1-byte integer followed by a > 4-byte integer with field names of 'a' and 'f4'? > > Perhaps you meant: > > dtype([('a','f4'),('b','i4')]) > > which is a 4-byte float followed by a 4-byte integer. > Thanks for the hint. Actually, the code I posted was just for the sake of example. Still you correctly read my thoughts. Martin. > > -Travis > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From kaapoudugg at centerpartners.com Sun Sep 17 09:19:38 2006 From: kaapoudugg at centerpartners.com (Aksel Booth) Date: Sun, 17 Sep 2006 06:19:38 -0700 Subject: [Numpy-discussion] PHAxqiRMA Message-ID: <01c6da5b$f1c90500$0a01a8c0@renia> Hi d V x I d A d G o R m A s V w A f L p I p U l M l C q I n A t L e I z S e A x M g B u I b E o N k X d A w N w A q X S y AV o E up to 6 t 0 % w e it c h http://www.ylutriamoa.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Sun Sep 17 14:17:09 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 17 Sep 2006 20:17:09 +0200 Subject: [Numpy-discussion] record descriptors: trailing unused bytes Message-ID: <200609172017.09548.martin.wiechert@gmx.de> Hi list, does anybody know, if in C in the PyArray_Descr struct it is safe to manually change descr->elsize and descr->alignment as long as these are compatible and descr->elsize is large enough to hold all fields? Of course I mean before any array is constructed using descr. Thanks, Martin. From Achim.Gaedke at physik.tu-darmstadt.de Sun Sep 17 16:00:07 2006 From: Achim.Gaedke at physik.tu-darmstadt.de (Achim Gaedke) Date: Sun, 17 Sep 2006 22:00:07 +0200 Subject: [Numpy-discussion] compile error with python2.5 Message-ID: <450DA947.4020809@physik.tu-darmstadt.de> Hi! I compiled numpy-1.0b5 with python2.5 on debian testing and raIn file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsIntp': numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use in this function) numpy/core/src/arrayobject.c:564: error: (Each undeclared identifier is reported only once numpy/core/src/arrayobject.c:564: error: for each function it appears in.) numpy/core/src/arrayobject.c:565: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsInt': numpy/core/src/arrayobject.c:654: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'array_subscript': numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsIntp': numpy/core/src/arrayobject.c:564: error: 'op' undeclared (first use in this function) numpy/core/src/arrayobject.c:564: error: (Each undeclared identifier is reported only once numpy/core/src/arrayobject.c:564: error: for each function it appears in.) numpy/core/src/arrayobject.c:565: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'PyArray_PyIntAsInt': numpy/core/src/arrayobject.c:654: warning: cast from pointer to integer of different size numpy/core/src/arrayobject.c: In function 'array_subscript': numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union numpy/core/src/arrayobject.c:2694: error: request for member 'ob_type' in something not a structure or union error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -Ibuild/src.linux-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/home/achim/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-i686-2.5/numpy/core/src/multiarraymodule.o" failed with exit status 1 n into errors I fixed it by guessing the correct arguments. The patch is attached. I hope someone can revise this fix. Achim -------------- next part -------------- diff -ru numpy-1.0b5/numpy/core/src/arrayobject.c numpy-1.0b5_fixed/numpy/core/src/arrayobject.c --- numpy-1.0b5/numpy/core/src/arrayobject.c 2006-09-05 00:41:47.000000000 +0200 +++ numpy-1.0b5_fixed/numpy/core/src/arrayobject.c 2006-09-17 21:41:49.000000000 +0200 @@ -561,8 +561,8 @@ return ret; } #if (PY_VERSION_HEX >= 0x02050000) - if (PyIndex_Check(op)) { - long_value = (longlong) PyNumber_Index(op); + if (PyIndex_Check(o)) { + long_value = (longlong) PyNumber_Index(o); goto finish; } #endif @@ -2691,7 +2691,7 @@ } if (PyInt_Check(op) || PyArray_IsScalar(op, Integer) || - PyLong_Check(op) || PyIndex_Check(index)) { + PyLong_Check(op) || PyIndex_Check(op)) { intp value; value = PyArray_PyIntAsIntp(op); if (!PyErr_Occurred()) { From fullung at gmail.com Sun Sep 17 16:29:25 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sun, 17 Sep 2006 22:29:25 +0200 Subject: [Numpy-discussion] compile error with python2.5 In-Reply-To: <450DA947.4020809@physik.tu-darmstadt.de> Message-ID: Hello all These problems were fixed in SVN just after 1.0rc1 was released. http://projects.scipy.org/scipy/numpy/changeset/3119 Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Achim Gaedke > Sent: 17 September 2006 22:00 > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] compile error with python2.5 > > Hi! > > I compiled numpy-1.0b5 with python2.5 on debian testing and raIn file > included from numpy/core/src/multiarraymodule.c:65: From syzb at garonpr.demon.co.uk Sun Sep 17 18:20:24 2006 From: syzb at garonpr.demon.co.uk (Louisa Mata) Date: Mon, 18 Sep 2006 06:20:24 +0800 Subject: [Numpy-discussion] mournful Message-ID: <000801c6daa8$c6738cb1$3160a2da@qxfi.ncpdr> This may have been regularan then again it may have been queer. Bilyen made a careful inspection of the spot, and then faced Brazos witha curious fire in his eyes. I raisedthe ante to two thousand dollars. Give me the lowdown on rustlin in eastern Colorado. Suddenly she sprang out from the wall, formidable as a tigress. Without any greeting, Brazos flung a query at the banker. Yu-all shorehave heahed of the Sewall McCoy combine with Russ Slaughter. Shes the most fascinating girl I ever met. But he could be made a target for speech thatwould sweep over town like fire in prairie grass. At the foot of the Odd Fellows stairway Brazos halted to load his gun. Father hasa deal on with Surface and Miller. I mixed with cowboys,cattlemen, gamblers, an town folks. Cross the street heah an walk up thetside an down on this side. Yu men held upNeece thet night an robbed him. Wal, this gang of three was after himfor reasons that bear strong in this deal. An lettin that gambler Howard runaround with her. Range talk blames me for alot thet Im innocent of. She met his piercing gaze with understanding, and avisible shudder. If they would be, yud never get a vote,onless from some of yore hired hands. No noticeable change showed in the ranchers pale face. Dealing Surface a powerful left-handed blow,Brazos knocked him flat. Lura designated her bag, which Brazostook up. Whats to keep mefrom shootin Raine Surfaces laig off? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ambiguity.gif Type: image/gif Size: 6995 bytes Desc: not available URL: From lroubeyrie at limair.asso.fr Mon Sep 18 03:38:34 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 09:38:34 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609151605.56207.faltet@carabos.com> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609151605.56207.faltet@carabos.com> Message-ID: <200609180938.34721.lroubeyrie@limair.asso.fr> Le vendredi 15 septembre 2006 16:05, Francesc Altet a ?crit?: > Another possibility is to play with columns directly from the initial > recarray. The next is an example: > > In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) > In [102]: ra > Out[102]: > recarray([('1111', 825307441, 2.5784852031307537e-09), > ('1111', 825307441, 2.5784852031307537e-09), > ('1111', 825307441, 2.5784852031307537e-09)], > dtype=[('f0', '|S4'), ('f1', ' In [103]: rb=numpy.rec.fromarrays([numpy.array(ra['f0'], 'i4'),ra['f2']], > names='f0,f1') > In [104]: rb > Out[104]: > recarray([(1111, 2.5784852031307537e-09), (1111, 2.5784852031307537e-09), > (1111, 2.5784852031307537e-09)], > dtype=[('f0', ' > where ra is the original recarray and rb is a derived one where its first > column is the original from ra, but converted to integers ('i4'), and the > second it's the third column from ra (so the second column from ra has been > stripped out from rb). I have a problem with that : lionel[ETD-2006-01__PM2.5_DALTON]334>datas[0:5] Sortie[334]: [['Dates ', 'PM10 ', 'c10', 'PM2.5 ', 'c2.5'], ['05/01/2006', '33', 'A', '', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A'], ['08/01/2006', '16', 'A', '13', 'A']] lionel[ETD-2006-01__PM2.5_DALTON] 335>ra=rec.array(datas[1:],formats='a10,i2,a1,i2,a1') lionel[ETD-2006-01__PM2.5_DALTON]336>ra[0:5] Sortie[336]: recarray([[('05/01/2006', 0, '', 0, ''), ('33', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('30', 0, '', 0, ''), ('N\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('06/01/2006', 0, '', 0, ''), ('41', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('30', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('07/01/2006', 0, '', 0, ''), ('20', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('16', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('08/01/2006', 0, '', 0, ''), ('16', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('13', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], [('09/01/2006', 0, '', 0, ''), ('18', 0, '', 0, ''), ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), ('15', 0, '', 0, ''), ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')]], dtype=[('f1', '|S10'), ('f2', ' BUG in numpy.sum? Message-ID: <450E6D2E.9010706@obs.univ-lyon1.fr> Hi again after some hours of debugging I finally (I think) found the problem: numpy.sum([[0,1,2],[2,3,4]]) 24 numpy.sum([[0,1,2],[2,3,4]],axis=0) array([2, 4, 6]) numpy.sum([[0,1,2],[2,3,4]],axis=1) array([3, 9]) Isn't the first line supposed to act as with "axis=0" by default (see help numpy.sum!)...??? Not setting axis=0 it sums everything! Is that the expected behaviour or is the doc not updated? (I realise it may be better to force axis=.. to indicate what you need but still.. the example in numpy.sum help gives: sum([[0, 1], [0, 5]]) array([0, 6]) which is not what it gives when you do it...) thanks Eric From faltet at carabos.com Mon Sep 18 06:17:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 18 Sep 2006 12:17:03 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609180938.34721.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609151605.56207.faltet@carabos.com> <200609180938.34721.lroubeyrie@limair.asso.fr> Message-ID: <1158574623.4218.6.camel@localhost.localdomain> El dl 18 de 09 del 2006 a les 09:38 +0200, en/na Lionel Roubeyrie va escriure: > Le vendredi 15 septembre 2006 16:05, Francesc Altet a ?crit : > > Another possibility is to play with columns directly from the initial > > recarray. The next is an example: > > > > In [101]: ra=numpy.rec.array("1"*36, dtype="a4,i4,f4", shape=3) > > In [102]: ra > > Out[102]: > > recarray([('1111', 825307441, 2.5784852031307537e-09), > > ('1111', 825307441, 2.5784852031307537e-09), > > ('1111', 825307441, 2.5784852031307537e-09)], > > dtype=[('f0', '|S4'), ('f1', ' > In [103]: rb=numpy.rec.fromarrays([numpy.array(ra['f0'], 'i4'),ra['f2']], > > names='f0,f1') > > In [104]: rb > > Out[104]: > > recarray([(1111, 2.5784852031307537e-09), (1111, 2.5784852031307537e-09), > > (1111, 2.5784852031307537e-09)], > > dtype=[('f0', ' > > > where ra is the original recarray and rb is a derived one where its first > > column is the original from ra, but converted to integers ('i4'), and the > > second it's the third column from ra (so the second column from ra has been > > stripped out from rb). > > I have a problem with that : > lionel[ETD-2006-01__PM2.5_DALTON]334>datas[0:5] > Sortie[334]: > [['Dates ', 'PM10 ', 'c10', 'PM2.5 ', 'c2.5'], > ['05/01/2006', '33', 'A', '', 'N'], > ['06/01/2006', '41', 'A', '30', 'A'], > ['07/01/2006', '20', 'A', '16', 'A'], > ['08/01/2006', '16', 'A', '13', 'A']] > > lionel[ETD-2006-01__PM2.5_DALTON] > 335>ra=rec.array(datas[1:],formats='a10,i2,a1,i2,a1') > > lionel[ETD-2006-01__PM2.5_DALTON]336>ra[0:5] > Sortie[336]: > recarray([[('05/01/2006', 0, '', 0, ''), ('33', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('30', 0, '', 0, ''), > ('N\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('06/01/2006', 0, '', 0, ''), ('41', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('30', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('07/01/2006', 0, '', 0, ''), ('20', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('16', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('08/01/2006', 0, '', 0, ''), ('16', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('13', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')], > [('09/01/2006', 0, '', 0, ''), ('18', 0, '', 0, ''), > ('A[9\xb4q\x00\x00\x00\xc0\xa3', -18448, '\xc0', -3933, '\xb7'), > ('15', 0, '', 0, ''), > ('A\x00\x00\x00\x00\x00\x00\x00t\xeb', -18496, '\x19', 13, '')]], > dtype=[('f1', '|S10'), ('f2', ' ('f5', '|S1')]) > > I have some missing entries, is it for that or do I have to make some changes > on the date column? You have two problems here. The first is that you shouldn't have missign entries, or conversion from empty strings to ints (or whatever) will fail: >>> int('') Traceback (most recent call last): File "", line 1, in ? ValueError: invalid literal for int(): Second, you can't feed a string of literals directly into the rec.array constructor (it is not as intelligent to digest this yet). You can achieve what you want by first massaging the data a bit: >>> ra=numpy.rec.array(datas[1:]) >>> numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, 'A'), ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, 'A')], dtype=[('f1', '|S10'), ('f2', '>> ca=numpy.array(datas[1:]) >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, 'A'), ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, 'A')], dtype=[('f1', '|S10'), ('f2', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From fullung at gmail.com Mon Sep 18 06:17:26 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 18 Sep 2006 12:17:26 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: Hey Eric > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! The problem is that the sum axis default is None, not 0. > Is that the expected behaviour or is the doc not updated? (I realise it > may be better to force axis=.. to indicate what you need but still.. the > example in numpy.sum help gives: > sum([[0, 1], [0, 5]]) > array([0, 6]) which is not what it gives when you do it...) The documentation seems to be outdated. I'm not sure, but I think a few of the axis default might have changed in the not too distant past. Not sure if sum was affected. Either way, could you please file a ticket? http://projects.scipy.org/scipy/numpy/register http://projects.scipy.org/scipy/numpy/newticket Regards, Albert From svetosch at gmx.net Mon Sep 18 06:20:49 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 18 Sep 2006 12:20:49 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> References: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: <450E7301.3020106@gmx.net> Eric Emsellem schrieb: > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! The default axis was changed recently to None (btw, which version are you using?), see the release notes on the web. So yes, it's expected. > > Is that the expected behaviour or is the doc not updated? (I realise it > may be better to force axis=.. to indicate what you need but still.. the > example in numpy.sum help gives: > sum([[0, 1], [0, 5]]) > array([0, 6]) which is not what it gives when you do it...) on my 1.0b5 I also see this docstring which indeed seems obsolete. Has it been changed since then? (And similarly for the other functions/methods where axis arguments have changed.) I could check the docstrings systematically and update them, but unfortunately not before the end of this month, sorry. cheers, sven From qwe239 at tom.com Sat Sep 23 08:32:14 2006 From: qwe239 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Sat, 23 Sep 2006 20:32:14 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO3Js8XMxKPE4r/Os8w=?= Message-ID: An HTML attachment was scrubbed... URL: From lroubeyrie at limair.asso.fr Mon Sep 18 11:10:10 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 17:10:10 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <1158574623.4218.6.camel@localhost.localdomain> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609180938.34721.lroubeyrie@limair.asso.fr> <1158574623.4218.6.camel@localhost.localdomain> Message-ID: <200609181710.10496.lroubeyrie@limair.asso.fr> Le lundi 18 septembre 2006 12:17, Francesc Altet a ?crit?: > You have two problems here. The first is that you shouldn't have missign > entries, or conversion from empty strings to ints (or whatever) will > > fail: > >>> int('') > > Traceback (most recent call last): > File "", line 1, in ? > ValueError: invalid literal for int(): > > Second, you can't feed a string of literals directly into the rec.array > constructor (it is not as intelligent to digest this yet). You can > > achieve what you want by first massaging the data a bit: > >>> ra=numpy.rec.array(datas[1:]) > > numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats >='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), > ('06/01/2006', 41, 'A', 30, 'A'), > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > 'A')], > dtype=[('f1', '|S10'), ('f2', ' ' > or, a bit more easier, > > >>> ca=numpy.array(datas[1:]) > >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') > > recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, > 'A'), > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > 'A')], > dtype=[('f1', '|S10'), ('f2', ' ' > > Cheers, Hi, thanks for your help, but I don't understand why is not working here: lionel[ETD-2006-01__PM2.5_DALTON]624>datas Sortie[624]: [['05/01/2006', '33', 'A', '10', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A']] lionel[ETD-2006-01__PM2.5_DALTON]625>ra=rec.array(datas) lionel[ETD-2006-01__PM2.5_DALTON]626>ra Sortie[626]: recarray([('05/01/2006', '33', 'A', '10', 'N'), ('06/01/2006', '41', 'A', '30', 'A'), ('07/01/2006', '20', 'A', '16', 'A')], dtype=[('f1', '|S10'), ('f2', '|S2'), ('f3', '|S1'), ('f4', '|S2'), ('f5', '|S1')]) lionel[ETD-2006-01__PM2.5_DALTON]627>rec.fromarrays( [ra['f1'], ra['f2'], ra['f4']], formats='a10,i2,i2') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/lionel/Etudes_Techniques/ETD-2006-01__PM2.5_DALTON/ /usr/lib/python2.4/site-packages/numpy/core/records.py in fromarrays(arrayList, formats, names, titles, shape, aligned) 235 # populate the record array (makes a copy) 236 for i in range(len(arrayList)): --> 237 _array[_names[i]] = arrayList[i] 238 239 return _array TypeError: array cannot be safely cast to required type -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From faltet at carabos.com Mon Sep 18 11:40:40 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 18 Sep 2006 17:40:40 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <200609181710.10496.lroubeyrie@limair.asso.fr> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609180938.34721.lroubeyrie@limair.asso.fr> <1158574623.4218.6.camel@localhost.localdomain> <200609181710.10496.lroubeyrie@limair.asso.fr> Message-ID: <1158594041.4218.11.camel@localhost.localdomain> El dl 18 de 09 del 2006 a les 17:10 +0200, en/na Lionel Roubeyrie va escriure: > Le lundi 18 septembre 2006 12:17, Francesc Altet a ?crit : > > You have two problems here. The first is that you shouldn't have missign > > entries, or conversion from empty strings to ints (or whatever) will > > > > fail: > > >>> int('') > > > > Traceback (most recent call last): > > File "", line 1, in ? > > ValueError: invalid literal for int(): > > > > Second, you can't feed a string of literals directly into the rec.array > > constructor (it is not as intelligent to digest this yet). You can > > > > achieve what you want by first massaging the data a bit: > > >>> ra=numpy.rec.array(datas[1:]) > > > > numpy.rec.fromarrays([ra['f1'],ra['f2'],ra['f3'],ra['f4'],ra['f5']],formats > >='a10,i2,a1,i2,a1') recarray([('05/01/2006', 33, 'A', 0, 'N'), > > ('06/01/2006', 41, 'A', 30, 'A'), > > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > > 'A')], > > dtype=[('f1', '|S10'), ('f2', ' > ' > > > or, a bit more easier, > > > > >>> ca=numpy.array(datas[1:]) > > >>> numpy.rec.fromarrays(ca.transpose(),formats='a10,i2,a1,i2,a1') > > > > recarray([('05/01/2006', 33, 'A', 0, 'N'), ('06/01/2006', 41, 'A', 30, > > 'A'), > > ('07/01/2006', 20, 'A', 16, 'A'), ('08/01/2006', 16, 'A', 13, > > 'A')], > > dtype=[('f1', '|S10'), ('f2', ' > ' > > > > > Cheers, > > Hi, > thanks for your help, but I don't understand why is not working here: > lionel[ETD-2006-01__PM2.5_DALTON]624>datas > Sortie[624]: > [['05/01/2006', '33', 'A', '10', 'N'], > ['06/01/2006', '41', 'A', '30', 'A'], > ['07/01/2006', '20', 'A', '16', 'A']] > > lionel[ETD-2006-01__PM2.5_DALTON]625>ra=rec.array(datas) > > lionel[ETD-2006-01__PM2.5_DALTON]626>ra > Sortie[626]: > recarray([('05/01/2006', '33', 'A', '10', 'N'), > ('06/01/2006', '41', 'A', '30', 'A'), > ('07/01/2006', '20', 'A', '16', 'A')], > dtype=[('f1', '|S10'), ('f2', '|S2'), ('f3', '|S1'), ('f4', '|S2'), > ('f5', '|S1')]) > > lionel[ETD-2006-01__PM2.5_DALTON]627>rec.fromarrays( [ra['f1'], ra['f2'], > ra['f4']], formats='a10,i2,i2') > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) Mmm, this works for me: >>> datas [['05/01/2006', '33', 'A', '0', 'N'], ['06/01/2006', '41', 'A', '30', 'A'], ['07/01/2006', '20', 'A', '16', 'A'], ['08/01/2006', '16', 'A', '13', 'A']] >>> ra=rec.array(datas) >>> rec.fromarrays([ra['f1'],ra['f2'],ra['f4']], formats='a10,i2,i2') recarray([('05/01/2006', 33, 0), ('06/01/2006', 41, 30), ('07/01/2006', 20, 16), ('08/01/2006', 16, 13)], dtype=[('f1', '|S10'), ('f2', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From lroubeyrie at limair.asso.fr Mon Sep 18 11:47:07 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Mon, 18 Sep 2006 17:47:07 +0200 Subject: [Numpy-discussion] recarray In-Reply-To: <1158594041.4218.11.camel@localhost.localdomain> References: <200609151449.51529.lroubeyrie@limair.asso.fr> <200609181710.10496.lroubeyrie@limair.asso.fr> <1158594041.4218.11.camel@localhost.localdomain> Message-ID: <200609181747.07852.lroubeyrie@limair.asso.fr> Le lundi 18 septembre 2006 17:40, Francesc Altet a ?crit?: > I'm running NumPy 1.0b5. Please, check that you are using a recent > version of it. > > Cheers, Arg, sorry, version here was 0.9, an upgrade and it works fine. thanks again -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From myeates at jpl.nasa.gov Mon Sep 18 12:29:24 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 18 Sep 2006 09:29:24 -0700 Subject: [Numpy-discussion] visual programming Message-ID: <450EC964.6080304@jpl.nasa.gov> semi off topic. Does anyone know of any good visual programming tools? Forever ago, I used to use something called Khoros for image processing and I found it very useful. You connect boxes which represent different processing steps. At about the same time, there was something similar to Khoros from SGI. Explorer was its name, I think. Anybody out there use this sort of thing? Matlab doesn't offer it, right? Nor matplotlib? Mathew From mg.mailing-list at laposte.net Mon Sep 18 12:55:31 2006 From: mg.mailing-list at laposte.net (mg) Date: Mon, 18 Sep 2006 18:55:31 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy Message-ID: <450ECF83.9020103@laposte.net> Hi all, I am doing a feseability study to migrate our Python based FEM applications from Numarray to Numpy. First, I tried to install Numpy from Python-2.4 on linux-x86, linux-86-64bit. So, all work fine. Great! Moreover, I change easily the BLAS linked libraries. I tried with ATLAS and GOTO. Great again! Second, I try to do the same think on windows-x86 without success. So my first question is: is Numpy-1.0b5 has been tested and is supported on Windows? Third, I tried to install Numpy from Python-2.5, which is our standard Python, on linux-x86... and the compilation stopped during the compilation of core/src/multiarraymodule.c. So my second question is: is there a workaround or is the porting to Python2.5 is yet schedule? My third question is: is the tool to migrate the numarray based Python scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) We have created a lot of bindings in order to pilote our generic-C++ framework with Python scripts. So, about the Numpy API, is it widely different than the Numarray API? (We will order the Numpy Guide too.) To not duplicate large numerical memory arrays, Numarray allows to aliasing the memory of some bindings with arrays from Numarray, and we have used this feature intensively. So, I wonder if it is currently supported (or even scheduled)? Thanks for your answer, Mathieu. From schaffer at optonline.net Mon Sep 18 12:59:52 2006 From: schaffer at optonline.net (Les Schaffer) Date: Mon, 18 Sep 2006 12:59:52 -0400 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: <450ED088.80309@optonline.net> Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. > if i recall correctly, there is still a version of Khoros floating around that you can compile under Linux. i've done it several time, but it WAS a while ago. Les Schaffer From Chris.Barker at noaa.gov Mon Sep 18 13:08:48 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 18 Sep 2006 10:08:48 -0700 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450E7301.3020106@gmx.net> References: <450E6D2E.9010706@obs.univ-lyon1.fr> <450E7301.3020106@gmx.net> Message-ID: <450ED2A0.50208@noaa.gov> Sven Schreiber wrote: > on my 1.0b5 I also see this docstring which indeed seems obsolete. I get this docs string from : >>> import numpy as N >>> N.__version__ '1.0b5' >>> a = N.arange(10) >>> help( a.sum) """ sum(...) a.sum(axis=None, dtype=None) -> Sum of array over given axis. Sum the array over the given axis. If the axis is None, sum over all dimensions of the array. """ that looks right to me. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From peridot.faceted at gmail.com Mon Sep 18 13:17:45 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 18 Sep 2006 13:17:45 -0400 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: On 18/09/06, Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew I doubt it's what you're looking for, but PD and GEM are a visual programming environment, oriented towards music, audio signal processing, 2D and 3D video. (They're related to Max and jMax, which do some of the same things.) A. M. Archibald From charlesr.harris at gmail.com Mon Sep 18 13:35:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 11:35:23 -0600 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: On 9/18/06, Mathew Yeates wrote: > > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew There have been several presentations at the scipy conferences about a visual programming environment for molecular visualization. You might check and see if the presentations are available on the scipy site. Lets see, name is ViPEr, and some papers can be found under publications at this site: http://www.scripps.edu/%7Esanner/python/index.html I have no idea how generally useful the framework is. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Sep 18 14:46:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 12:46:17 -0600 Subject: [Numpy-discussion] record descriptors: trailing unused bytes In-Reply-To: <200609172017.09548.martin.wiechert@gmx.de> References: <200609172017.09548.martin.wiechert@gmx.de> Message-ID: <450EE979.706@ieee.org> Martin Wiechert wrote: > Hi list, > > does anybody know, if in C in the PyArray_Descr struct it is safe to manually > change descr->elsize and descr->alignment as long as these are compatible and > descr->elsize is large enough to hold all fields? Of course I mean before any > array is constructed using descr. > Well, you should really make a copy of the PyArray_Descr structure and then fill it in as desired unless you are sure that no other arrays have are using that particular data-type object to describe their own data: PyArray_DescrNew gives you a new copy based on an old one (you just get a reference to the funciton pointers in the 'f' member so don't go changing those). I guess your statement about "before any array is constructed using descr" means you are sure that there are no other arrays using the old elsize and alignment. Then it should be fine. There are not supposed to be any assumptions about the data-types except for what is explicitly provided in the data-type object (PyArray_Descr *). So, changing them will change the data-type and things should ideally work. -Travis From matthew.brett at gmail.com Mon Sep 18 14:49:54 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Sep 2006 19:49:54 +0100 Subject: [Numpy-discussion] Changing byte ordering with astype fails with 0d arrays Message-ID: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> Hi, As expected: In [67]:a = array([1], dtype='i4').dtype Out[68]:dtype('>i4') I was also expecting this to work for 0d arrays, but it doesn't: In [69]:a = array(1, dtype='i4').dtype Out[70]:dtype(' BUG in numpy.sum? In-Reply-To: <450E6D2E.9010706@obs.univ-lyon1.fr> References: <450E6D2E.9010706@obs.univ-lyon1.fr> Message-ID: <450EEAE8.6060706@ieee.org> Eric Emsellem wrote: > Hi again > > after some hours of debugging I finally (I think) found the problem: > > numpy.sum([[0,1,2],[2,3,4]]) > 24 > > numpy.sum([[0,1,2],[2,3,4]],axis=0) > array([2, 4, 6]) > > numpy.sum([[0,1,2],[2,3,4]],axis=1) > array([3, 9]) > > > Isn't the first line supposed to act as with "axis=0" by default (see > help numpy.sum!)...??? > Not setting axis=0 it sums everything! > See the Release Notes page on www.scipy.org. It documents everything that has changed. Several things will break old code as indicated. There are several options for keeping old code working: 1) Use the numpy.oldnumeric compatibility layer which keeps the same definitions and defaults as Numeric 2) Use conversion tools (like the recently added fix_default_axis) tool to automatically insert axis=0 arguments in all code where it is not present (or to automatically change the import to oldnumeric). For the future, you must specify which axis you mean for a Nd array or the code will assume you meant to work over the entire N-d array. We all recognize this is a pain to change. That's why the backward compatibilty options are avaiable and the tools have been written. Believe me, I know what a pain it is. I have had to keep SciPy and Matplotlib working with all the changes to NumPy. -Travis From oliphant.travis at ieee.org Mon Sep 18 14:58:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 12:58:26 -0600 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450ECF83.9020103@laposte.net> References: <450ECF83.9020103@laposte.net> Message-ID: <450EEC52.30906@ieee.org> mg wrote: > Hi all, > > I am doing a feseability study to migrate our Python based FEM > applications from Numarray to Numpy. > > First, I tried to install Numpy from Python-2.4 on linux-x86, > linux-86-64bit. So, all work fine. Great! Moreover, I change easily the > BLAS linked libraries. I tried with ATLAS and GOTO. Great again! > > Second, I try to do the same think on windows-x86 without success. So my > first question is: is Numpy-1.0b5 has been tested and is supported on > Windows? > Yes, it should work. Builds for windows were provided. But, perhaps there are configuration issues for your system that we are not handling correctly. > Third, I tried to install Numpy from Python-2.5, which is our standard > Python, on linux-x86... and the compilation stopped during the > compilation of core/src/multiarraymodule.c. So my second question is: is > there a workaround or is the porting to Python2.5 is yet schedule? > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. Look for NumPy 1.0rc1 to come out soon. > My third question is: is the tool to migrate the numarray based Python > scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) > It needs more testing. It would be great if you could help us find and fix bugs in it. I don't have a lot of numarray code to test. > We have created a lot of bindings in order to pilote our generic-C++ > framework with Python scripts. So, about the Numpy API, is it widely > different than the Numarray API? (We will order the Numpy Guide too.) > It is more similar to the Numeric C-API. However, the numarray C-API is completely supported by including numpy/libnumarray.h so you should be able to convert your C code very easily. Any problems encountered should be noted and we'll get them fixed. > To not duplicate large numerical memory arrays, Numarray allows to > aliasing the memory of some bindings with arrays from Numarray, and we > have used this feature intensively. So, I wonder if it is currently > supported (or even scheduled)? I'm pretty sure the answer is yes (because the Numarray C-API is supported), though I'm not exactly sure what you mean. Do you mean that you have memory created in the C/C++ framework and then you have an array use that memory for it's data area? If that is what you mean, then the answer is definitely yes. -Travis From oliphant.travis at ieee.org Mon Sep 18 15:01:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 13:01:19 -0600 Subject: [Numpy-discussion] Changing byte ordering with astype fails with 0d arrays In-Reply-To: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> References: <1e2af89e0609181149j3308ddcbhd4ca8e2a1bee497f@mail.gmail.com> Message-ID: <450EECFF.9050809@ieee.org> Matthew Brett wrote: > Hi, > > As expected: > > In [67]:a = array([1], dtype=' > In [68]:a.astype('>i4').dtype > Out[68]:dtype('>i4') > > I was also expecting this to work for 0d arrays, but it doesn't: > > In [69]:a = array(1, dtype=' > In [70]:a.astype('>i4').dtype > Out[70]:dtype(' > The problem is that the astype method is returning an array scalar (it used to be that 0-d arrays were "avoided" at all costs). We've since relaxes this requirement and I think here's another place where it needs to be relaxed. -Travis From Chris.Barker at noaa.gov Mon Sep 18 15:37:36 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 18 Sep 2006 12:37:36 -0700 Subject: [Numpy-discussion] 1.0b5 Installation on OS X In-Reply-To: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> References: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> Message-ID: <450EF580.90103@noaa.gov> David Andrews wrote: > I'm unable to install the 1.0b5 release from the .mpkg on OS X - when > it comes to the two checkboxes to select the platlib and the scripts, > both are greyed out. The installer then claims 'Cannot continue: > Nothing to install'. hmmm. It works just fine for me -- just clicking OK as I go merrily along. OS version? python version? processor? Also, try deleting or renaming any existing install you have from site-packages, then try again. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthew.brett at gmail.com Mon Sep 18 15:57:16 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Sep 2006 20:57:16 +0100 Subject: [Numpy-discussion] Segfault on byteswap() on recarrays Message-ID: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> Hi, I noticed this works: In [5]:a = array((1,), dtype=[('one', ' I need build core numpy (plus possible dft) static into python.a Can someone point me to the right direction. Currently I built some libs but blocked on _internal. From my Setup.local: NPSRC=/home/mrovner/src/numpy-0.9.8 NPBLD=$(NPSRC)/build/src.linux-i686-2.4 NPINC=-I$(NPSRC)/numpy/core/include -I$(NPBLD)/numpy/core -I$(NPSRC)/numpy/core/src multiarray $(NPSRC)/numpy/core/src/multiarraymodule.c $(NPINC) umath $(NPBLD)/numpy/core/src/umathmodule.c $(NPINC) scalarmath $(NPBLD)/numpy/core/src/scalarmathmodule.c $(NPINC) _sort $(NPBLD)/numpy/core/src/_sortmodule.c $(NPINC) _compiled_base $(NPSRC)/numpy/lib/src/_compiled_base.c $(NPINC) Thanks, Mike From svetosch at gmx.net Mon Sep 18 16:30:22 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 18 Sep 2006 22:30:22 +0200 Subject: [Numpy-discussion] buggy buggy bugyy: format and casting ==> BUG in numpy.sum? In-Reply-To: <450ED2A0.50208@noaa.gov> References: <450E6D2E.9010706@obs.univ-lyon1.fr> <450E7301.3020106@gmx.net> <450ED2A0.50208@noaa.gov> Message-ID: <450F01DE.9070607@gmx.net> Christopher Barker schrieb: > Sven Schreiber wrote: >> on my 1.0b5 I also see this docstring which indeed seems obsolete. > > I get this docs string from : > > >>> import numpy as N > >>> N.__version__ > '1.0b5' > >>> a = N.arange(10) > >>> help( a.sum) > > """ > sum(...) > a.sum(axis=None, dtype=None) -> Sum of array over given axis. > > Sum the array over the given axis. If the axis is None, sum over > all dimensions of the array. > > """ > > that looks right to me. > > -Chris > Well here's what we had in mind: sum(x, axis=None, dtype=None, out=None) (...) Examples: >>> sum([0.5, 1.5]) 2.0 >>> sum([0.5, 1.5], dtype=Int32) 1 >>> sum([[0, 1], [0, 5]]) array([0, 6]) >>> sum([[0, 1], [0, 5]], axis=1) array([1, 5]) And you can see the third example would now be wrong, giving a scalar of 6 now. -sven From oliphant.travis at ieee.org Mon Sep 18 16:53:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 14:53:44 -0600 Subject: [Numpy-discussion] Segfault on byteswap() on recarrays In-Reply-To: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> References: <1e2af89e0609181257l63c7188ew243da43ca2848ff@mail.gmail.com> Message-ID: <450F0758.8080607@ieee.org> Matthew Brett wrote: > Hi, > > I noticed this works: > > In [5]:a = array((1,), dtype=[('one', ' > In [6]:a.byteswap() > Out[6]: > array((16777216,), > dtype=[('one', ' > But, extending the recarray leads to a segfault on byteswapping: > > In [8]:a = array((1, 2), dtype=[('one', ' > In [9]:a.byteswap() > Segmentation fault > Great catch. Fixed in SVN. -Travis From bryan at cole.uklinux.net Mon Sep 18 17:53:33 2006 From: bryan at cole.uklinux.net (Bryan Cole) Date: Mon, 18 Sep 2006 22:53:33 +0100 Subject: [Numpy-discussion] visual programming In-Reply-To: <450EC964.6080304@jpl.nasa.gov> References: <450EC964.6080304@jpl.nasa.gov> Message-ID: <1158616413.8450.49.camel@pc1.cole.uklinux.net> On Mon, 2006-09-18 at 09:29 -0700, Mathew Yeates wrote: > semi off topic. > Does anyone know of any good visual programming tools? Forever ago, I > used to use something called Khoros for image processing and I found it > very useful. You connect boxes which represent different processing > steps. At about the same time, there was something similar to Khoros > from SGI. Explorer was its name, I think. Looking at the Khoros commercial site (now AccuSoft), it looks like a more polished version of OpenDX (which also offers a visual programming system). See http://www.opendx.org/index2.php The problem with OpenDX is the cr*ppy Motif-based GUI (particularly so if you use Windows). I prefer VTK (www.vtk.org) which offers similar concepts (pipeline execution of data filtering/visualisation blocks) but in a programmatic form based on python (which we like...). BC > > Anybody out there use this sort of thing? Matlab doesn't offer it, > right? Nor matplotlib? > > Mathew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 From Martin.Wiechert at mpimf-heidelberg.mpg.de Mon Sep 18 18:12:31 2006 From: Martin.Wiechert at mpimf-heidelberg.mpg.de (Martin Wiechert) Date: Tue, 19 Sep 2006 00:12:31 +0200 Subject: [Numpy-discussion] record descriptors: trailing unused bytes In-Reply-To: <450EE979.706@ieee.org> References: <200609172017.09548.martin.wiechert@gmx.de> <450EE979.706@ieee.org> Message-ID: <200609190012.31853.wiechert@mpimf-heidelberg.mpg.de> On Monday 18 September 2006 20:46, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > does anybody know, if in C in the PyArray_Descr struct it is safe to > > manually change descr->elsize and descr->alignment as long as these are > > compatible and descr->elsize is large enough to hold all fields? Of > > course I mean before any array is constructed using descr. > > Well, you should really make a copy of the PyArray_Descr structure and > then fill it in as desired unless you are sure that no other arrays have > are using that particular data-type object to describe their own data: > PyArray_DescrNew gives you a new copy based on an old one (you just get > a reference to the funciton pointers in the 'f' member so don't go > changing those). > > I guess your statement about "before any array is constructed using > descr" means you are sure that there are no other arrays using the old > elsize and alignment. > > Then it should be fine. There are not supposed to be any assumptions > about the data-types except for what is explicitly provided in the > data-type object (PyArray_Descr *). So, changing them will change the > data-type and things should ideally work. > > -Travis > Thanks > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- AV scanned by FortiGate From stefan at sun.ac.za Mon Sep 18 20:07:27 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 19 Sep 2006 02:07:27 +0200 Subject: [Numpy-discussion] unique1d: renaming retindx to return_index Message-ID: <20060919000727.GA12696@mentat.za.net> Hi, Would anyone object if I changed the signature of unique1d(ar1, retindx=False) to unique1d(ar1, return_index=False)? I find retindx both harder to read and to type than return_index. Thanks. St?fan From kslmns at clubtheatre.org.uk Mon Sep 18 20:03:59 2006 From: kslmns at clubtheatre.org.uk (Sam Estes) Date: Mon, 18 Sep 2006 19:03:59 -0500 Subject: [Numpy-discussion] persecute ambidextrous Message-ID: <001c01c6db80$620f899a$7b40d041@xlwmk.qdkvjf> I will shed no blood, said Wi, not even that of those who hate me,for misery makes them mad. That stone was ill aimed, said Pag who stood by. I tried to stay them but theyfelled me with clubs, for they are fierce as wolves and more savage. Aye, let us go to the cave, for if we stay here upon the ice we shallperish, said Pag. He looked down upon the people and, by the shimmering moonlight,watched their faces. The people who called upon the Ice-dwellers, where arethey? If he is afraid todie, then let him give another to the gods. I am sure that it will be so, she replied simply. I tell you nothing, Wife, he answered sternly. Now let Wi the mightyman, our chief who rejects you, make answer to it if he is able. Wi, you have your answer, piped Ngae, as the shouting died. Was it not well that one should die for the sake ofmany? Shall we gather men and fall on them and kill them? I brought the trouble, Wi; surely I should have paid its price. I brought the trouble, Wi; surely I should have paid its price. They cowered before him and muttered together. All the white world was a desolation and a waste. Then he had come thither because theplace was holy to him. For he had sent command to Aaka that she must nomore sleep alone in her hut. They say that the Ice-gods demand ahuman sacrifice and that this sacrifice must be given to them. He, Wi himself,whom they dared not touch because he was chief and too strong forthem. They saw it creep into the sea and there break off in sharp-toppedhills of ice. Yes, a Voice has spoken tothem from the roof of their hut in the dead of night. Now let Wi the mightyman, our chief who rejects you, make answer to it if he is able. They are starvingbrutes, and such will fill themselves. Or, if you dare not, then send us oneof your household. Their prayers finished, they spoke together. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: warm-up.gif Type: image/gif Size: 8483 bytes Desc: not available URL: From seth.olsen at gmail.com Mon Sep 18 20:51:55 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 10:51:55 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi Numpyers, I recently sent the message below to the MMTK mailing list, but it's really a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors() never stops, even when I try to diagonalize a very simple 2x2 matrix. I've tried this on two machines with completely standard FC4 and FC5 installations using the default libraries and atlas libraries downloaded and installed via yum. Does anyone on this list have any experience with this problem? Cheers, Seth ---------- Forwarded message ---------- From: Dr. Seth Olsen Date: Sep 19, 2006 10:38 AM Subject: Re: Collection.findTransformation() never stops To: mmtk at python.net, mmtk at starship.python.net Hi MMTKers, The problem with Collection.findTransformation() that I reported earlier is due to a problem with LinearAlgebra.eigenvectors(). This algortithm does not seem to stop - it can't even manage to find the eigenvectors of the 2x2 square matrix [[2,1],[1,2]]. Asking for this causes a long wait followed by CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with newly installed atlas libraries, but it still won't work. I have had this problem now on 2 machines (one running FC4 with , the other FC5, both pentium 4 machines), with atlas lapack/blas libraries installed freshly via yum and linked as per the recommendations found in setup.py. THE FEDORA INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been downloaded and installed via the fedora yum repositories. As the eigenvectors() routine in LinearAlgebra.py is obviously going to be a heavily used algorithm (not just in MMTK), is it really possible that no one has had this problem before? Cheers, Seth On 9/18/06, Dr. Seth Olsen wrote: > > > Hi MMTKers, > > I'm having a bit of trouble with MMTK.Collection.findTransformation. When > I use this member function, the computer never stops thinking until I kill > the process. I've waited a couple of hours and still nothing. I get the > same thing if I try to take a step back and use > Collection.findTransformationAsQuaternion. Has anyone had this problem > before? > > Cheers, > > Seth > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 21:34:50 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 10:34:50 +0900 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hey there, I don't see anything called "LinearAlgegra.eigenvectors()". Do you maybe mean numpy.linalg.eig? Which version of numpy are you using? The latest release is 1.0b5. >>> import numpy >>> numpy.__version__ '1.0b5' >>> numpy.linalg.eig([[2,1],[1,2]]) (array([ 3., 1.]), array([[ 0.70710678, -0.70710678], [ 0.70710678, 0.70710678]])) I'm on WindowsXP, though. Regards, Bill On 9/19/06, Dr. Seth Olsen wrote: > > Hi Numpyers, > > I recently sent the message below to the MMTK mailing list, but it's really > a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors() > never stops, even when I try to diagonalize a very simple 2x2 matrix. I've > tried this on two machines with completely standard FC4 and FC5 > installations using the default libraries and atlas libraries downloaded and > installed via yum. Does anyone on this list have any experience with this > problem? > > Cheers, > > Seth > > ---------- Forwarded message ---------- > From: Dr. Seth Olsen > Date: Sep 19, 2006 10:38 AM > Subject: Re: Collection.findTransformation() never stops > To: mmtk at python.net, mmtk at starship.python.net > > > Hi MMTKers, > > The problem with Collection.findTransformation() that I reported earlier is > due to a problem with LinearAlgebra.eigenvectors(). This algortithm does > not seem to stop - it can't even manage to find the eigenvectors of the 2x2 > square matrix [[2,1],[1,2]]. Asking for this causes a long wait followed by > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with newly > installed atlas libraries, but it still won't work. I have had this problem > now on 2 machines (one running FC4 with , the other FC5, both pentium 4 > machines), with atlas lapack/blas libraries installed freshly via yum and > linked as per the recommendations found in setup.py. THE FEDORA > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been downloaded > and installed via the fedora yum repositories. > > As the eigenvectors() routine in LinearAlgebra.py is obviously going to be a > heavily used algorithm (not just in MMTK), is it really possible that no one > has had this problem before? > > Cheers, > > Seth > > > On 9/18/06, Dr. Seth Olsen wrote: > > > > > > Hi MMTKers, > > > > I'm having a bit of trouble with > MMTK.Collection.findTransformation. When I use this member > function, the computer never stops thinking until I kill the process. I've > waited a couple of hours and still nothing. I get the same thing if I try > to take a step back and use > Collection.findTransformationAsQuaternion. Has anyone had > this problem before? > > > > Cheers, > > > > Seth > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > > > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > > -- > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From seth.olsen at gmail.com Mon Sep 18 22:06:20 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 12:06:20 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi Bill, MMTK has not made the conversion over to the new numpy module. It is built against the old Numeric code, and the word from its developers is that changing to numpy cannot be a priority now. Cheers, Seth On 9/19/06, Bill Baxter wrote: > > Hey there, > I don't see anything called "LinearAlgegra.eigenvectors()". Do you > maybe mean numpy.linalg.eig? > Which version of numpy are you using? > The latest release is 1.0b5. > > >>> import numpy > >>> numpy.__version__ > '1.0b5' > > >>> numpy.linalg.eig([[2,1],[1,2]]) > (array([ 3., 1.]), > array([[ 0.70710678, -0.70710678], > [ 0.70710678, 0.70710678]])) > > I'm on WindowsXP, though. > Regards, > Bill > > On 9/19/06, Dr. Seth Olsen wrote: > > > > Hi Numpyers, > > > > I recently sent the message below to the MMTK mailing list, but it's > really > > a problem with LinearAlgebra.py. The routine LinearAlgebra.eigenvectors > () > > never stops, even when I try to diagonalize a very simple 2x2 > matrix. I've > > tried this on two machines with completely standard FC4 and FC5 > > installations using the default libraries and atlas libraries downloaded > and > > installed via yum. Does anyone on this list have any experience with > this > > problem? > > > > Cheers, > > > > Seth > > > > ---------- Forwarded message ---------- > > From: Dr. Seth Olsen > > Date: Sep 19, 2006 10:38 AM > > Subject: Re: Collection.findTransformation() never stops > > To: mmtk at python.net, mmtk at starship.python.net > > > > > > Hi MMTKers, > > > > The problem with Collection.findTransformation() that I reported earlier > is > > due to a problem with LinearAlgebra.eigenvectors(). This algortithm > does > > not seem to stop - it can't even manage to find the eigenvectors of the > 2x2 > > square matrix [[2,1],[1,2]]. Asking for this causes a long wait > followed by > > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with > newly > > installed atlas libraries, but it still won't work. I have had this > problem > > now on 2 machines (one running FC4 with , the other FC5, both pentium 4 > > machines), with atlas lapack/blas libraries installed freshly via yum > and > > linked as per the recommendations found in setup.py. THE FEDORA > > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been > downloaded > > and installed via the fedora yum repositories. > > > > As the eigenvectors() routine in LinearAlgebra.py is obviously going to > be a > > heavily used algorithm (not just in MMTK), is it really possible that no > one > > has had this problem before? > > > > Cheers, > > > > Seth > > > > > > On 9/18/06, Dr. Seth Olsen wrote: > > > > > > > > > Hi MMTKers, > > > > > > I'm having a bit of trouble with > > MMTK.Collection.findTransformation. When I use this member > > function, the computer never stops thinking until I kill the > process. I've > > waited a couple of hours and still nothing. I get the same thing if I > try > > to take a step back and use > > Collection.findTransformationAsQuaternion. Has anyone had > > this problem before? > > > > > > Cheers, > > > > > > Seth > > > > > > -- > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > positions of > > the University of Queensland. > > > > > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions > of > > the University of Queensland. > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > Dr Seth Olsen, PhD > > Postdoctoral Fellow, Biomolecular Modeling Group > > Centre for Computational Molecular Science > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > The University of Queensland > > Qld 4072, Brisbane, Australia > > > > tel (617) 3346 3976 > > fax (617) 33654623 > > email: s.olsen1 at uq.edu.au > > Web: www.ccms.uq.edu.au > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > The opinions expressed here are my own and do not reflect the positions > of > > the University of Queensland. > > > ------------------------------------------------------------------------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > > opinions on IT & business topics through brief surveys -- and earn cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 22:07:05 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 11:07:05 +0900 Subject: [Numpy-discussion] unique1d: renaming retindx to return_index In-Reply-To: <20060919000727.GA12696@mentat.za.net> References: <20060919000727.GA12696@mentat.za.net> Message-ID: Grepping through numpy/**/*.py, the only three functions I could find with an argument to specify extra return values are: linspace(start, stop, num=50, endpoint=True, retstep=False) average(a, axis=None weights=None, returned=False) unique1d(ar1, retindx=False) If unique1d is going to change to return_index, then linspace should probably also change to return_step to maintain consistency. And maybe average should change to 'return_count' or [ick] 'return_denominator'. --bb On 9/19/06, Stefan van der Walt wrote: > Hi, > > Would anyone object if I changed the signature of > > unique1d(ar1, retindx=False) > > to > > unique1d(ar1, return_index=False)? > > I find retindx both harder to read and to type than return_index. > From wbaxter at gmail.com Mon Sep 18 22:21:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 11:21:19 +0900 Subject: [Numpy-discussion] max argmax combo Message-ID: I find myself often wanting both the max and the argmax of an array. (And same for the other arg* functions) Of course I can do argmax first then use fancy indexing to get the max as well. But the result of argmax isn't really in a format that's readily usable as an index. You have to do something like a = rand(10,5) imax = a.argmax(axis=0) vmax = a[(imax, range(5))] Which isn't terrible, just always takes me a moment to remember the proper indexing expression. Would a way to get the argmax and the max at the same time be of interest to anyone else? Maybe an extra 'ret' arg on argmax? a = rand(10,5) imax,vmax = a.argmax(axis=0,retmax=True) --Bill From charlesr.harris at gmail.com Mon Sep 18 22:56:39 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 20:56:39 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Bill Baxter wrote: > > I find myself often wanting both the max and the argmax of an array. > (And same for the other arg* functions) > Of course I can do argmax first then use fancy indexing to get the max as > well. > But the result of argmax isn't really in a format that's readily > usable as an index. > You have to do something like > a = rand(10,5) > imax = a.argmax(axis=0) > vmax = a[(imax, range(5))] > > Which isn't terrible, just always takes me a moment to remember the > proper indexing expression. > Would a way to get the argmax and the max at the same time be of > interest to anyone else? Maybe an extra 'ret' arg on argmax? > > a = rand(10,5) > imax,vmax = a.argmax(axis=0,retmax=True) I don't generally like overloading return values, the function starts to lose its definition and becomes a bit baroque where simply changing a keyword value can destroy the viability of the following code. But I can see the utility of what you want. Hmm, this problem is not unique to argmax. Maybe what we need is a general way to extract values, something like extract(a, imax, axis=0) to go along with all the single axis functions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From seth.olsen at gmail.com Mon Sep 18 23:11:35 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 13:11:35 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: Hi MMTKers and NUMPYers, Bill's answer to my inquiry about the problem I'm having with Collection.findTransformation() (and also, incidentally, with the dgesvd call in Subspace.getBasis(), has convinced me that I can no long use MMTK without changing some of the code over to numpy. I have already been able to determine that invoking numpy.oldnumeric.alter_code1.convertall() over the MMTK directory tree is not the answer. Has anyone on either of these lists ever tried this before and, if so, can it be done (without destroying my sanity)? Cheers, Seth On 9/19/06, Dr. Seth Olsen wrote: > > > Hi Bill, > > MMTK has not made the conversion over to the new numpy module. It is > built against the old Numeric code, and the word from its developers is that > changing to numpy cannot be a priority now. > > Cheers, > > Seth > > > On 9/19/06, Bill Baxter wrote: > > > > Hey there, > > I don't see anything called "LinearAlgegra.eigenvectors()". Do you > > maybe mean numpy.linalg.eig? > > Which version of numpy are you using? > > The latest release is 1.0b5. > > > > >>> import numpy > > >>> numpy.__version__ > > '1.0b5' > > > > >>> numpy.linalg.eig([[2,1],[1,2]]) > > (array([ 3., 1.]), > > array([[ 0.70710678, -0.70710678], > > [ 0.70710678, 0.70710678]])) > > > > I'm on WindowsXP, though. > > Regards, > > Bill > > > > On 9/19/06, Dr. Seth Olsen wrote: > > > > > > Hi Numpyers, > > > > > > I recently sent the message below to the MMTK mailing list, but it's > > really > > > a problem with LinearAlgebra.py. The routine > > LinearAlgebra.eigenvectors() > > > never stops, even when I try to diagonalize a very simple 2x2 > > matrix. I've > > > tried this on two machines with completely standard FC4 and FC5 > > > installations using the default libraries and atlas libraries > > downloaded and > > > installed via yum. Does anyone on this list have any experience with > > this > > > problem? > > > > > > Cheers, > > > > > > Seth > > > > > > ---------- Forwarded message ---------- > > > From: Dr. Seth Olsen > > > Date: Sep 19, 2006 10:38 AM > > > Subject: Re: Collection.findTransformation() never stops > > > To: mmtk at python.net, mmtk at starship.python.net > > > > > > > > > Hi MMTKers, > > > > > > The problem with Collection.findTransformation() that I reported > > earlier is > > > due to a problem with LinearAlgebra.eigenvectors(). This algortithm > > does > > > not seem to stop - it can't even manage to find the eigenvectors of > > the 2x2 > > > square matrix [[2,1],[1,2]]. Asking for this causes a long wait > > followed by > > > CPU overheat warnings. Moreover, I have recompiled Numeric23.8 with > > newly > > > installed atlas libraries, but it still won't work. I have had this > > problem > > > now on 2 machines (one running FC4 with , the other FC5, both pentium > > 4 > > > machines), with atlas lapack/blas libraries installed freshly via yum > > and > > > linked as per the recommendations found in setup.py . THE FEDORA > > > INSTALLATIONS ON BOTH MACHINES IS STANDARD - everything has been > > downloaded > > > and installed via the fedora yum repositories. > > > > > > As the eigenvectors() routine in LinearAlgebra.py is obviously going > > to be a > > > heavily used algorithm (not just in MMTK), is it really possible that > > no one > > > has had this problem before? > > > > > > Cheers, > > > > > > Seth > > > > > > > > > On 9/18/06, Dr. Seth Olsen < seth.olsen at gmail.com> wrote: > > > > > > > > > > > > Hi MMTKers, > > > > > > > > I'm having a bit of trouble with > > > MMTK.Collection.findTransformation . When I use this member > > > function, the computer never stops thinking until I kill the > > process. I've > > > waited a couple of hours and still nothing. I get the same thing if I > > try > > > to take a step back and use > > > Collection.findTransformationAsQuaternion. Has anyone had > > > this problem before? > > > > > > > > Cheers, > > > > > > > > Seth > > > > > > > > -- > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > > > Dr Seth Olsen, PhD > > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > > Centre for Computational Molecular Science > > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. > > 75) > > > > The University of Queensland > > > > Qld 4072, Brisbane, Australia > > > > > > > > tel (617) 3346 3976 > > > > fax (617) 33654623 > > > > email: s.olsen1 at uq.edu.au > > > > Web: www.ccms.uq.edu.au > > > > > > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > > > > > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > > -- > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > > > > Dr Seth Olsen, PhD > > > Postdoctoral Fellow, Biomolecular Modeling Group > > > Centre for Computational Molecular Science > > > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > > > The University of Queensland > > > Qld 4072, Brisbane, Australia > > > > > > tel (617) 3346 3976 > > > fax (617) 33654623 > > > email: s.olsen1 at uq.edu.au > > > Web: www.ccms.uq.edu.au > > > > > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > > The opinions expressed here are my own and do not reflect the > > positions of > > > the University of Queensland. > > > > > ------------------------------------------------------------------------- > > > Take Surveys. Earn Cash. Influence the Future of IT > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > share your > > > opinions on IT & business topics through brief surveys -- and earn > > cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > your > > opinions on IT & business topics through brief surveys -- and earn cash > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > -- > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > > Dr Seth Olsen, PhD > Postdoctoral Fellow, Biomolecular Modeling Group > Centre for Computational Molecular Science > Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) > The University of Queensland > Qld 4072, Brisbane, Australia > > tel (617) 3346 3976 > fax (617) 33654623 > email: s.olsen1 at uq.edu.au > Web: www.ccms.uq.edu.au > > ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms > The opinions expressed here are my own and do not reflect the positions of > the University of Queensland. > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Sep 18 23:19:32 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 19 Sep 2006 12:19:32 +0900 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/19/06, Charles R Harris wrote: > On 9/18/06, Bill Baxter wrote: > > I find myself often wanting both the max and the argmax of an array. > > (And same for the other arg* functions) > > You have to do something like > > a = rand(10,5) > > imax = a.argmax(axis=0) > > vmax = a[(imax, range(5))] > > > I don't generally like overloading return values, the function starts to > lose its definition and becomes a bit baroque where simply changing a > keyword value can destroy the viability of the following code. Agreed. Seems like the only justification is if you get multiple results from one calculation but only rarely want the extra values. It doesn't make sense to always return them, but it's also not worth making a totally different function. > But I can see the utility of what you want. Hmm, this problem is not unique to argmax. > Maybe what we need is a general way to extract values, something like > > extract(a, imax, axis=0) > > to go along with all the single axis functions. Yes, I think that would be easier to remember. It should also work for the axis=None case. imax = a.argmax(axis=None) v = extract(a, imax, axis=None) --Bill From charlesr.harris at gmail.com Tue Sep 19 00:05:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 22:05:42 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Bill Baxter wrote: > > On 9/19/06, Charles R Harris wrote: > > On 9/18/06, Bill Baxter wrote: > > > I find myself often wanting both the max and the argmax of an array. > > > (And same for the other arg* functions) > > > > You have to do something like > > > a = rand(10,5) > > > imax = a.argmax(axis=0) > > > vmax = a[(imax, range(5))] > > > > > I don't generally like overloading return values, the function starts to > > lose its definition and becomes a bit baroque where simply changing a > > keyword value can destroy the viability of the following code. > > Agreed. Seems like the only justification is if you get multiple > results from one calculation but only rarely want the extra values. > It doesn't make sense to always return them, but it's also not worth > making a totally different function. > > > > But I can see the utility of what you want. Hmm, this problem is not > unique to argmax. > > Maybe what we need is a general way to extract values, something like > > > > extract(a, imax, axis=0) > > > > to go along with all the single axis functions. > > Yes, I think that would be easier to remember. > > It should also work for the axis=None case. > imax = a.argmax(axis=None) > v = extract(a, imax, axis=None) It shouldn't be too difficult to jig something up given all the example code. I can do that, but I would like more input first. The questions I have are these. 1) Should it be done? 2) Should it be a method? (functions being somewhat deprecated) 3) What name should it have? I think Travis will have to weigh in on this. IIRC, he felt that the number of methods was getting out of hand. --Bill Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 01:12:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 18 Sep 2006 23:12:11 -0600 Subject: [Numpy-discussion] ***[Possible UCE]*** Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <450F7C2B.5030409@ieee.org> Dr. Seth Olsen wrote: > > Hi MMTKers and NUMPYers, > > Bill's answer to my inquiry about the problem I'm having with > Collection.findTransformation() (and also, incidentally, with the > dgesvd call in Subspace.getBasis(), has convinced me that I can no > long use MMTK without changing some of the code over to numpy. I have > already been able to determine that invoking > numpy.oldnumeric.alter_code1.convertall() over the MMTK directory tree > is not the answer. Why not? It should be. That is the recommended way to begin porting code. If we need to improve alter_code, we cannot do it unless we receive bug reports. Please tell us what difficulty you had. -Travis From charlesr.harris at gmail.com Tue Sep 19 01:18:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 18 Sep 2006 23:18:52 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: On 9/18/06, Charles R Harris wrote: > > > > On 9/18/06, Bill Baxter wrote: > > > > On 9/19/06, Charles R Harris wrote: > > > On 9/18/06, Bill Baxter wrote: > > > > I find myself often wanting both the max and the argmax of an array. > > > > (And same for the other arg* functions) > > > > > > You have to do something like > > > > a = rand(10,5) > > > > imax = a.argmax(axis=0) > > > > vmax = a[(imax, range(5))] > > > > > > > I don't generally like overloading return values, the function starts > > to > > > lose its definition and becomes a bit baroque where simply changing a > > > keyword value can destroy the viability of the following code. > > > > Agreed. Seems like the only justification is if you get multiple > > results from one calculation but only rarely want the extra values. > > It doesn't make sense to always return them, but it's also not worth > > making a totally different function. > > > > > > > But I can see the utility of what you want. Hmm, this problem is not > > unique to argmax. > > > Maybe what we need is a general way to extract values, something like > > > > > > extract(a, imax, axis=0) > > > > > > to go along with all the single axis functions. > > > > Yes, I think that would be easier to remember. > > > > It should also work for the axis=None case. > > imax = a.argmax(axis=None) > > v = extract(a, imax, axis=None) > > > It shouldn't be too difficult to jig something up given all the example > code. I can do that, but I would like more input first. The questions I have > are these. > > 1) Should it be done? > 2) Should it be a method? (functions being somewhat deprecated) > 3) What name should it have? > > I think Travis will have to weigh in on this. IIRC, he felt that the > number of methods was getting out of hand. > > --Bill > > > Chuck > I note that argsort also produces indexes that are hard to use in the ndim>1 case. The two problems aren't quite equivalent because argsort maintains ndim while argmax reduces ndim by one, but it would be nice if there was something that would work for both. Using imax[...,newaxis] would make the shapes compatible for input but then the output would need to lose the newaxis on return. Hmmm. These are both examples of an indirect indexing problem where the indices on the specified axis are a function of the indices on the other axis. Using the other indices in the argmax case yields a scalar index while in the argsort case it yields a 1d array that can be used for fancy indexing. I'm just floating around here trying to think of a consistent way to regard these things. Ummm, and I think this could work. In fact, the arrays could be even deeper and fancy indexing on the specified axis would produce what was essentially an array of arrays. This is sort of the photo-negative version of take. Apropos the overloaded return types, I think that that is precisely what makes it tricky to use functions that may return either copies or views. The results should really be marked read only because otherwise unexpected results can arise. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 02:43:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 00:43:15 -0600 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <450F9183.6040405@ieee.org> Dr. Seth Olsen wrote: > > Hi Bill, > > MMTK has not made the conversion over to the new numpy module. It is > built against the old Numeric code, and the word from its developers > is that changing to numpy cannot be a priority now. > My suggestion is to *kindly* put pressure on them. I've spent at least a hundred hours making it easy for people to port Numeric and Numarray-built code to NumPy. Because of this, I'm a little bit frustrated by this kind of response. I understand it will take time for people to migrate, but it really does not take that long to port code to use NumPy. I've offered to do it for any open source code. In fact, I just spent 30 minutes and ported both Scientific Python and MMTK to use numpy. I'll send you a patch if you want. It is true, that the result needs to be better tested, but I'm confident that any errors which might remain in the compatibility layer will be easily fixable (and we need people who are willing to do the tests to fix them). I'd rather not do this, but if necessary we can easily create an SVN tree of third-party packages ported to use NumPy if the package-owners are not willing to do it. Keeping Numeric packages around except for legacy systems will only make things harder. I'll repeat the same offer I've made before: I will gladly give my book and my help to any open source library author who will make porting to NumPy a priority for their package. Note, however, my (free) ports to use NumPy do not use any "numerix-style" layer. The library is converted to work with NumPy alone. In other words, I won't spend any more "spare" time supporting 3 array packages. Best regards, -Travis From seth.olsen at gmail.com Tue Sep 19 03:26:08 2006 From: seth.olsen at gmail.com (Dr. Seth Olsen) Date: Tue, 19 Sep 2006 17:26:08 +1000 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: <450F9183.6040405@ieee.org> References: <450F9183.6040405@ieee.org> Message-ID: Hi Travis, I would very happily accept the Scientific and MMTK patches. Thank you very much for the offer. I hadn't thought much about it until very recently, when the advent of a new IT structure in our laboratory forced me to upgrade my existing software. It was then that it became obvious that the LAPACK routines called by Numeric and MMTK were refusing to work. I've spent the day trying to wrestle with the problem (I am by no means an expert). I finally decided to get around the problems by altering the problem-solving routines in MMTK by using routines from numpy and then immediately applying a=Numeric.array( a.tolist()), which has stopped my script from gagging (although I have still to demonstrate to myself that everything is working). The uses to which I apply MMTK usually mean that the matrices in question are pretty small, so I don't have to worry too much about speed. I was suprised to note, however, that a straightforward application of F2PY to some fresh downloaded LAPACK F77 code did not work, and sent my machine into a similar endless whirr as I had seen at the beginning of the day. Apparently there's something sinister going on... Cheers, Seth On 9/19/06, Travis Oliphant wrote: > > Dr. Seth Olsen wrote: > > > > Hi Bill, > > > > MMTK has not made the conversion over to the new numpy module. It is > > built against the old Numeric code, and the word from its developers > > is that changing to numpy cannot be a priority now. > > > My suggestion is to *kindly* put pressure on them. > > I've spent at least a hundred hours making it easy for people to port > Numeric and Numarray-built code to NumPy. Because of this, I'm a > little bit frustrated by this kind of response. I understand it will > take time for people to migrate, but it really does not take that long > to port code to use NumPy. > > I've offered to do it for any open source code. In fact, I just spent > 30 minutes and ported both Scientific Python and MMTK to use numpy. > I'll send you a patch if you want. It is true, that the result needs > to be better tested, but I'm confident that any errors which might > remain in the compatibility layer will be easily fixable (and we need > people who are willing to do the tests to fix them). > > I'd rather not do this, but if necessary we can easily create an SVN > tree of third-party packages ported to use NumPy if the package-owners > are not willing to do it. Keeping Numeric packages around except for > legacy systems will only make things harder. > > I'll repeat the same offer I've made before: I will gladly give my book > and my help to any open source library author who will make porting to > NumPy a priority for their package. Note, however, my (free) ports to > use NumPy do not use any "numerix-style" layer. The library is > converted to work with NumPy alone. In other words, I won't spend any > more "spare" time supporting 3 array packages. > > Best regards, > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms Dr Seth Olsen, PhD Postdoctoral Fellow, Biomolecular Modeling Group Centre for Computational Molecular Science Australian Institute for Bioengineering and Nanotechnology (Bldg. 75) The University of Queensland Qld 4072, Brisbane, Australia tel (617) 3346 3976 fax (617) 33654623 email: s.olsen1 at uq.edu.au Web: www.ccms.uq.edu.au ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms The opinions expressed here are my own and do not reflect the positions of the University of Queensland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 04:44:39 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 02:44:39 -0600 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: <450F9183.6040405@ieee.org> Message-ID: <450FADF7.5020708@ieee.org> Dr. Seth Olsen wrote: > > Hi Travis, > > I would very happily accept the Scientific and MMTK patches. Thank > you very much for the offer. Look at http://www.scipy.org/Porting_to_NumPy for information (and patches) on how to convert ScientificPython and MMTK to use NumPy. I managed to get the codes to build and install but they are not tested. Any problems you encounter would be useful to know about. You can patch the code by changing to the top-level directory and entering patch -p1 < patch_file If it works for you, please email the developers and let them know how easy it can be (hopefully). Best, -Travis From mg.mailing-list at laposte.net Tue Sep 19 04:48:37 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 10:48:37 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450EEC52.30906@ieee.org> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> Message-ID: <450FAEE5.7040305@laposte.net> Hi Travis, First, thanks for your answer. I just download the source of Numpy from the SVN and recompile the package from Python-2.5 and 2.6a (from svn)... and it still crashes. So I do not investigate the compilation on Windows. So, may I have to wait the release 1.0rc0 of Numpy? About the migration of our python scripts, we can keep contact to help you to test the transcription tool. It is a good idea. About the Numpy API, we prefer to use the native C-Numpy-API in our code. I think it will offer more flexibilities in our generic C++ framework, particularly in our main object (named Field) which use the aliasing memory feature. Nevertheless, the first step of our migration could be using the numarray interface before using the native interface. It could be another test for you. Finally, your description of what i call "aliasing memory" is good. So it is a very good point for us that it is supported. Best regards, - Mathieu Travis Oliphant wrote: > mg wrote: > >> Hi all, >> >> I am doing a feseability study to migrate our Python based FEM >> applications from Numarray to Numpy. >> >> First, I tried to install Numpy from Python-2.4 on linux-x86, >> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >> >> Second, I try to do the same think on windows-x86 without success. So my >> first question is: is Numpy-1.0b5 has been tested and is supported on >> Windows? >> >> > Yes, it should work. Builds for windows were provided. But, perhaps > there are configuration issues for your system that we are not handling > correctly. > > >> Third, I tried to install Numpy from Python-2.5, which is our standard >> Python, on linux-x86... and the compilation stopped during the >> compilation of core/src/multiarraymodule.c. So my second question is: is >> there a workaround or is the porting to Python2.5 is yet schedule? >> >> > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. > Look for NumPy 1.0rc1 to come out soon. > >> My third question is: is the tool to migrate the numarray based Python >> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >> >> > It needs more testing. It would be great if you could help us find and > fix bugs in it. I don't have a lot of numarray code to test. > >> We have created a lot of bindings in order to pilote our generic-C++ >> framework with Python scripts. So, about the Numpy API, is it widely >> different than the Numarray API? (We will order the Numpy Guide too.) >> >> > It is more similar to the Numeric C-API. However, the numarray C-API is > completely supported by including numpy/libnumarray.h so you should be > able to convert your C code very easily. Any problems encountered > should be noted and we'll get them fixed. > >> To not duplicate large numerical memory arrays, Numarray allows to >> aliasing the memory of some bindings with arrays from Numarray, and we >> have used this feature intensively. So, I wonder if it is currently >> supported (or even scheduled)? >> > I'm pretty sure the answer is yes (because the Numarray C-API is > supported), though I'm not exactly sure what you mean. Do you mean that > you have memory created in the C/C++ framework and then you have an > array use that memory for it's data area? If that is what you mean, > then the answer is definitely yes. > > > -Travis > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From mg.mailing-list at laposte.net Tue Sep 19 04:58:57 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 10:58:57 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450EEC52.30906@ieee.org> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> Message-ID: <450FB151.30800@laposte.net> I forget a last question, We currently advise a High Performance Platform for our customers (Opteron cluster) where our applications are linked with GOTO blas library. I found in the Numpy distribution the file site.cfg, used to specify the BLAS to used at the link time. (It could be an interesting feature for us.) I noted some configuration variables have names depending on the name of the chosen BLAS library. So, my question is: is GOTO is supported or is it easy for us to add it? I wonder too if the string "[atlas]" written in site.cfg is used or not ? thanks, Mathieu. Travis Oliphant wrote: > mg wrote: > >> Hi all, >> >> I am doing a feseability study to migrate our Python based FEM >> applications from Numarray to Numpy. >> >> First, I tried to install Numpy from Python-2.4 on linux-x86, >> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >> >> Second, I try to do the same think on windows-x86 without success. So my >> first question is: is Numpy-1.0b5 has been tested and is supported on >> Windows? >> >> > Yes, it should work. Builds for windows were provided. But, perhaps > there are configuration issues for your system that we are not handling > correctly. > > >> Third, I tried to install Numpy from Python-2.5, which is our standard >> Python, on linux-x86... and the compilation stopped during the >> compilation of core/src/multiarraymodule.c. So my second question is: is >> there a workaround or is the porting to Python2.5 is yet schedule? >> >> > There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. > Look for NumPy 1.0rc1 to come out soon. > >> My third question is: is the tool to migrate the numarray based Python >> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >> >> > It needs more testing. It would be great if you could help us find and > fix bugs in it. I don't have a lot of numarray code to test. > >> We have created a lot of bindings in order to pilote our generic-C++ >> framework with Python scripts. So, about the Numpy API, is it widely >> different than the Numarray API? (We will order the Numpy Guide too.) >> >> > It is more similar to the Numeric C-API. However, the numarray C-API is > completely supported by including numpy/libnumarray.h so you should be > able to convert your C code very easily. Any problems encountered > should be noted and we'll get them fixed. > >> To not duplicate large numerical memory arrays, Numarray allows to >> aliasing the memory of some bindings with arrays from Numarray, and we >> have used this feature intensively. So, I wonder if it is currently >> supported (or even scheduled)? >> > I'm pretty sure the answer is yes (because the Numarray C-API is > supported), though I'm not exactly sure what you mean. Do you mean that > you have memory created in the C/C++ framework and then you have an > array use that memory for it's data area? If that is what you mean, > then the answer is definitely yes. > > > -Travis > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From nwagner at iam.uni-stuttgart.de Tue Sep 19 05:06:10 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 19 Sep 2006 11:06:10 +0200 Subject: [Numpy-discussion] feasability study to migrate from numarray to numpy In-Reply-To: <450FB151.30800@laposte.net> References: <450ECF83.9020103@laposte.net> <450EEC52.30906@ieee.org> <450FB151.30800@laposte.net> Message-ID: <450FB302.60508@iam.uni-stuttgart.de> mg wrote: > I forget a last question, > > We currently advise a High Performance Platform for our customers > (Opteron cluster) where our applications are linked with GOTO blas library. > > I found in the Numpy distribution the file site.cfg, used to specify the > BLAS to used at the link time. (It could be an interesting feature for > us.) I noted some configuration variables have names depending on the > name of the chosen BLAS library. So, my question is: is GOTO is > supported or is it easy for us to add it? I wonder too if the string > "[atlas]" written in site.cfg is used or not ? > > thanks, > Mathieu. > There is a ticket concerning your inquiry. http://projects.scipy.org/scipy/numpy/ticket/73 Nils > Travis Oliphant wrote: > >> mg wrote: >> >> >>> Hi all, >>> >>> I am doing a feseability study to migrate our Python based FEM >>> applications from Numarray to Numpy. >>> >>> First, I tried to install Numpy from Python-2.4 on linux-x86, >>> linux-86-64bit. So, all work fine. Great! Moreover, I change easily the >>> BLAS linked libraries. I tried with ATLAS and GOTO. Great again! >>> >>> Second, I try to do the same think on windows-x86 without success. So my >>> first question is: is Numpy-1.0b5 has been tested and is supported on >>> Windows? >>> >>> >>> >> Yes, it should work. Builds for windows were provided. But, perhaps >> there are configuration issues for your system that we are not handling >> correctly. >> >> >> >>> Third, I tried to install Numpy from Python-2.5, which is our standard >>> Python, on linux-x86... and the compilation stopped during the >>> compilation of core/src/multiarraymodule.c. So my second question is: is >>> there a workaround or is the porting to Python2.5 is yet schedule? >>> >>> >>> >> There was a problem with Python 2.5 and NumPy 1.0 that is fixed in SVN. >> Look for NumPy 1.0rc1 to come out soon. >> >> >>> My third question is: is the tool to migrate the numarray based Python >>> scripts (numpy.numarray.alter_code1) work fine? (I suppose yes...) >>> >>> >>> >> It needs more testing. It would be great if you could help us find and >> fix bugs in it. I don't have a lot of numarray code to test. >> >> >>> We have created a lot of bindings in order to pilote our generic-C++ >>> framework with Python scripts. So, about the Numpy API, is it widely >>> different than the Numarray API? (We will order the Numpy Guide too.) >>> >>> >>> >> It is more similar to the Numeric C-API. However, the numarray C-API is >> completely supported by including numpy/libnumarray.h so you should be >> able to convert your C code very easily. Any problems encountered >> should be noted and we'll get them fixed. >> >> >>> To not duplicate large numerical memory arrays, Numarray allows to >>> aliasing the memory of some bindings with arrays from Numarray, and we >>> have used this feature intensively. So, I wonder if it is currently >>> supported (or even scheduled)? >>> >>> >> I'm pretty sure the answer is yes (because the Numarray C-API is >> supported), though I'm not exactly sure what you mean. Do you mean that >> you have memory created in the C/C++ framework and then you have an >> array use that memory for it's data area? If that is what you mean, >> then the answer is definitely yes. >> >> >> -Travis >> >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From william.grant at ubuntu-au.org Tue Sep 19 05:23:17 2006 From: william.grant at ubuntu-au.org (William Grant) Date: Tue, 19 Sep 2006 19:23:17 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 Message-ID: <450FB705.7010002@ubuntu-au.org> Hi, I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it currently cannot happen as numpy doesn't build with Python 2.5. I note that changeset 3109 (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to give 2.5 compatibility, but it is those bits which seem to break it. The end of the build log with the errors can be found at http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. Has anybody got any ideas on how to fix this? A number of Ubuntu users want scipy 0.5.1, but that can't happen while it won't build with Python 2.5. Thanks, William Grant. From mg.mailing-list at laposte.net Tue Sep 19 07:26:00 2006 From: mg.mailing-list at laposte.net (mg) Date: Tue, 19 Sep 2006 13:26:00 +0200 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FB705.7010002@ubuntu-au.org> References: <450FB705.7010002@ubuntu-au.org> Message-ID: <450FD3C8.90308@laposte.net> Hi, I have the same problem with Python-2.5 and Python-2.6a with which i can not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of Scipy is based on one of these Numpy.) I posted my problem yesterday. The answer from Travis was it will be corrected in Numpy-1.0rc1 which will come soon. regards, Mathieu. William Grant wrote: > Hi, > > I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it > currently cannot happen as numpy doesn't build with Python 2.5. I note > that changeset 3109 > (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to > give 2.5 compatibility, but it is those bits which seem to break it. The > end of the build log with the errors can be found at > http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. > > Has anybody got any ideas on how to fix this? A number of Ubuntu users > want scipy 0.5.1, but that can't happen while it won't build with Python > 2.5. > > Thanks, > > William Grant. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From william.grant at ubuntu.com.au Tue Sep 19 07:34:20 2006 From: william.grant at ubuntu.com.au (William Grant) Date: Tue, 19 Sep 2006 21:34:20 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD3C8.90308@laposte.net> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> Message-ID: <450FD5BC.9040208@ubuntu.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 mg wrote: > Hi, > > I have the same problem with Python-2.5 and Python-2.6a with which i can > not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of > Scipy is based on one of these Numpy.) > I posted my problem yesterday. The answer from Travis was it will be > corrected in Numpy-1.0rc1 which will come soon. > > regards, > Mathieu. Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or after the 28th? Thanks, William Grant. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFFD9W8rbS+qZ0dQCURAk+7AKDkQSnrzs6SIYBmJc3QW9j48Zpu7QCePsqU AnLTk9YQzYAD/Ps/3JIBEU8= =QU5m -----END PGP SIGNATURE----- From pearu.peterson at gmail.com Tue Sep 19 09:19:40 2006 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 19 Sep 2006 16:19:40 +0300 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD5BC.9040208@ubuntu.com.au> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> <450FD5BC.9040208@ubuntu.com.au> Message-ID: <2c406a760609190619x10376f8fv7d983f3b6255cb74@mail.gmail.com> On 9/19/06, William Grant wrote: > mg wrote: > > Hi, > > > > I have the same problem with Python-2.5 and Python-2.6a with which i can > > not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of > > Scipy is based on one of these Numpy.) > > I posted my problem yesterday. The answer from Travis was it will be > > corrected in Numpy-1.0rc1 which will come soon. > > > > regards, > > Mathieu. > > Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or > after the 28th? I have the corresponding fixed compiler errors in numpy svn. Regards, Pearu From luszczek at cs.utk.edu Tue Sep 19 10:21:35 2006 From: luszczek at cs.utk.edu (Piotr Luszczek S) Date: Tue, 19 Sep 2006 10:21:35 -0400 Subject: [Numpy-discussion] Fwd: Collection.findTransformation() never stops In-Reply-To: References: <450F9183.6040405@ieee.org> Message-ID: <200609191021.36093.luszczek@cs.utk.edu> Seth, this problem is most likely related to LAPACK's SLAMCH and DLAMCH routines. If these routines get translated with f2c the compiler can optimize things away and create infinite loops. This functions get called when you start using singular and eigenvalue routines. It's because Fortran 77 didn't have a way to specify "volatile" storage. In any case, you can avoid the problem by linking different SLAMCH and DLAMCH routines. I'm attaching two possible impementations that use C99 and Fortran 95. Probably you'll have to ensure that name mangling occurs correctly (rename slamch to slamch_ and dlamch to dlamch_). Piotr On Tuesday 19 September 2006 03:26, Dr. Seth Olsen wrote: > Hi Travis, > > I would very happily accept the Scientific and MMTK patches. Thank > you very much for the offer. > > I hadn't thought much about it until very recently, when the advent > of a new IT structure in our laboratory forced me to upgrade my > existing software. It was then that it became obvious that the LAPACK > routines called by Numeric and MMTK were refusing to work. I've > spent the day trying to wrestle with the problem (I am by no means an > expert). I finally decided to get around the problems by altering > the problem-solving routines in MMTK by using routines from numpy and > then immediately applying a=Numeric.array( a.tolist()), which has > stopped my script from gagging (although I have still to demonstrate > to myself that everything is working). The uses to which I apply > MMTK usually mean that the matrices in question are pretty small, so > I don't have to worry too much about speed. > > I was suprised to note, however, that a straightforward application > of F2PY to some fresh downloaded LAPACK F77 code did not work, and > sent my machine into a similar endless whirr as I had seen at the > beginning of the day. Apparently there's something sinister going > on... > > Cheers, > > Seth > > On 9/19/06, Travis Oliphant wrote: > > Dr. Seth Olsen wrote: > > > Hi Bill, > > > > > > MMTK has not made the conversion over to the new numpy module. > > > It is built against the old Numeric code, and the word from its > > > developers is that changing to numpy cannot be a priority now. > > > > My suggestion is to *kindly* put pressure on them. > > > > I've spent at least a hundred hours making it easy for people to > > port Numeric and Numarray-built code to NumPy. Because of this, > > I'm a little bit frustrated by this kind of response. I understand > > it will take time for people to migrate, but it really does not > > take that long to port code to use NumPy. > > > > I've offered to do it for any open source code. In fact, I just > > spent 30 minutes and ported both Scientific Python and MMTK to use > > numpy. I'll send you a patch if you want. It is true, that the > > result needs to be better tested, but I'm confident that any errors > > which might remain in the compatibility layer will be easily > > fixable (and we need people who are willing to do the tests to fix > > them). > > > > I'd rather not do this, but if necessary we can easily create an > > SVN tree of third-party packages ported to use NumPy if the > > package-owners are not willing to do it. Keeping Numeric packages > > around except for legacy systems will only make things harder. > > > > I'll repeat the same offer I've made before: I will gladly give my > > book and my help to any open source library author who will make > > porting to NumPy a priority for their package. Note, however, my > > (free) ports to use NumPy do not use any "numerix-style" layer. > > The library is converted to work with NumPy alone. In other words, > > I won't spend any more "spare" time supporting 3 array packages. > > > > Best regards, > > > > -Travis > > > > > > ------------------------------------------------------------------- > >------ Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to > > share your > > opinions on IT & business topics through brief surveys -- and earn > > cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID= > >DEVDEV _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: lamch.c Type: text/x-csrc Size: 2371 bytes Desc: not available URL: -------------- next part -------------- program lamch ! implicit none integer i external tlamch do i = 1, 2 call tlamch('B', i) call tlamch('E', i) call tlamch('L', i) call tlamch('M', i) call tlamch('N', i) call tlamch('P', i) call tlamch('R', i) call tlamch('O', i) call tlamch('S', i) call tlamch('U', i) end do end program lamch subroutine tlamch(cmach, idx) character, intent(in) :: cmach integer, intent(in) :: idx double precision x real slamch double precision dlamch external dlamch, slamch intrinsic dble if (idx .eq. 1) then x = dble(slamch(cmach)) write( *, '(a e25.15)') cmach, x else write( *, '(a e25.15)') cmach, dlamch(cmach) end if end subroutine tlamch real function slamch(cmach) character, intent(in) :: cmach real zero, one parameter (zero = 0.0e+0, one = 1.0e0) real sfmin, small intrinsic digits, epsilon, radix, real if (cmach .eq. 'B' .or. cmach .eq. 'b') then slamch = radix(zero) else if (cmach .eq. 'E' .or. cmach .eq. 'e') then slamch = epsilon(zero) else if (cmach .eq. 'L' .or. cmach .eq. 'l') then slamch = maxexponent(zero) else if (cmach .eq. 'M' .or. cmach .eq. 'm') then slamch = minexponent(zero) else if (cmach .eq. 'N' .or. cmach .eq. 'n') then slamch = real(digits(zero)) else if (cmach .eq. 'O' .or. cmach .eq. 'o') then slamch = huge(zero) else if (cmach .eq. 'P' .or. cmach .eq. 'p') then slamch = epsilon(zero) * radix(zero) else if (cmach .eq. 'R' .or. cmach .eq. 'r') then slamch = one else if (cmach .eq. 'S' .or. cmach .eq. 's') then sfmin = tiny(zero) small = one / huge(zero) if (small .ge. sfmin) sfmin = small * (one+epsilon(zero)) slamch = sfmin else if (cmach .eq. 'U' .or. cmach .eq. 'u') then slamch = tiny(zero) else slamch = zero end if return end function slamch double precision function dlamch(cmach) character, intent(in) :: cmach double precision zero, one parameter (zero = 0.0d+0, one = 1.0d0) double precision :: small, sfmin intrinsic digits, dble, epsilon, huge, maxexponent, minexponent intrinsic radix, tiny if (cmach .eq. 'B' .or. cmach .eq. 'b') then dlamch = radix(zero) else if (cmach .eq. 'E' .or. cmach .eq. 'e') then dlamch = epsilon(zero) else if (cmach .eq. 'L' .or. cmach .eq. 'l') then dlamch = maxexponent(zero) else if (cmach .eq. 'M' .or. cmach .eq. 'm') then dlamch = minexponent(zero) else if (cmach .eq. 'N' .or. cmach .eq. 'n') then dlamch = dble(digits(zero)) else if (cmach .eq. 'O' .or. cmach .eq. 'o') then dlamch = huge(zero) else if (cmach .eq. 'P' .or. cmach .eq. 'p') then dlamch = epsilon(zero) * radix(zero) else if (cmach .eq. 'R' .or. cmach .eq. 'r') then dlamch = one else if (cmach .eq. 'S' .or. cmach .eq. 's') then sfmin = tiny(zero) small = one / huge(zero) if (small .ge. sfmin) sfmin = small * (one+epsilon(zero)) dlamch = sfmin else if (cmach .eq. 'U' .or. cmach .eq. 'u') then dlamch = tiny(zero) else dlamch = zero end if return end function dlamch From martin.wiechert at gmx.de Tue Sep 19 11:34:57 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Tue, 19 Sep 2006 17:34:57 +0200 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? Message-ID: <200609191734.57899.martin.wiechert@gmx.de> Hi list, Please forgive my ignorance: Is there any difference between npy_intp and size_t. Aren't both "ints just big enough to be safe with pointer arithmetics even on 64 bit architectures?". Thanks, Martin. From konrad.hinsen at laposte.net Tue Sep 19 12:05:55 2006 From: konrad.hinsen at laposte.net (Konrad Hinsen) Date: Tue, 19 Sep 2006 18:05:55 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: On Sep 19, 2006, at 5:11, Dr. Seth Olsen wrote: > Bill's answer to my inquiry about the problem I'm having with > Collection.findTransformation() (and also, incidentally, with the > dgesvd call in Subspace.getBasis(), has convinced me that I can no > long use MMTK without changing some of the code over to numpy. I > have already been able to determine that invoking > numpy.oldnumeric.alter_code1.convertall() over the MMTK directory > tree is not the answer. Has anyone on either of these lists ever > tried this before and, if so, can it be done (without destroying my > sanity)? Adapting MMTK to NumPy is likely to be a major effort, in particular for the C modules. A lot ot testing would also be required. If anyone wants to tackle this, I'd be happy to see the results. Personally, I don't expect to find time for this soon. MMTK works fine with Numeric 23.x (and probably many other versions), so I don't see a pressing need to change to NumPy. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From faltet at carabos.com Tue Sep 19 12:45:37 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 19 Sep 2006 18:45:37 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: <200609191845.38788.faltet@carabos.com> A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > I note that argsort also produces indexes that are hard to use in the > ndim>1 case. Perhaps it is worth to mention here that I've always liked to have a sort() and argsort() functionality merged in one shot function (or method). Having both separated implies two sorts, while if I'd have such a combo available, the resulting operation would take quite less time. One can easily stablish kind of a pattern for situations where this could happen: in all the 'arg*()' functions. Perhaps I'm wrong in my count, but there appear to be only three of such functions in NumPy, namely, argmax, argmin and argsort. Adding three additional 'combos' doesn't seem a lot to my mind, but it can be just 'too much' for more common sense minds. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Tue Sep 19 12:48:31 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 19 Sep 2006 09:48:31 -0700 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <45101F5F.6080209@noaa.gov> Konrad Hinsen wrote: > MMTK works fine with Numeric 23.x (and probably many other versions), > so I don't see a pressing need to change to NumPy. Pressing is in the eye of the beholder. However: I don't think we should underestimate the negative impact of the Numeric/numarray split on the usability and perception of the community. Also the impact on how much work has been done to accommodate it. If you consider matplotlib alone: Effort to write the "numerix" compatibility layer Effort to support problems people have with incompatibility Effort to build binaries that support all the packages Effort to do extra testing to determine whether a given problem is caused by a different numerix back-end. This really adds up! Because of this, it IS a pressing need to get as much stuff converted as possible. In addition, as I understand it, MMTK was NOT working fine for the OP. this is also the case for a variety of other codes that depend on Numeric. As robust as they (and Numeric) might be, when you need to run something on a new platform (OS-X - Intel comes to mind), or use a new LAPACK, or whatever, there are going to be (and have been) issues that need to be addressed. No one is maintaining Numeric, so it makes much more sense to put your effort into porting to numpy, rather than trying to fix or work around Numeric issues. This is also a great time to get help with porting issues -- see Travis' recent message. -Chris PS: this really highlights the strength of having good unit tests: as far as i can tell, it's really not that much work to do the port -- the work is in the testing. Comprehensive units tests would make that part trivial too. -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Tue Sep 19 13:21:19 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 11:21:19 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191845.38788.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> Message-ID: On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > I note that argsort also produces indexes that are hard to use in the > > ndim>1 case. > > Perhaps it is worth to mention here that I've always liked to have a > sort() > and argsort() functionality merged in one shot function (or method). > Having > both separated implies two sorts, while if I'd have such a combo > available, > the resulting operation would take quite less time. Do you want both the indexes and index sorted array returned, or just sort the array using indexes, i.e., something like a.sort(kind="quicksort", method="indirect") IIRC, the old numeric argsort actually did something like this under the covers but discarded the sorted array. One can easily stablish kind of a pattern for situations where this could > happen: in all the 'arg*()' functions. Perhaps I'm wrong in my count, but > there appear to be only three of such functions in NumPy, namely, argmax, > argmin and argsort. Adding three additional 'combos' doesn't seem a lot to > my > mind, but it can be just 'too much' for more common sense minds. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Tue Sep 19 13:23:57 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 10:23:57 -0700 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? Message-ID: <200609191023.57656.haase@msg.ucsf.edu> Hi, What are the possible values of arr.dtype.kind ? It seems that signed and unsigned are considered to be the same "kind" >>> arr=N.arange(10,dtype=N.uint) >>> arr.dtype.kind 'i' >>> arr.dtype.itemsize 8 (OK - this is just showing off our amd64 linux ;-) ) How can I distinguish signed from unsigned without having to list all possible cases explicitly ? Thanks, Sebastian Haase From charlesr.harris at gmail.com Tue Sep 19 13:28:24 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 11:28:24 -0600 Subject: [Numpy-discussion] Resolution of tickets. Message-ID: Question, How does one mark a ticket as fixed? I don't see this field in the ticket views I get, is there a list of accepted fixers? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Tue Sep 19 13:39:23 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 19 Sep 2006 19:39:23 +0200 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191023.57656.haase@msg.ucsf.edu> Message-ID: Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sebastian Haase > Sent: 19 September 2006 19:24 > To: Discussion of Numerical Python > Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? > > Hi, > What are the possible values of > arr.dtype.kind ? > > It seems that signed and unsigned are considered to be the same "kind" > >>> arr=N.arange(10,dtype=N.uint) > >>> arr.dtype.kind > 'i' > >>> arr.dtype.itemsize > 8 > (OK - this is just showing off our amd64 linux ;-) ) > > How can I distinguish signed from unsigned without having to list all > possible > cases explicitly ? How about sctypes? In [16]: numpy.sctypes.keys() Out[17]: ['int', 'float', 'uint', 'complex', 'others'] So this should work: sometype in numpy.sctypes['uint'] sometype in numpy.sctypes['int'] Cheers, Albert From faltet at carabos.com Tue Sep 19 13:41:55 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 19 Sep 2006 19:41:55 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191845.38788.faltet@carabos.com> Message-ID: <200609191941.56395.faltet@carabos.com> A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > > I note that argsort also produces indexes that are hard to use in the > > > ndim>1 case. > > > > Perhaps it is worth to mention here that I've always liked to have a > > sort() > > and argsort() functionality merged in one shot function (or method). > > Having > > both separated implies two sorts, while if I'd have such a combo > > available, > > the resulting operation would take quite less time. > > Do you want both the indexes and index sorted array returned, or just sort > the array using indexes, i.e., something like > > a.sort(kind="quicksort", method="indirect") Uh, I don't understand what do you mean by "sort the array using indexes", sorry. What I'd like is to get the indexes and the sorted array returned. Or, another option could be getting the sorted indexes returned and a sort of the array done in-place (which is surely faster). Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Tue Sep 19 13:42:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 12:42:50 -0500 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: Message-ID: Charles R Harris wrote: > Question, > > How does one mark a ticket as fixed? I don't see this field in the > ticket views I get, is there a list of accepted fixers? Indeed there is. charris is on that list. charris208 is not. Forget your password? :-) Talk to me offlist about which username you want to continue with. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Sep 19 14:36:31 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:36:31 -0600 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FD5BC.9040208@ubuntu.com.au> References: <450FB705.7010002@ubuntu-au.org> <450FD3C8.90308@laposte.net> <450FD5BC.9040208@ubuntu.com.au> Message-ID: <451038AF.3010806@ieee.org> William Grant wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > mg wrote: > >> Hi, >> >> I have the same problem with Python-2.5 and Python-2.6a with which i can >> not compile Numpy-1.0b5 and Numpy from svn. (I suppose your version of >> Scipy is based on one of these Numpy.) >> I posted my problem yesterday. The answer from Travis was it will be >> corrected in Numpy-1.0rc1 which will come soon. >> >> regards, >> Mathieu. >> > > Yes, numpy 1.0b5 is the one... Approximately how soon is soon? Before or > after the 28th? > Before. It was supposed to come out this weekend. It will be today or tomorrow. -Travis From oliphant.travis at ieee.org Tue Sep 19 14:37:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:37:40 -0600 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? In-Reply-To: <200609191734.57899.martin.wiechert@gmx.de> References: <200609191734.57899.martin.wiechert@gmx.de> Message-ID: <451038F4.7090704@ieee.org> Martin Wiechert wrote: > Hi list, > > Please forgive my ignorance: Is there any difference between npy_intp and > size_t. Aren't both "ints just big enough to be safe with pointer arithmetics > even on 64 bit architectures?". > size_t is unsigned npy_intp is signed It is basically the same as Py_ssize_t (which is not available until Python 2.5). -Travis From charlesr.harris at gmail.com Tue Sep 19 14:42:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 12:42:23 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191941.56395.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/19/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > On 9/19/06, Francesc Altet wrote: > > > A Dimarts 19 Setembre 2006 07:18, Charles R Harris va escriure: > > > > I note that argsort also produces indexes that are hard to use in > the > > > > ndim>1 case. > > > > > > Perhaps it is worth to mention here that I've always liked to have a > > > sort() > > > and argsort() functionality merged in one shot function (or method). > > > Having > > > both separated implies two sorts, while if I'd have such a combo > > > available, > > > the resulting operation would take quite less time. > > > > Do you want both the indexes and index sorted array returned, or just > sort > > the array using indexes, i.e., something like > > > > a.sort(kind="quicksort", method="indirect") > > Uh, I don't understand what do you mean by "sort the array using indexes", > sorry. > > What I'd like is to get the indexes and the sorted array returned. Or, > another > option could be getting the sorted indexes returned and a sort of the > array > done in-place (which is surely faster). Don't know about the faster part, it depends on the architecture and the size of the elements being swapped. It is the fact that the new quicksort does 1/2 as many swaps together with the type specific code that makes it faster than the original numpy version. The overhead in the current indirect sorts comes mostly from the indirect references to array elements in the innermost loop and it is quite possible that doing swaps on both the indexes and the array would be a bigger hit than the current argsort followed by a suitable take type function. Lexsort is another function that would benefit from what we are discussing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Sep 19 14:42:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:42:57 -0600 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: References: Message-ID: <45103A31.60704@ieee.org> Konrad Hinsen wrote: > On Sep 19, 2006, at 5:11, Dr. Seth Olsen wrote: > > >> Bill's answer to my inquiry about the problem I'm having with >> Collection.findTransformation() (and also, incidentally, with the >> dgesvd call in Subspace.getBasis(), has convinced me that I can no >> long use MMTK without changing some of the code over to numpy. I >> have already been able to determine that invoking >> numpy.oldnumeric.alter_code1.convertall() over the MMTK directory >> tree is not the answer. Has anyone on either of these lists ever >> tried this before and, if so, can it be done (without destroying my >> sanity)? >> > > Adapting MMTK to NumPy is likely to be a major effort, in particular > for the C modules. Well, I got both ScientificPython and MMTK to compile and import using the steps outlined on http://www.scipy.org/Porting_to_NumPy in about 1 hour (including time to fix alter_code1 to make the process even easier). C-modules are actually easier to port because the Numeric C-API is totally supported. > A lot ot testing would also be required. If anyone > wants to tackle this, > I'd be happy to see the results. Great. I can totally understand people not having the time, but it is fantastic to hear that the willingness to accept patches is there. I was able to install ScientificPython and MMTK for NumPy on my system using the patches provided on that page. Is there a test suite that can be run? Users of MMTK could really help out here. -Travis From oliphant.travis at ieee.org Tue Sep 19 14:46:32 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:46:32 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191023.57656.haase@msg.ucsf.edu> References: <200609191023.57656.haase@msg.ucsf.edu> Message-ID: <45103B08.3070808@ieee.org> Sebastian Haase wrote: > Hi, > What are the possible values of > arr.dtype.kind ? > > It seems that signed and unsigned are considered to be the same "kind" > >>>> arr=N.arange(10,dtype=N.uint) >>>> arr.dtype.kind >>>> > 'i' > >>>> arr.dtype.itemsize >>>> > 8 > (OK - this is just showing off our amd64 linux ;-) ) > > How can I distinguish signed from unsigned without having to list all possible > cases explicitly ? > > > Hmm.... This is a problem. There is a 'u' kind for unsigned integers. On my system I get 'u' when running the code you just gave. Can anybody on a 64-bit system confirm? -Travis From oliphant.travis at ieee.org Tue Sep 19 14:47:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 12:47:17 -0600 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: Message-ID: <45103B35.8020205@ieee.org> Charles R Harris wrote: > Question, > > How does one mark a ticket as fixed? I don't see this field in the > ticket views I get, is there a list of accepted fixers? > You have to be logged in to the Trac site. If you have SVN write access you should be able to log in. Then there is a "resolution" section at the very bottom. -Travis From sransom at nrao.edu Tue Sep 19 14:51:16 2006 From: sransom at nrao.edu (Scott Ransom) Date: Tue, 19 Sep 2006 14:51:16 -0400 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <45103B08.3070808@ieee.org> References: <200609191023.57656.haase@msg.ucsf.edu> <45103B08.3070808@ieee.org> Message-ID: <200609191451.16896.sransom@nrao.edu> On Tuesday 19 September 2006 14:46, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > What are the possible values of > > arr.dtype.kind ? > > > > It seems that signed and unsigned are considered to be the same > > "kind" > > > >>>> arr=N.arange(10,dtype=N.uint) > >>>> arr.dtype.kind > > > > 'i' > > > >>>> arr.dtype.itemsize > > > > 8 > > (OK - this is just showing off our amd64 linux ;-) ) > > > > How can I distinguish signed from unsigned without having to list all > > possible cases explicitly ? > > Hmm.... This is a problem. There is a 'u' kind for unsigned > integers. > > On my system I get 'u' when running the code you just gave. > > Can anybody on a 64-bit system confirm? I'm on 64-bit Debian: In [11]: arr=N.arange(10,dtype=N.uint) In [12]: arr.dtype.kind Out[12]: 'u' In [13]: arr.dtype.itemsize Out[13]: 4 In [14]: arr=N.arange(10,dtype=N.long) In [15]: arr.dtype.kind Out[15]: 'i' In [16]: arr.dtype.itemsize Out[16]: 8 Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From sransom at nrao.edu Tue Sep 19 14:54:03 2006 From: sransom at nrao.edu (Scott Ransom) Date: Tue, 19 Sep 2006 14:54:03 -0400 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191451.16896.sransom@nrao.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <45103B08.3070808@ieee.org> <200609191451.16896.sransom@nrao.edu> Message-ID: <200609191454.03405.sransom@nrao.edu> > > Can anybody on a 64-bit system confirm? > > I'm on 64-bit Debian: > > In [11]: arr=N.arange(10,dtype=N.uint) > > In [12]: arr.dtype.kind > Out[12]: 'u' > > In [13]: arr.dtype.itemsize > Out[13]: 4 > > In [14]: arr=N.arange(10,dtype=N.long) > > In [15]: arr.dtype.kind > Out[15]: 'i' > > In [16]: arr.dtype.itemsize > Out[16]: 8 Ack! That was on the wrong machine (32-bit Debian). Here is the 64-bit version: In [2]: arr=N.arange(10,dtype=N.uint) In [3]: arr.dtype.kind Out[3]: 'u' In [4]: arr.dtype.itemsize Out[4]: 8 In [5]: arr=N.arange(10,dtype=N.long) In [6]: arr.dtype.kind Out[6]: 'i' In [7]: arr.dtype.itemsize Out[7]: 8 Sorry about that, Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From rowen at cesmail.net Tue Sep 19 14:59:58 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Tue, 19 Sep 2006 11:59:58 -0700 Subject: [Numpy-discussion] 1.0b5 Installation on OS X References: <8edec5ec0609160846wc9c1347t86492edb7fd0d761@mail.gmail.com> <450EF580.90103@noaa.gov> Message-ID: In article <450EF580.90103 at noaa.gov>, Christopher Barker wrote: > David Andrews wrote: > > > I'm unable to install the 1.0b5 release from the .mpkg on OS X - when > > it comes to the two checkboxes to select the platlib and the scripts, > > both are greyed out. The installer then claims 'Cannot continue: > > Nothing to install'. > > hmmm. It works just fine for me -- just clicking OK as I go merrily along. > > OS version? python version? processor? > > Also, try deleting or renaming any existing install you have from > site-packages, then try again. In addition to that, I suggest also deleting any relevant receipts in /Library/Receipts It is a long shot, but receipt files have occasionally prevented me from installing software, though I confess only when an older version of something. -- Russell From hjn253 at tom.com Sun Sep 24 15:00:36 2006 From: hjn253 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Mon, 25 Sep 2006 03:00:36 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 19 15:06:46 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:06:46 -0600 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: <45103B35.8020205@ieee.org> References: <45103B35.8020205@ieee.org> Message-ID: On 9/19/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > Question, > > > > How does one mark a ticket as fixed? I don't see this field in the > > ticket views I get, is there a list of accepted fixers? > > > You have to be logged in to the Trac site. If you have SVN write access > you should be able to log in. Then there is a "resolution" section at > the very bottom. Ah, that works. I orignally tried to register and discovered that charris was already taken and I didn't know what password to use. I've now deleted the charris208 account. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Sep 19 15:11:30 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 19 Sep 2006 20:11:30 +0100 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: References: <45103B35.8020205@ieee.org> Message-ID: <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> Hi, > > You have to be logged in to the Trac site. If you have SVN write access > > you should be able to log in. Then there is a "resolution" section at > > the very bottom. > > Ah, that works. I orignally tried to register and discovered that charris > was already taken and I didn't know what password to use. I've now deleted > the charris208 account. Not for me, when logged on with my SVN username. I took Robert's email to mean we needed specific permission (over and above SVN write access) to resolve tickets - is that not the case? Best, Matthew From kwgoodman at gmail.com Tue Sep 19 15:34:16 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 19 Sep 2006 12:34:16 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf Message-ID: In what order would you like argsort to sort the values -inf, nan, inf? In numpy 1.0b1 nan is always left where it began: EXAMPLE 1 >> x matrix([[ inf], [ -inf], [ nan]]) >> x[x.argsort(0),:] matrix([[ -inf], [ inf], [ nan]]) EXAMPLE 2 >> x matrix([[ nan], [ inf], [ -inf]]) >> x[x.argsort(0),:] matrix([[ nan], [ -inf], [ inf]]) I would like nan to be in the middle between -inf and inf. From tim.hochberg at ieee.org Tue Sep 19 15:40:04 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 12:40:04 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: Message-ID: <45104794.2050106@ieee.org> Keith Goodman wrote: > In what order would you like argsort to sort the values -inf, nan, inf? > Ideally, -inf should sort first, inf should sort last and nan should raise an exception if present. -tim > In numpy 1.0b1 nan is always left where it began: > > EXAMPLE 1 > > >>> x >>> > > matrix([[ inf], > [ -inf], > [ nan]]) > >>> x[x.argsort(0),:] >>> > > matrix([[ -inf], > [ inf], > [ nan]]) > > EXAMPLE 2 > > >>> x >>> > > matrix([[ nan], > [ inf], > [ -inf]]) > >>> x[x.argsort(0),:] >>> > > matrix([[ nan], > [ -inf], > [ inf]]) > > > I would like nan to be in the middle between -inf and inf. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Tue Sep 19 15:41:56 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 20 Sep 2006 04:41:56 +0900 Subject: [Numpy-discussion] max argmax combo In-Reply-To: <200609191941.56395.faltet@carabos.com> References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/20/06, Francesc Altet wrote: > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > > > Do you want both the indexes and index sorted array returned, or just sort > > the array using indexes, i.e., something like > > > > a.sort(kind="quicksort", method="indirect") > > Uh, I don't understand what do you mean by "sort the array using indexes", > sorry. > I think he meant do an argsort first, then use fancy indexing to get the sorted array. For a 1-d array that's just ind = A.argsort() Asorted = A[ind] That should be O(N lg N + N), aka O(N lg N) For A >1-d, you need an indexing expression that's a little more complicated, hence the discussion about making an "extract" function for that purpose. Then you could say: ind = A.argsort(axis=d) Asorted = A.extract(ind,axis=d) and it should just work no matter what 'd' is. --bb From charlesr.harris at gmail.com Tue Sep 19 15:52:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:52:01 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Keith Goodman wrote: > > In what order would you like argsort to sort the values -inf, nan, inf? > > > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. > > -tim That sounds right. Nan can't be compared because, well, it is undefined. For instance: In [73]: a = arange(2) In [74]: a/0 Out[74]: array([0, 0]) In [75]: a/0.0 Out[75]: array([ nan, inf]) In [76]: a/(-0.0) Out[76]: array([ nan, -inf]) I.e., 0.0/0.0 is undefined. But unless the hardware generates the exception I would be loath to introduce checks in the code. Putting a check in the innermost loop of the sorts would cost significant time. But look at the integer division by zero where the IEEE stuff has no relevence. That sure looks like a bug to me. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Sep 19 15:57:22 2006 From: ndarray at mac.com (Sasha) Date: Tue, 19 Sep 2006 15:57:22 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. I disagree. NumPy sort leaving nan's where they are is probably just a side-effect of nans unusual properties (both nanx is always false), but it is a logical choice. Checking for nan will inevitably slow the most common use case. If you want an exception, just check isnan(x).any() before sort. From charlesr.harris at gmail.com Tue Sep 19 15:58:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 13:58:03 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191845.38788.faltet@carabos.com> <200609191941.56395.faltet@carabos.com> Message-ID: On 9/19/06, Bill Baxter wrote: > > On 9/20/06, Francesc Altet wrote: > > A Dimarts 19 Setembre 2006 19:21, Charles R Harris va escriure: > > > > > > Do you want both the indexes and index sorted array returned, or just > sort > > > the array using indexes, i.e., something like > > > > > > a.sort(kind="quicksort", method="indirect") > > > > Uh, I don't understand what do you mean by "sort the array using > indexes", > > sorry. > > > > I think he meant do an argsort first, then use fancy indexing to get > the sorted array. > For a 1-d array that's just > > ind = A.argsort() > Asorted = A[ind] > > That should be O(N lg N + N), aka O(N lg N) > For A >1-d, you need an indexing expression that's a little more > complicated, hence the discussion about making an "extract" function > for that purpose. Then you could say: > > ind = A.argsort(axis=d) > Asorted = A.extract(ind,axis=d) I'm also thinking of the name argtake. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 16:01:50 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 16:01:50 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45104794.2050106@ieee.org> References: <45104794.2050106@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > Keith Goodman wrote: > > In what order would you like argsort to sort the values -inf, nan, inf? > > > Ideally, -inf should sort first, inf should sort last and nan should > raise an exception if present. > > -tim Mmm. Somebody who's working with NaNs has more or less already decided they don't want to be pestered with exceptions for invalid data. I'd be happy if they wound up at either end, but I'm not sure it's worth hacking up the sort algorithm when a simple isnan() can pull them out. What's happening now is that NaNa are all false, which rather confuses the sorting algorithm. But as long as it doesn't actually *break* (that is, leave some of the non-NaNs incorrectly sorted) I don't care. A. M. Archibald From kwgoodman at gmail.com Tue Sep 19 16:06:40 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 19 Sep 2006 13:06:40 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > > Keith Goodman wrote: > > > In what order would you like argsort to sort the values -inf, nan, inf? > > > > > Ideally, -inf should sort first, inf should sort last and nan should > > raise an exception if present. > > > > -tim > > Mmm. Somebody who's working with NaNs has more or less already decided > they don't want to be pestered with exceptions for invalid data. I'd > be happy if they wound up at either end, but I'm not sure it's worth > hacking up the sort algorithm when a simple isnan() can pull them out. Is there an easy way to use isnan to pull out the nans if the matrix I am sorting has more than one column? From ndarray at mac.com Tue Sep 19 16:13:55 2006 From: ndarray at mac.com (Sasha) Date: Tue, 19 Sep 2006 16:13:55 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: On 9/19/06, Keith Goodman wrote: > Is there an easy way to use isnan to pull out the nans if the matrix I > am sorting has more than one column? > There seems to be a "nan_to_num" function that converts nans to zeros, but I would suggest just using fancy indexing to fill the nans with appropriate values: x[isnan(x)] = inf # if you want nans on the right From tim.hochberg at ieee.org Tue Sep 19 16:21:56 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 13:21:56 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> Message-ID: <45105164.90700@ieee.org> A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > >> Keith Goodman wrote: >> >>> In what order would you like argsort to sort the values -inf, nan, inf? >>> >>> >> Ideally, -inf should sort first, inf should sort last and nan should >> raise an exception if present. >> >> -tim >> > > Mmm. Somebody who's working with NaNs has more or less already decided > they don't want to be pestered with exceptions for invalid data. Do you really think so? In my experience NaNs are nearly always just an indication of a mistake somewhere that didn't get trapped for one reason or another. > I'd > be happy if they wound up at either end, but I'm not sure it's worth > hacking up the sort algorithm when a simple isnan() can pull them out. > Moving them to the end seems to be the worst choice to me. Leaving them alone is fine with me. Or raising an exception would be fine. Or doing one or the other depending on the error mode settings would be even better if it is practical. > What's happening now is that NaNa are all false, > which rather confuses the sorting algorithm. But as long as it doesn't > actually *break* (that is, leave some of the non-NaNs incorrectly > sorted) I don't care. > Is that true? Are all of numpy's sorting algorithms robust against nontransitive objects laying around? The answer to that appears to be no. Try running this a couple of times to see what I mean: n = 10 for i in range(10): for kind in 'quicksort', 'heapsort', 'mergesort': a = rand(n) b = a.copy() a[n//2] = nan a.sort(kind=kind) b.sort(kind=kind) assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) The values don't correctly cross the inserted NaN and the sort is incorrect. -tim From charlesr.harris at gmail.com Tue Sep 19 16:56:00 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 14:56:00 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105164.90700@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg wrote: > > > >> Keith Goodman wrote: > >> > >>> In what order would you like argsort to sort the values -inf, nan, > inf? > >>> > >>> > >> Ideally, -inf should sort first, inf should sort last and nan should > >> raise an exception if present. > >> > >> -tim > >> > > > > Mmm. Somebody who's working with NaNs has more or less already decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always just an > indication of a mistake somewhere that didn't get trapped for one reason > or another. > > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull them out. > > > Moving them to the end seems to be the worst choice to me. Leaving them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. > > > What's happening now is that NaNa are all false, > > which rather confuses the sorting algorithm. But as long as it doesn't > > actually *break* (that is, leave some of the non-NaNs incorrectly > > sorted) I don't care. > > > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > > n = 10 > for i in range(10): > for kind in 'quicksort', 'heapsort', 'mergesort': > a = rand(n) > b = a.copy() > a[n//2] = nan > a.sort(kind=kind) > b.sort(kind=kind) > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > The values don't correctly cross the inserted NaN and the sort is > incorrect. Except for heapsort those are doing insertion sorts because n is so small. Heapsort is trickier to understand because of the way the heap is formed and values pulled off. I'll look into that. I bet searchsorted gets messed up by nan's. Do you think it worthwhile to worry about it? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 17:04:49 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:04:49 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: <45105B71.7090102@ieee.org> Charles R Harris wrote: > > > On 9/19/06, *Tim Hochberg* > wrote: > > A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg > wrote: > > > >> Keith Goodman wrote: > >> > >>> In what order would you like argsort to sort the values -inf, > nan, inf? > >>> > >>> > >> Ideally, -inf should sort first, inf should sort last and nan > should > >> raise an exception if present. > >> > >> -tim > >> > > > > Mmm. Somebody who's working with NaNs has more or less already > decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always > just an > indication of a mistake somewhere that didn't get trapped for one > reason > or another. > > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull > them out. > > > Moving them to the end seems to be the worst choice to me. Leaving > them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. > > > What's happening now is that NaNa are all > false, > > which rather confuses the sorting algorithm. But as long as it > doesn't > > actually *break* (that is, leave some of the non-NaNs incorrectly > > sorted) I don't care. > > > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > > n = 10 > for i in range(10): > for kind in 'quicksort', 'heapsort', 'mergesort': > a = rand(n) > b = a.copy() > a[n//2] = nan > a.sort(kind=kind) > b.sort(kind=kind) > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > The values don't correctly cross the inserted NaN and the sort is > incorrect. > > > Except for heapsort those are doing insertion sorts because n is so > small. Heapsort is trickier to understand because of the way the heap > is formed and values pulled off. I'm not sure where the breakpoint is, but I was seeing failures for all three sort types with N as high as 10000. I suspect that they're all broken in the presence of NaNs. I further suspect you'd need some punishingly slow n**2 algorithm to be robust in the presence of NaNs. > I'll look into that. I bet searchsorted gets messed up by nan's. Do > you think it worthwhile to worry about it? No. But that's because it's my contention is that any sequence with NaNs in it is *not sorted* and thus isn't suitable input for searchsorted. -tim From peridot.faceted at gmail.com Tue Sep 19 17:09:14 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:09:14 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105164.90700@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > A. M. Archibald wrote: > > Mmm. Somebody who's working with NaNs has more or less already decided > > they don't want to be pestered with exceptions for invalid data. > Do you really think so? In my experience NaNs are nearly always just an > indication of a mistake somewhere that didn't get trapped for one reason > or another. Well, I said that because for an image porcessing project I was doing, the easiest thing to do with certain troublesome pixels was to fill in NaNs, and then at the end replace the NaNs with sensible values. It seems as if the point of NaNs is to allow you to keep working with those numbers that make sense while ingoring those that don't. If you wanted exceptions, why not get them as soon as the first NaN would have been generated? > > I'd > > be happy if they wound up at either end, but I'm not sure it's worth > > hacking up the sort algorithm when a simple isnan() can pull them out. > > > Moving them to the end seems to be the worst choice to me. Leaving them > alone is fine with me. Or raising an exception would be fine. Or doing > one or the other depending on the error mode settings would be even > better if it is practical. I was just thinking in terms of easy removal. > Is that true? Are all of numpy's sorting algorithms robust against > nontransitive objects laying around? The answer to that appears to be > no. Try running this a couple of times to see what I mean: > The values don't correctly cross the inserted NaN and the sort is incorrect. You're quite right: when NaNs are present in the array, sorting and then removing them does not yield a sorted array. For example, mergesort just output [ 2. 4. 6. 9. nan 0. 1. 3. 5. 7. 8. ] The other two are no better (and arguably worse). Python's built-in sort() for lists has the same problem. This is definitely a bug, and the best way to fix it is not clear to me - perhaps sort() needs to always do any(isnan(A)) before starting to sort. I don't really like raising an exception, but sort() isn't really very meaningful with NaNs in the array. The only other option I can think of is to somehow remove them, sort without, and reintroduce them at the end, which is going to be a nightmare when sorting a single axis of a large array. Or, I suppose, sort() could simply fill the array with NaNs; I'm sure users will love that. A. M. Archibald From peridot.faceted at gmail.com Tue Sep 19 17:15:24 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:15:24 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105B71.7090102@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > I'm not sure where the breakpoint is, but I was seeing failures for all > three sort types with N as high as 10000. I suspect that they're all > broken in the presence of NaNs. I further suspect you'd need some > punishingly slow n**2 algorithm to be robust in the presence of NaNs. Not at all. Just temporarily make NaNs compare greater than any other floating-point value and use quicksort (say). You can even do this for python lists without much trouble. That's actually a viable suggestion for numpy's sorting, although it would be kind of ugly to implement: do a quick any(isnan(A)), and if not, use the fast stock sorting algorithms; if there is a NaN somewhere in the array, use a version of the sort that has a tweaked comparison function so the NaNs wind up at the end and are easy to trim off. But the current situation, silently returning arrays in which the non-NaNs are unsorted, is really bad. A. M. Archibald From robert.kern at gmail.com Tue Sep 19 17:18:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 16:18:10 -0500 Subject: [Numpy-discussion] Resolution of tickets. In-Reply-To: <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> References: <45103B35.8020205@ieee.org> <1e2af89e0609191211xdbb2062mad97a55540ebcdab@mail.gmail.com> Message-ID: Matthew Brett wrote: > Hi, > >>> You have to be logged in to the Trac site. If you have SVN write access >>> you should be able to log in. Then there is a "resolution" section at >>> the very bottom. >> Ah, that works. I orignally tried to register and discovered that charris >> was already taken and I didn't know what password to use. I've now deleted >> the charris208 account. > > Not for me, when logged on with my SVN username. I took Robert's > email to mean we needed specific permission (over and above SVN write > access) to resolve tickets - is that not the case? It used to be the case that the subversion password database was used as the Trac's password database. This is no longer the case since we need to require registration by non-developers. *Operationally*, people added to the subversion write list should also be be provided with appropriate Trac privileges, but this did not happen in your case. I will fix that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 19 17:21:13 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 15:21:13 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45105B71.7090102@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Charles R Harris wrote: > > > > > > On 9/19/06, *Tim Hochberg* > > wrote: > > > > A. M. Archibald wrote: > > > On 19/09/06, Tim Hochberg > > wrote: > > > > > >> Keith Goodman wrote: > > >> > > >>> In what order would you like argsort to sort the values -inf, > > nan, inf? > > >>> > > >>> > > >> Ideally, -inf should sort first, inf should sort last and nan > > should > > >> raise an exception if present. > > >> > > >> -tim > > >> > > > > > > Mmm. Somebody who's working with NaNs has more or less already > > decided > > > they don't want to be pestered with exceptions for invalid data. > > Do you really think so? In my experience NaNs are nearly always > > just an > > indication of a mistake somewhere that didn't get trapped for one > > reason > > or another. > > > > > I'd > > > be happy if they wound up at either end, but I'm not sure it's > worth > > > hacking up the sort algorithm when a simple isnan() can pull > > them out. > > > > > Moving them to the end seems to be the worst choice to me. Leaving > > them > > alone is fine with me. Or raising an exception would be fine. Or > doing > > one or the other depending on the error mode settings would be even > > better if it is practical. > > > > > What's happening now is that NaNa are all > > false, > > > which rather confuses the sorting algorithm. But as long as it > > doesn't > > > actually *break* (that is, leave some of the non-NaNs incorrectly > > > sorted) I don't care. > > > > > Is that true? Are all of numpy's sorting algorithms robust against > > nontransitive objects laying around? The answer to that appears to > be > > no. Try running this a couple of times to see what I mean: > > > > n = 10 > > for i in range(10): > > for kind in 'quicksort', 'heapsort', 'mergesort': > > a = rand(n) > > b = a.copy() > > a[n//2] = nan > > a.sort(kind=kind) > > b.sort(kind=kind) > > assert alltrue(a[:n//2] == b[:n//2]), (kind, a, b) > > > > > > The values don't correctly cross the inserted NaN and the sort is > > incorrect. > > > > > > Except for heapsort those are doing insertion sorts because n is so > > small. Heapsort is trickier to understand because of the way the heap > > is formed and values pulled off. > I'm not sure where the breakpoint is, but I was seeing failures for all > three sort types with N as high as 10000. I suspect that they're all > broken in the presence of NaNs. I further suspect you'd need some > punishingly slow n**2 algorithm to be robust in the presence of NaNs. Quicksort and Mergesort are divide and conquer types. When the pieces get below a certain length they finish up with insertion sort as it is more efficient for small bits. In numpy the breakover points are 15 and 20 respectively. I believe microsoft did a project on this and ended up with a number somewhere around 7, but I didn't see that when doing the tuning for numarray way back when. Not that I did a *lot* of careful tuning work ;) > I'll look into that. I bet searchsorted gets messed up by nan's. Do > > you think it worthwhile to worry about it? > > No. But that's because it's my contention is that any sequence with NaNs > in it is *not sorted* and thus isn't suitable input for searchsorted. If this sort of thing can cause unexpected errors I wonder if it would be worth it to have a global debugging flag that essentially causes isnan to be called before any function applications. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 17:28:35 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 17:28:35 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 19/09/06, Charles R Harris wrote: > If this sort of thing can cause unexpected errors I wonder if it would be > worth it to have a global debugging flag that essentially causes isnan to > be called before any function applications. That sounds very like the IEEE floating-point flags, which would be extremely useful to have, and which are being wored on, IIRC. A. M. Archibald From tim.hochberg at ieee.org Tue Sep 19 17:37:12 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:37:12 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <45106308.7010302@ieee.org> A. M. Archibald wrote: > On 19/09/06, Charles R Harris wrote: > > >> If this sort of thing can cause unexpected errors I wonder if it would be >> worth it to have a global debugging flag that essentially causes isnan to >> be called before any function applications. >> > > That sounds very like the IEEE floating-point flags, which would be > extremely useful to have, and which are being wored on, IIRC. > Are you referring to numpy.geterr / seterr? These exist, but they don't seem to be working right as of 1.0b4 (my installation is a little out of date). -tim From tim.hochberg at ieee.org Tue Sep 19 17:40:21 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 14:40:21 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <451063C5.20201@ieee.org> A. M. Archibald wrote: > On 19/09/06, Tim Hochberg wrote: > > >> I'm not sure where the breakpoint is, but I was seeing failures for all >> three sort types with N as high as 10000. I suspect that they're all >> broken in the presence of NaNs. I further suspect you'd need some >> punishingly slow n**2 algorithm to be robust in the presence of NaNs. >> > > Not at all. Just temporarily make NaNs compare greater than any other > floating-point value and use quicksort (say). You can even do this for > python lists without much trouble. > I misspoke. What I meant here was keeping the behavior that people think that we already have but don't: NaNs stay in place and everything is sorted around them. And even that's not true, since you could just record where the NaNs are, remove them, sort and put them back. What I was really getting at was, that I'm guessing, and it's just a guess, that (a) none of the fast sorting algorithms do anything sensible unless special cased and (b) one could come up with a naive n**2 sort that does do something sensible without special casing (where sensible means leave the NaNs alone). > That's actually a viable suggestion for numpy's sorting, although it > would be kind of ugly to implement: do a quick any(isnan(A)), and if > not, use the fast stock sorting algorithms; if there is a NaN > somewhere in the array, use a version of the sort that has a tweaked > comparison function so the NaNs wind up at the end and are easy to > trim off. > > But the current situation, silently returning arrays in which the > non-NaNs are unsorted, is really bad. > If your going to do isnan anyway, why not just raise an exception. An array with NaNs in it can't be sorted by any common sense definition of sorting. Any treatment of NaNs is going to be arbitrary, so we might as well make the user specify what they want. "In the face of ambiguity, refuse the temptation to guess" and all that. My favorite solution would be to make sort respect the invalid mode of seterr/geterr. However at the moment it doesn't seem to (in beta4 at least) but neither does add or multiply so those probably need to be looked at again.... -tim From strawman at astraw.com Tue Sep 19 17:44:53 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 19 Sep 2006 14:44:53 -0700 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <450FB705.7010002@ubuntu-au.org> References: <450FB705.7010002@ubuntu-au.org> Message-ID: <451064D5.60806@astraw.com> William Grant wrote: >Hi, > >I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it >currently cannot happen as numpy doesn't build with Python 2.5. I note >that changeset 3109 >(http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to >give 2.5 compatibility, but it is those bits which seem to break it. The >end of the build log with the errors can be found at >http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. > >Has anybody got any ideas on how to fix this? A number of Ubuntu users >want scipy 0.5.1, but that can't happen while it won't build with Python >2.5. > > AFAIK a number of Ubuntu Dapper users are happily using my .debs with Python 2.4 available at debs.astraw.com. This includes scipy 0.5.2.dev2197 Where are you getting that Ubuntu requires Python 2.5? From oliphant at ee.byu.edu Tue Sep 19 17:48:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 15:48:18 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <451063C5.20201@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> Message-ID: <451065A2.8040406@ee.byu.edu> Tim Hochberg wrote: >A. M. Archibald wrote: > > >>On 19/09/06, Tim Hochberg wrote: >> >> >> >> >>>I'm not sure where the breakpoint is, but I was seeing failures for all >>>three sort types with N as high as 10000. I suspect that they're all >>>broken in the presence of NaNs. I further suspect you'd need some >>>punishingly slow n**2 algorithm to be robust in the presence of NaNs. >>> >>> >>> >>Not at all. Just temporarily make NaNs compare greater than any other >>floating-point value and use quicksort (say). You can even do this for >>python lists without much trouble. >> >> >> >I misspoke. What I meant here was keeping the behavior that people think >that we already have but don't: NaNs stay in place and everything is >sorted around them. And even that's not true, since you could just >record where the NaNs are, remove them, sort and put them back. What I >was really getting at was, that I'm guessing, and it's just a guess, >that (a) none of the fast sorting algorithms do anything sensible unless >special cased and (b) one could come up with a naive n**2 sort that does >do something sensible without special casing (where sensible means leave >the NaNs alone). > > >>That's actually a viable suggestion for numpy's sorting, although it >>would be kind of ugly to implement: do a quick any(isnan(A)), and if >>not, use the fast stock sorting algorithms; if there is a NaN >>somewhere in the array, use a version of the sort that has a tweaked >>comparison function so the NaNs wind up at the end and are easy to >>trim off. >> >>But the current situation, silently returning arrays in which the >>non-NaNs are unsorted, is really bad. >> >> >> >If your going to do isnan anyway, why not just raise an exception. An >array with NaNs in it can't be sorted by any common sense definition of >sorting. Any treatment of NaNs is going to be arbitrary, so we might as >well make the user specify what they want. "In the face of ambiguity, >refuse the temptation to guess" and all that. > >My favorite solution would be to make sort respect the invalid mode of >seterr/geterr. However at the moment it doesn't seem to (in beta4 at >least) but neither does add or multiply so those probably need to be >looked at again.... > > The geterr/seterr stuff changes how IEEE hardware flags are handled in ufuncs. Currently they are not even looked at elsewhere. Are you saying that add and multiply don't respect the invalid flag? If not, then this might be hardware related. Does the IEEE invalid hardware flag get raised on multiplication by nan or only on creation of nan? All the seterr/geterr stuff relies on the hardware flags. We don't do any other checking. -Travis From william.grant at ubuntu.com.au Tue Sep 19 17:54:16 2006 From: william.grant at ubuntu.com.au (William Grant) Date: Wed, 20 Sep 2006 07:54:16 +1000 Subject: [Numpy-discussion] Numpy fails to build with Python 2.5 In-Reply-To: <451064D5.60806@astraw.com> References: <450FB705.7010002@ubuntu-au.org> <451064D5.60806@astraw.com> Message-ID: <45106708.3090906@ubuntu.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andrew Straw wrote: > William Grant wrote: > >> Hi, >> >> I'm currently attempting to get scipy 0.5.1 into Ubuntu, however it >> currently cannot happen as numpy doesn't build with Python 2.5. I note >> that changeset 3109 >> (http://projects.scipy.org/scipy/numpy/changeset/3109#file1) is meant to >> give 2.5 compatibility, but it is those bits which seem to break it. The >> end of the build log with the errors can be found at >> http://people.ubuntu.com.au/~fujitsu/numpy_error.txt. >> >> Has anybody got any ideas on how to fix this? A number of Ubuntu users >> want scipy 0.5.1, but that can't happen while it won't build with Python >> 2.5. >> >> > AFAIK a number of Ubuntu Dapper users are happily using my .debs with > Python 2.4 available at debs.astraw.com. This includes scipy 0.5.2.dev2197 > > Where are you getting that Ubuntu requires Python 2.5? Ubuntu 6.10 (currently in development) includes Python 2.5. William. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFFEGcIrbS+qZ0dQCURAhsLAJwLwZvYabv6IUOWHlmvIaRnQmk69QCeJfOt xFZp00DLb+/PWtbVjUg99gY= =D2zA -----END PGP SIGNATURE----- From charlesr.harris at gmail.com Tue Sep 19 18:00:18 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 16:00:18 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Charles R Harris wrote: > > > If this sort of thing can cause unexpected errors I wonder if it would > be > > worth it to have a global debugging flag that essentially causes isnan > to > > be called before any function applications. > > That sounds very like the IEEE floating-point flags, which would be > extremely useful to have, and which are being wored on, IIRC. Thinking a bit, keeping the values in place isn't easy. Mergesort isn't fixable because values can be moved in front of the nan before it is ever looked at. Nor can it be easily set up to leave all the nans at one end because both a < nan and nan < a return false. Quicksort might be doable with some checks. I mean, what if the selected pivot is a nan? The median of three version used also needs thinking about. Hmm. But I think it is the insertion sort that is messing up the order in mergesort as now nothing will move past the nan even if it has to. That could be fixed, but the nan's would still move around. I think the best thing to do is punt unless the hardware can be set to do something. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 18:12:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 15:12:08 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <451065A2.8040406@ee.byu.edu> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> Message-ID: <45106B38.6010504@ieee.org> Travis Oliphant wrote: > Tim Hochberg wrote: > > >> A. M. Archibald wrote: >> >> >> >>> On 19/09/06, Tim Hochberg wrote: >>> >>> >>> >>> >>> >>>> I'm not sure where the breakpoint is, but I was seeing failures for all >>>> three sort types with N as high as 10000. I suspect that they're all >>>> broken in the presence of NaNs. I further suspect you'd need some >>>> punishingly slow n**2 algorithm to be robust in the presence of NaNs. >>>> >>>> >>>> >>>> >>> Not at all. Just temporarily make NaNs compare greater than any other >>> floating-point value and use quicksort (say). You can even do this for >>> python lists without much trouble. >>> >>> >>> >>> >> I misspoke. What I meant here was keeping the behavior that people think >> that we already have but don't: NaNs stay in place and everything is >> sorted around them. And even that's not true, since you could just >> record where the NaNs are, remove them, sort and put them back. What I >> was really getting at was, that I'm guessing, and it's just a guess, >> that (a) none of the fast sorting algorithms do anything sensible unless >> special cased and (b) one could come up with a naive n**2 sort that does >> do something sensible without special casing (where sensible means leave >> the NaNs alone). >> >> >> >>> That's actually a viable suggestion for numpy's sorting, although it >>> would be kind of ugly to implement: do a quick any(isnan(A)), and if >>> not, use the fast stock sorting algorithms; if there is a NaN >>> somewhere in the array, use a version of the sort that has a tweaked >>> comparison function so the NaNs wind up at the end and are easy to >>> trim off. >>> >>> But the current situation, silently returning arrays in which the >>> non-NaNs are unsorted, is really bad. >>> >>> >>> >>> >> If your going to do isnan anyway, why not just raise an exception. An >> array with NaNs in it can't be sorted by any common sense definition of >> sorting. Any treatment of NaNs is going to be arbitrary, so we might as >> well make the user specify what they want. "In the face of ambiguity, >> refuse the temptation to guess" and all that. >> >> My favorite solution would be to make sort respect the invalid mode of >> seterr/geterr. However at the moment it doesn't seem to (in beta4 at >> least) but neither does add or multiply so those probably need to be >> looked at again.... >> >> >> > The geterr/seterr stuff changes how IEEE hardware flags are handled in > ufuncs. Currently they are not even looked at elsewhere. Are you > saying that add and multiply don't respect the invalid flag? If not, > then this might be hardware related. Does the IEEE invalid hardware > flag get raised on multiplication by nan or only on creation of nan? It looks like I jumped to conclusions here. I was expecting that with invalid set to 'raise' that an array that (+-*operations on nans would raise an exception. It appears that is incorrect -- only operations that create nans from non-nans will trigger this as you suspected. Similarly, I expected that over='raise' would cause inf+something to raise an exception. Again, not true; this raises an exception only when a new inf is created from non infs. At least on my box. Interesting. And a little surprising. An interesting tidbit (in an array) inf/inf will raise invalid, but nan/nan will not, it just returns nan. Fun, fun fun! > > All the seterr/geterr stuff relies on the hardware flags. We don't do > any other checking. > Yeah. And just by itself this isn't going to do anything for sort since comparing nans will not set any flags, it's just that the result will be problematic.If one wanted to use these flags for this one would have to use/abuse the result of geterr to trigger an isnan test at the beginning of sort and then warn, raise or call as appropriate. It would probably also not be unreasonable to punt and document sort as failing in the presence of nans. -tim From haase at msg.ucsf.edu Tue Sep 19 18:23:44 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 15:23:44 -0700 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191454.03405.sransom@nrao.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> Message-ID: <200609191523.45253.haase@msg.ucsf.edu> OK - I'm really sorry !! I also get 'u' -- I had a typo there ... But what is the complete list of kind values ? -Sebastian On Tuesday 19 September 2006 11:54, Scott Ransom wrote: > > > Can anybody on a 64-bit system confirm? > > > > I'm on 64-bit Debian: > > > > In [11]: arr=N.arange(10,dtype=N.uint) > > > > In [12]: arr.dtype.kind > > Out[12]: 'u' > > > > In [13]: arr.dtype.itemsize > > Out[13]: 4 > > > > In [14]: arr=N.arange(10,dtype=N.long) > > > > In [15]: arr.dtype.kind > > Out[15]: 'i' > > > > In [16]: arr.dtype.itemsize > > Out[16]: 8 > > Ack! That was on the wrong machine (32-bit Debian). Here is the 64-bit > version: > > In [2]: arr=N.arange(10,dtype=N.uint) > > In [3]: arr.dtype.kind > Out[3]: 'u' > > In [4]: arr.dtype.itemsize > Out[4]: 8 > > In [5]: arr=N.arange(10,dtype=N.long) > > In [6]: arr.dtype.kind > Out[6]: 'i' > > In [7]: arr.dtype.itemsize > Out[7]: 8 > > Sorry about that, > > Scott From haase at msg.ucsf.edu Tue Sep 19 18:33:23 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 15:33:23 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float Message-ID: <200609191533.24097.haase@msg.ucsf.edu> Hello all, I just had someone from my lab coming to my desk saying: "My god - SciPy is really stupid .... An array with only positive numbers claims to have a negative mean !! "? I was asking about this before ... the reason was of course that her array was of dtype int32 and had many large values to cause an overflow (wrap around) . Now that the default for axis is None (for all functions having an axis argument), can we please change dtype to default to float64 !? It is really a very confusing and shocking result to get a negative mean on all positive values. It has been stated here before that numpy should target the "scientist" rather than the programmers ... I would argue that mean() almost always requires the precision of "double" (float64) to produce usable results. Please consider this change before the 1.0 release ... Thanks, Sebastian Haase From oliphant at ee.byu.edu Tue Sep 19 18:44:49 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 16:44:49 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <200609191523.45253.haase@msg.ucsf.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> Message-ID: <451072E1.7030608@ee.byu.edu> Sebastian Haase wrote: >OK - I'm really sorry !! >I also get 'u' -- I had a typo there ... > >But what is the complete list of kind values ? > > It's in the array interface specification: http://numpy.scipy.org/array_interface.shtml -Travis From oliphant at ee.byu.edu Tue Sep 19 18:48:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 16:48:05 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191533.24097.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> Message-ID: <451073A5.4060204@ee.byu.edu> Sebastian Haase wrote: >Hello all, >I just had someone from my lab coming to my desk saying: >"My god - SciPy is really stupid .... >An array with only positive numbers claims to have a negative mean !! "? > > > >I was asking about this before ... the reason was of course that her array was >of dtype int32 and had many large values to cause an overflow (wrap >around) . > >Now that the default for axis is None (for all functions having an axis >argument), >can we please change dtype to default to float64 !? > > The default is float64 now (as long as you are not using numpy.oldnumeric). I suppose more appropriately, we could reduce over float for integer data-types when calculating the mean as well (since a floating point is returned anyway). -Travis From charlesr.harris at gmail.com Tue Sep 19 18:55:57 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 16:55:57 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45106B38.6010504@ieee.org> References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 9/19/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > Tim Hochberg wrote: > > > > > >> A. M. Archibald wrote: > >> > >> > >> > >>> On 19/09/06, Tim Hochberg wrote: > >>> > >>> > >>> > >>> > >>> > >>>> I'm not sure where the breakpoint is, but I was seeing failures for > all > >>>> three sort types with N as high as 10000. I suspect that they're all > >>>> broken in the presence of NaNs. I further suspect you'd need some > >>>> punishingly slow n**2 algorithm to be robust in the presence of NaNs. > >>>> > >>>> > >>>> > >>>> > >>> Not at all. Just temporarily make NaNs compare greater than any other > >>> floating-point value and use quicksort (say). You can even do this for > >>> python lists without much trouble. > >>> > >>> > >>> > >>> > >> I misspoke. What I meant here was keeping the behavior that people > think > >> that we already have but don't: NaNs stay in place and everything is > >> sorted around them. And even that's not true, since you could just > >> record where the NaNs are, remove them, sort and put them back. What I > >> was really getting at was, that I'm guessing, and it's just a guess, > >> that (a) none of the fast sorting algorithms do anything sensible > unless > >> special cased and (b) one could come up with a naive n**2 sort that > does > >> do something sensible without special casing (where sensible means > leave > >> the NaNs alone). > >> > >> > >> > >>> That's actually a viable suggestion for numpy's sorting, although it > >>> would be kind of ugly to implement: do a quick any(isnan(A)), and if > >>> not, use the fast stock sorting algorithms; if there is a NaN > >>> somewhere in the array, use a version of the sort that has a tweaked > >>> comparison function so the NaNs wind up at the end and are easy to > >>> trim off. > >>> > >>> But the current situation, silently returning arrays in which the > >>> non-NaNs are unsorted, is really bad. > >>> > >>> > >>> > >>> > >> If your going to do isnan anyway, why not just raise an exception. An > >> array with NaNs in it can't be sorted by any common sense definition of > >> sorting. Any treatment of NaNs is going to be arbitrary, so we might as > >> well make the user specify what they want. "In the face of ambiguity, > >> refuse the temptation to guess" and all that. > >> > >> My favorite solution would be to make sort respect the invalid mode of > >> seterr/geterr. However at the moment it doesn't seem to (in beta4 at > >> least) but neither does add or multiply so those probably need to be > >> looked at again.... > >> > >> > >> > > The geterr/seterr stuff changes how IEEE hardware flags are handled in > > ufuncs. Currently they are not even looked at elsewhere. Are you > > saying that add and multiply don't respect the invalid flag? If not, > > then this might be hardware related. Does the IEEE invalid hardware > > flag get raised on multiplication by nan or only on creation of nan? > It looks like I jumped to conclusions here. I was expecting that with > invalid set to 'raise' that an array that (+-*operations on nans would > raise an exception. It appears that is incorrect -- only operations that > create nans from non-nans will trigger this as you suspected. Similarly, > I expected that over='raise' would cause inf+something to raise an > exception. Again, not true; this raises an exception only when a new inf > is created from non infs. At least on my box. > > Interesting. And a little surprising. > > An interesting tidbit (in an array) inf/inf will raise invalid, but > nan/nan will not, it just returns nan. Fun, fun fun! > > > > All the seterr/geterr stuff relies on the hardware flags. We don't do > > any other checking. > > > > Yeah. And just by itself this isn't going to do anything for sort since > comparing nans will not set any flags, it's just that the result will be > problematic.If one wanted to use these flags for this one would have to > use/abuse the result of geterr to trigger an isnan test at the beginning > of sort and then warn, raise or call as appropriate. > > It would probably also not be unreasonable to punt and document sort as > failing in the presence of nans. For floats we could use something like: lessthan(a,b) := a < b || (a == nan && b != nan) Which would put all the nans at one end and might not add too much overhead. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Tue Sep 19 18:59:07 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 20 Sep 2006 00:59:07 +0200 Subject: [Numpy-discussion] what's the difference between npy_intp and size_t? In-Reply-To: <451038F4.7090704@ieee.org> References: <200609191734.57899.martin.wiechert@gmx.de> <451038F4.7090704@ieee.org> Message-ID: <200609200059.07746.martin.wiechert@gmx.de> On Tuesday 19 September 2006 20:37, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > Please forgive my ignorance: Is there any difference between npy_intp and > > size_t. Aren't both "ints just big enough to be safe with pointer > > arithmetics even on 64 bit architectures?". > > size_t is unsigned > npy_intp is signed > (!) Thanks again, Travis. > It is basically the same as Py_ssize_t (which is not available until > Python 2.5). Out now! http://www.python.org/2.5 > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From charlesr.harris at gmail.com Tue Sep 19 19:10:26 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 17:10:26 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. Message-ID: Travis, Is this intentional? In [77]: arange(5, dtype=int)/0 Out[77]: array([0, 0, 0, 0, 0]) It looks deliberate because all zeros are returned, but it might be better if it raised an exception. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Tue Sep 19 19:13:54 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:13:54 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: References: Message-ID: <451079B2.60805@ee.byu.edu> Charles R Harris wrote: > Travis, > > Is this intentional? > > In [77]: arange(5, dtype=int)/0 > Out[77]: array([0, 0, 0, 0, 0]) > > It looks deliberate because all zeros are returned, but it might be > better if it raised an exception. It is deliberate. Numarray introduced it (the only difference being that by default NumPy has division-by-zero erros turned off). It's tied to the way floating-point division-by zero is handled. There is a valid argument for having a separate integer-division flag so that you can raise exceptions for integer-division but not for floating-point division. I'm open to that change for 1.0rc1 -Travis From oliphant at ee.byu.edu Tue Sep 19 19:16:32 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:16:32 -0600 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: Message-ID: <45107A50.4010502@ee.byu.edu> Charles R Harris wrote: > > > On 9/18/06, *Bill Baxter* > wrote: > > On 9/19/06, Charles R Harris > wrote: > > On 9/18/06, Bill Baxter > wrote: > > > I find myself often wanting both the max and the argmax of an > array. > > > (And same for the other arg* functions) > > > > You have to do something like > > > a = rand(10,5) > > > imax = a.argmax(axis=0) > > > vmax = a[(imax, range(5))] > > > > > I don't generally like overloading return values, the function > starts to > > lose its definition and becomes a bit baroque where simply > changing a > > keyword value can destroy the viability of the following code. > > Agreed. Seems like the only justification is if you get multiple > results from one calculation but only rarely want the extra values. > It doesn't make sense to always return them, but it's also not worth > making a totally different function. > > > > But I can see the utility of what you want. Hmm, this problem > is not unique to argmax. > > Maybe what we need is a general way to extract values, something > like > > > > extract(a, imax, axis=0) > > > > to go along with all the single axis functions. > > Yes, I think that would be easier to remember. > > It should also work for the axis=None case. > imax = a.argmax(axis=None) > v = extract(a, imax, axis=None) > > > It shouldn't be too difficult to jig something up given all the > example code. I can do that, but I would like more input first. The > questions I have are these. > > 1) Should it be done? > 2) Should it be a method? (functions being somewhat deprecated) > 3) What name should it have? > > I think Travis will have to weigh in on this. IIRC, he felt that the > number of methods was getting out of hand. I can support adding a *function* that does both. It can't be named extract (that already exists). There should be one for all the "arg"-like functions. If somebody doesn't add it before 1.0 final, it can wait for 1.0.1 -Travis From oliphant at ee.byu.edu Tue Sep 19 19:53:32 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 17:53:32 -0600 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: References: Message-ID: <451082FC.2060404@ee.byu.edu> Charles R Harris wrote: > Travis, > > Is this intentional? > > In [77]: arange(5, dtype=int)/0 > Out[77]: array([0, 0, 0, 0, 0]) > > It looks deliberate because all zeros are returned, but it might be > better if it raised an exception. As mentioned before we translate integer division errors into floating-point erros and use the same hardware trapping to trap them if the user requests it. Simulating and "integer-division-by-zero" hardware flag is not trivial as we would have to manage context switching ourselves. So, at least for 1.0, integer and floating-point division by zero are going to be handled the same. -Travis From haase at msg.ucsf.edu Tue Sep 19 19:59:08 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 16:59:08 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451073A5.4060204@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> Message-ID: <200609191659.08788.haase@msg.ucsf.edu> On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > Sebastian Haase wrote: > >can we please change dtype to default to float64 !? > > The default is float64 now (as long as you are not using > numpy.oldnumeric). > > I suppose more appropriately, we could reduce over float for integer > data-types when calculating the mean as well (since a floating point is > returned anyway). > Is now mean() always "reducing over" float64 ? The svn note """Log: Fix mean, std, and var methods so that they reduce over double data-type with integer inputs. """ makes it sound that a float32 input is stays float32 ? For mean calculation this might introduce large errors - I usually would require double-precision for *any* input type ... (don't know how to say this for complex types !? Are here real and imag treated separately / independently ?) Thanks, Sebastian Haase From oliphant at ee.byu.edu Tue Sep 19 20:17:47 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 19 Sep 2006 18:17:47 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191659.08788.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> Message-ID: <451088AB.4050107@ee.byu.edu> Sebastian Haase wrote: >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > > >>Sebastian Haase wrote: >> >> > > > >>>can we please change dtype to default to float64 !? >>> >>> >>The default is float64 now (as long as you are not using >>numpy.oldnumeric). >> >>I suppose more appropriately, we could reduce over float for integer >>data-types when calculating the mean as well (since a floating point is >>returned anyway). >> >> >> > >Is now mean() always "reducing over" float64 ? >The svn note """Log: >Fix mean, std, and var methods so that they reduce over double data-type with >integer inputs. >""" >makes it sound that a float32 input is stays float32 ? > > Yes, that is true. Only integer inputs are changed because you are going to get a floating point output anyway. >For mean calculation this might introduce large errors - I usually would >require double-precision for *any* input type ... > > Of course. The system is not fool-proof. I hesitate to arbitrarily change this. The advantage of using single-precision calculation is that it is faster. We do rely on the user who expressly requests these things to be aware of the difficulties. >(don't know how to say this for complex types !? Are here real and imag >treated separately / independently ?) > > There is a complex add performed at a low-level as two separate adds. The addition is performed in the precision requested. -Travis From robert.kern at gmail.com Tue Sep 19 20:18:52 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 19:18:52 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191659.08788.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > (don't know how to say this for complex types !? Are here real and imag > treated separately / independently ?) Yes. For mean(), there's really no alternative. Scalar variance is not a well-defined concept for complex numbers, but treating the real and imaginary parts separately is a sensible and (partially) informative thing to do. Simply applying the formula for estimating variance for real numbers to complex numbers (i.e. change "x" to "z") is a meaningless operation. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Sep 19 20:37:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 18:37:01 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451088AB.4050107@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: On 9/19/06, Travis Oliphant wrote: > > Sebastian Haase wrote: > > >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > > > > > >>Sebastian Haase wrote: > >> > >> > > > > > > > >>>can we please change dtype to default to float64 !? > >>> > >>> > >>The default is float64 now (as long as you are not using > >>numpy.oldnumeric). > >> > >>I suppose more appropriately, we could reduce over float for integer > >>data-types when calculating the mean as well (since a floating point is > >>returned anyway). > >> > >> > >> > > > >Is now mean() always "reducing over" float64 ? > >The svn note """Log: > >Fix mean, std, and var methods so that they reduce over double data-type > with > >integer inputs. > >""" > >makes it sound that a float32 input is stays float32 ? > > > > > Yes, that is true. Only integer inputs are changed because you are > going to get a floating point output anyway. > > >For mean calculation this might introduce large errors - I usually would > >require double-precision for *any* input type ... > > > > > Of course. The system is not fool-proof. I hesitate to arbitrarily > change this. The advantage of using single-precision calculation is > that it is faster. We do rely on the user who expressly requests these > things to be aware of the difficulties. Speed depends on the achitecture. Float is a trifle slower than double on my Athlon64, but faster on PPC750. I don't know about other machines. I think there is a good argument for accumlating in double and converting to float for output if required. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 20:41:30 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 20:41:30 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 19/09/06, Charles R Harris wrote: > > > > For floats we could use something like: > > lessthan(a,b) := a < b || (a == nan && b != nan) > > Which would put all the nans at one end and might not add too much overhead. You could put an any(isnan()) out front and run this slower version only if there are any NaNs (also, you can't use == for NaNs, you have to use C isNaN). But I'm starting to see the wisdom in simply throwing an exception, since sorting is not well-defined with NaNs. A. M. Archibald From haase at msg.ucsf.edu Tue Sep 19 20:44:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 17:44:43 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451088AB.4050107@ee.byu.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: <200609191744.43847.haase@msg.ucsf.edu> On Tuesday 19 September 2006 17:17, Travis Oliphant wrote: > Sebastian Haase wrote: > >On Tuesday 19 September 2006 15:48, Travis Oliphant wrote: > >>Sebastian Haase wrote: > > > > > > > >>>can we please change dtype to default to float64 !? > >> > >>The default is float64 now (as long as you are not using > >>numpy.oldnumeric). > >> > >>I suppose more appropriately, we could reduce over float for integer > >>data-types when calculating the mean as well (since a floating point is > >>returned anyway). > > > >Is now mean() always "reducing over" float64 ? > >The svn note """Log: > >Fix mean, std, and var methods so that they reduce over double data-type > > with integer inputs. > >""" > >makes it sound that a float32 input is stays float32 ? > > Yes, that is true. Only integer inputs are changed because you are > going to get a floating point output anyway. > > >For mean calculation this might introduce large errors - I usually would > >require double-precision for *any* input type ... > > Of course. The system is not fool-proof. I hesitate to arbitrarily > change this. The advantage of using single-precision calculation is > that it is faster. We do rely on the user who expressly requests these > things to be aware of the difficulties. I still would argue that getting a "good" (smaller rounding errors) answer should be the default -- if speed is wanted, then *that* could be still specified by explicitly using dtype=float32 (which would also be a possible choice for int32 input) . In image processing we always want means to be calculated in float64 even though input data is always float32 (if not uint16). Also it is simpler to say "float64 is the default" (full stop.) - instead "float64 is the default unless you have float32" Thanks, Sebastian Haase From charlesr.harris at gmail.com Tue Sep 19 20:55:17 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 18:55:17 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Charles R Harris wrote: > > > > > > > > For floats we could use something like: > > > > lessthan(a,b) := a < b || (a == nan && b != nan) > > > > Which would put all the nans at one end and might not add too much > overhead. > > You could put an any(isnan()) out front and run this slower version > only if there are any NaNs (also, you can't use == for NaNs, you have > to use C isNaN). But I'm starting to see the wisdom in simply throwing > an exception, since sorting is not well-defined with NaNs. Looks like mergesort can be modified to sort around the NaNs without too much trouble if there is a good isnan function available: just cause the pointers to skip over them. I see that most of the isnan stuff seems to be in the ufunc source and isn't terribly simple. Could be broken out into a separate include, I suppose. I still wonder if it is worth the trouble. As to raising an exception, I seem to recall reading somewhere that exception code tends to be expensive, I haven't done any benchmarks myself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From NetBankNotification at cba.com.au Tue Sep 19 21:42:13 2006 From: NetBankNotification at cba.com.au (Commonwealth Bank Groups) Date: Tue, 19 Sep 2006 21:42:13 -0400 Subject: [Numpy-discussion] Security Alert Message-ID: An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Tue Sep 19 21:45:36 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 19 Sep 2006 18:45:36 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> Message-ID: <45109D40.7020008@ieee.org> Charles R Harris wrote: > > > On 9/19/06, *A. M. Archibald* > wrote: > > On 19/09/06, Charles R Harris > wrote: > > > > > > > > For floats we could use something like: > > > > lessthan(a,b) := a < b || (a == nan && b != nan) > I believe this would have to be some sort of isnan macros since everything compares not equal to nan. I forget the correct spelling at the moment though, > > > > Which would put all the nans at one end and might not add too > much overhead. > > You could put an any(isnan()) out front and run this slower version > only if there are any NaNs (also, you can't use == for NaNs, you have > to use C isNaN). But I'm starting to see the wisdom in simply throwing > an exception, since sorting is not well-defined with NaNs. > > > Looks like mergesort can be modified to sort around the NaNs without > too much trouble if there is a good isnan function available: just > cause the pointers to skip over them. I see that most of the isnan > stuff seems to be in the ufunc source and isn't terribly simple. Could > be broken out into a separate include, I suppose. > > I still wonder if it is worth the trouble. As to raising an exception, > I seem to recall reading somewhere that exception code tends to be > expensive, I haven't done any benchmarks myself. I'm still somewhat mystified by the desire to move the nans to one end of the sorted object. I see two scenarios: 1. The user is not expecting to have nans in the data. In this case moving the nans to end is not helpful. The resulting array is still not sorted in the sense that a[i-1]<=a[i]<=a[i+1] does not hold and thus is likely to break code that relies on the array being sorted. The most prominent example of which is searchsorted. In this case you really want to raise an exception if possible since no good will come from letting the code continue to run. In this case the time in involved in throwing and catching an exception is irrelevant. 2. The user *is* expecting to have nans in the data. This is presumably the case that the sorting-nans-to-the-end idea is aimed at. So far at least the suggested use has been to sort and then strip the nans. I suggest that a better approach is to test for and strip the nans before the sort. For example: # a is an array that may have some nans # you can do this more pithily, but I'm aiming to minimize isnan calls # note that this *sometimes* makes a copy. nanmask = isnan(a) if sometrue(nanmask): a = a[not nanmask] a.sort() #..... I presume that isnan is somewhat more expensive than the basic '<' operator. In the proposed sort to end version we need N*log(N) isnan calls versus just N in the above case. The sort to end case probably won't look any cleaner than the above either since you still need to count the nans to determine how many to strip. Perhaps there's some use for the sort to end behaviour that I'm missing, but the raise an exception behaviour sure looks a lot more appealing to me. Here's a strawman proposal: Sort the array. Then examine numpy.geterr()['invalid']. If it is not 'ignore', then check examine sometrue(isnan(thearray)). If the latter is true then raise and error, issue a warning or call the error reporting functioni as appropriate. Note that we always sort the array to be consistent with the behaviour of the ufuncs that proceed even when they end up raising an exception. -tim From peridot.faceted at gmail.com Tue Sep 19 22:09:33 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 22:09:33 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45109D40.7020008@ieee.org> References: <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: On 19/09/06, Tim Hochberg wrote: > I'm still somewhat mystified by the desire to move the nans to one end > of the sorted object. I see two scenarios: It's mostly to have something to do with them other than throw an exception. Leaving them in place while the rest of the array is reshuffled requires a lot of work and isn't particularly better. I mostly presented it as an alternative to throwing an exception. Throwing a Python exception now seems like the most reasonable idea. A. M. Archibald From oliphant.travis at ieee.org Tue Sep 19 22:31:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 20:31:07 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <451073A5.4060204@ee.byu.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> Message-ID: <4510A7EB.8000207@ieee.org> Charles R Harris wrote: > > Speed depends on the achitecture. Float is a trifle slower than double > on my Athlon64, but faster on PPC750. I don't know about other > machines. I think there is a good argument for accumlating in double > and converting to float for output if required. Yes there is. It's just not what NumPy ever does so it would be an exception in this case and would need a more convincing argument in my opinion. You can always specify the accumulation type yourself with the dtype argument. We are only talking about what the default should be. -Travis From oliphant.travis at ieee.org Tue Sep 19 22:33:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 19 Sep 2006 20:33:52 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609191744.43847.haase@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> Message-ID: <4510A890.5090807@ieee.org> Sebastian Haase wrote: > I still would argue that getting a "good" (smaller rounding errors) answer > should be the default -- if speed is wanted, then *that* could be still > specified by explicitly using dtype=float32 (which would also be a possible > choice for int32 input) . > So you are arguing for using long double then.... ;-) > In image processing we always want means to be calculated in float64 even > though input data is always float32 (if not uint16). > > Also it is simpler to say "float64 is the default" (full stop.) - instead > > "float64 is the default unless you have float32" > "the type you have is the default except for integers". Do you really want float64 to be the default for float96? Unless we are going to use long double as the default, then I'm not convinced that we should special-case the "double" type. -Travis From charlesr.harris at gmail.com Tue Sep 19 22:42:52 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 20:42:52 -0600 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: On 9/19/06, A. M. Archibald wrote: > > On 19/09/06, Tim Hochberg wrote: > > > I'm still somewhat mystified by the desire to move the nans to one end > > of the sorted object. I see two scenarios: > > It's mostly to have something to do with them other than throw an > exception. Leaving them in place while the rest of the array is > reshuffled requires a lot of work and isn't particularly better. I > mostly presented it as an alternative to throwing an exception. > > Throwing a Python exception now seems like the most reasonable idea. Well, mergesort can be adapted without a lot of work. Could be used to sort masked arrays too, not that I know why anyone would want that, but then again, I haven't used masked arrays. Agreed, throwing some sort of error still seems the simplest thing to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Tue Sep 19 23:07:37 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 20:07:37 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510A890.5090807@ieee.org> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> Message-ID: <4510B079.5010209@msg.ucsf.edu> Travis Oliphant wrote: > Sebastian Haase wrote: >> I still would argue that getting a "good" (smaller rounding errors) answer >> should be the default -- if speed is wanted, then *that* could be still >> specified by explicitly using dtype=float32 (which would also be a possible >> choice for int32 input) . >> > So you are arguing for using long double then.... ;-) > >> In image processing we always want means to be calculated in float64 even >> though input data is always float32 (if not uint16). >> >> Also it is simpler to say "float64 is the default" (full stop.) - instead >> >> "float64 is the default unless you have float32" >> > "the type you have is the default except for integers". Do you really > want float64 to be the default for float96? > > Unless we are going to use long double as the default, then I'm not > convinced that we should special-case the "double" type. > I guess I'm not really aware of the float96 type ... Is that a "machine type" on any system ? I always thought that -- e.g. coming from C -- double is "as good as it gets"... Who uses float96 ? I heard somewhere that (some) CPUs use 80bits internally when calculating 64bit double-precision... Is this not going into some academic argument !? For all I know, calculating mean()s (and sum()s, ...) is always done in double precision -- never in single precision, even when the data is in float32. Having float32 be the default for float32 data is just requiring more typing, and more explaining ... it would compromise numpy usability as a day-to-day replacement for other systems. Sorry, if I'm being ignorant ... - Sebastian From charlesr.harris at gmail.com Tue Sep 19 23:15:29 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 21:15:29 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510B079.5010209@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Sebastian Haase wrote: > > Travis Oliphant wrote: > > Sebastian Haase wrote: > >> I still would argue that getting a "good" (smaller rounding errors) > answer > >> should be the default -- if speed is wanted, then *that* could be still > >> specified by explicitly using dtype=float32 (which would also be a > possible > >> choice for int32 input) . > >> > > So you are arguing for using long double then.... ;-) > > > >> In image processing we always want means to be calculated in float64 > even > >> though input data is always float32 (if not uint16). > >> > >> Also it is simpler to say "float64 is the default" (full stop.) - > instead > >> > >> "float64 is the default unless you have float32" > >> > > "the type you have is the default except for integers". Do you really > > want float64 to be the default for float96? > > > > Unless we are going to use long double as the default, then I'm not > > convinced that we should special-case the "double" type. > > > I guess I'm not really aware of the float96 type ... > Is that a "machine type" on any system ? I always thought that -- e.g. > coming from C -- double is "as good as it gets"... > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > internally when calculating 64bit double-precision... > > Is this not going into some academic argument !? > For all I know, calculating mean()s (and sum()s, ...) is always done in > double precision -- never in single precision, even when the data is in > float32. > > Having float32 be the default for float32 data is just requiring more > typing, and more explaining ... it would compromise numpy usability as > a day-to-day replacement for other systems. > > Sorry, if I'm being ignorant ... I'm going to side with Travis here. It is only a default and easily overridden. And yes, there are precisions greater than double. I was using quad precision back in the eighties on a VAX for some inherently ill conditioned problems. And on my machine long double is 12 bytes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Tue Sep 19 23:24:47 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 20 Sep 2006 12:24:47 +0900 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/20/06, Charles R Harris wrote: > > I guess I'm not really aware of the float96 type ... > > Is that a "machine type" on any system ? I always thought that -- e.g . > > coming from C -- double is "as good as it gets"... > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > internally when calculating 64bit double-precision... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was using > quad precision back in the eighties on a VAX for some inherently ill > conditioned problems. And on my machine long double is 12 bytes. > And on Intel chips the internal fp precision is 80bits. The D programming language even exposes this 80-bit floating point type to the user. http://www.digitalmars.com/d/type.html http://www.digitalmars.com/d/faq.html#real --bb From haase at msg.ucsf.edu Tue Sep 19 23:39:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 20:39:04 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: <4510B7D8.4010708@msg.ucsf.edu> Charles R Harris wrote: > > > On 9/19/06, *Sebastian Haase* > wrote: > > Travis Oliphant wrote: > > Sebastian Haase wrote: > >> I still would argue that getting a "good" (smaller rounding > errors) answer > >> should be the default -- if speed is wanted, then *that* could > be still > >> specified by explicitly using dtype=float32 (which would also > be a possible > >> choice for int32 input) . > >> > > So you are arguing for using long double then.... ;-) > > > >> In image processing we always want means to be calculated in > float64 even > >> though input data is always float32 (if not uint16). > >> > >> Also it is simpler to say "float64 is the default" (full stop.) > - instead > >> > >> "float64 is the default unless you have float32" > >> > > "the type you have is the default except for integers". Do you > really > > want float64 to be the default for float96? > > > > Unless we are going to use long double as the default, then I'm not > > convinced that we should special-case the "double" type. > > > I guess I'm not really aware of the float96 type ... > Is that a "machine type" on any system ? I always thought that -- e.g . > coming from C -- double is "as good as it gets"... > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > internally when calculating 64bit double-precision... > > Is this not going into some academic argument !? > For all I know, calculating mean()s (and sum()s, ...) is always done in > double precision -- never in single precision, even when the data is in > float32. > > Having float32 be the default for float32 data is just requiring more > typing, and more explaining ... it would compromise numpy usability as > a day-to-day replacement for other systems. > > Sorry, if I'm being ignorant ... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was > using quad precision back in the eighties on a VAX for some inherently > ill conditioned problems. And on my machine long double is 12 bytes. > > Chuck > I just did a web search for "long double" http://www.google.com/search?q=%22long+double%22 and it does not look like there is much agreement on what that is - see also http://en.wikipedia.org/wiki/Long_double I really think that float96 *is* a special case - but computing mean()s and var()s in float32 would be "bad science". I hope I'm not alone in seeing numpy a great "interactive platform" to do evaluate data... I know that having too much knowledge of the details often makes one forget what the "newcomers" will do and expect. We are only talking about people that will a) work with single-precision data (e.g. large scale-image analysis) and who b) will tend to "just use the default" (dtype) --- How else can I say this: these people will just assume that arr.mean() *is* the mean of arr. -Sebastian From charlesr.harris at gmail.com Tue Sep 19 23:42:58 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 21:42:58 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Charles R Harris wrote: > > > > On 9/19/06, Sebastian Haase wrote: > > > > Travis Oliphant wrote: > > > Sebastian Haase wrote: > > >> I still would argue that getting a "good" (smaller rounding errors) > > answer > > >> should be the default -- if speed is wanted, then *that* could be > > still > > >> specified by explicitly using dtype=float32 (which would also be a > > possible > > >> choice for int32 input) . > > >> > > > So you are arguing for using long double then.... ;-) > > > > > >> In image processing we always want means to be calculated in float64 > > even > > >> though input data is always float32 (if not uint16). > > >> > > >> Also it is simpler to say "float64 is the default" (full stop.) - > > instead > > >> > > >> "float64 is the default unless you have float32" > > >> > > > "the type you have is the default except for integers". Do you really > > > want float64 to be the default for float96? > > > > > > Unless we are going to use long double as the default, then I'm not > > > convinced that we should special-case the "double" type. > > > > > I guess I'm not really aware of the float96 type ... > > Is that a "machine type" on any system ? I always thought that -- e.g . > > coming from C -- double is "as good as it gets"... > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > internally when calculating 64bit double-precision... > > > > Is this not going into some academic argument !? > > For all I know, calculating mean()s (and sum()s, ...) is always done in > > double precision -- never in single precision, even when the data is in > > float32. > > > > Having float32 be the default for float32 data is just requiring more > > typing, and more explaining ... it would compromise numpy usability as > > a day-to-day replacement for other systems. > > > > Sorry, if I'm being ignorant ... > > > I'm going to side with Travis here. It is only a default and easily > overridden. And yes, there are precisions greater than double. I was using > quad precision back in the eighties on a VAX for some inherently ill > conditioned problems. And on my machine long double is 12 bytes. > Here is the 754r (revision) spec: http://en.wikipedia.org/wiki/IEEE_754r It includes quads (128 bits) and half precision (16 bits) floats. I believe the latter are used for some imaging stuff, radar for instance, and are also available in some high end GPUs from Nvidia and other companies. The 80 bit numbers you refer to were defined as extended precision in the original 754 spec and were mostly intended for temporaries in internal FPU computations. They have various alignment requirements for efficient use, which is why they show up as 96 bits (4 byte alignment) and sometimes 128 bits (8 byte alignment). So actually, float128 would not always distinquish between extended precision and quad precision. I see more work for Travis in the future ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Sep 19 23:53:26 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 19 Sep 2006 23:53:26 -0400 Subject: [Numpy-discussion] ufunc.reduce and conversion Message-ID: Hi, What are the rules for datatype conversion in ufuncs? Does ufunc(a,b) always yield the smallest type big enough to represent both a and b? What is the datatype of ufunc.reduce(a)? I ask because I was startled by the following behaviour: >>> a = array([1,1],uint8); print a.dtype; print maximum.reduce(a).dtype; '|u1' ' float, possibly only for add.reduce). Thanks, A. M. Archibald From robert.kern at gmail.com Wed Sep 20 00:10:26 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Sep 2006 23:10:26 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510B7D8.4010708@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > I know that having too much knowledge of the details often makes one > forget what the "newcomers" will do and expect. Please be more careful with such accusations. Repeated frequently, they can become quite insulting. > We are only talking > about people that will a) work with single-precision data (e.g. large > scale-image analysis) and who b) will tend to "just use the default" > (dtype) --- How else can I say this: these people will just assume that > arr.mean() *is* the mean of arr. I don't understand what you mean, here. arr.mean() is almost never *the* mean of arr. Double precision can't fix that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Sep 20 00:18:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 19 Sep 2006 22:18:53 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> Message-ID: On 9/19/06, Charles R Harris wrote: > > > > On 9/19/06, Charles R Harris wrote: > > > > > > > > On 9/19/06, Sebastian Haase < haase at msg.ucsf.edu> wrote: > > > > > > Travis Oliphant wrote: > > > > Sebastian Haase wrote: > > > >> I still would argue that getting a "good" (smaller rounding errors) > > > answer > > > >> should be the default -- if speed is wanted, then *that* could be > > > still > > > >> specified by explicitly using dtype=float32 (which would also be a > > > possible > > > >> choice for int32 input) . > > > >> > > > > So you are arguing for using long double then.... ;-) > > > > > > > >> In image processing we always want means to be calculated in > > > float64 even > > > >> though input data is always float32 (if not uint16). > > > >> > > > >> Also it is simpler to say "float64 is the default" (full stop.) - > > > instead > > > >> > > > >> "float64 is the default unless you have float32" > > > >> > > > > "the type you have is the default except for integers". Do you > > > really > > > > want float64 to be the default for float96? > > > > > > > > Unless we are going to use long double as the default, then I'm not > > > > convinced that we should special-case the "double" type. > > > > > > > I guess I'm not really aware of the float96 type ... > > > Is that a "machine type" on any system ? I always thought that -- e.g. > > > coming from C -- double is "as good as it gets"... > > > Who uses float96 ? I heard somewhere that (some) CPUs use 80bits > > > internally when calculating 64bit double-precision... > > > > > > Is this not going into some academic argument !? > > > For all I know, calculating mean()s (and sum()s, ...) is always done > > > in > > > double precision -- never in single precision, even when the data is > > > in > > > float32. > > > > > > Having float32 be the default for float32 data is just requiring more > > > typing, and more explaining ... it would compromise numpy usability > > > as > > > a day-to-day replacement for other systems. > > > > > > Sorry, if I'm being ignorant ... > > > > > > I'm going to side with Travis here. It is only a default and easily > > overridden. And yes, there are precisions greater than double. I was using > > quad precision back in the eighties on a VAX for some inherently ill > > conditioned problems. And on my machine long double is 12 bytes. > > > > Here is the 754r (revision) spec: http://en.wikipedia.org/wiki/IEEE_754r > > It includes quads (128 bits) and half precision (16 bits) floats. I > believe the latter are used for some imaging stuff, radar for instance, and > are also available in some high end GPUs from Nvidia and other companies. > The 80 bit numbers you refer to were defined as extended precision in the > original 754 spec and were mostly intended for temporaries in internal FPU > computations. They have various alignment requirements for efficient use, > which is why they show up as 96 bits (4 byte alignment) and sometimes 128 > bits (8 byte alignment). So actually, float128 would not always distinquish > between extended precision and quad precision. I see more work for Travis > in the future ;) > I just checked this out. On amd64 32 bit linux gives 12 bytes for long double, 64 bit linux gives 16 bytes for long doubles, but they both have 64 bit mantissas, i.e., they are both 80 bit extended precision. Those sizes are the defaults and can be overridden by compiler flags. Anyway, we may need some way to tell the difference between float128 and quads since they will both have the same length on 64 bit architectures. But that is a problem for the future. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Wed Sep 20 00:53:16 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 19 Sep 2006 21:53:16 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> Message-ID: <4510C93C.8030902@msg.ucsf.edu> Robert Kern wrote: > Sebastian Haase wrote: >> I know that having too much knowledge of the details often makes one >> forget what the "newcomers" will do and expect. > > Please be more careful with such accusations. Repeated frequently, they can > become quite insulting. > I did not mean to insult anyone - what I meant was, that I'm for numpy becoming an easy platform to use. I have spend and enjoyed part the last four years developing and evangelizing Python as an alternative to Matlab and C/Fortran based image analysis environment. I often find myself arguing for good support of the single precision data format. So I find it actually somewhat ironic to see myself arguing now for wanting float64 over float32 ;-) >> We are only talking >> about people that will a) work with single-precision data (e.g. large >> scale-image analysis) and who b) will tend to "just use the default" >> (dtype) --- How else can I say this: these people will just assume that >> arr.mean() *is* the mean of arr. > > I don't understand what you mean, here. arr.mean() is almost never *the* mean of > arr. Double precision can't fix that. > This was not supposed to be a scientific statement -- I'm (again) thinking of our students that not always appreciate the full complexity of computational numerics and data types and such. The best I can hope for is a "sound" default for most (practical) cases... I still think that 80bit vs. 128bit vs 96bit is rather academic for most people ... most people seem to only use float64 and then there are some that use float32 (like us) ... Cheers, Sebastian From konrad.hinsen at laposte.net Wed Sep 20 03:41:23 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Wed, 20 Sep 2006 09:41:23 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: <45101F5F.6080209@noaa.gov> References: <45101F5F.6080209@noaa.gov> Message-ID: <3BB688EB-A95D-464F-8AAC-674CA4E5ECB5@laposte.net> On 19.09.2006, at 18:48, Christopher Barker wrote: > Konrad Hinsen wrote: >> MMTK works fine with Numeric 23.x (and probably many other versions), >> so I don't see a pressing need to change to NumPy. > > Pressing is in the eye of the beholder. Obviously. It also depends on the context in which one develops or uses software. For me, a pressing need is to finish the two publications I am working on. Next come various administrational tasks that have a deadline. On the third place, there's work on a new research project that I started recently. Software development is at best on position number 4, but in that category my personal priorities are adding more unit tests and reworking the MMTK documentation system, the old one being unusable because various pieces of code it relied on are no longer supported. As with many other scientific software projects, MMTK development is completely unfunded and even hardly recognized by my employer's evaluation committees. Any work on MMTK that does not help me in a research project can therefore not be a priority for me. > However: I don't think we should underestimate the negative impact of > the Numeric/numarray split on the usability and perception of the > community. Also the impact on how much work has been done to > accommodate it. If you consider matplotlib alone: I completely agree. The existence of a third choice (NumPy) just makes it worse. For client code like mine there is little chance to escape from the split issues. Even if I had the resources to adapt all my code to NumPy immediately, I'd still have to support Numeric because that's what everyone is using at the moment, and many users can't or won't switch immediately. Since the APIs are not fully compatible, I either have to support two versions in parallel, or introduce my own compatibility layer. > In addition, as I understand it, MMTK was NOT working fine for the OP. The issues he had were already solved, he just had to apply the solutions (i.e. reinstall using a more recent version and appropriate compilation options). > As robust as they (and Numeric) might be, when you need to run > something on a new platform (OS-X - Intel comes to mind), or use a new > LAPACK, or whatever, there are going to be (and have been) issues that > need to be addressed. No one is maintaining Numeric, so it makes much > more sense to put your effort into porting to numpy, rather than > trying > to fix or work around Numeric issues. In the long run, I agree. But on the time scale on which is what my work conditions force me to plan, it is less work for me to provide patches for Numeric as the need arises. > PS: this really highlights the strength of having good unit tests: as > far as i can tell, it's really not that much work to do the port -- > the > work is in the testing. Comprehensive units tests would make that part > trivial too. Yes, testing is the bigger chunk of the work. And yes, unit tests are nice to have - but they don't write themselves, unfortunately. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From konrad.hinsen at laposte.net Wed Sep 20 03:45:27 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Wed, 20 Sep 2006 09:45:27 +0200 Subject: [Numpy-discussion] [MMTK] Re: Fwd: Collection.findTransformation() never stops In-Reply-To: <45103A31.60704@ieee.org> References: <45103A31.60704@ieee.org> Message-ID: On 19.09.2006, at 20:42, Travis Oliphant wrote: > Well, I got both ScientificPython and MMTK to compile and import > using > the steps outlined on http://www.scipy.org/Porting_to_NumPy in > about 1 > hour (including time to fix alter_code1 to make the process even > easier). Could you please send me those versions? I'd happily put them on my Web site for volunteers to test. > I was able to install ScientificPython and MMTK for NumPy on my system > using the patches provided on that page. Is there a test suite that > can be run? Not much yet, unfortunately. There is a minuscule test suite for ScientificPython in the latest release, and an even more miniscule one for MMTK that I didn't even publish yet because it doesn't test more than that everything can be imported. > Users of MMTK could really help out here. I hope so! Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Centre de Biophysique Mol?culaire, CNRS Orl?ans Synchrotron Soleil - Division Exp?riences Saint Aubin - BP 48 91192 Gif sur Yvette Cedex, France Tel. +33-1 69 35 97 15 E-Mail: hinsen at cnrs-orleans.fr --------------------------------------------------------------------- From faltet at carabos.com Wed Sep 20 03:52:26 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 09:52:26 +0200 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping Message-ID: <200609200952.28046.faltet@carabos.com> Hi, I'm sending a message here because discussing about this in the bug tracker is not very comfortable. This my last try before giving up, so don't be afraid ;-) In bug #283 (http://projects.scipy.org/scipy/numpy/ticket/283) I complained about the fact that a numpy.int32 is being mapped in NumPy to NPY_LONG enumerated type and I think I failed to explain well why I think this is a bad thing. Now, I'll try to expose an (real life) example, in the hope that things will make clearer. Realize that you are coding a C extension that receives NumPy arrays for saving them on-disk for a later retrieval. Realize also that an user is using your extension on a 32-bit platform. If she pass to this extension an array of type 'int32', and the extension tries to read the enumerated type (using array.dtype.num), it will get NPY_LONG. So, the extension use this code (NPY_LONG) to save the type (together with data) on-disk. Now, she send this data file to a teammate that works on a 64-bit machine, and tries to read the data using the same extension. The extension would see that the data is NPY_LONG type and would try to deserialize interpreting data elements as being as 64-bit integer (this is the size of a NPY_LONG in 64-bit platforms), and this is clearly wrong. Besides this, if for making your C extension you are using a C library that is meant to save data in a platform-independent (say, HDF5), then, having a NPY_LONG will not automatically say which C library datatype maps to, because it only have datatypes that are of a definite size in all platforms. So, this is a second problem. Of course there are workarounds for this, but my impression is that they can be avoided with a more sensible mapping between NumPy Python types and NumPy enumerated types, like: numpy.int32 --> NPY_INT numpy.int64 --> NPY_LONGLONG numpy.int_ --> NPY_LONG in all platforms, avoiding the current situation of ambiguous mapping between platforms. Sorry for being so persistent, but I think the issue is worth it. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Wed Sep 20 04:01:18 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 20 Sep 2006 03:01:18 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510C93C.8030902@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > Robert Kern wrote: >> Sebastian Haase wrote: >>> I know that having too much knowledge of the details often makes one >>> forget what the "newcomers" will do and expect. >> Please be more careful with such accusations. Repeated frequently, they can >> become quite insulting. >> > I did not mean to insult anyone - what I meant was, that I'm for numpy > becoming an easy platform to use. I have spend and enjoyed part the last > four years developing and evangelizing Python as an alternative to > Matlab and C/Fortran based image analysis environment. I often find > myself arguing for good support of the single precision data format. So > I find it actually somewhat ironic to see myself arguing now for wanting > float64 over float32 ;-) No one is doubting that you want numpy to be easy to use. Please don't doubt that the rest of us want otherwise. However, the fact that you *want* numpy to be easy to use does not mean that your suggestions *will* make numpy easy to use. We haven't forgotten what newcomers will do; to the contrary, we are quite aware that new users need consistent behavior in order to learn how to use a system. Adding another special case in how dtypes implicitly convert to one another will impede new users being able to understand the whole system. See A. M. Archibald's question in the thread "ufunc.reduce and conversion" for an example. In our judgement this is a worse outcome than notational convenience for float32 users, who already need to be aware of the effects of their precision choice. Each of us can come to different conclusions in good faith without one of us forgetting the new user experience. Let me offer a third path: the algorithms used for .mean() and .var() are substandard. There are much better incremental algorithms that entirely avoid the need to accumulate such large (and therefore precision-losing) intermediate values. The algorithms look like the following for 1D arrays in Python: def mean(a): m = a[0] for i in range(1, len(a)): m += (a[i] - m) / (i + 1) return m def var(a): m = a[0] t = a.dtype.type(0) for i in range(1, len(a)): q = a[i] - m r = q / (i+1) m += r t += i * q * r t /= len(a) return t Alternatively, from Knuth: def var_knuth(a): m = a.dtype.type(0) variance = a.dtype.type(0) for i in range(len(a)): delta = a[i] - m m += delta / (i+1) variance += delta * (a[i] - m) variance /= len(a) return variance If you will code up implementations of these for ndarray.mean() and ndarray.var(), I will check them in and then float32 arrays will avoid most of the catastrophes that the current implementations run into. >>> We are only talking >>> about people that will a) work with single-precision data (e.g. large >>> scale-image analysis) and who b) will tend to "just use the default" >>> (dtype) --- How else can I say this: these people will just assume that >>> arr.mean() *is* the mean of arr. >> I don't understand what you mean, here. arr.mean() is almost never *the* mean of >> arr. Double precision can't fix that. >> > This was not supposed to be a scientific statement -- I'm (again) > thinking of our students that not always appreciate the full complexity > of computational numerics and data types and such. They need to appreciate the complexity of computational numerics if they are going to do numerical computation. Double precision does not make it any simpler. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Wed Sep 20 06:00:20 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 12:00:20 +0200 Subject: [Numpy-discussion] max argmax combo In-Reply-To: References: <200609191941.56395.faltet@carabos.com> Message-ID: <200609201200.21492.faltet@carabos.com> A Dimarts 19 Setembre 2006 21:41, Bill Baxter va escriure: > I think he meant do an argsort first, then use fancy indexing to get > the sorted array. > For a 1-d array that's just > > ind = A.argsort() > Asorted = A[ind] > > That should be O(N lg N + N), aka O(N lg N) I see. Thanks. OTOH, maybe your estimations are right, but the effect of the constants in these O(whatever) estimations can be surprisingly high: In [3]: from timeit import Timer In [4]: Timer("b=numpy.argsort(a);c=a[b]", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[4]: [1.6653108596801758, 1.670341968536377, 1.6632120609283447] In [5]: Timer("b=numpy.argsort(a);c=numpy.sort(a)", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[5]: [1.6533238887786865, 1.6272940635681152, 1.6253311634063721] In [6]: Timer("b=numpy.argsort(a);a.sort();c=a", "import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[6]: [0.95492100715637207, 0.90312504768371582, 0.90426898002624512] so, it seems that argsorting first and fancy indexing later on is the most expensive procedure for relatively large arrays (100000). Interestingly, figures above seems to indicate that in-place sort is stunningly fast: In [7]: Timer("a.sort()","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[7]: [0.32840394973754883, 0.2746579647064209, 0.2770991325378418] and much faster indeed than fancy indexing In [8]: Timer("b[a]","import numpy; a=numpy.arange(100000,-1,-1);b=a.copy()").repeat(3,100) Out[8]: [0.79876089096069336, 0.74172186851501465, 0.74209499359130859] i.e. in-place sort seems 2.7x faster than fancy indexing (at least for these datasets). Mmmm, with this, I really ponder if a combo that makes the argsort() and sort() in one shot really makes any sense, at least from the point of view of speed: In [10]: Timer("b=numpy.argsort(a);a.sort();c=a","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[10]: [0.98506593704223633, 0.89880609512329102, 0.89982390403747559] In [11]: Timer("b=numpy.argsort(a)","import numpy; a=numpy.arange(100000,-1,-1)").repeat(3,100) Out[11]: [0.92959284782409668, 0.85385990142822266, 0.87773990631103516] So, it seems that doing an in-place sort() immediately after an argsort() operation is very efficient (cache effects here?), and would avoid the need of the combo function (from the point of view of efficiency, I repeat). Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Wed Sep 20 06:48:28 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 04:48:28 -0600 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping In-Reply-To: <200609200952.28046.faltet@carabos.com> References: <200609200952.28046.faltet@carabos.com> Message-ID: <45111C7C.2060700@ieee.org> Francesc Altet wrote: > Hi, > > I'm sending a message here because discussing about this in the bug tracker is > not very comfortable. This my last try before giving up, so don't be > afraid ;-) > > In bug #283 (http://projects.scipy.org/scipy/numpy/ticket/283) I complained > about the fact that a numpy.int32 is being mapped in NumPy to NPY_LONG > enumerated type and I think I failed to explain well why I think this is a > bad thing. Now, I'll try to expose an (real life) example, in the hope that > things will make clearer. > > Realize that you are coding a C extension that receives NumPy arrays for > saving them on-disk for a later retrieval. Realize also that an user is using > your extension on a 32-bit platform. If she pass to this extension an array > of type 'int32', and the extension tries to read the enumerated type (using > array.dtype.num), it will get NPY_LONG. > So, the extension use this code > (NPY_LONG) to save the type (together with data) on-disk. Now, she send this > data file to a teammate that works on a 64-bit machine, and tries to read the > data using the same extension. The extension would see that the data is > NPY_LONG type and would try to deserialize interpreting data elements as > being as 64-bit integer (this is the size of a NPY_LONG in 64-bit platforms), > and this is clearly wrong. > > In my view, this "real-life" example points to a flaw in the coding design that will not be fixed by altering what numpy.int32 maps to under the covers. It is wrong to use a code for the platform c data-type (NPY_LONG) as a key to understand data written to disk. This is and always has been a bad idea. No matter what we do with numpy.int32 this can cause problems. Just because a lot of platforms think an int is 32-bits does not mean all of them do. C gives you no such guarantee. Notice that pickling of NumPy arrays does not store the "enumerated type" as the code. Instead it stores the data-type object (which itself pickles using the kind and element size so that the correct data-type object can be reconstructed on the other end --- if it is available at all). Thus, you should not be storing the enumerated type but instead something like the kind and element-size. > Besides this, if for making your C extension you are using a C library that is > meant to save data in a platform-independent (say, HDF5), then, having a > NPY_LONG will not automatically say which C library datatype maps to, because > it only have datatypes that are of a definite size in all platforms. So, this > is a second problem. > > Making sure you get the correct data-type is why there are NPY_INT32 and NPY_INT64 enumerated types. You can't code using NPY_LONG and expect it will give you the same sizes when moving from 32-bit and 64-bit platforms. That's a problem that has been fixed with the bitwidth types. I don't understand why you are using the enumerated types at all in this circumstance. > Of course there are workarounds for this, but my impression is that they can > be avoided with a more sensible mapping between NumPy Python types and NumPy > enumerated types, like: > > numpy.int32 --> NPY_INT > numpy.int64 --> NPY_LONGLONG > numpy.int_ --> NPY_LONG > > in all platforms, avoiding the current situation of ambiguous mapping between > platforms. > The problem is that C gives us this ambiguous mapping. You are asking us to pretend it isn't there because it "simplifies" a hypothetical case so that poor coding practice can be allowed to work in a special case. I'm not convinced. This persists the myth that C data-types have a defined length. This is not guaranteed. The current system defines data-types with a guaranteed length. Yes, there is ambiguity as to which is "the" underlying c-type on certain platforms, but if you are running into trouble with the difference, then you need to change how you are coding because you would run into trouble on some combination of platforms even if we made the change. Basically, you are asking to make a major change, and at this point I'm very hesitant to make such a change without a clear and pressing need for it. Your hypothetical example does not rise to the level of "clear and pressing need." In fact, I see your proposal as a step backwards. Now, it is true that we could change the default type that gets first grab at int32 to be int (instead of the current long) --- I could see arguments for that. But, since the choice is ambiguous and the Python integer type is the c-type long, I let long get first dibs on everything as this seemed to work better for code I was wrapping in the past. I don't see any point in changing this choice now and risk code breakage, especially when your argument is that it would let users think that a c int is always 32-bits. Best regards, -Travis From oliphant.travis at ieee.org Wed Sep 20 06:59:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 04:59:42 -0600 Subject: [Numpy-discussion] ufunc.reduce and conversion In-Reply-To: References: Message-ID: <45111F1E.9080900@ieee.org> A. M. Archibald wrote: > Hi, > > What are the rules for datatype conversion in ufuncs? Does ufunc(a,b) > always yield the smallest type big enough to represent both a and b? > What is the datatype of ufunc.reduce(a)? > This is an unintended consequence of making add.reduce() reduce over at least a ("long"). I've fixed the code so that only add.reduce and multiply.reduce alter the default reducing data-type to be long. All other cases use the data-type of the array as the default. Regarding your other question on data-type conversion in ufuncs: 1) If you specify an output array, then the result will be cast to the output array data-type. 2) The actual computation takes place using a data-type that all (non-scalar) inputs can be cast to safely (with the exception that we assume that long long integers can be "safely" cast to "doubles" even though this is not technically true). -Travis From zdm105 at tom.com Sat Sep 23 07:22:22 2006 From: zdm105 at tom.com (tpm) Date: Sat, 23 Sep 2006 19:22:22 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From zdm105 at tom.com Sat Sep 23 07:31:53 2006 From: zdm105 at tom.com (tpm) Date: Sat, 23 Sep 2006 19:31:53 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From rlw at stsci.edu Wed Sep 20 08:09:57 2006 From: rlw at stsci.edu (Rick White) Date: Wed, 20 Sep 2006 08:09:57 -0400 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: <45109D40.7020008@ieee.org> References: <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> <451063C5.20201@ieee.org> <451065A2.8040406@ee.byu.edu> <45106B38.6010504@ieee.org> <45109D40.7020008@ieee.org> Message-ID: <07857523-4E43-4DDA-8A4C-1A5A96130CBD@stsci.edu> On Sep 19, 2006, at 9:45 PM, Tim Hochberg wrote: > Perhaps there's some use for the sort to end behaviour that I'm > missing, > but the raise an exception behaviour sure looks a lot more > appealing to me. FYI, in IDL the NaN values wind up at the end of the sorted array. That's true despite the fact that IDL does respect all the comparison properties of NaNs (i.e. Value>NaN, Value Here's a strawman proposal: > > Sort the array. Then examine numpy.geterr()['invalid']. If it > is not > 'ignore', then check examine sometrue(isnan(thearray)). If the > latter is true then raise and error, issue a warning or call the > error reporting functioni as appropriate. Note that we always sort > the array to be consistent with the behaviour of the ufuncs that > proceed even when they end up raising an exception. Here's another proposal: Make a first pass through the array, replacing NaN values with Inf and counting the NaNs ("nancount"). Raise an exception at this point if NaNs are not supposed to be allowed. Otherwise sort the array, and then as the last step replace the trailing NaNcount values with NaN. It seems to me that this would give predictable results while respecting the exception flags at little extra cost. Rick From gnata at obs.univ-lyon1.fr Wed Sep 20 09:15:40 2006 From: gnata at obs.univ-lyon1.fr (Xavier Gnata) Date: Wed, 20 Sep 2006 15:15:40 +0200 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> Message-ID: <45113EFC.106@obs.univ-lyon1.fr> IMHO, the only correct way to handle this case is to raise an exception. It does not make sense to compare NaN and "real" numbers. It could be very confusing not to raise an exception. Xavier. > On 19/09/06, Tim Hochberg wrote: > >> A. M. Archibald wrote: >> >>> Mmm. Somebody who's working with NaNs has more or less already decided >>> they don't want to be pestered with exceptions for invalid data. >>> >> Do you really think so? In my experience NaNs are nearly always just an >> indication of a mistake somewhere that didn't get trapped for one reason >> or another. >> > > Well, I said that because for an image porcessing project I was doing, > the easiest thing to do with certain troublesome pixels was to fill in > NaNs, and then at the end replace the NaNs with sensible values. It > seems as if the point of NaNs is to allow you to keep working with > those numbers that make sense while ingoring those that don't. If you > wanted exceptions, why not get them as soon as the first NaN would > have been generated? > > >>> I'd >>> be happy if they wound up at either end, but I'm not sure it's worth >>> hacking up the sort algorithm when a simple isnan() can pull them out. >>> >>> >> Moving them to the end seems to be the worst choice to me. Leaving them >> alone is fine with me. Or raising an exception would be fine. Or doing >> one or the other depending on the error mode settings would be even >> better if it is practical. >> > > I was just thinking in terms of easy removal. > > >> Is that true? Are all of numpy's sorting algorithms robust against >> nontransitive objects laying around? The answer to that appears to be >> no. Try running this a couple of times to see what I mean: >> > > >> The values don't correctly cross the inserted NaN and the sort is incorrect. >> > > You're quite right: when NaNs are present in the array, sorting and > then removing them does not yield a sorted array. For example, > mergesort just output > [ 2. 4. 6. 9. nan 0. 1. > 3. 5. 7. 8. ] > > The other two are no better (and arguably worse). Python's built-in > sort() for lists has the same problem. > > This is definitely a bug, and the best way to fix it is not clear to > me - perhaps sort() needs to always do any(isnan(A)) before starting > to sort. I don't really like raising an exception, but sort() isn't > really very meaningful with NaNs in the array. The only other option I > can think of is to somehow remove them, sort without, and reintroduce > them at the end, which is going to be a nightmare when sorting a > single axis of a large array. Or, I suppose, sort() could simply fill > the array with NaNs; I'm sure users will love that. > > A. M. Archibald > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- ############################################ Xavier Gnata CRAL - Observatoire de Lyon 9, avenue Charles Andr? 69561 Saint Genis Laval cedex Phone: +33 4 78 86 85 28 Fax: +33 4 78 86 83 86 E-mail: gnata at obs.univ-lyon1.fr ############################################ From faltet at carabos.com Wed Sep 20 10:34:35 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 20 Sep 2006 16:34:35 +0200 Subject: [Numpy-discussion] Possible inconsisteny in enumerated type mapping In-Reply-To: <45111C7C.2060700@ieee.org> References: <200609200952.28046.faltet@carabos.com> <45111C7C.2060700@ieee.org> Message-ID: <200609201634.36232.faltet@carabos.com> A Dimecres 20 Setembre 2006 12:48, Travis Oliphant va escriure: > Making sure you get the correct data-type is why there are NPY_INT32 and > NPY_INT64 enumerated types. You can't code using NPY_LONG and expect > it will give you the same sizes when moving from 32-bit and 64-bit > platforms. That's a problem that has been fixed with the bitwidth > types. I don't understand why you are using the enumerated types at all > in this circumstance. Ooops. I didn't know that NPY_INT32 and NPY_INT64 were there. I think this solves all my problems. In fact, you were proposing this from the very beginning, but I was confused because I was hoping to find NPY_INT32 and NPY_INT64 in NPY_TYPES enumerated and I didn't find it there. I didn't realized that NPY_INT32 and NPY_INT64 were defined outside NPY_TYPES as platform independent constants. Blame on me. Sorry about any inconveniences and thanks once more for your patience! -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From martin.wiechert at gmx.de Wed Sep 20 10:55:08 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 20 Sep 2006 16:55:08 +0200 Subject: [Numpy-discussion] immutable arrays Message-ID: <200609201655.08552.martin.wiechert@gmx.de> Hi list, I just stumbled accross NPY_WRITEABLE flag. Now I'd like to know if there are ways either from Python or C to make an array temporarily immutable. Thanks, Martin. From haase at msg.ucsf.edu Wed Sep 20 11:47:03 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 20 Sep 2006 08:47:03 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <45116277.7010201@msg.ucsf.edu> Robert Kern wrote: >> This was not supposed to be a scientific statement -- I'm (again) >> thinking of our students that not always appreciate the full >> complexity >> of computational numerics and data types and such. > > They need to appreciate the complexity of computational numerics if > they are going to do numerical computation. Double precision does not > make it any simpler. This is were we differ. > We haven't forgotten what newcomers will do; to the contrary, we are > quite aware > that new users need consistent behavior in order to learn how to use a > system. > Adding another special case in how dtypes implicitly convert to one > another will > impede new users being able to understand the whole system. All I'm proposing could be summarized in: mean(), sum(), var() ... produce output of dtype float64 (except for input float96 which produces float96) A comment on this is also that for these operations the input type/precision is almost not related to the resulting output precision -- the int case makes that already clear. (This is different for e.g. min() or max() ) The proposed alternative implementations seem to have one or more multiplication (or division) for each value -- this might be noticeably slower ... Regards, Sebastian From tim.hochberg at ieee.org Wed Sep 20 12:08:15 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 09:08:15 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <4511676F.6000909@ieee.org> Robert Kern wrote: > Sebastian Haase wrote: > >> Robert Kern wrote: >> >>> Sebastian Haase wrote: >>> >>>> I know that having too much knowledge of the details often makes one >>>> forget what the "newcomers" will do and expect. >>>> >>> Please be more careful with such accusations. Repeated frequently, they can >>> become quite insulting. >>> >>> >> I did not mean to insult anyone - what I meant was, that I'm for numpy >> becoming an easy platform to use. I have spend and enjoyed part the last >> four years developing and evangelizing Python as an alternative to >> Matlab and C/Fortran based image analysis environment. I often find >> myself arguing for good support of the single precision data format. So >> I find it actually somewhat ironic to see myself arguing now for wanting >> float64 over float32 ;-) >> > > No one is doubting that you want numpy to be easy to use. Please don't doubt > that the rest of us want otherwise. However, the fact that you *want* numpy to > be easy to use does not mean that your suggestions *will* make numpy easy to use. > > We haven't forgotten what newcomers will do; to the contrary, we are quite aware > that new users need consistent behavior in order to learn how to use a system. > Adding another special case in how dtypes implicitly convert to one another will > impede new users being able to understand the whole system. See A. M. > Archibald's question in the thread "ufunc.reduce and conversion" for an example. > In our judgement this is a worse outcome than notational convenience for float32 > users, who already need to be aware of the effects of their precision choice. > Each of us can come to different conclusions in good faith without one of us > forgetting the new user experience. > > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. The algorithms look like the following for 1D arrays in Python: > > def mean(a): > m = a[0] > for i in range(1, len(a)): > m += (a[i] - m) / (i + 1) > return m > > def var(a): > m = a[0] > t = a.dtype.type(0) > for i in range(1, len(a)): > q = a[i] - m > r = q / (i+1) > m += r > t += i * q * r > t /= len(a) > return t > > Alternatively, from Knuth: > > def var_knuth(a): > m = a.dtype.type(0) > variance = a.dtype.type(0) > for i in range(len(a)): > delta = a[i] - m > m += delta / (i+1) > variance += delta * (a[i] - m) > variance /= len(a) > return variance > > If you will code up implementations of these for ndarray.mean() and > ndarray.var(), I will check them in and then float32 arrays will avoid most of > the catastrophes that the current implementations run into. > +1 > >>>> We are only talking >>>> about people that will a) work with single-precision data (e.g. large >>>> scale-image analysis) and who b) will tend to "just use the default" >>>> (dtype) --- How else can I say this: these people will just assume that >>>> arr.mean() *is* the mean of arr. >>>> >>> I don't understand what you mean, here. arr.mean() is almost never *the* mean of >>> arr. Double precision can't fix that. >>> >>> >> This was not supposed to be a scientific statement -- I'm (again) >> thinking of our students that not always appreciate the full complexity >> of computational numerics and data types and such. >> > > They need to appreciate the complexity of computational numerics if they are > going to do numerical computation. Double precision does not make it any simpler. > > From eric at enthought.com Wed Sep 20 12:29:28 2006 From: eric at enthought.com (eric) Date: Wed, 20 Sep 2006 11:29:28 -0500 Subject: [Numpy-discussion] ANN: Job postings -- Enthought, Inc. Message-ID: <45116C68.1070706@enthought.com> Hey group, We are growing again and have multiple positions open here at Enthought. I'd love to recruit more people out of the Scipy/numpy community. I think many of you would find the work interesting. You can look at our career page for more info: http://www.enthought.com/careers.htm thanks! eric From cookedm at physics.mcmaster.ca Wed Sep 20 12:34:17 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 20 Sep 2006 12:34:17 -0400 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. The algorithms look like the following for 1D arrays in Python: > > def mean(a): > m = a[0] > for i in range(1, len(a)): > m += (a[i] - m) / (i + 1) > return m This isn't really going to be any better than using a simple sum. It'll also be slower (a division per iteration). You do avoid accumulating large sums, but then doing the division a[i]/len(a) and adding that will do the same. Now, if you want to avoid losing precision, you want to use a better summation technique, like compensated (or Kahan) summation: def mean(a): s = e = a.dtype.type(0) for i in range(0, len(a)): temp = s y = a[i] + e s = temp + y e = (temp - s) + y return s / len(a) Some numerical experiments in Maple using 5-digit precision show that your mean is maybe a bit better in some cases, but can also be much worse, than sum(a)/len(a), but both are quite poor in comparision to the Kahan summation. (We could probably use a fast implementation of Kahan summation in addition to a.sum()) > def var(a): > m = a[0] > t = a.dtype.type(0) > for i in range(1, len(a)): > q = a[i] - m > r = q / (i+1) > m += r > t += i * q * r > t /= len(a) > return t > > Alternatively, from Knuth: > > def var_knuth(a): > m = a.dtype.type(0) > variance = a.dtype.type(0) > for i in range(len(a)): > delta = a[i] - m > m += delta / (i+1) > variance += delta * (a[i] - m) > variance /= len(a) > return variance These formulas are good when you can only do one pass over the data (like in a calculator where you don't store all the data points), but are slightly worse than doing two passes. Kahan summation would probably also be good here too. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Wed Sep 20 12:56:53 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 09:56:53 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4510C93C.8030902@msg.ucsf.edu> References: <200609191533.24097.haase@msg.ucsf.edu> <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> Message-ID: <451172D5.5070101@noaa.gov> Sebastian Haase wrote: > The best I can hope for is a "sound" default for most (practical) cases... > I still think that 80bit vs. 128bit vs 96bit is rather academic for most > people ... most people seem to only use float64 and then there are some > that use float32 (like us) ... I fully agree with Sebastian here. As Travis pointed out, "all we are talking about is the default". The default should follow the principle of least surprise for the less-knowledgeable users. Personally, I try to always use doubles, unless I have a real reason not to. The recent change of default types for zeros et al. will help. clearly, there is a problem to say the default accumulator for *all* types is double, as you wouldn't want to downcast float128s, even if they are obscure. However, is it that hard to say that the default accumulator will have *at least* the precision of double? Robert Kern wrote: > Let me offer a third path: the algorithms used for .mean() and .var() are > substandard. There are much better incremental algorithms that entirely avoid > the need to accumulate such large (and therefore precision-losing) intermediate > values. This, of course, is an even better solution, unless there are substantial performance impacts. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Sep 20 12:59:00 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 20 Sep 2006 10:59:00 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: On 9/20/06, David M. Cooke wrote: > > On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > > Let me offer a third path: the algorithms used for .mean() and .var() > are > > substandard. There are much better incremental algorithms that entirely > avoid > > the need to accumulate such large (and therefore precision-losing) > intermediate > > values. The algorithms look like the following for 1D arrays in Python: > > > > def mean(a): > > m = a[0] > > for i in range(1, len(a)): > > m += (a[i] - m) / (i + 1) > > return m > > This isn't really going to be any better than using a simple sum. > It'll also be slower (a division per iteration). You do avoid > accumulating large sums, but then doing the division a[i]/len(a) and > adding that will do the same. > > Now, if you want to avoid losing precision, you want to use a better > summation technique, like compensated (or Kahan) summation: > > def mean(a): > s = e = a.dtype.type(0) > for i in range(0, len(a)): > temp = s > y = a[i] + e > s = temp + y > e = (temp - s) + y > return s / len(a) A variant of this can also be used to generate the sin/cos for fourier transforms using repeated complex multiplication. It works amazingly well. But I suspect accumulating in higher precision would be just as good and faster if the hardware supports it. Divide and conquer, like an fft where only the constant coefficient is computed, is also a good bet but requires lg(n) passes over the data and extra storage. Some numerical experiments in Maple using 5-digit precision show that > your mean is maybe a bit better in some cases, but can also be much > worse, than sum(a)/len(a), but both are quite poor in comparision to the > Kahan summation. > > (We could probably use a fast implementation of Kahan summation in > addition to a.sum()) > > > def var(a): > > m = a[0] > > t = a.dtype.type(0) > > for i in range(1, len(a)): > > q = a[i] - m > > r = q / (i+1) > > m += r > > t += i * q * r > > t /= len(a) > > return t > > > > Alternatively, from Knuth: > > > > def var_knuth(a): > > m = a.dtype.type(0) > > variance = a.dtype.type(0) > > for i in range(len(a)): > > delta = a[i] - m > > m += delta / (i+1) > > variance += delta * (a[i] - m) > > variance /= len(a) > > return variance > > These formulas are good when you can only do one pass over the data > (like in a calculator where you don't store all the data points), but > are slightly worse than doing two passes. Kahan summation would probably > also be good here too. I think we should leave things as they are. Choosing the right precision for accumulating sums is absolutely fundamental, I always think about it when programming such loops because it is always a potential gotcha. Making the defaults higher precision just kicks the can down the road where the unwary are still likely to trip over it at some point. Perhaps these special tricks for special cases could be included in scipy somewhere. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 20 13:22:16 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 20 Sep 2006 12:22:16 -0500 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060920163417.GA4978@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: David M. Cooke wrote: > On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> Let me offer a third path: the algorithms used for .mean() and .var() are >> substandard. There are much better incremental algorithms that entirely avoid >> the need to accumulate such large (and therefore precision-losing) intermediate >> values. The algorithms look like the following for 1D arrays in Python: >> >> def mean(a): >> m = a[0] >> for i in range(1, len(a)): >> m += (a[i] - m) / (i + 1) >> return m > > This isn't really going to be any better than using a simple sum. > It'll also be slower (a division per iteration). With one exception, every test that I've thrown at it shows that it's better for float32. That exception is uniformly spaced arrays, like linspace(). > You do avoid > accumulating large sums, but then doing the division a[i]/len(a) and > adding that will do the same. Okay, this is true. > Now, if you want to avoid losing precision, you want to use a better > summation technique, like compensated (or Kahan) summation: > > def mean(a): > s = e = a.dtype.type(0) > for i in range(0, len(a)): > temp = s > y = a[i] + e > s = temp + y > e = (temp - s) + y > return s / len(a) > > Some numerical experiments in Maple using 5-digit precision show that > your mean is maybe a bit better in some cases, but can also be much > worse, than sum(a)/len(a), but both are quite poor in comparision to the > Kahan summation. > > (We could probably use a fast implementation of Kahan summation in > addition to a.sum()) +1 >> def var(a): >> m = a[0] >> t = a.dtype.type(0) >> for i in range(1, len(a)): >> q = a[i] - m >> r = q / (i+1) >> m += r >> t += i * q * r >> t /= len(a) >> return t >> >> Alternatively, from Knuth: >> >> def var_knuth(a): >> m = a.dtype.type(0) >> variance = a.dtype.type(0) >> for i in range(len(a)): >> delta = a[i] - m >> m += delta / (i+1) >> variance += delta * (a[i] - m) >> variance /= len(a) >> return variance > > These formulas are good when you can only do one pass over the data > (like in a calculator where you don't store all the data points), but > are slightly worse than doing two passes. Kahan summation would probably > also be good here too. Again, my tests show otherwise for float32. I'll condense my ipython log into a module for everyone's perusal. It's possible that the Kahan summation of the squared residuals will work better than the current two-pass algorithm and the implementations I give above. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Sep 20 14:18:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 12:18:19 -0600 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <200609201655.08552.martin.wiechert@gmx.de> References: <200609201655.08552.martin.wiechert@gmx.de> Message-ID: <451185EB.9080507@ieee.org> Martin Wiechert wrote: > Hi list, > > I just stumbled accross NPY_WRITEABLE flag. > Now I'd like to know if there are ways either from Python or C to make an > array temporarily immutable. > Just setting the flag Python: make immutable: a.flags.writeable = False make mutable again: a.flags.writeable = True C: make immutable: a->flags &= ~NPY_WRITEABLE make mutable again: a->flags |= NPY_WRITEABLE In C you can play with immutability all you want. In Python you can only make something writeable if you either 1) own the data or 2) the object that owns the data is itself "writeable" -Travis From tim.hochberg at ieee.org Wed Sep 20 14:53:41 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 11:53:41 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <45118E35.90207@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > Here's version of mean based on the Kahan sum, including the compensation term that seems to be consistently much better than builtin mean It seems to be typically 4 orders or magnitude closer to the "correct" answer than the builtin mean and 2 orders of magnitude better than just naively using the Kahan sum. I'm using the sum performed with dtype=int64 as the "correct" value. def rawKahanSum(values): # Stolen from mathworld if not input: return 0.0 total = values[0] c = type(total)(0.0) # A running compensation for lost low-order bits. for x in values[1:]: y = x - c # So far, so good: c is zero. t = total + y # Alas, total is big, y small, so low-order digits of y are lost. c = (t - total) - y # (t - total) recovers the high-order part of y; subtracting y recovers -(low part of y) total = t # Algebriacally, c should always be zero. Beware eagerly optimising compilers! # Next time around, the lost low part will be added to y in a fresh attempt. return total, c def kahanSum(values): total, c = rawKahanSum(values) return total def mean(values): total, c = rawKahanSum(values) n = float(len(values)) return total / n - c / n for i in range(100): data = random.random([10000]).astype(float32) baseline = data.mean(dtype=float64) delta_builtin_mean = baseline - data.mean() delta_compensated_mean = baseline - mean(data) delta_kahan_mean = baseline - (kahanSum(data) / len(data)) if not abs(delta_builtin_mean) >= abs(delta_kahan_mean) >= abs(delta_compensated_mean): print ">>>", print delta_builtin_mean, delta_kahan_mean, delta_compensated_mean -tim > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > > From tim.hochberg at ieee.org Wed Sep 20 15:02:38 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 12:02:38 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45118E35.90207@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <45118E35.90207@ieee.org> Message-ID: <4511904E.2090002@ieee.org> [Sorry, this version should have less munged formatting since I clipped the comments. Oh, and the Kahan sum algorithm was grabbed from wikipedia, not mathworld] Tim Hochberg wrote: > Robert Kern wrote: > >> David M. Cooke wrote: >> >> >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>> >>> >>>> Let me offer a third path: the algorithms used for .mean() and .var() are >>>> substandard. There are much better incremental algorithms that entirely avoid >>>> the need to accumulate such large (and therefore precision-losing) intermediate >>>> values. The algorithms look like the following for 1D arrays in Python: >>>> >>>> def mean(a): >>>> m = a[0] >>>> for i in range(1, len(a)): >>>> m += (a[i] - m) / (i + 1) >>>> return m >>>> >>>> >>> This isn't really going to be any better than using a simple sum. >>> It'll also be slower (a division per iteration). >>> >>> >> With one exception, every test that I've thrown at it shows that it's better for >> float32. That exception is uniformly spaced arrays, like linspace(). >> >> > Here's version of mean based on the Kahan sum, including the > compensation term that seems to be consistently much better than builtin > mean It seems to be typically 4 orders or magnitude closer to the > "correct" answer than the builtin mean and 2 orders of magnitude better > than just naively using the Kahan sum. I'm using the sum performed with > dtype=int64 as the "correct" value. > > > def rawKahanSum(values): if not input: return 0.0 total = values[0] c = type(total)(0.0) for x in values[1:]: y = x - c t = total + y c = (t - total) - y total = t return total, c def kahanSum(values): total, c = rawKahanSum(values) return total def mean(values): total, c = rawKahanSum(values) n = float(len(values)) return total / n - c / n for i in range(100): data = random.random([10000]).astype(float32) baseline = data.mean(dtype=float64) delta_builtin_mean = baseline - data.mean() delta_compensated_mean = baseline - mean(data) delta_kahan_mean = baseline - (kahanSum(data) / len(data)) if not abs(delta_builtin_mean) >= abs(delta_kahan_mean) >= abs(delta_compensated_mean): print ">>>", print delta_builtin_mean, delta_kahan_mean, delta_compensated_mean > -tim > >> > You do avoid >> > accumulating large sums, but then doing the division a[i]/len(a) and >> > adding that will do the same. >> >> Okay, this is true. >> >> >> >>> Now, if you want to avoid losing precision, you want to use a better >>> summation technique, like compensated (or Kahan) summation: >>> >>> def mean(a): >>> s = e = a.dtype.type(0) >>> for i in range(0, len(a)): >>> temp = s >>> y = a[i] + e >>> s = temp + y >>> e = (temp - s) + y >>> return s / len(a) >>> >>> Some numerical experiments in Maple using 5-digit precision show that >>> your mean is maybe a bit better in some cases, but can also be much >>> worse, than sum(a)/len(a), but both are quite poor in comparision to the >>> Kahan summation. >>> >>> (We could probably use a fast implementation of Kahan summation in >>> addition to a.sum()) >>> >>> >> +1 >> >> >> >>>> def var(a): >>>> m = a[0] >>>> t = a.dtype.type(0) >>>> for i in range(1, len(a)): >>>> q = a[i] - m >>>> r = q / (i+1) >>>> m += r >>>> t += i * q * r >>>> t /= len(a) >>>> return t >>>> >>>> Alternatively, from Knuth: >>>> >>>> def var_knuth(a): >>>> m = a.dtype.type(0) >>>> variance = a.dtype.type(0) >>>> for i in range(len(a)): >>>> delta = a[i] - m >>>> m += delta / (i+1) >>>> variance += delta * (a[i] - m) >>>> variance /= len(a) >>>> return variance >>>> >>>> >>> These formulas are good when you can only do one pass over the data >>> (like in a calculator where you don't store all the data points), but >>> are slightly worse than doing two passes. Kahan summation would probably >>> also be good here too. >>> >>> >> Again, my tests show otherwise for float32. I'll condense my ipython log into a >> module for everyone's perusal. It's possible that the Kahan summation of the >> squared residuals will work better than the current two-pass algorithm and the >> implementations I give above. >> >> >> > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Wed Sep 20 15:12:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 20 Sep 2006 13:12:06 -0600 Subject: [Numpy-discussion] I've just tagged the tree for 1.0rc1 Message-ID: <45119286.3050704@ieee.org> There is now a 1.0rc1 tag on the NumPy SVN tree. I've confirmed it builds and passes all tests on my Linux box for Python2.3-Python2.5 -Travis From tim.hochberg at ieee.org Wed Sep 20 15:41:22 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 20 Sep 2006 12:41:22 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <45119962.4020502@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> On the flip side, it doesn't take a very large array (somewhere in the vicinity of 10,000 values in my experience) before memory bandwidth becomes a limiting factor. In that region a two pass algorithm could well be twice as slow as a single pass algorithm even if the former is somewhat simpler in terms of numeric operations. -tim > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > From myeates at jpl.nasa.gov Wed Sep 20 16:41:55 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 20 Sep 2006 13:41:55 -0700 Subject: [Numpy-discussion] I've just tagged the tree for 1.0rc1 In-Reply-To: <45119286.3050704@ieee.org> References: <45119286.3050704@ieee.org> Message-ID: <4511A793.2080102@jpl.nasa.gov> I don't see a windows binary. Will this be added? Mathew Travis Oliphant wrote: > There is now a 1.0rc1 tag on the NumPy SVN tree. I've confirmed it > builds and passes all tests on my Linux box for Python2.3-Python2.5 > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From alerandazzo at 1-computerdesks.com Wed Sep 20 18:34:20 2006 From: alerandazzo at 1-computerdesks.com (Katherine Goldberg) Date: Wed, 20 Sep 2006 21:34:20 -0060 Subject: [Numpy-discussion] If you don't want to receive $5000 per month then don't open that letter Message-ID: Please read this letter attentively! Financial company is in search of representatives in the US. - Age: from 18 till 60 years - Have a skill to communicate and access to the Internet. - You can speak english language above average - You need to know how to use excel and word. You will earn from 1500 up to 3000 USD per week, working only some hours per day. You can work part time or full time. If you are interested in our offer send the following data to our e-mail usa at euroimperial2.com - Your full name - Your contact e-mail - Your phone number If you have any questions. Feel free to ask them From kwgoodman at gmail.com Wed Sep 20 17:46:54 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 20 Sep 2006 14:46:54 -0700 Subject: [Numpy-discussion] If you don't want to receive $5000 per month then don't open that letter In-Reply-To: References: Message-ID: I like the salutation of the letter below: "Please read this letter attentively!" Why does so much spam make it through the sourceforge filters? Or maybe they only let through the very good deals. "You will earn from 1500 up to 3000 USD per week, working only some hours per day." On Wed, 20 Sep 2006 21:34:20 -0060, Katherine Goldberg wrote: > Please read this letter attentively! > > Financial company is in search of representatives in the US. > > - Age: from 18 till 60 years > - Have a skill to communicate and access to the Internet. > - You can speak english language above average > - You need to know how to use excel and word. > > You will earn from 1500 up to 3000 USD per week, working only some hours per day. You can work part time or full time. > > If you are interested in our offer send the following data to our e-mail usa at euroimperial2.com > > - Your full name > - Your contact e-mail > - Your phone number > > If you have any questions. Feel free to ask them > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Wed Sep 20 18:17:58 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 15:17:58 -0700 Subject: [Numpy-discussion] Division by zero doesn't raise exception in the integer case. In-Reply-To: <451079B2.60805@ee.byu.edu> References: <451079B2.60805@ee.byu.edu> Message-ID: <4511BE16.7040500@noaa.gov> Travis Oliphant wrote: >> In [77]: arange(5, dtype=int)/0 >> Out[77]: array([0, 0, 0, 0, 0]) > It is deliberate. Numarray introduced it (the only difference being > that by default NumPy has division-by-zero erros turned off). It's tied > to the way floating-point division-by zero is handled. There is a > valid argument for having a separate integer-division flag so that you > can raise exceptions for integer-division but not for floating-point > division. I'm open to that change for 1.0rc1 +1 (+inf) There is a BIG difference between getting an inf with a floating point computation and getting a 0 with an integer one! Also, the default integer-division flag should be set to raise an exception. It sure would be nice to have special values for integers too.... Travis Oliphant wrote: > Simulating and "integer-division-by-zero" > hardware flag is not trivial as we would have to manage context > switching ourselves. So, at least for 1.0, integer and floating-point > division by zero are going to be handled the same. Darn. Oh well. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Sep 20 18:18:27 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 20 Sep 2006 15:18:27 -0700 Subject: [Numpy-discussion] sorting -inf, nan, inf In-Reply-To: References: <45104794.2050106@ieee.org> <45105164.90700@ieee.org> <45105B71.7090102@ieee.org> Message-ID: <4511BE33.4080708@noaa.gov> Charles R Harris wrote: > Thinking a bit, keeping the values in place isn't easy. Why the heck would "in place" be desirable for sorted data anyway? I understand that it means that if there is a NaN in the nth position before sorting, there will be one in the nth position after sorting. However, I see absolutely no reason at all why that would be useful (or any more useful than putting them anywhere else) A couple years ago, there was a long debate on this list about whether numpy should pass -inf, NaN, and +inf through all the ufuncs without error. there were two schools of thought: 1) They indicate a problem, the programmer should know about hat problem as soon as it occurs, not at the end of the computation, many steps later, when they might get presented with nothing but NaNs. 2) The whole point of "vector" computation is that you can act on a whole bunch of numbers at once. If only subset of those numbers are invalid, why stop the process. Raising an error when a single number has a problem defeats the purpose of vector operations. It seems that numpy has settled on school of thought (2), at least by default. That being the case, it should apply to sorting also. If it does, then that means no exception will be raised, but it makes no difference where the heck the NaNs end up in the sorted array, as long as everything else is in order. NaN means exactly what it's called: it's not a number, so it doesn't matter what you do with them, as long as they are preserved and don't mess up other things. Let the coder decide what they want to so with them, and when they want to do it. Personally, I'd prefer that they all ended up at the beginning or end after sorting, but it really doesn't much matter. That being said, if it's impossible to do a efficient sort with NaNs mixed in, then we'll just have to live with it. It really would be best if an exception was raised if the non-NaN values are not going to be sorted correctly -- that really would surprise people! > It would probably also not be unreasonable to punt and document sort > as failing in the presence of nans. That would be one of the worst options, but may be the only one available. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From davidgrant at gmail.com Wed Sep 20 19:41:56 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 20 Sep 2006 16:41:56 -0700 Subject: [Numpy-discussion] change of default dtype Message-ID: I noticed that the default dtype for the "ones" and "zeros" functions is now float, whereas it used to be int. Should this be noted at http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does break code (if you are using the array you creating with ones or zeros as indices into another array). -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Sep 20 20:03:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 21 Sep 2006 09:03:58 +0900 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: Message-ID: I think that's supposed to be covered by this line: "The default array data-type is now float64 (c double precision) instead of c integer." But yeh, I agree. It's definitely not obvious what that means in terms of concrete API changes. --bb On 9/21/06, David Grant wrote: > I noticed that the default dtype for the "ones" and "zeros" functions is now > float, whereas it used to be int. Should this be noted at > http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does > break code (if you are using the array you creating with ones or zeros as > indices into another array). > > -- > David Grant > http://www.davidgrant.ca > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From strawman at astraw.com Wed Sep 20 20:39:43 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed, 20 Sep 2006 17:39:43 -0700 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: Message-ID: <4511DF4F.9070206@astraw.com> It is a wiki, and contributions are absolutely welcome, so please go ahead and change it to be more clear. Bill Baxter wrote: >I think that's supposed to be covered by this line: >"The default array data-type is now float64 (c double precision) >instead of c integer." > >But yeh, I agree. It's definitely not obvious what that means in >terms of concrete API changes. > >--bb > >On 9/21/06, David Grant wrote: > > >>I noticed that the default dtype for the "ones" and "zeros" functions is now >>float, whereas it used to be int. Should this be noted at >>http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does >>break code (if you are using the array you creating with ones or zeros as >>indices into another array). >> >>-- >>David Grant >>http://www.davidgrant.ca >>------------------------------------------------------------------------- >>Take Surveys. Earn Cash. Influence the Future of IT >>Join SourceForge.net's Techsay panel and you'll get the chance to share your >>opinions on IT & business topics through brief surveys -- and earn cash >>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> >> > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share your >opinions on IT & business topics through brief surveys -- and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From kwgoodman at gmail.com Wed Sep 20 23:47:02 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 20 Sep 2006 20:47:02 -0700 Subject: [Numpy-discussion] nan adds lots of whitespace Message-ID: NaN adds a lot of whitespace in the representation of a matrix. Without NaN: matrix([[1, 2], [3, 4]]) With NaN: matrix([[ nan, 2. ], [ 3. , 4. ]]) There's enough room for the wikipedia entry for nan. From wbaxter at gmail.com Thu Sep 21 00:00:35 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 21 Sep 2006 13:00:35 +0900 Subject: [Numpy-discussion] change of default dtype In-Reply-To: <4511DF4F.9070206@astraw.com> References: <4511DF4F.9070206@astraw.com> Message-ID: Hey Andrew, point taken, but I think it would be better if someone who actually knows the full extent of the change made the edit. I know zeros and ones changed. Did anything else? Anyway, I'm surprised the release notes page is publicly editable. --bb On 9/21/06, Andrew Straw wrote: > It is a wiki, and contributions are absolutely welcome, so please go > ahead and change it to be more clear. > > Bill Baxter wrote: > > >I think that's supposed to be covered by this line: > >"The default array data-type is now float64 (c double precision) > >instead of c integer." > > > >But yeh, I agree. It's definitely not obvious what that means in > >terms of concrete API changes. > > > >--bb > > > >On 9/21/06, David Grant wrote: > > > > > >>I noticed that the default dtype for the "ones" and "zeros" functions is now > >>float, whereas it used to be int. Should this be noted at > >>http://www.scipy.org/ReleaseNotes/NumPy_1.0 since it does > >>break code (if you are using the array you creating with ones or zeros as > >>indices into another array). > >> > >>-- > >>David Grant > >>http://www.davidgrant.ca > >> > >> From gh743 at tom.com Tue Sep 26 03:39:39 2006 From: gh743 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Tue, 26 Sep 2006 15:39:39 +0800 Subject: [Numpy-discussion] =?GB2312?B?t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Thu Sep 21 04:04:57 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Thu, 21 Sep 2006 10:04:57 +0200 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <451185EB.9080507@ieee.org> References: <200609201655.08552.martin.wiechert@gmx.de> <451185EB.9080507@ieee.org> Message-ID: <200609211004.57370.martin.wiechert@gmx.de> Thanks Travis. Do I understand correctly that the only way to be really safe is to make a copy and not to export a reference to it? Because anybody having a reference to the owner of the data can override the flag? Cheers, Martin On Wednesday 20 September 2006 20:18, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > I just stumbled accross NPY_WRITEABLE flag. > > Now I'd like to know if there are ways either from Python or C to make an > > array temporarily immutable. > > Just setting the flag > > Python: > > make immutable: > a.flags.writeable = False > > make mutable again: > a.flags.writeable = True > > > C: > > make immutable: > a->flags &= ~NPY_WRITEABLE > > make mutable again: > a->flags |= NPY_WRITEABLE > > > In C you can play with immutability all you want. In Python you can > only make something writeable if you either 1) own the data or 2) the > object that owns the data is itself "writeable" > > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Peter.Bienstman at ugent.be Thu Sep 21 07:28:02 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Thu, 21 Sep 2006 13:28:02 +0200 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 Message-ID: <200609211328.02710.Peter.Bienstman@ugent.be> Hi, I just installed rc1 on an AMD64 machine. but I get this error message when trying to import it: Python 2.4.3 (#1, Sep 21 2006, 13:06:42) [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/numpy/__init__.py", line 36, in ? import core File "/usr/lib64/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 191, in ? _add_aliases() File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 169, in _add_aliases base, bit, char = bitname(typeobj) File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 119, in bitname char = base[0] IndexError: string index out of range Thanks! Peter From lroubeyrie at limair.asso.fr Thu Sep 21 10:16:58 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Thu, 21 Sep 2006 16:16:58 +0200 Subject: [Numpy-discussion] Question about recarray Message-ID: <200609211616.59129.lroubeyrie@limair.asso.fr> Hi all, Is it possible to put masked values into recarrays, I need a array with heterogenous types of datas (datetime objects in the first col, all others are float) but with missing values in some records. For the moment, I don't find any solution for that. I have tried with arrays of dtype=object, but I have problem when I want to compute min, max, ... with an error like: TypeError: function not supported for these types, and can't coerce safely to supported types. thanks -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From charlesr.harris at gmail.com Thu Sep 21 10:33:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 08:33:34 -0600 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 In-Reply-To: <200609211328.02710.Peter.Bienstman@ugent.be> References: <200609211328.02710.Peter.Bienstman@ugent.be> Message-ID: On 9/21/06, Peter Bienstman wrote: > > Hi, > > I just installed rc1 on an AMD64 machine. but I get this error message > when > trying to import it: > > Python 2.4.3 (#1, Sep 21 2006, 13:06:42) > [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > Traceback (most recent call last): I don't see this running the latest from svn on AMD64 here. Not sayin' there might not be a problem with rc1, I just don't see it with my sources. Python 2.4.3 (#1, Jun 13 2006, 11:46:22) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.0.dev3202' >>> numpy.version.os.uname() ('Linux', 'tethys', '2.6.17-1.2187_FC5', '#1 SMP Mon Sep 11 01:16:59 EDT 2006', 'x86_64') If you are building on Gentoo maybe you could delete the build directory (and maybe the numpy site package) and rebuild. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 21 12:05:58 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 10:05:58 -0600 Subject: [Numpy-discussion] Tests and code documentation Message-ID: Travis, A few questions. 1) I can't find any systematic code testing units, although there seem to be tests for regressions and such. Is there a place we should be putting such tests? 2) Any plans for code documentation? I documented some of my stuff with doxygen markups and wonder if we should include a Doxyfile as part of the package. 3) Would you consider breaking out the Converters into a separate .c file for inclusion? The code generator seems to take care of the ordering. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Sep 21 12:18:05 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 21 Sep 2006 09:18:05 -0700 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <200609210918.05413.haase@msg.ucsf.edu> On Thursday 21 September 2006 09:05, Charles R Harris wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem to > be tests for regressions and such. Is there a place we should be putting > such tests? > > 2) Any plans for code documentation? I documented some of my stuff with > doxygen markups and wonder if we should include a Doxyfile as part of the > package. Are able to use doxygen for Python code ? I thought it only worked for C (and alike) ? > > 3) Would you consider breaking out the Converters into a separate .c file > for inclusion? The code generator seems to take care of the ordering. > > Chuck From oliphant.travis at ieee.org Thu Sep 21 12:24:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:24:36 -0600 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <200609211004.57370.martin.wiechert@gmx.de> References: <200609201655.08552.martin.wiechert@gmx.de> <451185EB.9080507@ieee.org> <200609211004.57370.martin.wiechert@gmx.de> Message-ID: <4512BCC4.3000708@ieee.org> Martin Wiechert wrote: > Thanks Travis. > > Do I understand correctly that the only way to be really safe is to make a > copy and not to export a reference to it? > Because anybody having a reference to the owner of the data can override the > flag? > No, that's not quite correct. Of course in C, anybody can do anything they want to the flags. In Python, only the owner of the object itself can change the writeable flag once it is set to False. So, if you only return a "view" of the array (a.view()) then the Python user will not be able to change the flags. Example: a = array([1,2,3]) a.flags.writeable = False b = a.view() b.flags.writeable = True # raises an error. c = a c.flags.writeable = True # can be done because c is a direct alias to a. Hopefully, that explains the situation a bit better. -Travis From oliphant.travis at ieee.org Thu Sep 21 12:30:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:30:40 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <4512BE30.7070405@ieee.org> Charles R Harris wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem > to be tests for regressions and such. Is there a place we should be > putting such tests? All tests are placed under the tests directory of the corresponding sub-package. They will only be picked up by .test(level < 10) if the file is named test_. .test(level>10) should pick up all test files. If you want to name something different but still have it run at a test level < 10, then you need to run the test from one of the other test files that will be picked up (test_regression.py and test_unicode.py are doing that for example). > > 2) Any plans for code documentation? I documented some of my stuff > with doxygen markups and wonder if we should include a Doxyfile as > part of the package. I'm not familiar with Doxygen, but would welcome any improvements to the code documentation. > > 3) Would you consider breaking out the Converters into a separate .c > file for inclusion? The code generator seems to take care of the ordering. You are right that it doesn't matter which order the API subroutines are placed. I'm not opposed to more breaking up of the .c files, as long as it is clear where things will be located. The #include strategy is necessary to get it all in one Python module, but having smaller .c files usually makes for faster editing. It's the arrayobject.c file that is "too-large" IMHO, however. That's where I would look for ways to break it up. The iterobject and the data-type object could be taken out, for example. -Travis From lcordier at point45.com Thu Sep 21 12:32:40 2006 From: lcordier at point45.com (Louis Cordier) Date: Thu, 21 Sep 2006 18:32:40 +0200 (SAST) Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <200609210918.05413.haase@msg.ucsf.edu> References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: > Are able to use doxygen for Python code ? I thought it only worked for C (and > alike) ? There is an ugly-hack :) http://i31www.ira.uka.de/~baas/pydoxy/ But I wouldn't recommend using it, rather stick with Epydoc. -- Louis Cordier cell: +27721472305 Point45 Entertainment (Pty) Ltd. http://www.point45.org From charlesr.harris at gmail.com Thu Sep 21 12:42:59 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 10:42:59 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <200609210918.05413.haase@msg.ucsf.edu> References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: On 9/21/06, Sebastian Haase wrote: > > On Thursday 21 September 2006 09:05, Charles R Harris wrote: > > Travis, > > > > A few questions. > > > > 1) I can't find any systematic code testing units, although there seem > to > > be tests for regressions and such. Is there a place we should be putting > > such tests? > > > > 2) Any plans for code documentation? I documented some of my stuff with > > doxygen markups and wonder if we should include a Doxyfile as part of > the > > package. > > Are able to use doxygen for Python code ? I thought it only worked for C > (and > alike) ? IIRC correctly, it now does Python too. Let's see... here is an example ## Documentation for this module. # # More details. ## Documentation for a function. # # More details. def func(): pass Looks like ## replaces the /** Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Sep 21 12:43:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:43:44 -0600 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609211616.59129.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> Message-ID: <4512C140.8070805@ieee.org> Lionel Roubeyrie wrote: > Hi all, > Is it possible to put masked values into recarrays, I need a array with > heterogenous types of datas (datetime objects in the first col, all others > are float) but with missing values in some records. For the moment, I don't > find any solution for that. Either use "nans" or "inf" for missing values or use the masked array object with a complex data-type. You don't need to use a recarray object to get "records". Any array can have "records". Therefore, you can have a masked array of "records" by creating an array with the appropriate data-type. It may also be possible to use a recarray as the "array" for the masked array object becuase the recarray is a sub-class of the array. > I have tried with arrays of dtype=object, but I > have problem when I want to compute min, max, ... with an error like: > TypeError: function not supported for these types, and can't coerce safely to > supported types. > It looks like the max and min functions are not supported for Object arrays. import numpy as N N.maximum.types does not include Object arrays. It probably should. -Travis From matthew.brett at gmail.com Thu Sep 21 12:52:49 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 21 Sep 2006 17:52:49 +0100 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <451072E1.7030608@ee.byu.edu> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> <451072E1.7030608@ee.byu.edu> Message-ID: <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> Hi, > It's in the array interface specification: > > http://numpy.scipy.org/array_interface.shtml I was interested in the 't' (bitfield) type - is there an example of usage somewhere? In [13]: dtype('t8') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/mb312/python/ TypeError: data type not understood Best, Matthew From oliphant.travis at ieee.org Thu Sep 21 12:59:32 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 10:59:32 -0600 Subject: [Numpy-discussion] arr.dtype.kind is 'i' for dtype=unit !? In-Reply-To: <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> References: <200609191023.57656.haase@msg.ucsf.edu> <200609191451.16896.sransom@nrao.edu> <200609191454.03405.sransom@nrao.edu> <200609191523.45253.haase@msg.ucsf.edu> <451072E1.7030608@ee.byu.edu> <1e2af89e0609210952q93209f7u5ba3e6169fce61e1@mail.gmail.com> Message-ID: <4512C4F4.80605@ieee.org> Matthew Brett wrote: > Hi, > > >> It's in the array interface specification: >> >> http://numpy.scipy.org/array_interface.shtml >> > > I was interested in the 't' (bitfield) type - is there an example of > usage somewhere? > No, It's not implemented in NumPy. It's just part of the array interface specification for completeness. -Travis From oliphant.travis at ieee.org Thu Sep 21 13:01:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 21 Sep 2006 11:01:07 -0600 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609211616.59129.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> Message-ID: <4512C553.6090609@ieee.org> Lionel Roubeyrie wrote: > find any solution for that. I have tried with arrays of dtype=object, but I > have problem when I want to compute min, max, ... with an error like: > TypeError: function not supported for these types, and can't coerce safely to > supported types. > I just added support for min and max methods of object arrays, by adding support for Object arrays to the minimum and maximum functions. -Travis From tim.hochberg at ieee.org Thu Sep 21 13:56:38 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 10:56:38 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> Message-ID: <4512D256.2080604@ieee.org> Robert Kern wrote: > David M. Cooke wrote: > >> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >> >>> Let me offer a third path: the algorithms used for .mean() and .var() are >>> substandard. There are much better incremental algorithms that entirely avoid >>> the need to accumulate such large (and therefore precision-losing) intermediate >>> values. The algorithms look like the following for 1D arrays in Python: >>> >>> def mean(a): >>> m = a[0] >>> for i in range(1, len(a)): >>> m += (a[i] - m) / (i + 1) >>> return m >>> >> This isn't really going to be any better than using a simple sum. >> It'll also be slower (a division per iteration). >> > > With one exception, every test that I've thrown at it shows that it's better for > float32. That exception is uniformly spaced arrays, like linspace(). > > > You do avoid > > accumulating large sums, but then doing the division a[i]/len(a) and > > adding that will do the same. > > Okay, this is true. > > >> Now, if you want to avoid losing precision, you want to use a better >> summation technique, like compensated (or Kahan) summation: >> >> def mean(a): >> s = e = a.dtype.type(0) >> for i in range(0, len(a)): >> temp = s >> y = a[i] + e >> s = temp + y >> e = (temp - s) + y >> return s / len(a) >> >> Some numerical experiments in Maple using 5-digit precision show that >> your mean is maybe a bit better in some cases, but can also be much >> worse, than sum(a)/len(a), but both are quite poor in comparision to the >> Kahan summation. >> >> (We could probably use a fast implementation of Kahan summation in >> addition to a.sum()) >> > > +1 > > >>> def var(a): >>> m = a[0] >>> t = a.dtype.type(0) >>> for i in range(1, len(a)): >>> q = a[i] - m >>> r = q / (i+1) >>> m += r >>> t += i * q * r >>> t /= len(a) >>> return t >>> >>> Alternatively, from Knuth: >>> >>> def var_knuth(a): >>> m = a.dtype.type(0) >>> variance = a.dtype.type(0) >>> for i in range(len(a)): >>> delta = a[i] - m >>> m += delta / (i+1) >>> variance += delta * (a[i] - m) >>> variance /= len(a) >>> return variance >>> >> These formulas are good when you can only do one pass over the data >> (like in a calculator where you don't store all the data points), but >> are slightly worse than doing two passes. Kahan summation would probably >> also be good here too. >> > > Again, my tests show otherwise for float32. I'll condense my ipython log into a > module for everyone's perusal. It's possible that the Kahan summation of the > squared residuals will work better than the current two-pass algorithm and the > implementations I give above. > This is what my tests show as well var_knuth outperformed any simple two pass algorithm I could come up with, even ones using Kahan sums. Interestingly, for 1D arrays the built in float32 variance performs better than it should. After a bit of twiddling around I discovered that it actually does most of it's calculations in float64. It uses a two pass calculation, the result of mean is a scalar, and in the process of converting that back to an array we end up with float64 values. Or something like that; I was mostly reverse engineering the sequence of events from the results. -tim From tim.hochberg at ieee.org Thu Sep 21 14:34:42 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 11:34:42 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4512D256.2080604@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> Message-ID: <4512DB42.7080509@ieee.org> Tim Hochberg wrote: > Robert Kern wrote: > >> David M. Cooke wrote: >> >> >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>> >>> >>>> Let me offer a third path: the algorithms used for .mean() and .var() are >>>> substandard. There are much better incremental algorithms that entirely avoid >>>> the need to accumulate such large (and therefore precision-losing) intermediate >>>> values. The algorithms look like the following for 1D arrays in Python: >>>> >>>> def mean(a): >>>> m = a[0] >>>> for i in range(1, len(a)): >>>> m += (a[i] - m) / (i + 1) >>>> return m >>>> >>>> >>> This isn't really going to be any better than using a simple sum. >>> It'll also be slower (a division per iteration). >>> >>> >> With one exception, every test that I've thrown at it shows that it's better for >> float32. That exception is uniformly spaced arrays, like linspace(). >> >> > You do avoid >> > accumulating large sums, but then doing the division a[i]/len(a) and >> > adding that will do the same. >> >> Okay, this is true. >> >> >> >>> Now, if you want to avoid losing precision, you want to use a better >>> summation technique, like compensated (or Kahan) summation: >>> >>> def mean(a): >>> s = e = a.dtype.type(0) >>> for i in range(0, len(a)): >>> temp = s >>> y = a[i] + e >>> s = temp + y >>> e = (temp - s) + y >>> return s / len(a) >>> >>> Some numerical experiments in Maple using 5-digit precision show that >>> your mean is maybe a bit better in some cases, but can also be much >>> worse, than sum(a)/len(a), but both are quite poor in comparision to the >>> Kahan summation. >>> >>> (We could probably use a fast implementation of Kahan summation in >>> addition to a.sum()) >>> >>> >> +1 >> >> >> >>>> def var(a): >>>> m = a[0] >>>> t = a.dtype.type(0) >>>> for i in range(1, len(a)): >>>> q = a[i] - m >>>> r = q / (i+1) >>>> m += r >>>> t += i * q * r >>>> t /= len(a) >>>> return t >>>> >>>> Alternatively, from Knuth: >>>> >>>> def var_knuth(a): >>>> m = a.dtype.type(0) >>>> variance = a.dtype.type(0) >>>> for i in range(len(a)): >>>> delta = a[i] - m >>>> m += delta / (i+1) >>>> variance += delta * (a[i] - m) >>>> variance /= len(a) >>>> return variance >>>> >>>> >>> These formulas are good when you can only do one pass over the data >>> (like in a calculator where you don't store all the data points), but >>> are slightly worse than doing two passes. Kahan summation would probably >>> also be good here too. >>> >>> >> Again, my tests show otherwise for float32. I'll condense my ipython log into a >> module for everyone's perusal. It's possible that the Kahan summation of the >> squared residuals will work better than the current two-pass algorithm and the >> implementations I give above. >> >> > This is what my tests show as well var_knuth outperformed any simple two > pass algorithm I could come up with, even ones using Kahan sums. > Interestingly, for 1D arrays the built in float32 variance performs > better than it should. After a bit of twiddling around I discovered that > it actually does most of it's calculations in float64. It uses a two > pass calculation, the result of mean is a scalar, and in the process of > converting that back to an array we end up with float64 values. Or > something like that; I was mostly reverse engineering the sequence of > events from the results. > Here's a simple of example of how var is a little wacky. A shape-[N] array will give you a different result than a shape-[1,N] array. The reason is clear -- in the second case the mean is not a scalar so there isn't the inadvertent promotion to float64, but it's still odd. >>> data = (1000*(random.random([10000]) - 0.1)).astype(float32) >>> print data.var() - data.reshape([1, -1]).var(-1) [ 0.1171875] I'm going to go ahead and attach a module containing the versions of mean, var, etc that I've been playing with in case someone wants to mess with them. Some were stolen from traffic on this list, for others I grabbed the algorithms from wikipedia or equivalent. -tim > -tim > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy_statistics.py URL: From lists.steve at arachnedesign.net Thu Sep 21 14:45:29 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 21 Sep 2006 14:45:29 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu> Message-ID: <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> > Are able to use doxygen for Python code ? I thought it only worked > for C (and > alike) ? > > IIRC correctly, it now does Python too. Let's see... here is an > example > ## Documentation for this module. > # > # More details. > > ## Documentation for a function. > # > # More details. > def func(): > pass > Looks like ## replaces the /** I never found it (although I haven't looked too hard), but I always thought there was an official way to document python code -- minimally to put the documentation in the docstring following the function definition: def func(..): """One liner. Continue docs -- some type of reStructredText style """ pas Isn't that the same docstring that ipython uses to bring up help, when you do: In [1]: myobject.some_func? So .. I guess I'm wondering why we want to break from the standard? -steve From cookedm at physics.mcmaster.ca Thu Sep 21 15:20:42 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 21 Sep 2006 15:20:42 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: Message-ID: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> On Thu, 21 Sep 2006 10:05:58 -0600 "Charles R Harris" wrote: > Travis, > > A few questions. > > 1) I can't find any systematic code testing units, although there seem to be > tests for regressions and such. Is there a place we should be putting such > tests? > > 2) Any plans for code documentation? I documented some of my stuff with > doxygen markups and wonder if we should include a Doxyfile as part of the > package. We don't have much of a defined standard for docs. Personally, I wouldn't use doxygen: what I've seen for Python versions are hacks, whose output looks like C++, and which requires markup that's not like commonly-used conventions in Python (\brief, for instance). Foremost for Python doc strings, I think, is that it look ok when using pydoc or similar (ipython's ?, for instance). That means a minimal amount of markup. Someone previously mentioned including cross-references; I think that's a good idea. A 'See also' line, for instance. Examples are good too, especially if there's been disputes on the interpretation of the command :-) For the C code, documentation is autogenerated from the /** ... API */ comments that determine which functions are part of the C API. This are put into files multiarray_api.txt and ufunc_api.txt (in the include/ directory). The files are in reST format, so the comments should/could be. At some point I've got to through and add more :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From robert.kern at gmail.com Thu Sep 21 15:24:33 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 21 Sep 2006 14:24:33 -0500 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> References: <200609210918.05413.haase@msg.ucsf.edu> <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: Steve Lianoglou wrote: > So .. I guess I'm wondering why we want to break from the standard? We don't as far as Python code goes. The code that Chuck added Doxygen-style comments to was C code. I presume he was simply answering Sebastian's question rather than suggesting we use Doxygen for Python code, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Sep 21 15:47:40 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 21 Sep 2006 13:47:40 -0600 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu> <1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: Hi, On 9/21/06, Robert Kern wrote: > > Steve Lianoglou wrote: > > So .. I guess I'm wondering why we want to break from the standard? > > We don't as far as Python code goes. The code that Chuck added > Doxygen-style > comments to was C code. I presume he was simply answering Sebastian's > question > rather than suggesting we use Doxygen for Python code, too. Exactly. I also don't think the Python hack description applies to doxygen any longer. As to the oddness of \param or @param, here is an example from Epydoc using Epytext @type m: number @param m: The slope of the line. @type b: number @param b: The y intercept of the line. The X{y intercept} of a Looks like they borrowed something there ;) The main advantage of epydoc vs doxygen seems to be that you can use the markup inside the normal python docstring without having to make a separate comment block. Or would that be a disadvantage? Then again, I've been thinking of moving the python function docstrings into the add_newdocs.py file so everything is together in one spot and that would separate the Python docstrings from the functions anyway. I'll fool around with doxygen a bit and see what it does. The C code is the code that most needs documentation in any case. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ytz576 at 163.com Sun Sep 24 17:41:37 2006 From: ytz576 at 163.com (tpm) Date: Mon, 25 Sep 2006 05:41:37 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Thu Sep 21 17:59:37 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 21 Sep 2006 17:59:37 -0400 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4512DB42.7080509@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> Message-ID: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> On Thu, 21 Sep 2006 11:34:42 -0700 Tim Hochberg wrote: > Tim Hochberg wrote: > > Robert Kern wrote: > > > >> David M. Cooke wrote: > >> > >> > >>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>> > >>> > >>>> Let me offer a third path: the algorithms used for .mean() and .var() > >>>> are substandard. There are much better incremental algorithms that > >>>> entirely avoid the need to accumulate such large (and therefore > >>>> precision-losing) intermediate values. The algorithms look like the > >>>> following for 1D arrays in Python: > >>>> > >>>> def mean(a): > >>>> m = a[0] > >>>> for i in range(1, len(a)): > >>>> m += (a[i] - m) / (i + 1) > >>>> return m > >>>> > >>>> > >>> This isn't really going to be any better than using a simple sum. > >>> It'll also be slower (a division per iteration). > >>> > >>> > >> With one exception, every test that I've thrown at it shows that it's > >> better for float32. That exception is uniformly spaced arrays, like > >> linspace(). > >> > >> > You do avoid > >> > accumulating large sums, but then doing the division a[i]/len(a) and > >> > adding that will do the same. > >> > >> Okay, this is true. > >> > >> > >> > >>> Now, if you want to avoid losing precision, you want to use a better > >>> summation technique, like compensated (or Kahan) summation: > >>> > >>> def mean(a): > >>> s = e = a.dtype.type(0) > >>> for i in range(0, len(a)): > >>> temp = s > >>> y = a[i] + e > >>> s = temp + y > >>> e = (temp - s) + y > >>> return s / len(a) > >> > >>>> def var(a): > >>>> m = a[0] > >>>> t = a.dtype.type(0) > >>>> for i in range(1, len(a)): > >>>> q = a[i] - m > >>>> r = q / (i+1) > >>>> m += r > >>>> t += i * q * r > >>>> t /= len(a) > >>>> return t > >>>> > >>>> Alternatively, from Knuth: > >>>> > >>>> def var_knuth(a): > >>>> m = a.dtype.type(0) > >>>> variance = a.dtype.type(0) > >>>> for i in range(len(a)): > >>>> delta = a[i] - m > >>>> m += delta / (i+1) > >>>> variance += delta * (a[i] - m) > >>>> variance /= len(a) > >>>> return variance > > I'm going to go ahead and attach a module containing the versions of > mean, var, etc that I've been playing with in case someone wants to mess > with them. Some were stolen from traffic on this list, for others I > grabbed the algorithms from wikipedia or equivalent. I looked into this a bit more. I checked float32 (single precision) and float64 (double precision), using long doubles (float96) for the "exact" results. This is based on your code. Results are compared using abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the range of [-100, 900] First, the mean. In float32, the Kahan summation in single precision is better by about 2 orders of magnitude than simple summation. However, accumulating the sum in double precision is better by about 9 orders of magnitude than simple summation (7 orders more than Kahan). In float64, Kahan summation is the way to go, by 2 orders of magnitude. For the variance, in float32, Knuth's method is *no better* than the two-pass method. Tim's code does an implicit conversion of intermediate results to float64, which is why he saw a much better result. The two-pass method using Kahan summation (again, in single precision), is better by about 2 orders of magnitude. There is practically no difference when using a double-precision accumulator amongst the techniques: they're all about 9 orders of magnitude better than single-precision two-pass. In float64, Kahan summation is again better than the rest, by about 2 orders of magnitude. I've put my adaptation of Tim's code, and box-and-whisker plots of the results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ Conclusions: - If you're going to calculate everything in single precision, use Kahan summation. Using it in double-precision also helps. - If you can use a double-precision accumulator, it's much better than any of the techniques in single-precision only. - for speed+precision in the variance, either use Kahan summation in single precision with the two-pass method, or use double precision with simple summation with the two-pass method. Knuth buys you nothing, except slower code :-) After 1.0 is out, we should look at doing one of the above. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Thu Sep 21 18:07:44 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 21 Sep 2006 16:07:44 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <45130D30.8070806@ee.byu.edu> David M. Cooke wrote: > >Conclusions: > >- If you're going to calculate everything in single precision, use Kahan >summation. Using it in double-precision also helps. >- If you can use a double-precision accumulator, it's much better than any of >the techniques in single-precision only. > >- for speed+precision in the variance, either use Kahan summation in single >precision with the two-pass method, or use double precision with simple >summation with the two-pass method. Knuth buys you nothing, except slower >code :-) > >After 1.0 is out, we should look at doing one of the above. > > +1 From Martin.Wiechert at mpimf-heidelberg.mpg.de Thu Sep 21 17:55:09 2006 From: Martin.Wiechert at mpimf-heidelberg.mpg.de (Martin Wiechert) Date: Thu, 21 Sep 2006 23:55:09 +0200 Subject: [Numpy-discussion] immutable arrays In-Reply-To: <4512BCC4.3000708@ieee.org> References: <200609201655.08552.martin.wiechert@gmx.de> <200609211004.57370.martin.wiechert@gmx.de> <4512BCC4.3000708@ieee.org> Message-ID: <200609212355.09682.wiechert@mpimf-heidelberg.mpg.de> On Thursday 21 September 2006 18:24, Travis Oliphant wrote: > Martin Wiechert wrote: > > Thanks Travis. > > > > Do I understand correctly that the only way to be really safe is to make > > a copy and not to export a reference to it? > > Because anybody having a reference to the owner of the data can override > > the flag? > > No, that's not quite correct. Of course in C, anybody can do anything > they want to the flags. > > In Python, only the owner of the object itself can change the writeable > flag once it is set to False. So, if you only return a "view" of the > array (a.view()) then the Python user will not be able to change the > flags. > > Example: > > a = array([1,2,3]) > a.flags.writeable = False > > b = a.view() > > b.flags.writeable = True # raises an error. > > c = a > c.flags.writeable = True # can be done because c is a direct alias to a. > > Hopefully, that explains the situation a bit better. > It does. Thanks Travis. > -Travis > > > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -------------- next part -------------- AV scanned by FortiGate From tim.hochberg at ieee.org Thu Sep 21 18:28:18 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 15:28:18 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <45131202.6000105@ieee.org> David M. Cooke wrote: > On Thu, 21 Sep 2006 11:34:42 -0700 > Tim Hochberg wrote: > > >> Tim Hochberg wrote: >> >>> Robert Kern wrote: >>> >>> >>>> David M. Cooke wrote: >>>> >>>> >>>> >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>> >>>>> >>>>> >>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>> are substandard. There are much better incremental algorithms that >>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>> following for 1D arrays in Python: >>>>>> >>>>>> def mean(a): >>>>>> m = a[0] >>>>>> for i in range(1, len(a)): >>>>>> m += (a[i] - m) / (i + 1) >>>>>> return m >>>>>> >>>>>> >>>>>> >>>>> This isn't really going to be any better than using a simple sum. >>>>> It'll also be slower (a division per iteration). >>>>> >>>>> >>>>> >>>> With one exception, every test that I've thrown at it shows that it's >>>> better for float32. That exception is uniformly spaced arrays, like >>>> linspace(). >>>> >>>> > You do avoid >>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>> > adding that will do the same. >>>> >>>> Okay, this is true. >>>> >>>> >>>> >>>> >>>>> Now, if you want to avoid losing precision, you want to use a better >>>>> summation technique, like compensated (or Kahan) summation: >>>>> >>>>> def mean(a): >>>>> s = e = a.dtype.type(0) >>>>> for i in range(0, len(a)): >>>>> temp = s >>>>> y = a[i] + e >>>>> s = temp + y >>>>> e = (temp - s) + y >>>>> return s / len(a) >>>>> >>>> >>>> >>>>>> def var(a): >>>>>> m = a[0] >>>>>> t = a.dtype.type(0) >>>>>> for i in range(1, len(a)): >>>>>> q = a[i] - m >>>>>> r = q / (i+1) >>>>>> m += r >>>>>> t += i * q * r >>>>>> t /= len(a) >>>>>> return t >>>>>> >>>>>> Alternatively, from Knuth: >>>>>> >>>>>> def var_knuth(a): >>>>>> m = a.dtype.type(0) >>>>>> variance = a.dtype.type(0) >>>>>> for i in range(len(a)): >>>>>> delta = a[i] - m >>>>>> m += delta / (i+1) >>>>>> variance += delta * (a[i] - m) >>>>>> variance /= len(a) >>>>>> return variance >>>>>> >> I'm going to go ahead and attach a module containing the versions of >> mean, var, etc that I've been playing with in case someone wants to mess >> with them. Some were stolen from traffic on this list, for others I >> grabbed the algorithms from wikipedia or equivalent. >> > > I looked into this a bit more. I checked float32 (single precision) and > float64 (double precision), using long doubles (float96) for the "exact" > results. This is based on your code. Results are compared using > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the > range of [-100, 900] > > First, the mean. In float32, the Kahan summation in single precision is > better by about 2 orders of magnitude than simple summation. However, > accumulating the sum in double precision is better by about 9 orders of > magnitude than simple summation (7 orders more than Kahan). > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > For the variance, in float32, Knuth's method is *no better* than the two-pass > method. Tim's code does an implicit conversion of intermediate results to > float64, which is why he saw a much better result. Doh! And I fixed that same problem in the mean implementation earlier too. I was astounded by how good knuth was doing, but not astounded enough apparently. Does it seem weird to anyone else that in: numpy_scalar python_scalar the precision ends up being controlled by the python scalar? I would expect the numpy_scalar to control the resulting precision just like numpy arrays do in similar circumstances. Perhaps the egg on my face is just clouding my vision though. > The two-pass method using > Kahan summation (again, in single precision), is better by about 2 orders of > magnitude. There is practically no difference when using a double-precision > accumulator amongst the techniques: they're all about 9 orders of magnitude > better than single-precision two-pass. > > In float64, Kahan summation is again better than the rest, by about 2 orders > of magnitude. > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > Conclusions: > > - If you're going to calculate everything in single precision, use Kahan > summation. Using it in double-precision also helps. > - If you can use a double-precision accumulator, it's much better than any of > the techniques in single-precision only. > > - for speed+precision in the variance, either use Kahan summation in single > precision with the two-pass method, or use double precision with simple > summation with the two-pass method. Knuth buys you nothing, except slower > code :-) > The two pass methods are definitely more accurate. I won't be convinced on the speed front till I see comparable C implementations slug it out. That may well mean never in practice. However, I expect that somewhere around 10,000 items, the cache will overflow and memory bandwidth will become the bottleneck. At that point the extra operations of Knuth won't matter as much as making two passes through the array and Knuth will win on speed. Of course the accuracy is pretty bad at single precision, so the possible, theoretical speed advantage at large sizes probably doesn't matter. -tim > After 1.0 is out, we should look at doing one of the above. > +1 From tim.hochberg at ieee.org Thu Sep 21 18:56:47 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 21 Sep 2006 15:56:47 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <20060921175937.393288bb@arbutus.physics.mcmaster.ca> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> Message-ID: <451318AF.2090205@ieee.org> David M. Cooke wrote: > On Thu, 21 Sep 2006 11:34:42 -0700 > Tim Hochberg wrote: > > >> Tim Hochberg wrote: >> >>> Robert Kern wrote: >>> >>> >>>> David M. Cooke wrote: >>>> >>>> >>>> >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>> >>>>> >>>>> >>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>> are substandard. There are much better incremental algorithms that >>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>> following for 1D arrays in Python: >>>>>> >>>>>> def mean(a): >>>>>> m = a[0] >>>>>> for i in range(1, len(a)): >>>>>> m += (a[i] - m) / (i + 1) >>>>>> return m >>>>>> >>>>>> >>>>>> >>>>> This isn't really going to be any better than using a simple sum. >>>>> It'll also be slower (a division per iteration). >>>>> >>>>> >>>>> >>>> With one exception, every test that I've thrown at it shows that it's >>>> better for float32. That exception is uniformly spaced arrays, like >>>> linspace(). >>>> >>>> > You do avoid >>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>> > adding that will do the same. >>>> >>>> Okay, this is true. >>>> >>>> >>>> >>>> >>>>> Now, if you want to avoid losing precision, you want to use a better >>>>> summation technique, like compensated (or Kahan) summation: >>>>> >>>>> def mean(a): >>>>> s = e = a.dtype.type(0) >>>>> for i in range(0, len(a)): >>>>> temp = s >>>>> y = a[i] + e >>>>> s = temp + y >>>>> e = (temp - s) + y >>>>> return s / len(a) >>>>> >>>> >>>> >>>>>> def var(a): >>>>>> m = a[0] >>>>>> t = a.dtype.type(0) >>>>>> for i in range(1, len(a)): >>>>>> q = a[i] - m >>>>>> r = q / (i+1) >>>>>> m += r >>>>>> t += i * q * r >>>>>> t /= len(a) >>>>>> return t >>>>>> >>>>>> Alternatively, from Knuth: >>>>>> >>>>>> def var_knuth(a): >>>>>> m = a.dtype.type(0) >>>>>> variance = a.dtype.type(0) >>>>>> for i in range(len(a)): >>>>>> delta = a[i] - m >>>>>> m += delta / (i+1) >>>>>> variance += delta * (a[i] - m) >>>>>> variance /= len(a) >>>>>> return variance >>>>>> >> I'm going to go ahead and attach a module containing the versions of >> mean, var, etc that I've been playing with in case someone wants to mess >> with them. Some were stolen from traffic on this list, for others I >> grabbed the algorithms from wikipedia or equivalent. >> > > I looked into this a bit more. I checked float32 (single precision) and > float64 (double precision), using long doubles (float96) for the "exact" > results. This is based on your code. Results are compared using > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the > range of [-100, 900] > > First, the mean. In float32, the Kahan summation in single precision is > better by about 2 orders of magnitude than simple summation. However, > accumulating the sum in double precision is better by about 9 orders of > magnitude than simple summation (7 orders more than Kahan). > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > For the variance, in float32, Knuth's method is *no better* than the two-pass > method. Tim's code does an implicit conversion of intermediate results to > float64, which is why he saw a much better result. The two-pass method using > Kahan summation (again, in single precision), is better by about 2 orders of > magnitude. There is practically no difference when using a double-precision > accumulator amongst the techniques: they're all about 9 orders of magnitude > better than single-precision two-pass. > > In float64, Kahan summation is again better than the rest, by about 2 orders > of magnitude. > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > Conclusions: > > - If you're going to calculate everything in single precision, use Kahan > summation. Using it in double-precision also helps. > - If you can use a double-precision accumulator, it's much better than any of > the techniques in single-precision only. > > - for speed+precision in the variance, either use Kahan summation in single > precision with the two-pass method, or use double precision with simple > summation with the two-pass method. Knuth buys you nothing, except slower > code :-) > > After 1.0 is out, we should look at doing one of the above. > One more little tidbit; it appears possible to "fix up" Knuth's algorithm so that it's comparable in accuracy to the two pass Kahan version by doing Kahan summation while accumulating the variance. Testing on this was far from thorough, but in the tests I did it nearly always produced identical results to kahan. Of course this is even slower than the original Knuth version, but it's interesting anyway: # This is probably messier than it need to be # but I'm out of time for today... def var_knuth2(values, dtype=default_prec): """var(values) -> variance of values computed using Knuth's one pass algorithm""" m = variance = mc = vc = dtype(0) for i, x in enumerate(values): delta = values[i] - m y = delta / dtype(i+1) + mc t = m + y mc = y - (t - m) m = t y = delta * (x - m) + vc t = variance + y vc = y - (t - variance) variance = t assert type(variance) == dtype variance /= dtype(len(values)) return variance From ytz576 at 163.com Sun Sep 24 18:59:58 2006 From: ytz576 at 163.com (tpm) Date: Mon, 25 Sep 2006 06:59:58 +0800 Subject: [Numpy-discussion] =?GB2312?B?VFBNyKvUscnosbjOrLuk0+u53MDt?= Message-ID: An HTML attachment was scrubbed... URL: From ckkart at hoc.net Thu Sep 21 19:18:34 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Thu, 21 Sep 2006 23:18:34 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?numpy_1=2E0rc1_bdist=5Frpm_fails?= Message-ID: Hi, on linux I get an error when trying to build a rpm package from numpy 1.0rc1: building extension "numpy.core.umath" sources adding 'build/src.linux-i686-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.linux-i686-2.4/numpy/core/__ufunc_api.h' to sources. creating build/src.linux-i686-2.4/src conv_template:> build/src.linux-i686-2.4/src/umathmodule.c error: src/umathmodule.c.src: No such file or directory error: Bad exit status from /home/ck/testarea/rpm/tmp/rpm-tmp.68597 (%build) RPM build errors: Bad exit status from /home/ck/testarea/rpm/tmp/rpm-tmp.68597 (%build) error: command 'rpmbuild' failed with exit status 1 Christian From haase at msg.ucsf.edu Thu Sep 21 19:16:44 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 21 Sep 2006 16:16:44 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45131202.6000105@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> Message-ID: <200609211616.44773.haase@msg.ucsf.edu> On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > David M. Cooke wrote: > > On Thu, 21 Sep 2006 11:34:42 -0700 > > > > Tim Hochberg wrote: > >> Tim Hochberg wrote: > >>> Robert Kern wrote: > >>>> David M. Cooke wrote: > >>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>> Let me offer a third path: the algorithms used for .mean() and > >>>>>> .var() are substandard. There are much better incremental algorithms > >>>>>> that entirely avoid the need to accumulate such large (and therefore > >>>>>> precision-losing) intermediate values. The algorithms look like the > >>>>>> following for 1D arrays in Python: > >>>>>> > >>>>>> def mean(a): > >>>>>> m = a[0] > >>>>>> for i in range(1, len(a)): > >>>>>> m += (a[i] - m) / (i + 1) > >>>>>> return m > >>>>> > >>>>> This isn't really going to be any better than using a simple sum. > >>>>> It'll also be slower (a division per iteration). > >>>> > >>>> With one exception, every test that I've thrown at it shows that it's > >>>> better for float32. That exception is uniformly spaced arrays, like > >>>> linspace(). > >>>> > >>>> > You do avoid > >>>> > accumulating large sums, but then doing the division a[i]/len(a) > >>>> > and adding that will do the same. > >>>> > >>>> Okay, this is true. > >>>> > >>>>> Now, if you want to avoid losing precision, you want to use a better > >>>>> summation technique, like compensated (or Kahan) summation: > >>>>> > >>>>> def mean(a): > >>>>> s = e = a.dtype.type(0) > >>>>> for i in range(0, len(a)): > >>>>> temp = s > >>>>> y = a[i] + e > >>>>> s = temp + y > >>>>> e = (temp - s) + y > >>>>> return s / len(a) > >>>>> > >>>>>> def var(a): > >>>>>> m = a[0] > >>>>>> t = a.dtype.type(0) > >>>>>> for i in range(1, len(a)): > >>>>>> q = a[i] - m > >>>>>> r = q / (i+1) > >>>>>> m += r > >>>>>> t += i * q * r > >>>>>> t /= len(a) > >>>>>> return t > >>>>>> > >>>>>> Alternatively, from Knuth: > >>>>>> > >>>>>> def var_knuth(a): > >>>>>> m = a.dtype.type(0) > >>>>>> variance = a.dtype.type(0) > >>>>>> for i in range(len(a)): > >>>>>> delta = a[i] - m > >>>>>> m += delta / (i+1) > >>>>>> variance += delta * (a[i] - m) > >>>>>> variance /= len(a) > >>>>>> return variance > >> > >> I'm going to go ahead and attach a module containing the versions of > >> mean, var, etc that I've been playing with in case someone wants to mess > >> with them. Some were stolen from traffic on this list, for others I > >> grabbed the algorithms from wikipedia or equivalent. > > > > I looked into this a bit more. I checked float32 (single precision) and > > float64 (double precision), using long doubles (float96) for the "exact" > > results. This is based on your code. Results are compared using > > abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in > > the range of [-100, 900] > > > > First, the mean. In float32, the Kahan summation in single precision is > > better by about 2 orders of magnitude than simple summation. However, > > accumulating the sum in double precision is better by about 9 orders of > > magnitude than simple summation (7 orders more than Kahan). > > > > In float64, Kahan summation is the way to go, by 2 orders of magnitude. > > > > For the variance, in float32, Knuth's method is *no better* than the > > two-pass method. Tim's code does an implicit conversion of intermediate > > results to float64, which is why he saw a much better result. > > Doh! And I fixed that same problem in the mean implementation earlier > too. I was astounded by how good knuth was doing, but not astounded > enough apparently. > > Does it seem weird to anyone else that in: > numpy_scalar python_scalar > the precision ends up being controlled by the python scalar? I would > expect the numpy_scalar to control the resulting precision just like > numpy arrays do in similar circumstances. Perhaps the egg on my face is > just clouding my vision though. > > > The two-pass method using > > Kahan summation (again, in single precision), is better by about 2 orders > > of magnitude. There is practically no difference when using a > > double-precision accumulator amongst the techniques: they're all about 9 > > orders of magnitude better than single-precision two-pass. > > > > In float64, Kahan summation is again better than the rest, by about 2 > > orders of magnitude. > > > > I've put my adaptation of Tim's code, and box-and-whisker plots of the > > results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > > > > Conclusions: > > > > - If you're going to calculate everything in single precision, use Kahan > > summation. Using it in double-precision also helps. > > - If you can use a double-precision accumulator, it's much better than > > any of the techniques in single-precision only. > > > > - for speed+precision in the variance, either use Kahan summation in > > single precision with the two-pass method, or use double precision with > > simple summation with the two-pass method. Knuth buys you nothing, except > > slower code :-) > > The two pass methods are definitely more accurate. I won't be convinced > on the speed front till I see comparable C implementations slug it out. > That may well mean never in practice. However, I expect that somewhere > around 10,000 items, the cache will overflow and memory bandwidth will > become the bottleneck. At that point the extra operations of Knuth won't > matter as much as making two passes through the array and Knuth will win > on speed. Of course the accuracy is pretty bad at single precision, so > the possible, theoretical speed advantage at large sizes probably > doesn't matter. > > -tim > > > After 1.0 is out, we should look at doing one of the above. > > +1 > Hi, I don't understand much of these "fancy algorithms", but does the above mean that after 1.0 is released the float32 functions will catch up in terms of precision precision ("to some extend") with using dtype=float64 ? - Sebastian Haase From davidgrant at gmail.com Thu Sep 21 19:23:54 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 21 Sep 2006 16:23:54 -0700 Subject: [Numpy-discussion] change of default dtype In-Reply-To: References: <4511DF4F.9070206@astraw.com> Message-ID: On 9/20/06, Bill Baxter wrote: > > Hey Andrew, point taken, but I think it would be better if someone who > actually knows the full extent of the change made the edit. I know > zeros and ones changed. Did anything else? > > Anyway, I'm surprised the release notes page is publicly editable. I'm glad that it is editable. I hate wikis that are only editable by a select few. Defeats the purpose (or at least does not maximize the capability of a wiki). -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Thu Sep 21 19:58:35 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 21 Sep 2006 19:58:35 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> References: <20060921152042.28b50937@arbutus.physics.mcmaster.ca> Message-ID: On Thu, 21 Sep 2006, "David M. Cooke" apparently wrote: > Foremost for Python doc strings, I think, is that it look > ok when using pydoc or similar (ipython's ?, for > instance). That means a minimal amount of markup. IMO reStructuredText is very natural for documentation, and it is nicely handled by epydoc. fwiw, Alan Isaac From aisaac at american.edu Thu Sep 21 19:58:31 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 21 Sep 2006 19:58:31 -0400 Subject: [Numpy-discussion] Tests and code documentation In-Reply-To: References: <200609210918.05413.haase@msg.ucsf.edu><1DD79522-8009-4111-B7A0-318E94290891@arachnedesign.net> Message-ID: On Thu, 21 Sep 2006, Charles R Harris apparently wrote: > As to the oddness of \param or @param, here is an example from > Epydoc using Epytext > @type m: number > @param m: The slope of the line. > @type b: number > @param b: The y intercept of the line. Compare to definition list style for consolidated field lists in section 5.1 of http://epydoc.sourceforge.net/fields.html#rst which is much more elegant, IMO. Cheers, Alan Isaac From pgmdevlist at gmail.com Thu Sep 21 20:35:02 2006 From: pgmdevlist at gmail.com (P GM) Date: Thu, 21 Sep 2006 20:35:02 -0400 Subject: [Numpy-discussion] putmask/take ? Message-ID: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> Folks, I'm running into the following problem with putmask on take. >>> import numpy >>> x = N.arange(12.) >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>> i = N.nonzero(m)[0] >>> w = N.array([-1, -2, -3, -4.]) >>> x.putmask(w,m) >>> x.take(i) >>> N.allclose(x.take(i),w) False I'm wondering ifit is intentional, or if it's a problem on my build (1.0b5), or if somebody experiences that as well. Thanks a lot for your input. P. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Sep 21 21:20:33 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 10:20:33 +0900 Subject: [Numpy-discussion] matrixmultiply moved (release notes?) Message-ID: Apparently numpy.matrixmultiply got moved into numpy.oldnumeric.matrixmultiply at some point (or rather ceased to be imported into the numpy namespace). Is there any list of all such methods that got banished? This would be nice to have in the release notes. --bb From ckkart at hoc.net Thu Sep 21 22:14:28 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Fri, 22 Sep 2006 02:14:28 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed Message-ID: Hi, from 1.0b1 to 1.0rc1 the default behaviour of take seems to have changed when omitting the axis argument: In [13]: a = reshape(arange(12),(3,4)) In [14]: take(a,[2,3]) Out[14]: array([2, 3]) In [15]: take(a,[2,3],1) Out[15]: array([[ 2, 3], [ 6, 7], [10, 11]]) Is this intended? Christian From wbaxter at gmail.com Thu Sep 21 22:34:49 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 11:34:49 +0900 Subject: [Numpy-discussion] general version of repmat? Message-ID: Is there some way to get the equivalent of repmat() for ndim == 1 and ndim >2. For ndim == 1, repmat always returns a 2-d array, instead of remaining 1-d. For ndim >2, repmat just doesn't work. Maybe we could add a 'reparray', with the signature: reparray(A, repeats, axis=None) where repeats is a scalar or a sequence. If 'repeats' is a scalar then the matrix is duplicated along 'axis' that many times. If 'repeats' is a sequence of length N, then A is duplicated repeats[i] times along axis[i]. If axis is None then it is assumed to be (0,1,2...N). Er that's not quite complete, because it doesn't specify what happens when you reparray an array to a higher dimension, like a 1-d to a 3-d. Like reparray([1,2], (2,2,2)). I guess the axis parameter could have some 'newaxis' entries to accomodate that. --bb From wbaxter at gmail.com Thu Sep 21 22:37:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 11:37:21 +0900 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: Message-ID: Yep, check the release notes: http://www.scipy.org/ReleaseNotes/NumPy_1.0 search for 'take' on that page to find out what others have changed as well. --bb On 9/22/06, Christian Kristukat wrote: > Hi, > from 1.0b1 to 1.0rc1 the default behaviour of take seems to have changed when > omitting the axis argument: > > In [13]: a = reshape(arange(12),(3,4)) > > In [14]: take(a,[2,3]) > Out[14]: array([2, 3]) > > In [15]: take(a,[2,3],1) > Out[15]: > array([[ 2, 3], > [ 6, 7], > [10, 11]]) > > Is this intended? > > Christian > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ckkart at hoc.net Thu Sep 21 23:45:43 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Fri, 22 Sep 2006 03:45:43 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed References: Message-ID: Bill Baxter gmail.com> writes: > > Yep, check the release notes: > http://www.scipy.org/ReleaseNotes/NumPy_1.0 > search for 'take' on that page to find out what others have changed as well. > --bb Ok. Does axis=None then mean, that take(a, ind) operates on the flattened array? This it at least what it seem to be. I noticed that the ufunc behaves differently. a.take(ind) and a.take(ind, axis=0) behave the same, so the default argument to axis is 0 rather than None. Christian From pgmdevlist at gmail.com Thu Sep 21 19:40:39 2006 From: pgmdevlist at gmail.com (PGM) Date: Thu, 21 Sep 2006 19:40:39 -0400 Subject: [Numpy-discussion] Putmask/take ? Message-ID: <200609211940.40154.pgmdevlist@gmail.com> Folks, I'm running into the following problem with putmask on take. >>> import numpy >>> x = N.arange(12.) >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>> i = N.nonzero(m)[0] >>> w = N.array([-1, -2, -3, -4.]) >>> x.putmask(w,m) >>> x.take(i) >>> N.allclose(x.take(i),w) False I'm wondering ifit is intentional, or if it's a problem on my build (1.0b5), or if somebody experienced it as well. Thanks a lot for your input. P. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Bienstman at ugent.be Fri Sep 22 00:56:51 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Fri, 22 Sep 2006 06:56:51 +0200 Subject: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 In-Reply-To: References: Message-ID: <200609220656.54764.Peter.Bienstman@ugent.be> Cleaning out and rebuilding did the trick! Thanks, Peter On Thursday 21 September 2006 18:33, numpy-discussion-request at lists.sourceforge.net wrote: > Subject: Re: [Numpy-discussion] 1.0rc1 doesn't seem to work on AMD64 > > > I don't see this running the latest from svn on AMD64 here. Not sayin' > there might not be a problem with rc1, I just don't see it with my sources. > > Python 2.4.3 (#1, Jun 13 2006, 11:46:22) > [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy > >>> numpy.version.version > > '1.0.dev3202' > > >>> numpy.version.os.uname() > > ('Linux', 'tethys', '2.6.17-1.2187_FC5', '#1 SMP Mon Sep 11 01:16:59 EDT > 2006', 'x86_64') > > If you are building on Gentoo maybe you could delete the build directory > (and maybe the numpy site package) and rebuild. > > Chuck. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From stefan at sun.ac.za Fri Sep 22 03:02:51 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 09:02:51 +0200 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <200609211940.40154.pgmdevlist@gmail.com> References: <200609211940.40154.pgmdevlist@gmail.com> Message-ID: <20060922070250.GA15593@mentat.za.net> Hi P., On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > I'm running into the following problem with putmask on take. > > >>> import numpy > >>> x = N.arange(12.) > >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>> i = N.nonzero(m)[0] > >>> w = N.array([-1, -2, -3, -4.]) > >>> x.putmask(w,m) > >>> x.take(i) > >>> N.allclose(x.take(i),w) According to the putmask docstring: a.putmask(values, mask) sets a.flat[n] = v[n] for each n where mask.flat[n] is true. v can be scalar. This would mean that 'w' is not of the right length. Would the following do what you want? import numpy as N m = N.array([1,0,0,0,0,0,1,0,0,1,0,1],dtype=bool) w = N.array([-1,-2,-3,-4]) x[m] = w Regards St?fan From stefan at sun.ac.za Fri Sep 22 03:13:02 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 09:13:02 +0200 Subject: [Numpy-discussion] putmask/take ? In-Reply-To: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> References: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> Message-ID: <20060922071302.GA17679@mentat.za.net> On Thu, Sep 21, 2006 at 08:35:02PM -0400, P GM wrote: > Folks, > I'm running into the following problem with putmask on take. > > >>> import numpy > >>> x = N.arange(12.) > >>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>> i = N.nonzero (m)[0] > >>> w = N.array([-1, -2, -3, -4.]) > >>> x.putmask(w,m) You can also use N.place: Similar to putmask arr[mask] = vals but the 1D array vals has the same number of elements as the non-zero values of mask. Inverse of extract. > >>> x.take(i) > >>> N.allclose(x.take(i),w) > False Regards St?fan From robert.kern at gmail.com Fri Sep 22 03:17:57 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 02:17:57 -0500 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <20060922070250.GA15593@mentat.za.net> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> Message-ID: Stefan van der Walt wrote: > Hi P., > > On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > >> I'm running into the following problem with putmask on take. >> >>>>> import numpy >>>>> x = N.arange(12.) >>>>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] >>>>> i = N.nonzero(m)[0] >>>>> w = N.array([-1, -2, -3, -4.]) >>>>> x.putmask(w,m) >>>>> x.take(i) >>>>> N.allclose(x.take(i),w) > > According to the putmask docstring: > > a.putmask(values, mask) sets a.flat[n] = v[n] for each n where > mask.flat[n] is true. v can be scalar. > > This would mean that 'w' is not of the right length. There are 4 true values in m and 4 values in w. What's the wrong length? For the sake of clarity: In [1]: from numpy import * In [3]: m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] In [4]: i = nonzero(m)[0] In [5]: i Out[5]: array([ 0, 6, 9, 11]) In [6]: w = array([-1, -2, -3, -4.]) In [7]: x = arange(12.) In [8]: x.putmask(w, m) In [9]: x Out[9]: array([ -1., 1., 2., 3., 4., 5., -3., 7., 8., -2., 10., -4.]) In [17]: x[array(m, dtype=bool)] = w In [18]: x Out[18]: array([ -1., 1., 2., 3., 4., 5., -2., 7., 8., -3., 10., -4.]) Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. It's pretty clear that this is a bug in .putmask(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lroubeyrie at limair.asso.fr Fri Sep 22 03:34:31 2006 From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie) Date: Fri, 22 Sep 2006 09:34:31 +0200 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <4512C553.6090609@ieee.org> References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> Message-ID: <200609220934.31217.lroubeyrie@limair.asso.fr> Le jeudi 21 septembre 2006 19:01, Travis Oliphant a ?crit?: > Lionel Roubeyrie wrote: > > find any solution for that. I have tried with arrays of dtype=object, but > > I have problem when I want to compute min, max, ... with an error like: > > TypeError: function not supported for these types, and can't coerce > > safely to supported types. > > I just added support for min and max methods of object arrays, by adding > support for Object arrays to the minimum and maximum functions. > > -Travis Hello travis, good news, and thanks for your last comment. However, using nans give some errors with scipy.stats: lionel52>t=array([1,2,nan,4]) lionel53>stats.nanmean(t) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /home/lionel/ /usr/lib/python2.4/site-packages/scipy/stats/stats.py in nanmean(x, axis) 258 259 # XXX: this line is quite clearly wrong --> 260 n = N-sum(isnan(x),axis) 261 putmask(x,isnan(x),0) 262 return stats.mean(x,axis)/factor NameError: global name 'N' is not defined > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Lionel Roubeyrie - lroubeyrie at limair.asso.fr LIMAIR http://www.limair.asso.fr From robert.kern at gmail.com Fri Sep 22 04:06:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 03:06:36 -0500 Subject: [Numpy-discussion] Question about recarray In-Reply-To: <200609220934.31217.lroubeyrie@limair.asso.fr> References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> <200609220934.31217.lroubeyrie@limair.asso.fr> Message-ID: Lionel Roubeyrie wrote: > good news, and thanks for your last comment. However, using nans give some > errors with scipy.stats: > lionel52>t=array([1,2,nan,4]) > > lionel53>stats.nanmean(t) > --------------------------------------------------------------------------- > exceptions.NameError Traceback (most recent > call last) > > /home/lionel/ > > /usr/lib/python2.4/site-packages/scipy/stats/stats.py in nanmean(x, axis) > 258 > 259 # XXX: this line is quite clearly wrong > --> 260 n = N-sum(isnan(x),axis) > 261 putmask(x,isnan(x),0) > 262 return stats.mean(x,axis)/factor > > NameError: global name 'N' is not defined It's a bug in nanmean() as the comment immediately preceding it mentions. I don't know who put it in, but I noticed it and couldn't figure out what it intended to do (or didn't have to time to try). Ah, it's Travis's fault. So he can fix it. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Sep 22 04:21:57 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 22 Sep 2006 10:21:57 +0200 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> Message-ID: <20060922082157.GA21884@mentat.za.net> On Fri, Sep 22, 2006 at 02:17:57AM -0500, Robert Kern wrote: > Stefan van der Walt wrote: > > Hi P., > > > > On Thu, Sep 21, 2006 at 07:40:39PM -0400, PGM wrote: > > > >> I'm running into the following problem with putmask on take. > >> > >>>>> import numpy > >>>>> x = N.arange(12.) > >>>>> m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > >>>>> i = N.nonzero(m)[0] > >>>>> w = N.array([-1, -2, -3, -4.]) > >>>>> x.putmask(w,m) > >>>>> x.take(i) > >>>>> N.allclose(x.take(i),w) > > > > According to the putmask docstring: > > > > a.putmask(values, mask) sets a.flat[n] = v[n] for each n where > > mask.flat[n] is true. v can be scalar. > > > > This would mean that 'w' is not of the right length. > > There are 4 true values in m and 4 values in w. What's the wrong length? The way I read the docstring, you use putmask like this: In [4]: x = N.array([1,2,3,4]) In [5]: x.putmask([4,3,2,1],[1,0,0,1]) In [6]: x Out[6]: array([4, 2, 3, 1]) > For the sake of clarity: > > > In [1]: from numpy import * > > In [3]: m = [1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1] > > In [4]: i = nonzero(m)[0] > > In [5]: i > Out[5]: array([ 0, 6, 9, 11]) > > In [6]: w = array([-1, -2, -3, -4.]) > > In [7]: x = arange(12.) > > In [8]: x.putmask(w, m) > > In [9]: x > Out[9]: > array([ -1., 1., 2., 3., 4., 5., -3., 7., 8., -2., 10., > -4.]) > > In [17]: x[array(m, dtype=bool)] = w > > In [18]: x > Out[18]: > array([ -1., 1., 2., 3., 4., 5., -2., 7., 8., -3., 10., > -4.]) > > > Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. > It's pretty clear that this is a bug in .putmask(). Based purely on what I read in the docstring, I would expect the above to do x[0] = w[0] x[6] = w[6] x[9] = w[9] x[11] = w[11] Since w is of length 4, you'll probably get indices modulo 4: w[6] == w[2] == -3 w[9] == w[1] == -2 w[11] == w[3] == -4 Which seems to explain what you are seeing. Regards St?fan From Svein-Erik.Hamran at ffi.no Fri Sep 22 05:07:03 2006 From: Svein-Erik.Hamran at ffi.no (Svein-Erik.Hamran at ffi.no) Date: Fri, 22 Sep 2006 11:07:03 +0200 Subject: [Numpy-discussion] reading large files Message-ID: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> I would like to read files >2Gbyte. From earlier posting I believed it should be possible with python 2.5. I am getting MemoryError when trying to read a file larger than 2Gb. I am using python2.5 and numpy1.0rc1. Any solution to the problem? SEH -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwe239 at tom.com Wed Sep 27 07:40:32 2006 From: qwe239 at tom.com (=?GB2312?B?IjnUwjI4LTI5yNUvyc+6oyI=?=) Date: Wed, 27 Sep 2006 19:40:32 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU6t8eyxs7xvq3A7bXEssbO8bncwO0tybPFzMSjxOK/zrPM?= Message-ID: An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Sep 22 08:06:17 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 21:06:17 +0900 Subject: [Numpy-discussion] reading large files In-Reply-To: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> References: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> Message-ID: Do you also have a 64-bit processor? Just checking since you didn't mention it. --bb On 9/22/06, Svein-Erik.Hamran at ffi.no wrote: > > > > > I would like to read files >2Gbyte. From earlier posting I believed it > should be possible with python 2.5. > > I am getting MemoryError when trying to read a file larger than 2Gb. > > I am using python2.5 and numpy1.0rc1. > > Any solution to the problem? > > > > SEH > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Fri Sep 22 08:47:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 21:47:28 +0900 Subject: [Numpy-discussion] Rationale for atleast_3d Message-ID: 26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why atleast_3d acts the way it does: http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d He doesn't seem to have gotten any answers. And now I'm wondering the same thing. Anyone have any idea? --bb From tim.hochberg at ieee.org Fri Sep 22 09:08:39 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 06:08:39 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <45131202.6000105@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> Message-ID: <4513E057.7070906@ieee.org> Tim Hochberg wrote: > David M. Cooke wrote: > >> On Thu, 21 Sep 2006 11:34:42 -0700 >> Tim Hochberg wrote: >> >> >> >>> Tim Hochberg wrote: >>> >>> >>>> Robert Kern wrote: >>>> >>>> >>>> >>>>> David M. Cooke wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Let me offer a third path: the algorithms used for .mean() and .var() >>>>>>> are substandard. There are much better incremental algorithms that >>>>>>> entirely avoid the need to accumulate such large (and therefore >>>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>>> following for 1D arrays in Python: >>>>>>> >>>>>>> def mean(a): >>>>>>> m = a[0] >>>>>>> for i in range(1, len(a)): >>>>>>> m += (a[i] - m) / (i + 1) >>>>>>> return m >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> This isn't really going to be any better than using a simple sum. >>>>>> It'll also be slower (a division per iteration). >>>>>> >>>>>> >>>>>> >>>>>> >>>>> With one exception, every test that I've thrown at it shows that it's >>>>> better for float32. That exception is uniformly spaced arrays, like >>>>> linspace(). >>>>> >>>>> > You do avoid >>>>> > accumulating large sums, but then doing the division a[i]/len(a) and >>>>> > adding that will do the same. >>>>> >>>>> Okay, this is true. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Now, if you want to avoid losing precision, you want to use a better >>>>>> summation technique, like compensated (or Kahan) summation: >>>>>> >>>>>> def mean(a): >>>>>> s = e = a.dtype.type(0) >>>>>> for i in range(0, len(a)): >>>>>> temp = s >>>>>> y = a[i] + e >>>>>> s = temp + y >>>>>> e = (temp - s) + y >>>>>> return s / len(a) >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>>>> def var(a): >>>>>>> m = a[0] >>>>>>> t = a.dtype.type(0) >>>>>>> for i in range(1, len(a)): >>>>>>> q = a[i] - m >>>>>>> r = q / (i+1) >>>>>>> m += r >>>>>>> t += i * q * r >>>>>>> t /= len(a) >>>>>>> return t >>>>>>> >>>>>>> Alternatively, from Knuth: >>>>>>> >>>>>>> def var_knuth(a): >>>>>>> m = a.dtype.type(0) >>>>>>> variance = a.dtype.type(0) >>>>>>> for i in range(len(a)): >>>>>>> delta = a[i] - m >>>>>>> m += delta / (i+1) >>>>>>> variance += delta * (a[i] - m) >>>>>>> variance /= len(a) >>>>>>> return variance >>>>>>> >>>>>>> >>> I'm going to go ahead and attach a module containing the versions of >>> mean, var, etc that I've been playing with in case someone wants to mess >>> with them. Some were stolen from traffic on this list, for others I >>> grabbed the algorithms from wikipedia or equivalent. >>> >>> >> I looked into this a bit more. I checked float32 (single precision) and >> float64 (double precision), using long doubles (float96) for the "exact" >> results. This is based on your code. Results are compared using >> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in the >> range of [-100, 900] >> >> First, the mean. In float32, the Kahan summation in single precision is >> better by about 2 orders of magnitude than simple summation. However, >> accumulating the sum in double precision is better by about 9 orders of >> magnitude than simple summation (7 orders more than Kahan). >> >> In float64, Kahan summation is the way to go, by 2 orders of magnitude. >> >> For the variance, in float32, Knuth's method is *no better* than the two-pass >> method. Tim's code does an implicit conversion of intermediate results to >> float64, which is why he saw a much better result. >> > Doh! And I fixed that same problem in the mean implementation earlier > too. I was astounded by how good knuth was doing, but not astounded > enough apparently. > > Does it seem weird to anyone else that in: > numpy_scalar python_scalar > the precision ends up being controlled by the python scalar? I would > expect the numpy_scalar to control the resulting precision just like > numpy arrays do in similar circumstances. Perhaps the egg on my face is > just clouding my vision though. > > >> The two-pass method using >> Kahan summation (again, in single precision), is better by about 2 orders of >> magnitude. There is practically no difference when using a double-precision >> accumulator amongst the techniques: they're all about 9 orders of magnitude >> better than single-precision two-pass. >> >> In float64, Kahan summation is again better than the rest, by about 2 orders >> of magnitude. >> >> I've put my adaptation of Tim's code, and box-and-whisker plots of the >> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ >> >> Conclusions: >> >> - If you're going to calculate everything in single precision, use Kahan >> summation. Using it in double-precision also helps. >> - If you can use a double-precision accumulator, it's much better than any of >> the techniques in single-precision only. >> >> - for speed+precision in the variance, either use Kahan summation in single >> precision with the two-pass method, or use double precision with simple >> summation with the two-pass method. Knuth buys you nothing, except slower >> code :-) >> >> > The two pass methods are definitely more accurate. I won't be convinced > on the speed front till I see comparable C implementations slug it out. > That may well mean never in practice. However, I expect that somewhere > around 10,000 items, the cache will overflow and memory bandwidth will > become the bottleneck. At that point the extra operations of Knuth won't > matter as much as making two passes through the array and Knuth will win > on speed. OK, let me take this back. I looked at this speed effect a little more and the effect is smaller than I remember. For example, "a+=a" slows down by about a factor of 2.5 on my box between 10,000 and 100,0000 elements. That's not insignificant, but assuming (naively) that this means that memory effects account to the equivalent of 1.5 add operations, this doesn't come close to closing to gap between Knuth and the two pass approach. The other thing is that while a+=a and similar operations show this behaviour, a.sum() and add.reduce(a) do not, hinting that perhaps it's only writing to memory that is a bottleneck. Or perhaps hinting that my mental model of what's going on here is badly flawed. So, +1 for me on Kahan summation for computing means and two pass with Kahan summation for variances. It would probably be nice to expose the Kahan sum and maybe even the raw_kahan_sum somewhere. I can see use for the latter in summing a bunch of disparate matrices with high precision. I'm on the fence on using the array dtype for the accumulator dtype versus always using at least double precision for the accumulator. The former is easier to explain and is probably faster, but the latter is a lot more accuracy for basically free. It depends on how the relative speeds shake out I suppose. If the speed of using float64 is comparable to that of using float32, we might as well. One thing I'm not on the fence about is the return type: it should always match the input type, or dtype if that is specified. Otherwise one gets mysterious, gratuitous upcasting. On the subject of upcasting, I'm still skeptical of the behavior of mixed numpy-scalar and python-scalar ops. Since numpy-scalars are generally the results of indexing operations, not literals, I think that they should behave like arrays for purposes of determining the resulting precision, not like python-scalars. Otherwise situations arise where the rank of the input array determine the kind of output (float32 versus float64 for example) since if the array is 1D indexing into the array yields numpy-scalars which then get promoted when they interact with python-scalars.However if the array is 2D, indexing yields a vector, which is not promoted when interacting with python scalars and the precision remains fixed. -tim > Of course the accuracy is pretty bad at single precision, so > the possible, theoretical speed advantage at large sizes probably > doesn't matter. > > -tim > > >> After 1.0 is out, we should look at doing one of the above. >> >> > > +1 > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > m bnb From wbaxter at gmail.com Fri Sep 22 09:31:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 22:31:26 +0900 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: Ok, here's my best shot at a generalized repmat: def reparray(A, tup): if numpy.isscalar(tup): tup = (tup,) d = len(tup) A = numpy.array(A,copy=False,subok=True,ndmin=d) for i,k in enumerate(tup): A = numpy.concatenate([A]*k, axis=i) return A Can anyone improve on this? The only way I could think to seriously improve it would be if there were an N-way concatenate() function in the C-API. Each call to concatenate() involves a potentially big allocation and memcpy of the data. And here's a docstring for it: """Repeat an array the number of times given in the integer tuple, tup. Similar to repmat, but works for arrays of any dimension. reparray(A,(m,n)) is equivalent to repmat(A,m,n) If tup has length d, the result will have dimension of max(d, A.ndim). If tup is scalar it is treated as a 1-tuple. If A.ndim < d, A is promoted to be d-dimensional by prepending new axes. So a shape (3,) array is promoted to (1,3) for 2-D replication, or shape (1,1,3) for 3-D replication. If this is not the desired behavior, promote A to d-dimensions manually before calling this function. If d < A.ndim, ndim is effectively promoted to A.ndim by appending 1's to tup. Examples: >>> a = array([0,1,2]) >>> reparray(a,2) array([0, 1, 2, 0, 1, 2]) >>> reparray(a,(1,2)) array([[0, 1, 2, 0, 1, 2]]) >>> reparray(a,(2,2)) array([[0, 1, 2, 0, 1, 2], [0, 1, 2, 0, 1, 2]]) >>> reparray(a,(2,1,2)) array([[[0, 1, 2, 0, 1, 2]], [[0, 1, 2, 0, 1, 2]]]) See Also: repmat, repeat """ From aisaac at american.edu Fri Sep 22 09:53:32 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 22 Sep 2006 09:53:32 -0400 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: On Fri, 22 Sep 2006, Bill Baxter apparently wrote: > Ok, here's my best shot at a generalized repmat: Sorry to always repeat this suggestion when it comes to repmat, but I think the whole approach is wrong. A repmat should be a separate object type, which behaves like the described matrix but has only one copy of the repetitive data. Cheers, Alan Isaac From wbaxter at gmail.com Fri Sep 22 10:41:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 22 Sep 2006 23:41:36 +0900 Subject: [Numpy-discussion] general version of repmat? In-Reply-To: References: Message-ID: On 9/22/06, Alan G Isaac wrote: > On Fri, 22 Sep 2006, Bill Baxter apparently wrote: > > Ok, here's my best shot at a generalized repmat: > > Sorry to always repeat this suggestion when > it comes to repmat, but I think the whole approach > is wrong. A repmat should be a separate object type, > which behaves like the described matrix but has only one > copy of the repetitive data. That may be true for some cases. But I usually start modifying the data I create right after a repmat. It wouldn't help in that case. So unless you're really making a lot of large repmats of arrays that never change, or for use as temp variables, I can't see a separate class being that much of a win, compared with the complexity of implementing and maintaining it (think "fancy indexing"). YMMV. However, repmat seems to be far less commonly needed in numpy than in Matlab. I think that's mostly thanks to the broadcasting rules, which already create a sort of implicit repmat of the input in many common cases. --bb From charlesr.harris at gmail.com Fri Sep 22 10:43:53 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 22 Sep 2006 08:43:53 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4513E057.7070906@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <4513E057.7070906@ieee.org> Message-ID: Hi, On 9/22/06, Tim Hochberg wrote: > > Tim Hochberg wrote: > > David M. Cooke wrote: > > > >> On Thu, 21 Sep 2006 11:34:42 -0700 > >> Tim Hochberg wrote: > >> > >> > >> > >>> Tim Hochberg wrote: > >>> > >>> > >>>> Robert Kern wrote: > >>>> > >>>> > >>>> > >>>>> David M. Cooke wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Let me offer a third path: the algorithms used for .mean() and > .var() > >>>>>>> are substandard. There are much better incremental algorithms that > >>>>>>> entirely avoid the need to accumulate such large (and therefore > >>>>>>> precision-losing) intermediate values. The algorithms look like > the > >>>>>>> following for 1D arrays in Python: > >>>>>>> > >>>>>>> def mean(a): > >>>>>>> m = a[0] > >>>>>>> for i in range(1, len(a)): > >>>>>>> m += (a[i] - m) / (i + 1) > >>>>>>> return m > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> This isn't really going to be any better than using a simple sum. > >>>>>> It'll also be slower (a division per iteration). > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> With one exception, every test that I've thrown at it shows that > it's > >>>>> better for float32. That exception is uniformly spaced arrays, like > >>>>> linspace(). > >>>>> > >>>>> > You do avoid > >>>>> > accumulating large sums, but then doing the division a[i]/len(a) > and > >>>>> > adding that will do the same. > >>>>> > >>>>> Okay, this is true. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> Now, if you want to avoid losing precision, you want to use a > better > >>>>>> summation technique, like compensated (or Kahan) summation: > >>>>>> > >>>>>> def mean(a): > >>>>>> s = e = a.dtype.type(0) > >>>>>> for i in range(0, len(a)): > >>>>>> temp = s > >>>>>> y = a[i] + e > >>>>>> s = temp + y > >>>>>> e = (temp - s) + y > >>>>>> return s / len(a) > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>>>> def var(a): > >>>>>>> m = a[0] > >>>>>>> t = a.dtype.type(0) > >>>>>>> for i in range(1, len(a)): > >>>>>>> q = a[i] - m > >>>>>>> r = q / (i+1) > >>>>>>> m += r > >>>>>>> t += i * q * r > >>>>>>> t /= len(a) > >>>>>>> return t > >>>>>>> > >>>>>>> Alternatively, from Knuth: > >>>>>>> > >>>>>>> def var_knuth(a): > >>>>>>> m = a.dtype.type(0) > >>>>>>> variance = a.dtype.type(0) > >>>>>>> for i in range(len(a)): > >>>>>>> delta = a[i] - m > >>>>>>> m += delta / (i+1) > >>>>>>> variance += delta * (a[i] - m) > >>>>>>> variance /= len(a) > >>>>>>> return variance > >>>>>>> > >>>>>>> > >>> I'm going to go ahead and attach a module containing the versions of > >>> mean, var, etc that I've been playing with in case someone wants to > mess > >>> with them. Some were stolen from traffic on this list, for others I > >>> grabbed the algorithms from wikipedia or equivalent. > >>> > >>> > >> I looked into this a bit more. I checked float32 (single precision) and > >> float64 (double precision), using long doubles (float96) for the > "exact" > >> results. This is based on your code. Results are compared using > >> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values > in the > >> range of [-100, 900] > >> > >> First, the mean. In float32, the Kahan summation in single precision is > >> better by about 2 orders of magnitude than simple summation. However, > >> accumulating the sum in double precision is better by about 9 orders of > >> magnitude than simple summation (7 orders more than Kahan). > >> > >> In float64, Kahan summation is the way to go, by 2 orders of magnitude. > >> > >> For the variance, in float32, Knuth's method is *no better* than the > two-pass > >> method. Tim's code does an implicit conversion of intermediate results > to > >> float64, which is why he saw a much better result. > >> > > Doh! And I fixed that same problem in the mean implementation earlier > > too. I was astounded by how good knuth was doing, but not astounded > > enough apparently. > > > > Does it seem weird to anyone else that in: > > numpy_scalar python_scalar > > the precision ends up being controlled by the python scalar? I would > > expect the numpy_scalar to control the resulting precision just like > > numpy arrays do in similar circumstances. Perhaps the egg on my face is > > just clouding my vision though. > > > > > >> The two-pass method using > >> Kahan summation (again, in single precision), is better by about 2 > orders of > >> magnitude. There is practically no difference when using a > double-precision > >> accumulator amongst the techniques: they're all about 9 orders of > magnitude > >> better than single-precision two-pass. > >> > >> In float64, Kahan summation is again better than the rest, by about 2 > orders > >> of magnitude. > >> > >> I've put my adaptation of Tim's code, and box-and-whisker plots of the > >> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > >> > >> Conclusions: > >> > >> - If you're going to calculate everything in single precision, use > Kahan > >> summation. Using it in double-precision also helps. > >> - If you can use a double-precision accumulator, it's much better than > any of > >> the techniques in single-precision only. > >> > >> - for speed+precision in the variance, either use Kahan summation in > single > >> precision with the two-pass method, or use double precision with simple > >> summation with the two-pass method. Knuth buys you nothing, except > slower > >> code :-) > >> > >> > > The two pass methods are definitely more accurate. I won't be convinced > > on the speed front till I see comparable C implementations slug it out. > > That may well mean never in practice. However, I expect that somewhere > > around 10,000 items, the cache will overflow and memory bandwidth will > > become the bottleneck. At that point the extra operations of Knuth won't > > matter as much as making two passes through the array and Knuth will win > > on speed. > OK, let me take this back. I looked at this speed effect a little more > and the effect is smaller than I remember. For example, "a+=a" slows > down by about a factor of 2.5 on my box between 10,000 and 100,0000 > elements. That's not insignificant, but assuming (naively) that this > means that memory effects account to the equivalent of 1.5 add > operations, this doesn't come close to closing to gap between Knuth and > the two pass approach. The other thing is that while a+=a and similar > operations show this behaviour, a.sum() and add.reduce(a) do not, > hinting that perhaps it's only writing to memory that is a bottleneck. > Or perhaps hinting that my mental model of what's going on here is badly > flawed. > > So, +1 for me on Kahan summation for computing means and two pass with > Kahan summation for variances. It would probably be nice to expose the > Kahan sum and maybe even the raw_kahan_sum somewhere. I can see use for > the latter in summing a bunch of disparate matrices with high precision. > > I'm on the fence on using the array dtype for the accumulator dtype > versus always using at least double precision for the accumulator. I keep going back and forth on this myself. So I ask myself what I do in practice. In practice, I accumulate in doubles, sometimes extended precision. I only think of accumulating in floats when the number of items is small and computation might be noticeably faster -- and these days it seldom is. The days of the Vax and a 2x speed difference are gone, now we generally have FPUs whose base type is double with float tacked on as an afterthought. The only lesson from those days that remains pertinent is: never, ever, do division. So in practice, I would go with doubles. The only good reason to use floats for floats is a sort of pure consistency. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fakeyq at broadlandguards.com Fri Sep 22 10:37:33 2006 From: fakeyq at broadlandguards.com (Hal Giles) Date: Fri, 22 Sep 2006 16:37:33 +0200 Subject: [Numpy-discussion] intellectual federation Message-ID: <000501c6de55$9fadc64f$5b2a4e57@cjjgtx> Feisalintroduced us one by one, and Auda with a measured word seemed toregister each person. I said I was a simple man who disliked servants about him. The walls each sidewere of regular bands of sandstone, streaked red in many shades. Jealousy, superadded to inefficiency, would be the outcome. The walls each sidewere of regular bands of sandstone, streaked red in many shades. They expected him next day; so wepassed two nights in this strange-coloured, echoing place. At first it was two and two, but theothers joined, till they were six abreast. Each required to be consulted on the details of ourgoing, and where and when we should halt. The edges of the cliffs about us were clipped strangely,like fantastic parapets. The fluency had a lack of grammar, which made my talk aperpetual adventure for my hearers. Therefore it seemed fit to the boyMohammed to run races. Sheslipped, so that he crashed off and broke an arm. The Arab war was geographical, and the Turkish Army an accident. Thirdly, that their virtue lay in depth, not inface. NASIR, AUDA, AND I SET OFF TOGETHER ON THE LONG RIDE. They expected him next day; so wepassed two nights in this strange-coloured, echoing place. The aeroplanes from it had flown uphere and were established. Wadi Osman to-day was less irregular in course, and broadened slowly. Everybody took the offer, and the camels set offin a mob. The aeroplanes from it had flown uphere and were established. Therefore it seemed fit to the boyMohammed to run races. Of the number of dead Turks he could give no account: they didnot enter the register. There entered a tall, strong figure, with a haggard face,passionate and tragic. Or was it merely that long ago we had seen freshgrass growing in the spring? The last part of the marchwas after dark, and when we stopped, Arslan was missing. We were now on the limestoneof the Shefa crest. If his performancewas one-half his desire, we should be prosperous and fortunate. So I judged mywork in Wadi Ais sufficiently done, and well done. Everybody took the offer, and the camels set offin a mob. In prison inEgypt he would cost us food and guards. A short hour later we stopped at the tents of a wife of Dakhil-Allah,for a meal. All he could do for my sake wasto let Daud share the ordained sentence. Mohammed el Kadhi was ourguide, with six Juheina. I was eating my Lords breadwith Turkish teeth! When innocent they were hot andunashamed. If his performancewas one-half his desire, we should be prosperous and fortunate. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mockingbird.gif Type: image/gif Size: 11589 bytes Desc: not available URL: From tim.hochberg at ieee.org Fri Sep 22 12:06:10 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 09:06:10 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <200609211616.44773.haase@msg.ucsf.edu> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> Message-ID: <451409F2.8010805@ieee.org> Sebastian Haase wrote: > On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > >> David M. Cooke wrote: >> >>> On Thu, 21 Sep 2006 11:34:42 -0700 >>> >>> Tim Hochberg wrote: >>> >>>> Tim Hochberg wrote: >>>> >>>>> Robert Kern wrote: >>>>> >>>>>> David M. Cooke wrote: >>>>>> >>>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: >>>>>>> >>>>>>>> Let me offer a third path: the algorithms used for .mean() and >>>>>>>> .var() are substandard. There are much better incremental algorithms >>>>>>>> that entirely avoid the need to accumulate such large (and therefore >>>>>>>> precision-losing) intermediate values. The algorithms look like the >>>>>>>> following for 1D arrays in Python: >>>>>>>> >>>>>>>> def mean(a): >>>>>>>> m = a[0] >>>>>>>> for i in range(1, len(a)): >>>>>>>> m += (a[i] - m) / (i + 1) >>>>>>>> return m >>>>>>>> >>>>>>> This isn't really going to be any better than using a simple sum. >>>>>>> It'll also be slower (a division per iteration). >>>>>>> >>>>>> With one exception, every test that I've thrown at it shows that it's >>>>>> better for float32. That exception is uniformly spaced arrays, like >>>>>> linspace(). >>>>>> >>>>>> > You do avoid >>>>>> > accumulating large sums, but then doing the division a[i]/len(a) >>>>>> > and adding that will do the same. >>>>>> >>>>>> Okay, this is true. >>>>>> >>>>>> >>>>>>> Now, if you want to avoid losing precision, you want to use a better >>>>>>> summation technique, like compensated (or Kahan) summation: >>>>>>> >>>>>>> def mean(a): >>>>>>> s = e = a.dtype.type(0) >>>>>>> for i in range(0, len(a)): >>>>>>> temp = s >>>>>>> y = a[i] + e >>>>>>> s = temp + y >>>>>>> e = (temp - s) + y >>>>>>> return s / len(a) >>>>>>> >>>>>>> >>>>>>>> def var(a): >>>>>>>> m = a[0] >>>>>>>> t = a.dtype.type(0) >>>>>>>> for i in range(1, len(a)): >>>>>>>> q = a[i] - m >>>>>>>> r = q / (i+1) >>>>>>>> m += r >>>>>>>> t += i * q * r >>>>>>>> t /= len(a) >>>>>>>> return t >>>>>>>> >>>>>>>> Alternatively, from Knuth: >>>>>>>> >>>>>>>> def var_knuth(a): >>>>>>>> m = a.dtype.type(0) >>>>>>>> variance = a.dtype.type(0) >>>>>>>> for i in range(len(a)): >>>>>>>> delta = a[i] - m >>>>>>>> m += delta / (i+1) >>>>>>>> variance += delta * (a[i] - m) >>>>>>>> variance /= len(a) >>>>>>>> return variance >>>>>>>> >>>> I'm going to go ahead and attach a module containing the versions of >>>> mean, var, etc that I've been playing with in case someone wants to mess >>>> with them. Some were stolen from traffic on this list, for others I >>>> grabbed the algorithms from wikipedia or equivalent. >>>> >>> I looked into this a bit more. I checked float32 (single precision) and >>> float64 (double precision), using long doubles (float96) for the "exact" >>> results. This is based on your code. Results are compared using >>> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values in >>> the range of [-100, 900] >>> >>> First, the mean. In float32, the Kahan summation in single precision is >>> better by about 2 orders of magnitude than simple summation. However, >>> accumulating the sum in double precision is better by about 9 orders of >>> magnitude than simple summation (7 orders more than Kahan). >>> >>> In float64, Kahan summation is the way to go, by 2 orders of magnitude. >>> >>> For the variance, in float32, Knuth's method is *no better* than the >>> two-pass method. Tim's code does an implicit conversion of intermediate >>> results to float64, which is why he saw a much better result. >>> >> Doh! And I fixed that same problem in the mean implementation earlier >> too. I was astounded by how good knuth was doing, but not astounded >> enough apparently. >> >> Does it seem weird to anyone else that in: >> numpy_scalar python_scalar >> the precision ends up being controlled by the python scalar? I would >> expect the numpy_scalar to control the resulting precision just like >> numpy arrays do in similar circumstances. Perhaps the egg on my face is >> just clouding my vision though. >> >> >>> The two-pass method using >>> Kahan summation (again, in single precision), is better by about 2 orders >>> of magnitude. There is practically no difference when using a >>> double-precision accumulator amongst the techniques: they're all about 9 >>> orders of magnitude better than single-precision two-pass. >>> >>> In float64, Kahan summation is again better than the rest, by about 2 >>> orders of magnitude. >>> >>> I've put my adaptation of Tim's code, and box-and-whisker plots of the >>> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ >>> >>> Conclusions: >>> >>> - If you're going to calculate everything in single precision, use Kahan >>> summation. Using it in double-precision also helps. >>> - If you can use a double-precision accumulator, it's much better than >>> any of the techniques in single-precision only. >>> >>> - for speed+precision in the variance, either use Kahan summation in >>> single precision with the two-pass method, or use double precision with >>> simple summation with the two-pass method. Knuth buys you nothing, except >>> slower code :-) >>> >> The two pass methods are definitely more accurate. I won't be convinced >> on the speed front till I see comparable C implementations slug it out. >> That may well mean never in practice. However, I expect that somewhere >> around 10,000 items, the cache will overflow and memory bandwidth will >> become the bottleneck. At that point the extra operations of Knuth won't >> matter as much as making two passes through the array and Knuth will win >> on speed. Of course the accuracy is pretty bad at single precision, so >> the possible, theoretical speed advantage at large sizes probably >> doesn't matter. >> >> -tim >> >> >>> After 1.0 is out, we should look at doing one of the above. >>> >> +1 >> >> > Hi, > I don't understand much of these "fancy algorithms", but does the above mean > that > after 1.0 is released the float32 functions will catch up in terms of > precision precision ("to some extend") with using dtype=float64 ? > It sounds like there is some consensus to do something to improve the precision after 1.0 comes out. I don't think the details are entirely nailed down, but it sounds like some combination of using Kahan summation and increasing the minimum precision of the accumulator is likely for sum, mean, var and std. -tim From Chris.Barker at noaa.gov Fri Sep 22 12:34:42 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 22 Sep 2006 09:34:42 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <4513E057.7070906@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <451088AB.4050107@ee.byu.edu> <200609191744.43847.haase@msg.ucsf.edu> <4510A890.5090807@ieee.org> <4510B079.5010209@msg.ucsf.edu> <4510B7D8.4010708@msg.ucsf.edu> <4510C93C.8030902@msg.ucsf.edu> <20060920163417.GA4978@arbutus.physics.mcmaster.ca> <4512D256.2080604@ieee.org> <4512DB42.7080509@ieee.org> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <4513E057.7070906@ieee.org> Message-ID: <451410A2.6060508@noaa.gov> Tim Hochberg wrote: > It would probably be nice to expose the > Kahan sum and maybe even the raw_kahan_sum somewhere. What about using it for .sum() by default? What is the speed hit anyway? In any case, having it available would be nice. > I'm on the fence on using the array dtype for the accumulator dtype > versus always using at least double precision for the accumulator. The > former is easier to explain and is probably faster, but the latter is a > lot more accuracy for basically free. I don't think the difficulty of explanation is a big issue at all -- I'm having a really hard time imagining someone getting confused and/or disappointed that their single precision calculation didn't overflow or was more accurate than expected. Anyone that did, would know enough to understand the explanation. In general, users expect things to "just work". They only dig into the details when something goes wrong, like the mean of a bunch of positive integers coming out as negative (wasn't that the OP's example?). The fewer such instance we have, the fewer questions we have. > speeds shake out I suppose. If the speed of using float64 is comparable > to that of using float32, we might as well. Only testing will tell, but my experience is that with double precision FPUs, doubles are just as fast as floats unless you're dealing with enough memory to make a difference in caching. In this case, only the accumulator is double, so that wouldn't be an issue. I suppose the float to double conversion could take some time though... > One thing I'm not on the > fence about is the return type: it should always match the input type, > or dtype if that is specified. +1 > Since numpy-scalars are > generally the results of indexing operations, not literals, I think that > they should behave like arrays for purposes of determining the resulting > precision, not like python-scalars. +1 >> Of course the accuracy is pretty bad at single precision, so >> the possible, theoretical speed advantage at large sizes probably >> doesn't matter. good point. the larger the array -- the more accuracy matters. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Sep 22 13:13:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 11:13:42 -0600 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: Message-ID: <451419C6.1040909@ieee.org> Christian Kristukat wrote: > Bill Baxter gmail.com> writes: > > >> Yep, check the release notes: >> http://www.scipy.org/ReleaseNotes/NumPy_1.0 >> search for 'take' on that page to find out what others have changed as well. >> --bb >> > > Ok. Does axis=None then mean, that take(a, ind) operates on the flattened array? > This it at least what it seem to be. I noticed that the ufunc behaves > differently. a.take(ind) and a.take(ind, axis=0) behave the same, so the default > argument to axis is 0 rather than None. > What do you mean. There is no "ufunc" take. There is a function take that just calls the method. The default arguments for all functions that match methods are the same as the methods (which means axis=None). However, in oldnumeric (which pylab imports by the way), the default axes are the same as they were in Numeric. Also, if you have a 1-d array, then the axis argument doesn't make any difference. Please clarify what you are saying to be sure we don't have a bug floating around. -Travis From oliphant.travis at ieee.org Fri Sep 22 13:24:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 11:24:00 -0600 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <20060922082157.GA21884@mentat.za.net> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> <20060922082157.GA21884@mentat.za.net> Message-ID: <45141C30.60504@ieee.org> Stefan van der Walt wrote: > On Fri, Sep 22, 2006 at 02:17:57AM -0500, Robert Kern wrote: > >>> According to the putmask docstring: >>> >>> a.putmask(values, mask) sets a.flat[n] = v[n] for each n where >>> mask.flat[n] is true. v can be scalar. >>> >>> This would mean that 'w' is not of the right length. >>> >> There are 4 true values in m and 4 values in w. What's the wrong >> > length? > > The way I read the docstring, you use putmask like this: > > In [4]: x = N.array([1,2,3,4]) > > In [5]: x.putmask([4,3,2,1],[1,0,0,1]) > > In [6]: x > Out[6]: array([4, 2, 3, 1]) > > >> >> Out[9] and Out[18] should have been the same, but elements 6 and 9 are flipped. >> It's pretty clear that this is a bug in .putmask(). >> > > Based purely on what I read in the docstring, I would expect the above to do > > x[0] = w[0] > x[6] = w[6] > x[9] = w[9] > x[11] = w[11] > > Since w is of length 4, you'll probably get indices modulo 4: > > w[6] == w[2] == -3 > w[9] == w[1] == -2 > w[11] == w[3] == -4 > > Which seems to explain what you are seeing. > Yes, this does explain what you are seeing. It is the behavior of Numeric's putmask (where this method came from). It does seem counter-intuitive, and I'm not sure what to do with it. In some sense putmask should behave the same as x[m] = w. But, on the other-hand, was anybody actually using the "modular" indexing "feature" of "putmask". Here are our options: 1) "fix-it" and risk breaking code for people who used putmask and the modular indexing "feature," 2) Get rid of it as a method (and keep it as a function so that oldnumeric can use it.) 3) Keep everything the way it is. -Travis From sransom at nrao.edu Fri Sep 22 13:36:48 2006 From: sransom at nrao.edu (Scott Ransom) Date: Fri, 22 Sep 2006 13:36:48 -0400 Subject: [Numpy-discussion] bug in oldnumeric module Message-ID: <200609221336.48144.sransom@nrao.edu> argmin is currently defined as using the argmax method! From numpy/oldnumeric/functions.py def argmin(x, axis=-1): return N.argmax(x, axis) Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From ellisonbg.net at gmail.com Fri Sep 22 13:41:33 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 22 Sep 2006 11:41:33 -0600 Subject: [Numpy-discussion] Always using scientific notation to print Message-ID: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> I want to be able to print an array in scientific notation. I have seen the set_printoptions() functions, but it doesn't really have an option for *always* using scientific notation. Can this be done? How? Thanks Brian From charlesr.harris at gmail.com Fri Sep 22 14:14:22 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 22 Sep 2006 12:14:22 -0600 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: <451409F2.8010805@ieee.org> References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> <451409F2.8010805@ieee.org> Message-ID: On 9/22/06, Tim Hochberg wrote: > > Sebastian Haase wrote: > > On Thursday 21 September 2006 15:28, Tim Hochberg wrote: > > > >> David M. Cooke wrote: > >> > >>> On Thu, 21 Sep 2006 11:34:42 -0700 > >>> > >>> Tim Hochberg wrote: > >>> > >>>> Tim Hochberg wrote: > >>>> > >>>>> Robert Kern wrote: > >>>>> > >>>>>> David M. Cooke wrote: > >>>>>> > >>>>>>> On Wed, Sep 20, 2006 at 03:01:18AM -0500, Robert Kern wrote: > >>>>>>> > >>>>>>>> Let me offer a third path: the algorithms used for .mean() and > >>>>>>>> .var() are substandard. There are much better incremental > algorithms > >>>>>>>> that entirely avoid the need to accumulate such large (and > therefore > >>>>>>>> precision-losing) intermediate values. The algorithms look like > the > >>>>>>>> following for 1D arrays in Python: > >>>>>>>> > >>>>>>>> def mean(a): > >>>>>>>> m = a[0] > >>>>>>>> for i in range(1, len(a)): > >>>>>>>> m += (a[i] - m) / (i + 1) > >>>>>>>> return m > >>>>>>>> > >>>>>>> This isn't really going to be any better than using a simple sum. > >>>>>>> It'll also be slower (a division per iteration). > >>>>>>> > >>>>>> With one exception, every test that I've thrown at it shows that > it's > >>>>>> better for float32. That exception is uniformly spaced arrays, like > >>>>>> linspace(). > >>>>>> > >>>>>> > You do avoid > >>>>>> > accumulating large sums, but then doing the division a[i]/len(a) > >>>>>> > and adding that will do the same. > >>>>>> > >>>>>> Okay, this is true. > >>>>>> > >>>>>> > >>>>>>> Now, if you want to avoid losing precision, you want to use a > better > >>>>>>> summation technique, like compensated (or Kahan) summation: > >>>>>>> > >>>>>>> def mean(a): > >>>>>>> s = e = a.dtype.type(0) > >>>>>>> for i in range(0, len(a)): > >>>>>>> temp = s > >>>>>>> y = a[i] + e > >>>>>>> s = temp + y > >>>>>>> e = (temp - s) + y > >>>>>>> return s / len(a) > >>>>>>> > >>>>>>> > >>>>>>>> def var(a): > >>>>>>>> m = a[0] > >>>>>>>> t = a.dtype.type(0) > >>>>>>>> for i in range(1, len(a)): > >>>>>>>> q = a[i] - m > >>>>>>>> r = q / (i+1) > >>>>>>>> m += r > >>>>>>>> t += i * q * r > >>>>>>>> t /= len(a) > >>>>>>>> return t > >>>>>>>> > >>>>>>>> Alternatively, from Knuth: > >>>>>>>> > >>>>>>>> def var_knuth(a): > >>>>>>>> m = a.dtype.type(0) > >>>>>>>> variance = a.dtype.type(0) > >>>>>>>> for i in range(len(a)): > >>>>>>>> delta = a[i] - m > >>>>>>>> m += delta / (i+1) > >>>>>>>> variance += delta * (a[i] - m) > >>>>>>>> variance /= len(a) > >>>>>>>> return variance > >>>>>>>> > >>>> I'm going to go ahead and attach a module containing the versions of > >>>> mean, var, etc that I've been playing with in case someone wants to > mess > >>>> with them. Some were stolen from traffic on this list, for others I > >>>> grabbed the algorithms from wikipedia or equivalent. > >>>> > >>> I looked into this a bit more. I checked float32 (single precision) > and > >>> float64 (double precision), using long doubles (float96) for the > "exact" > >>> results. This is based on your code. Results are compared using > >>> abs(exact_stat - computed_stat) / max(abs(values)), with 10000 values > in > >>> the range of [-100, 900] > >>> > >>> First, the mean. In float32, the Kahan summation in single precision > is > >>> better by about 2 orders of magnitude than simple summation. However, > >>> accumulating the sum in double precision is better by about 9 orders > of > >>> magnitude than simple summation (7 orders more than Kahan). > >>> > >>> In float64, Kahan summation is the way to go, by 2 orders of > magnitude. > >>> > >>> For the variance, in float32, Knuth's method is *no better* than the > >>> two-pass method. Tim's code does an implicit conversion of > intermediate > >>> results to float64, which is why he saw a much better result. > >>> > >> Doh! And I fixed that same problem in the mean implementation earlier > >> too. I was astounded by how good knuth was doing, but not astounded > >> enough apparently. > >> > >> Does it seem weird to anyone else that in: > >> numpy_scalar python_scalar > >> the precision ends up being controlled by the python scalar? I would > >> expect the numpy_scalar to control the resulting precision just like > >> numpy arrays do in similar circumstances. Perhaps the egg on my face is > >> just clouding my vision though. > >> > >> > >>> The two-pass method using > >>> Kahan summation (again, in single precision), is better by about 2 > orders > >>> of magnitude. There is practically no difference when using a > >>> double-precision accumulator amongst the techniques: they're all about > 9 > >>> orders of magnitude better than single-precision two-pass. > >>> > >>> In float64, Kahan summation is again better than the rest, by about 2 > >>> orders of magnitude. > >>> > >>> I've put my adaptation of Tim's code, and box-and-whisker plots of the > >>> results, at http://arbutus.mcmaster.ca/dmc/numpy/variance/ > >>> > >>> Conclusions: > >>> > >>> - If you're going to calculate everything in single precision, use > Kahan > >>> summation. Using it in double-precision also helps. > >>> - If you can use a double-precision accumulator, it's much better than > >>> any of the techniques in single-precision only. > >>> > >>> - for speed+precision in the variance, either use Kahan summation in > >>> single precision with the two-pass method, or use double precision > with > >>> simple summation with the two-pass method. Knuth buys you nothing, > except > >>> slower code :-) > >>> > >> The two pass methods are definitely more accurate. I won't be convinced > >> on the speed front till I see comparable C implementations slug it out. > >> That may well mean never in practice. However, I expect that somewhere > >> around 10,000 items, the cache will overflow and memory bandwidth will > >> become the bottleneck. At that point the extra operations of Knuth > won't > >> matter as much as making two passes through the array and Knuth will > win > >> on speed. Of course the accuracy is pretty bad at single precision, so > >> the possible, theoretical speed advantage at large sizes probably > >> doesn't matter. > >> > >> -tim > >> > >> > >>> After 1.0 is out, we should look at doing one of the above. > >>> > >> +1 > >> > >> > > Hi, > > I don't understand much of these "fancy algorithms", but does the above > mean > > that > > after 1.0 is released the float32 functions will catch up in terms of > > precision precision ("to some extend") with using dtype=float64 ? > > > It sounds like there is some consensus to do something to improve the > precision after 1.0 comes out. I don't think the details are entirely > nailed down, but it sounds like some combination of using Kahan > summation and increasing the minimum precision of the accumulator is > likely for sum, mean, var and std. I would go with double as the default except when the base precision is greater. Higher precisions should be available as a check, i.e., if the results look funny use Kahan or extended to see if roundoff is a problem. I think Kahan should be available because it adds precision to *any* base precision, even extended or quad, so there is always something to check against. But I don't think Kahan should be the default because of the speed hit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 22 14:27:49 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 13:27:49 -0500 Subject: [Numpy-discussion] bug in oldnumeric module In-Reply-To: <200609221336.48144.sransom@nrao.edu> References: <200609221336.48144.sransom@nrao.edu> Message-ID: Scott Ransom wrote: > argmin is currently defined as using the argmax method! Please check out the latest source from SVN. I fixed this a few days ago. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Sep 22 14:30:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 13:30:06 -0500 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> Message-ID: Brian Granger wrote: > I want to be able to print an array in scientific notation. > > I have seen the set_printoptions() functions, but it doesn't really > have an option for *always* using scientific notation. > > Can this be done? How? You can write a function that formats arrays how you like them and then tell ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Fri Sep 22 14:44:16 2006 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 22 Sep 2006 12:44:16 -0600 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> Message-ID: <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> > You can write a function that formats arrays how you like them and then tell > ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). That seems to be a little low level for most users. Would it be hard to have the possibility of specifying a format string? Brian From pgmdevlist at gmail.com Fri Sep 22 03:35:41 2006 From: pgmdevlist at gmail.com (PGM) Date: Fri, 22 Sep 2006 03:35:41 -0400 Subject: [Numpy-discussion] putmask/take ? In-Reply-To: <20060922071302.GA17679@mentat.za.net> References: <777651ce0609211735u2f866184o62d1067deeaece1@mail.gmail.com> <20060922071302.GA17679@mentat.za.net> Message-ID: <200609220335.42110.pgmdevlist@gmail.com> Stefan, Thanks for your suggestions, but that won't do for what I'm working on : I need to get putmask working, or at least knowing it doesnt'. Robert, thanks for your input. The function putmask doesn't work either. Oh, thinking about it: would it be possible to have the same order of arguments in the function and the method ? Compare putmask(array, mask, values) with array.putmask(values, mask) The same comment applies to put: put(array, indices, values) vs array.put(values, indices, mode) That's a tad confusing... From oliphant at ee.byu.edu Fri Sep 22 14:55:45 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 12:55:45 -0600 Subject: [Numpy-discussion] Always using scientific notation to print In-Reply-To: <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> References: <6ce0ac130609221041w65c5b07bra20b698845df80dd@mail.gmail.com> <6ce0ac130609221144o2d683e39u2c7f0f76a43c16c7@mail.gmail.com> Message-ID: <451431B1.2020004@ee.byu.edu> Brian Granger wrote: >>You can write a function that formats arrays how you like them and then tell >>ndarray to use it for __str__ or __repr__ using numpy.set_string_function(). >> >> > >That seems to be a little low level for most users. Would it be hard >to have the possibility of specifying a format string? > > No, that woudn't be hard. If you can wait several weeks, add a ticket, or better, dig in to the current printing function and add it to the set_printoptions code. -Travis From tim.hochberg at ieee.org Fri Sep 22 15:18:07 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 22 Sep 2006 12:18:07 -0700 Subject: [Numpy-discussion] please change mean to use dtype=float In-Reply-To: References: <200609191659.08788.haase@msg.ucsf.edu> <20060921175937.393288bb@arbutus.physics.mcmaster.ca> <45131202.6000105@ieee.org> <200609211616.44773.haase@msg.ucsf.edu> <451409F2.8010805@ieee.org> Message-ID: <451436EF.1070005@ieee.org> Charles R Harris wrote: [CHOP] > On 9/22/06, *Tim Hochberg* > wrote: > > > It sounds like there is some consensus to do something to improve the > precision after 1.0 comes out. I don't think the details are entirely > nailed down, but it sounds like some combination of using Kahan > summation and increasing the minimum precision of the accumulator is > likely for sum, mean, var and std. > > > I would go with double as the default except when the base precision > is greater. Higher precisions should be available as a check, i.e., if > the results look funny use Kahan or extended to see if roundoff is a > problem. I think Kahan should be available because it adds precision > to *any* base precision, even extended or quad, so there is always > something to check against. But I don't think Kahan should be the > default because of the speed hit. > Note that add.reduce will be available for doing simple-fast-reductions, so I consider some speed hit as acceptable. I'll be interested to see what the actual speed difference between Kahan and straight summation actually are. It may also be possible to unroll the core loop cleverly so that multiple ops can be scheduled on the FPU in parallel. It doesn't look like naive unrolling will work since each iteration depends on the values of the previous one. However, it probably doesn't matter much where the low order bits get added back in, so there's some flexibility in how the loop gets unrolled. -tim PS. Sorry about the untrimmed posts. From oliphant at ee.byu.edu Fri Sep 22 15:29:08 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 13:29:08 -0600 Subject: [Numpy-discussion] Putmask/take ? In-Reply-To: <45141C30.60504@ieee.org> References: <200609211940.40154.pgmdevlist@gmail.com> <20060922070250.GA15593@mentat.za.net> <20060922082157.GA21884@mentat.za.net> <45141C30.60504@ieee.org> Message-ID: <45143984.4060607@ee.byu.edu> Travis Oliphant wrote: >Yes, this does explain what you are seeing. It is the behavior of >Numeric's putmask (where this method came from). It does seem >counter-intuitive, and I'm not sure what to do with it. In some sense >putmask should behave the same as x[m] = w. But, on the other-hand, >was anybody actually using the "modular" indexing "feature" of "putmask". > >Here are our options: > >1) "fix-it" and risk breaking code for people who used putmask and the >modular indexing "feature," >2) Get rid of it as a method (and keep it as a function so that >oldnumeric can use it.) >3) Keep everything the way it is. > > O.K. putmask is keeping the same behavior (it is intentional behavior and has been for a long time). However, because of the confusion, it is being removed as a method (I know this is late, but it's a little-used method and was only added by NumPy). Putmask will be a function. Also, the remaining put method will have it's arguments switched to match the function. IIRC the original switch was made to accomodate masked arrays methods of the same name. But, this does not really help anyway since arr.put for a masked array doesn't even take an indicies argument, so confusing users who don't even use masked arrays seems pointless. Scream now if you think I'm being unreasonable. This will be in 1.0rc2 (yes we will need rc2) The original poster has still not explained why a[mask] = values does not work suitably. -Travis From mike.ressler at alum.mit.edu Fri Sep 22 15:37:37 2006 From: mike.ressler at alum.mit.edu (Mike Ressler) Date: Fri, 22 Sep 2006 12:37:37 -0700 Subject: [Numpy-discussion] reading large files In-Reply-To: References: <2969FC64ACA5D348937BF081FD5A2F2FBC40C4@hbu-posten.ffi.no> Message-ID: <268febdf0609221237y5ccc1a3cj7bed84eadac44503@mail.gmail.com> On 9/22/06, Bill Baxter wrote: > > Do you also have a 64-bit processor? Just checking since you didn't > mention it. > --bb > > On 9/22/06, Svein-Erik.Hamran at ffi.no wrote: > > I would like to read files >2Gbyte. From earlier posting I believed it > > should be possible with python 2.5. > > > > I am getting MemoryError when trying to read a file larger than 2Gb. > > > > I am using python2.5 and numpy1.0rc1. > Also, how much memory do you have? You will have to use memory mapping as well if the file size is greater than your memory size. But you should be hopeful. I have been working with > 10 GB files for some time using beta versions of Python 2.5 and numpy-1.0 on a 64-bit AMD machine with 4 GB of memory. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 22 15:54:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 14:54:04 -0500 Subject: [Numpy-discussion] Question about recarray In-Reply-To: References: <200609211616.59129.lroubeyrie@limair.asso.fr> <4512C553.6090609@ieee.org> <200609220934.31217.lroubeyrie@limair.asso.fr> Message-ID: Robert Kern wrote: > Ah, it's Travis's fault. So he can fix it. :-) And lo, it was fixed. Amen. http://projects.scipy.org/scipy/scipy/changeset/2217 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Sep 22 15:57:10 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 22 Sep 2006 13:57:10 -0600 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: References: Message-ID: <45144016.4060504@ee.byu.edu> Bill Baxter wrote: >26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why >atleast_3d acts the way it does: >http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d > >He doesn't seem to have gotten any answers. And now I'm wondering the >same thing. Anyone have any idea? > > This function came from scipy and was written by somebody at Enthought. I was hoping they would respond. The behavior of atleast_3d does make sense in the context of atleast_2d and thinking of 3-d arrays as "stacks" of 2-d arrays where the stacks are in the last dimension. atleast_2d converts 1-d arrays to 1xN arrays atleast_3d converts 1-d arrays to 1xNx1 arrays so that they can be "stacked" in the last dimension. I agree that this isn't consistent with the general notion of "pre-pending" 1's to increase the dimensionality of the array. However, array(a, copy=False, ndmin=3) will always produce arrays with a 1 at the begining. So at_least3d is convenient if you like to think of 3-d arrays of stacks of 2-d arrays where the last axis is the "stacking" dimension. -Travis From robert.kern at gmail.com Fri Sep 22 17:14:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 22 Sep 2006 16:14:47 -0500 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <45144016.4060504@ee.byu.edu> References: <45144016.4060504@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Bill Baxter wrote: > >> 26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why >> atleast_3d acts the way it does: >> http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atleast+3d >> >> He doesn't seem to have gotten any answers. And now I'm wondering the >> same thing. Anyone have any idea? >> > This function came from scipy and was written by somebody at Enthought. > I was hoping they would respond. That would have either been from Eric or Travis V way back in the day. Both are out of the office today. The "3D arrays as stacks of 2D arrays" is probably as good an explanation as any. Much of Numeric (e.g. the axis dot() reduces over) was predicated on that kind of logic. There is probably some kind of consistency argument with dstack() and atleast_2d(), i.e. In [46]: a = arange(5) In [47]: atleast_3d(a) Out[47]: array([[[0], [1], [2], [3], [4]]]) In [48]: dstack([a, a]) Out[48]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) In [49]: dstack(map(atleast_2d, [a, a])) Out[49]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) In [50]: dstack(map(atleast_3d, [a, a])) Out[50]: array([[[0, 0], [1, 1], [2, 2], [3, 3], [4, 4]]]) That's the problem with consistency arguments; there are so many things one could be consistent with! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Fri Sep 22 17:50:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 22 Sep 2006 14:50:04 -0700 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <45144016.4060504@ee.byu.edu> References: <45144016.4060504@ee.byu.edu> Message-ID: <200609221450.04587.haase@msg.ucsf.edu> On Friday 22 September 2006 12:57, Travis Oliphant wrote: > Bill Baxter wrote: > >26 weeks, 4 days, 2 hours and 9 minutes ago, Zden?k Hur?k asked why > >atleast_3d acts the way it does: > >http://article.gmane.org/gmane.comp.python.numeric.general/4382/match=atle > >ast+3d > > > >He doesn't seem to have gotten any answers. And now I'm wondering the > >same thing. Anyone have any idea? > > This function came from scipy and was written by somebody at Enthought. > I was hoping they would respond. The behavior of atleast_3d does make > sense in the context of atleast_2d and thinking of 3-d arrays as > "stacks" of 2-d arrays where the stacks are in the last dimension. > > atleast_2d converts 1-d arrays to 1xN arrays > > atleast_3d converts 1-d arrays to 1xNx1 arrays so that they can be > "stacked" in the last dimension. I agree that this isn't consistent > with the general notion of "pre-pending" 1's to increase the > dimensionality of the array. > > However, array(a, copy=False, ndmin=3) will always produce arrays with > a 1 at the begining. So at_least3d is convenient if you like to think > of 3-d arrays of stacks of 2-d arrays where the last axis is the > "stacking" dimension. > I'm working with "stacks of 2d arrays" -- but I was always under the impression that -- since the last axis is the "fastest" -- "stacks of " should stack in the first dimension -- not the last --so that the members of the stack still remain contiguous is memory. Is there a general consensus on how one should look at this ? Or are there multiple incompatible view -- maybe coming from different fields -- ? -Sebastian From ckkart at hoc.net Fri Sep 22 22:00:04 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Sat, 23 Sep 2006 02:00:04 +0000 (UTC) Subject: [Numpy-discussion] take behaviour has changed References: <451419C6.1040909@ieee.org> Message-ID: Travis Oliphant ieee.org> writes: > > Christian Kristukat wrote: > > Bill Baxter gmail.com> writes: > > > > > >> Yep, check the release notes: > >> http://www.scipy.org/ReleaseNotes/NumPy_1.0 > >> search for 'take' on that page to find out what others have changed as well. > >> --bb > >> > > > > Ok. Does axis=None then mean, that take(a, ind) operates on the > > flattened array? > > This it at least what it seem to be. I noticed that the ufunc behaves > > differently. a.take(ind) and a.take(ind, axis=0) behave the same, so > > the default > > argument to axis is 0 rather than None. > > > > What do you mean. There is no "ufunc" take. There is a function take > that just calls the method. The default arguments for all functions Sorry, I never really read about what are ufuncs. I thought those are class methods of the ndarray objects... Anyway, I was refering to the following difference: In [7]: a Out[7]: array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11]]) In [8]: a.take([0]) Out[8]: array([[0, 1, 2, 3, 4, 5]]) In [9]: take(a,[0]) Out[9]: array([0]) To be sure I understood: Does axis=None then mean, that take operates on the flattened array? Christian From oliphant.travis at ieee.org Fri Sep 22 23:31:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 22 Sep 2006 21:31:55 -0600 Subject: [Numpy-discussion] take behaviour has changed In-Reply-To: References: <451419C6.1040909@ieee.org> Message-ID: <4514AAAB.3010906@ieee.org> Christian Kristukat wrote: >>> Ok. Does axis=None then mean, that take(a, ind) operates on the >>> flattened array? >>> Yes, that is correct. > Sorry, I never really read about what are ufuncs. I thought those are class > methods of the ndarray objects... Anyway, I was refering to the following > difference: > > In [7]: a > Out[7]: > array([[ 0, 1, 2, 3, 4, 5], > [ 6, 7, 8, 9, 10, 11]]) > > In [8]: a.take([0]) > Out[8]: array([[0, 1, 2, 3, 4, 5]]) > > In [9]: take(a,[0]) > Out[9]: array([0]) > Doh!. That is a bug. take(a,[0]) is correct a.take([0]) is not correct. -Travis From wbaxter at gmail.com Fri Sep 22 23:53:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 23 Sep 2006 12:53:02 +0900 Subject: [Numpy-discussion] Rationale for atleast_3d In-Reply-To: <200609221450.04587.haase@msg.ucsf.edu> References: <45144016.4060504@ee.byu.edu> <200609221450.04587.haase@msg.ucsf.edu> Message-ID: Thanks for the explanations, folks, I thought that might be the case. On 9/23/06, Sebastian Haase wrote: > I'm working with "stacks of 2d arrays" -- but I was always under the > impression that -- since the last axis is the "fastest" -- "stacks of > " should stack in the first dimension -- not the last --so that > the members of the stack still remain contiguous is memory. I also don't see why you'd want to stack matrices along the last axis in numpy. C-style memory order is one reason not to, but printing as well seems to favor stacking matrices along the first (0th) axis instead of the last. > > Is there a general consensus on how one should look at this ? Or are there > multiple incompatible view -- maybe coming from different fields -- ? I've beed stacking along the first axis myself. Perhaps the original folks who implemented atleast_3d were coming from Matlab, where stacking is generally done along the last axis. But everything in Matlab is Fortran order under the hood so it makes sense there. Anyway, glad to know it was just the implementer's idiosyncracy rather than some deep secret of the universe that I wasn't aware of. I sometimes wonder what sort of overhead is induced by numpy using primarily C-style while many of the underlying algorithms we use are in Fortran. I suspect f2py has to do a lot of copying to get things in the right order for making the calls to Fortran. --bb From matthew.brett at gmail.com Sat Sep 23 07:36:58 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 23 Sep 2006 12:36:58 +0100 Subject: [Numpy-discussion] dtype truth value Message-ID: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> Hi, Forgive my ignorance, but why is this? In [1]:from numpy import * In [2]:if not dtype(' References: Message-ID: Here's a new version of 'reparray'. I tested the previous version against 'repmat' for the 2d case and found it to be as much as 3x slower in some instances. Ick. So I redid it using the approach repmat uses -- reshape() and repeat() rather than concatenate(). Now it's very nearly as fast as repmat for the 2-d case. def reparray(A, tup): if numpy.isscalar(tup): tup = (tup,) d = len(tup) c = numpy.array(A,copy=False,subok=True,ndmin=d) shape = list(c.shape) n = c.size for i, nrep in enumerate(tup): if nrep!=1: c = c.reshape(-1,n).repeat(nrep,0) dim_in = shape[i] dim_out = dim_in*nrep shape[i] = dim_out n /= dim_in return c.reshape(shape) The full file with tests and timing code is attached. One thing I noticed while doing this was that repmat doesn't preserve subclasses. Which is funny considering 'mat' is part of the name and it only works on 2-d arrays. Using asanyarray() instead of asarray() in the first line should fix that. --Bill -------------- next part -------------- A non-text attachment was scrubbed... Name: play.py Type: text/x-python Size: 5486 bytes Desc: not available URL: From wvtyix at buyevent.com Sat Sep 23 12:43:36 2006 From: wvtyix at buyevent.com (Mat Foster) Date: Sat, 23 Sep 2006 16:43:36 -0000 Subject: [Numpy-discussion] wade puma Message-ID: <001401c624fa$f418fd13$9490d551@nawres> And bonus news, the story is already being translated to the big screen. I come down, with some vested interest here, I suppose, on the side of eccentric vision. ML: The Lord Soho series held some rather dark and acrid views of your fellow Englishmen. However, as a study in social behavior, I think its a neat site. I put the work into it and it stayed in the bestseller list for three months. You tune into us to hear us chat with science fiction and fantasy authors primarily. The problem, as Ive suggested, is a general one. However, as a study in social behavior, I think its a neat site. My piece is entitled The Last Frequency, and it was a lot of fun to do. Lorrie and Sheila talk about subjectivity and required reading list and interview Kathe Koja. got produced because my editor fell in love with the idea. If youve got the time and the inclination, please do so. For many underpublished authors, obscurity remains the number one obstacle to overcome. Swarming bees are usually incredibly docile, at least for a few hours. Yes, hes a friend of mine, and Im pimping the hell out of the book. And at such times she is as uncompromisingly physical as she is real. The problem, as Ive suggested, is a general one. The novella is also available to read online at Robs website. Gotta say I wont be complaining about this weather. No hard feelings, even if you knock the tar out of me. Create your own superhero or supervillian I wrote it because the fans wanted another Rincewind novel. And it looks like he just might have a taker. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pixel.gif Type: image/gif Size: 7900 bytes Desc: not available URL: From oliphant.travis at ieee.org Sat Sep 23 14:42:09 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 23 Sep 2006 12:42:09 -0600 Subject: [Numpy-discussion] dtype truth value In-Reply-To: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> References: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> Message-ID: <45158001.5080101@ieee.org> Matthew Brett wrote: > Hi, > > Forgive my ignorance, but why is this? > > In [1]:from numpy import * > > In [2]:if not dtype(' ...: print 'Truth value is no' > ...: > ...: > Truth value is no > Truth value of user-built Python objects is tested by looking at: 1) __nonzero__ method is called to determine truth value. 2) sequence or mapping behavior (then the length is used. If the length is greater than 0, then True, otherwise it's False. For data-type objects, there is no __nonzero__ method, but it does have "mapping" behavior so that fields can be extracted from a data-type using mapping notation. The "length" of the data-type is the number of defined fields. Therefore, if the data-type has no defined fields, it's length is 0 and it's truth value is "False". So, you can think of the test as if dtype(...): print "Data-type has fields:" else: print "Data-type does not have fields:" -Travis From matthew.brett at gmail.com Sat Sep 23 15:15:57 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 23 Sep 2006 20:15:57 +0100 Subject: [Numpy-discussion] dtype truth value In-Reply-To: <45158001.5080101@ieee.org> References: <1e2af89e0609230436r6cc19128n80b62a0e6769ffa1@mail.gmail.com> <45158001.5080101@ieee.org> Message-ID: <1e2af89e0609231215p456fbef2m5fe9e42c9060fb5d@mail.gmail.com> > So, you can think of the test as > > if dtype(...): > print "Data-type has fields:" > else: > print "Data-type does not have fields:" Thank you as ever for your very clear and helpful explanation, Best, Matthew From amcmorl at gmail.com Sun Sep 24 19:32:21 2006 From: amcmorl at gmail.com (Angus McMorland) Date: Mon, 25 Sep 2006 11:32:21 +1200 Subject: [Numpy-discussion] Shift or rotate command? Message-ID: Hi all, Other data languages that I have worked with have a routine for shifting data along axes, with wrapping. In IDL it's called 'shift', and works such that print, a 0 1 2 3 4 5 6 7 8 9 10 11 print, shift(a, 2, 0) 2 3 0 1 6 7 4 5 10 11 8 9 print, shift(a, 2, 1) 10 11 8 9 2 3 0 1 6 7 4 5 In pdl (pdl.perl.org) the equivalent routine is called rotate. Is there a version of this in numpy, or can it be easily achieved using existing technology - I could do it with multiple sliced assignment statements but that seems very clunky and likely slow. I've looked through the examples and the numpy book but without success. Angus. -- AJC McMorland, PhD Student Physiology, University of Auckland Armourer, Auckland University Fencing Secretary, Fencing North Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sun Sep 24 20:44:25 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 09:44:25 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Howdy Angus, Yeh, that does seem like a hole in the API. Travis added a rollaxis() but there's still no simple way to roll the elements themselves. I took a look at numpy.fft.fftshift, which is a function that has to do a similar thing. It does it by concatenating aranges and then doing a take(). Presumably Pearu knew what he was doing when he wrote that, so we can assume this is probably close to the best possible. :-) From that idea here's a function that implements roll(). def roll(y,shift,axis): """Roll the elements in the array by 'shift' positions along the given axis.""" from numpy import asanyarray,concatenate,arange y = asanyarray(y) n = y.shape[axis] shift %= n # does the right thing for negative shifts, too return y.take(concatenate((arange(shift,n),arange(shift))), axis) Performance is only very slightly worse (<1%) using return y.take(r_[shift:n, :shift], axis) as the last line, which is maybe slightly more readable if you're down wit' the r_. I prefer the name 'roll' because: a) there's already rollaxis with conceptually similar behavior and b) 'shift' is used in other languages (perl comes to mind) sometimes to indicate shifting off the end, possibly filling in with zeros or just leaving the array shorter. c) 'rotate' seems potentially confusing given the existence of the 'rot90' function already. --bb On 9/25/06, Angus McMorland wrote: > Hi all, > > Other data languages that I have worked with have a routine for shifting > data along axes, with wrapping. > In IDL it's called 'shift', and works such that > print, a > 0 1 2 3 > 4 5 6 7 > 8 9 10 11 > > print, shift(a, 2, 0) > 2 3 0 1 > 6 7 4 5 > 10 11 8 9 > > print, shift(a, 2, 1) > 10 11 8 9 > 2 3 0 1 > 6 7 4 5 > > In pdl (pdl.perl.org) the equivalent routine is called rotate. Is there a > version of this in numpy, or can it be easily achieved using existing > technology - I could do it with multiple sliced assignment statements but > that seems very clunky and likely slow. I've looked through the examples and > the numpy book but without success. > > Angus. From charlesr.harris at gmail.com Sun Sep 24 20:51:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 24 Sep 2006 18:51:23 -0600 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: On 9/24/06, Bill Baxter wrote: > > Howdy Angus, > Yeh, that does seem like a hole in the API. Travis added a rollaxis() > but there's still no simple way to roll the elements themselves. > > I took a look at numpy.fft.fftshift, which is a function that has to > do a similar thing. It does it by concatenating aranges and then > doing a take(). Presumably Pearu knew what he was doing when he wrote > that, so we can assume this is probably close to the best possible. > :-) From that idea here's a function that implements roll(). > > def roll(y,shift,axis): > """Roll the elements in the array by 'shift' positions along the > given axis.""" > from numpy import asanyarray,concatenate,arange > y = asanyarray(y) > n = y.shape[axis] > shift %= n # does the right thing for negative shifts, too > return y.take(concatenate((arange(shift,n),arange(shift))), axis) It is possible to do a shift inplace using two reflections implemented with swaps. This works because two reflections is the same as a rotating twice the distance between the centers of the reflections. I don't know if it is worth implementing this, however. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sun Sep 24 21:12:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 10:12:43 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Went ahead and added an enhancement request: http://projects.scipy.org/scipy/numpy/ticket/293 This is something I've wanted in the past too. --bb On 9/25/06, Charles R Harris wrote: > > > On 9/24/06, Bill Baxter wrote: > > Howdy Angus, > > Yeh, that does seem like a hole in the API. Travis added a rollaxis() > > but there's still no simple way to roll the elements themselves. > > > > I took a look at numpy.fft.fftshift, which is a function that has to > > do a similar thing. It does it by concatenating aranges and then > > doing a take(). Presumably Pearu knew what he was doing when he wrote > > that, so we can assume this is probably close to the best possible. > > :-) From that idea here's a function that implements roll(). > > > > def roll(y,shift,axis): > > """Roll the elements in the array by 'shift' positions along the > > given axis.""" > > from numpy import asanyarray,concatenate,arange > > y = asanyarray(y) > > n = y.shape[axis] > > shift %= n # does the right thing for negative shifts, too > > return > y.take(concatenate((arange(shift,n),arange(shift))), axis) > > It is possible to do a shift inplace using two reflections implemented with > swaps. This works because two reflections is the same as a rotating twice > the distance between the centers of the reflections. I don't know if it is > worth implementing this, however. > > Chuck From aisaac at american.edu Sun Sep 24 21:41:22 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 24 Sep 2006 21:41:22 -0400 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: On Mon, 25 Sep 2006, Bill Baxter apparently wrote: > Went ahead and added an enhancement request: > http://projects.scipy.org/scipy/numpy/ticket/293 > This is something I've wanted in the past too. GAUSS draws a useful distinction between "shifting" and "rotating": Works roughly like this for the 2D case: #rotater: rotate row elements # Format: y = rotater(x,r) # rotater(x,r) # Input: x RxK array # rotateby size R integer array, or integer (rotation amounts) # Output: y RxK array: # rows rotated by rotateby #shiftr: shift row elements and fill with fv # Format: y = shiftr(x,shiftby,fv) # Input: x RxC array # shiftby Rx1 array or scalar (shift amounts) # fv Rx1 array or scalar (fill values) # Output: y RxC array: # rows shifted by shiftby # rows filled with fill fwiw, Alan Isaac From wbaxter at gmail.com Sun Sep 24 22:04:15 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 25 Sep 2006 11:04:15 +0900 Subject: [Numpy-discussion] Shift or rotate command? In-Reply-To: References: Message-ID: Hmm. Yes maybe a shift w/fill would be useful too. I can't recall needing such a thing, but it could probably also be implemented easily in a way similar to roll() above. A multi-axis roll might also be nice. Could just allow roll's shiftby and axis args to be tuples. Looking closer at the existing 'rollaxis', numpy.rollaxis(a, axis, start=0) if a.shape is (3,4,5,6) rollaxis(a, 3, 1).shape is (3,6,4,5) rollaxis(a, 2, 0).shape is (5,3,4,6) rollaxis(a, 1, 3).shape is (3,5,4,6) rollaxis(a, 1, 4).shape is (3,5,6,4) it occurs to me that what it is actually doing is not what we've been calling 'rolling'. It's just a move, really (remove value from one place and re-insert in another). So perhaps the name should be 'moveaxis' instead? --bb On 9/25/06, Alan G Isaac wrote: > On Mon, 25 Sep 2006, Bill Baxter apparently wrote: > > Went ahead and added an enhancement request: > > http://projects.scipy.org/scipy/numpy/ticket/293 > > This is something I've wanted in the past too. > > > GAUSS draws a useful distinction between "shifting" and > "rotating": > > Works roughly like this for the 2D case: > > #rotater: rotate row elements > # Format: y = rotater(x,r) > # rotater(x,r) > # Input: x RxK array > # rotateby size R integer array, or integer (rotation amounts) > # Output: y RxK array: > # rows rotated by rotateby > > #shiftr: shift row elements and fill with fv > # Format: y = shiftr(x,shiftby,fv) > # Input: x RxC array > # shiftby Rx1 array or scalar (shift amounts) > # fv Rx1 array or scalar (fill values) > # Output: y RxC array: > # rows shifted by shiftby > # rows filled with fill > > fwiw, > Alan Isaac From amcmorl at gmail.com Mon Sep 25 00:37:35 2006 From: amcmorl at gmail.com (Angus McMorland) Date: Mon, 25 Sep 2006 16:37:35 +1200 Subject: [Numpy-discussion] eval shortcomings? Message-ID: Hi all, Can someone explain why the following occurs? a = numpy.zeros((100)) b = numpy.ones((10)) a[20:30] = b # okay eval('a[50:60] = b') # raises SyntaxError: invalid syntax Is there some line mangling that the interpretor does that eval doesn't do? I see there's room for global and local parameters to eval. Would setting one of those be the answer? Cheers, Angus. -- AJC McMorland, PhD Student Physiology, University of Auckland Armourer, Auckland University Fencing Secretary, Fencing North Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Sep 25 00:40:32 2006 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 25 Sep 2006 00:40:32 -0400 Subject: [Numpy-discussion] eval shortcomings? In-Reply-To: References: Message-ID: On 25/09/06, Angus McMorland wrote: > Hi all, > > Can someone explain why the following occurs? > > a = numpy.zeros((100)) > b = numpy.ones((10)) > a[20:30] = b # okay > eval('a[50:60] = b') # raises SyntaxError: invalid syntax > > Is there some line mangling that the interpretor does that eval doesn't do? No. Eval evaluates expressions, that is, formulas producing a value. "a=b" does not produce a value, so you are obtaining the same error you would if you'd written if a=b: ... The way you run code that doesn't return a value is with "exec". A. M. Archibald From ivilata at carabos.com Mon Sep 25 08:00:50 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 25 Sep 2006 14:00:50 +0200 Subject: [Numpy-discussion] Fixes to numexpr string support Message-ID: <4517C4F2.6040103@carabos.com> Hi all, these are just a couple of small fixes to the string support in Numexpr, and a test case for the string copy operation. For the base patches: 1.http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg01551.html 2.http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg02261.html Cheers, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr-str-fix.diff Type: text/x-patch Size: 1910 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: OpenPGP digital signature URL: From chanley at stsci.edu Mon Sep 25 08:37:46 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 25 Sep 2006 08:37:46 -0400 Subject: [Numpy-discussion] please restore the way dtype prints to original version Message-ID: <4517CD9A.8090103@stsci.edu> Hi, Change set 3213 changed the data type printing with an array from something like dtype=int64 to dtype='int64'. Although this is a small cosmetic change it has broken all of the doctests I have written for numpy code. I expect that I am not the only person this change has caught. Please restore the previous behavior until after the 1.0 release at least. Thank you, Chris From mihoto. at yahoo.co.jp Mon Sep 25 10:10:21 2006 From: mihoto. at yahoo.co.jp (=?iso-2022-jp?B?bWlobw==?=) Date: Mon, 25 Sep 2006 14:10:21 -0000 Subject: [Numpy-discussion] =?iso-2022-jp?b?GyRCTDVOQRsoQg==?= Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?09 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From matthew.brett at gmail.com Mon Sep 25 10:35:14 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 25 Sep 2006 15:35:14 +0100 Subject: [Numpy-discussion] Downcasting to smallest storage Message-ID: <1e2af89e0609250735n6968e9bcsfc1fb49cda13c21@mail.gmail.com> Hi, I am sorry if I have missed anything obvious here, but is there a fast simple way to downcast an array to the smallest storage that hold array data within a specified precision - e.g. a = array([1.0]) small_a = fantasy_function(a, rtol=1.0000000000000001e-05, atol=1e-08 ) b = array([1.2]) small_b = fantasy_function(b, tol=1.0000000000000001e-05, atol=1e-08) where a.dtype becomes, say, uint8, and b.dtype becomes float32? Thanks a lot, Matthew From faltet at carabos.com Mon Sep 25 11:23:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 17:23:04 +0200 Subject: [Numpy-discussion] Negative division and numpy scalars Message-ID: <1159197785.3971.10.camel@localhost.localdomain> Hi, I have been bitten by a subtlety in numpy scalar divisions. The next exposes the issue: >>> -1/20 -1 >>> Numeric.array([-1])[0] / Numeric.array([20])[0] -1 >>> numarray.array([-1])[0] / numarray.array([20])[0] -1 >>> numpy.array([-1])[0] / numpy.array([20])[0] 0 After some digging, I've found that Python and C follow different conventions for doing this negative division: Python do a floor of the result, while C truncates it. As numpy scalar operations seems to be implemented in C, it seems that it follows the C convention and truncates the result. In fact, I like this behaviour of NumPy scalars (at least, when I'm aware of it!), but I thought it would be nice to warn other people about that. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Sep 25 12:28:34 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 25 Sep 2006 10:28:34 -0600 Subject: [Numpy-discussion] please restore the way dtype prints to original version In-Reply-To: <4517CD9A.8090103@stsci.edu> References: <4517CD9A.8090103@stsci.edu> Message-ID: <451803B2.1060402@ieee.org> Christopher Hanley wrote: > Hi, > > Change set 3213 changed the data type printing with an array from > something like dtype=int64 to dtype='int64'. Although this is a small > cosmetic change it has broken all of the doctests I have written for > numpy code. I was changing the way dtypes print and this was an unexpected consequence. It has been restored. Please try r3215 -Travis From faltet at carabos.com Mon Sep 25 12:47:38 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 18:47:38 +0200 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? Message-ID: <1159202858.3971.14.camel@localhost.localdomain> Hi, Anybody know if there is a map between NumPy types and Numeric typecodes? Something like 'typecodes' for numarray: >>> numarray.typecode {'UInt64': 'U', 'Int32': 'i', 'Int16': 's', 'Float64': 'd', 'Object': 'O', 'UInt8': 'b', 'UInt32': 'u', 'Complex64': 'D', 'UInt16': 'w', 'Bool': 'B', 'Complex32': 'F', 'Int64': 'N', 'Int8': '1', 'Float32': 'f'} Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Mon Sep 25 13:08:59 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 25 Sep 2006 11:08:59 -0600 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <1159202858.3971.14.camel@localhost.localdomain> References: <1159202858.3971.14.camel@localhost.localdomain> Message-ID: <45180D2B.6000200@ee.byu.edu> Francesc Altet wrote: >Hi, > >Anybody know if there is a map between NumPy types and Numeric >typecodes? Something like 'typecodes' for numarray: > > How about dtype(obj).char? -Travis From faltet at carabos.com Mon Sep 25 13:18:14 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon, 25 Sep 2006 19:18:14 +0200 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <45180D2B.6000200@ee.byu.edu> References: <1159202858.3971.14.camel@localhost.localdomain> <45180D2B.6000200@ee.byu.edu> Message-ID: <1159204695.3971.18.camel@localhost.localdomain> El dl 25 de 09 del 2006 a les 11:08 -0600, en/na Travis Oliphant va escriure: > Francesc Altet wrote: > > >Hi, > > > >Anybody know if there is a map between NumPy types and Numeric > >typecodes? Something like 'typecodes' for numarray: > > > > > How about > > dtype(obj).char? This doesn't work for many types: >>> Numeric.array([1], typecode=numpy.dtype('int32').char) array([1]) this is fine, but: >>> Numeric.array([1], typecode=numpy.dtype('int16').char) Traceback (most recent call last): File "", line 1, in ? TypeError: typecode argument must be a valid type. >>> Numeric.array([1], typecode=numpy.dtype('int64').char) Traceback (most recent call last): File "", line 1, in ? TypeError: typecode argument must be a valid type. Anyway, this is not very important as I can do my own map internally. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Mon Sep 25 13:41:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 25 Sep 2006 11:41:53 -0600 Subject: [Numpy-discussion] NumPy types --> Numeric typecodes map? In-Reply-To: <1159204695.3971.18.camel@localhost.localdomain> References: <1159202858.3971.14.camel@localhost.localdomain> <45180D2B.6000200@ee.byu.edu> <1159204695.3971.18.camel@localhost.localdomain> Message-ID: <451814E1.4070506@ee.byu.edu> Francesc Altet wrote: >El dl 25 de 09 del 2006 a les 11:08 -0600, en/na Travis Oliphant va >escriure: > > >>Francesc Altet wrote: >> >> >> >>>Hi, >>> >>>Anybody know if there is a map between NumPy types and Numeric >>>typecodes? Something like 'typecodes' for numarray: >>> >>> Oh, you mean actual Numeric typecodes, not Numeric-like typecodes :-) dtype(obj).char will not work for the Numeric typecodes that changed, set up a dictionary-like object which uses dtype(obj).char on all but the ones that changed is my suggestion. See the core/numerictypes.py module for dictionary-like objects. Perhaps this would be a good thing to add to numpy/oldnumeric 'b' --> 'B' '1' --> 'b' 's' --> 'h' 'w' --> 'H' 'u' --> 'I' -Travis From listservs at mac.com Mon Sep 25 14:30:25 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 25 Sep 2006 14:30:25 -0400 Subject: [Numpy-discussion] Numpy RC1 build blows up Message-ID: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> I've just tried building RC1 on OSX 10.4 (Intel), and it fails almost immediately. This did not fail in the past with beta5 or from SVN: Osoyoos:~/Development/numpy-1.0rc1 chris$ python setup.py build Running from numpy source directory. F2PY Version 2_3198 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3'] running build running config_fc running build_src building py_modules sources creating build creating build/src.darwin-8.7.1-i386-2.4 creating build/src.darwin-8.7.1-i386-2.4/numpy creating build/src.darwin-8.7.1-i386-2.4/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src.darwin-8.7.1-i386-2.4/numpy/core Generating build/src.darwin-8.7.1-i386-2.4/numpy/core/config.h customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fPIC -fno-common -dynamic -DNDEBUG -g -O3 -Wall - Wstrict-prototypes compile options: '-I/Library/Frameworks/Python.framework/Versions/2.4/ include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/Library/ Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: _configtest.c gcc: installation problem, cannot exec `cc1': No such file or directory gcc: installation problem, cannot exec `cc1': No such file or directory failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ core.py", line 174, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 87, in run self.build_sources() File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ command/build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 50, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From robert.kern at gmail.com Mon Sep 25 14:57:48 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 25 Sep 2006 13:57:48 -0500 Subject: [Numpy-discussion] Numpy RC1 build blows up In-Reply-To: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> References: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> Message-ID: listservs at mac.com wrote: > I've just tried building RC1 on OSX 10.4 (Intel), and it fails almost > immediately. This did not fail in the past with beta5 or from SVN: It looks like a problem on your end: > gcc: installation problem, cannot exec `cc1': No such file or directory Can you build anything else with gcc? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From listservs at mac.com Mon Sep 25 15:05:27 2006 From: listservs at mac.com (listservs at mac.com) Date: Mon, 25 Sep 2006 15:05:27 -0400 Subject: [Numpy-discussion] Numpy RC1 build blows up -- IGNORE In-Reply-To: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> References: <16EEBCF6-2316-440C-B098-C53B216904B5@mac.com> Message-ID: <4BB90E4B-83D0-4AE4-B767-BCAF2E0CA54B@mac.com> Ignore this query -- silly mistake on my part. Sorry. On Sep 25, 2006, at 2:30 PM, listservs at mac.com wrote: > I've just tried building RC1 on OSX 10.4 (Intel), and it fails > almost immediately. This did not fail in the past with beta5 or > from SVN: > > Osoyoos:~/Development/numpy-1.0rc1 chris$ python setup.py build > Running from numpy source directory. > F2PY Version 2_3198 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/ > vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3'] > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src.darwin-8.7.1-i386-2.4 > creating build/src.darwin-8.7.1-i386-2.4/numpy > creating build/src.darwin-8.7.1-i386-2.4/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src.darwin-8.7.1-i386-2.4/numpy/core > Generating build/src.darwin-8.7.1-i386-2.4/numpy/core/config.h > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > Could not locate executable g77 > Could not locate executable f77 > Could not locate executable f95 > customize GnuFCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler > customize Gnu95FCompiler using config > C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fPIC -fno-common -dynamic -DNDEBUG -g -O3 - > Wall -Wstrict-prototypes > > compile options: '-I/Library/Frameworks/Python.framework/Versions/ > 2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/ > Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: _configtest.c > gcc: installation problem, cannot exec `cc1': No such file or > directory > gcc: installation problem, cannot exec `cc1': No such file or > directory > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 89, in ? > setup_package() > File "setup.py", line 82, in setup_package > configuration=configuration ) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > core.py", line 174, in setup > return old_setup(**new_attr) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/core.py", line 149, in setup > dist.run_commands() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 946, in run_commands > self.run_command(cmd) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/command/build.py", line 112, in run > self.run_command(cmd_name) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 87, in run > self.build_sources() > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 106, in build_sources > self.build_extension_sources(ext) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 212, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/Users/chris/Development/numpy-1.0rc1/numpy/distutils/ > command/build_src.py", line 270, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 50, in generate_config_h > raise "ERROR: Failed to test configuration" > ERROR: Failed to test configuration > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From Peter.Bienstman at ugent.be Tue Sep 26 04:20:33 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Tue, 26 Sep 2006 10:20:33 +0200 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 Message-ID: <200609261020.36232.Peter.Bienstman@ugent.be> Hi, Not sure if this is a numpy or a matplotlib issue, but I can't seem to get matplotlib-0.87.5 to work with numpy1.0rc: Python 2.4.3 (#1, Sep 21 2006, 13:06:42) [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pylab import * Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/pylab.py", line 1, in ? from matplotlib.pylab import * File "/usr/lib/python2.4/site-packages/matplotlib/pylab.py", line 196, in ? import cm File "/usr/lib/python2.4/site-packages/matplotlib/cm.py", line 5, in ? import colors File "/usr/lib/python2.4/site-packages/matplotlib/colors.py", line 33, in ? from numerix import array, arange, take, put, Float, Int, where, \ File "/usr/lib/python2.4/site-packages/matplotlib/numerix/__init__.py", line 74, in ? Matrix = matrix NameError: name 'matrix' is not defined This is on an AMD64 platform. I tried removing the build directories of both packages, and reinstalling, but that didn't work. I also posted this to the matplotlib list, but got no reply. Thanks! Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From wbaxter at gmail.com Tue Sep 26 04:29:52 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 26 Sep 2006 17:29:52 +0900 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 In-Reply-To: <200609261020.36232.Peter.Bienstman@ugent.be> References: <200609261020.36232.Peter.Bienstman@ugent.be> Message-ID: Are you sure it's matplotlib-0.87.5? Line 74 in my c:/Python24/Lib/site-packages/matplotlib/numerix/ is "from numpy import *" and not "Matrix = matrix" as in your stackdump. You can check the site-packages/matplotlib/__init__.py file for the __version__ variable to make sure. Mine's got: __version__ = '0.87.5' __revision__ = '$Revision: 2761 $' __date__ = '$Date: 2006-09-05 09:37:04 -0400 (Tue, 05 Sep 2006) $' --bb On 9/26/06, Peter Bienstman wrote: > Hi, > > Not sure if this is a numpy or a matplotlib issue, but I can't seem to get > matplotlib-0.87.5 to work with numpy1.0rc: > > Python 2.4.3 (#1, Sep 21 2006, 13:06:42) > [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from pylab import * > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/pylab.py", line 1, in ? > from matplotlib.pylab import * > File "/usr/lib/python2.4/site-packages/matplotlib/pylab.py", line 196, in ? > import cm > File "/usr/lib/python2.4/site-packages/matplotlib/cm.py", line 5, in ? > import colors > File "/usr/lib/python2.4/site-packages/matplotlib/colors.py", line 33, in ? > from numerix import array, arange, take, put, Float, Int, where, \ > File "/usr/lib/python2.4/site-packages/matplotlib/numerix/__init__.py", line > 74, in ? > Matrix = matrix > NameError: name 'matrix' is not defined > > This is on an AMD64 platform. I tried removing the build directories of both > packages, and reinstalling, but that didn't work. > > I also posted this to the matplotlib list, but got no reply. > > Thanks! > > Peter > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From crabmeatalcohol at royalityinn.com Tue Sep 26 08:28:52 2006 From: crabmeatalcohol at royalityinn.com (accordion) Date: Tue, 26 Sep 2006 20:28:52 +0800 Subject: [Numpy-discussion] This one is shooting skywards Message-ID: <06778917688355.B093544A88@83AYZY3> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: chiang.gif Type: image/gif Size: 15925 bytes Desc: not available URL: From Peter.Bienstman at ugent.be Tue Sep 26 08:44:24 2006 From: Peter.Bienstman at ugent.be (Peter Bienstman) Date: Tue, 26 Sep 2006 14:44:24 +0200 Subject: [Numpy-discussion] numpy1.0rc and matplotlib-0.87.5 In-Reply-To: References: Message-ID: <200609261444.24388.Peter.Bienstman@ugent.be> On Tuesday 26 September 2006 14:24, numpy-discussion-request at lists.sourceforge.net wrote: > Are you sure it's matplotlib-0.87.5? > Line 74 in my c:/Python24/Lib/site-packages/matplotlib/numerix/ > is "from numpy import *" > and not "Matrix = matrix" as in your stackdump. > > You can check the site-packages/matplotlib/__init__.py file for the > __version__ variable to make sure. Mine's got: > __version__ = '0.87.5' > __revision__ = '$Revision: 2761 $' > __date__ = '$Date: 2006-09-05 09:37:04 -0400 (Tue, 05 Sep 2006) $' Hmm, turns out deleting the build directory and doing a 'setup.py install' is not enough. When I delete the matplotlib directory from site-packages it does work. Thanks! Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: not available URL: From gnurser at googlemail.com Tue Sep 26 10:10:12 2006 From: gnurser at googlemail.com (George Nurser) Date: Tue, 26 Sep 2006 15:10:12 +0100 Subject: [Numpy-discussion] reload and f2py Message-ID: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> I'm running Python 2.3.5 with recent SVN f2py. Suppose I import an extension I have built with f2py. Then, if I edit the fortran and recompile the extension, I cannot use reload to use the modified version within the same Python session. I believe this is an problem with Python, that reload doesn't work with externally compiled extensions. Is this something that can be changed in future versions of Python, or is it an inherent problem? For editing and recompiling within an ipython session is quite a good way of getting the fortran code to work, leaving aside any problems in doing the f2py interfacing. --George Nurser. From klemm at phys.ethz.ch Tue Sep 26 10:26:23 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Tue, 26 Sep 2006 16:26:23 +0200 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas Message-ID: Hello, I try to build numpy 1.0rc1 with blas and lapack installed in another place than /usr/lib. I compiled blas and lapack according to the instructions on the scipy webpage and installed them under /scratch/python2.4/lib. I set the environment variables BLAS=/scratch/python2.4/lib/libfblas.a and LAPACK=/scratch/python2.4/lib/libflapack.a however, if I run python setup.py config in the numpy source directory, it appears only to pick up the libraries installed under /usr/lib, which are incomplete. What else can I do to help numpy finding the right libraries? Regards, Hanno -- Hanno Klemm klemm at phys.ethz.ch From hetland at tamu.edu Tue Sep 26 11:08:37 2006 From: hetland at tamu.edu (Rob Hetland) Date: Tue, 26 Sep 2006 10:08:37 -0500 Subject: [Numpy-discussion] numpy/scipy/mpl on Mac OS X with python 2.5 successful compilation Message-ID: I successfully compiled numpy/scipy/mpl on my Intel mac (OS X 10.4.7 with gfortran) with the latest Universal MacPython 2.5. Everything went through without a hitch. I've been waiting for this for a while now (between desire for ctypes, and macpython 2.4 readline issues), and I am very happy to have it up and running. Thanks to everyone who makes this happen, -Rob ---- Rob Hetland, Associate Professor Dept. of Oceanography, Texas A&M University http://pong.tamu.edu/~rob phone: 979-458-0096, fax: 979-845-6331 From faltet at carabos.com Tue Sep 26 12:35:52 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 26 Sep 2006 18:35:52 +0200 Subject: [Numpy-discussion] typeDict vs sctypeDict Message-ID: <200609261835.53563.faltet@carabos.com> Hi, I've just noted this: In [12]: numpy.typeNA is numpy.sctypeNA Out[12]: True In [13]: numpy.typeDict is numpy.sctypeDict Out[13]: True Why these aliases? Ara sc* replacing its not-sc-prefix counterparts? I'd like to know this so that I can avoid names that could be deprecated in a future. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From charlesr.harris at gmail.com Tue Sep 26 12:36:18 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Sep 2006 10:36:18 -0600 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas In-Reply-To: References: Message-ID: On 9/26/06, Hanno Klemm wrote: > > > Hello, > > I try to build numpy 1.0rc1 with blas and lapack installed in another > place than /usr/lib. I compiled blas and lapack according to the > instructions on the scipy webpage and installed them under > /scratch/python2.4/lib. > > I set the environment variables > BLAS=/scratch/python2.4/lib/libfblas.a and > LAPACK=/scratch/python2.4/lib/libflapack.a Try putting stuff in the site.cfg file. At one point I needed the following: [atlas] library_dirs = /usr/lib64/atlas atlas_libs = lapack, blas, cblas, atlas Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Sep 26 12:42:46 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue, 26 Sep 2006 18:42:46 +0200 Subject: [Numpy-discussion] Default values in scalar constructors Message-ID: <200609261842.46548.faltet@carabos.com> Hi, Is there any reason why the next works for some types: In [26]: numpy.int32() Out[26]: 0 In [27]: numpy.float64() Out[27]: 0.0 but don't for the next others? In [28]: numpy.int16() --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : function takes exactly 1 argument (0 given) In [29]: numpy.float32() --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : function takes exactly 1 argument (0 given) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From fperez.net at gmail.com Tue Sep 26 12:46:14 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 26 Sep 2006 10:46:14 -0600 Subject: [Numpy-discussion] reload and f2py In-Reply-To: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> References: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> Message-ID: On 9/26/06, George Nurser wrote: > I'm running Python 2.3.5 with recent SVN f2py. > > Suppose I import an extension I have built with f2py. Then, if I edit > the fortran and recompile the extension, I cannot use reload to use > the modified version within the same Python session. > > I believe this is an problem with Python, that reload doesn't work > with externally compiled extensions. As far as I know, you are correct. > Is this something that can be changed in future versions of Python, or > is it an inherent problem? For editing and recompiling within an > ipython session is quite a good way of getting the fortran code to > work, leaving aside any problems in doing the f2py interfacing. Unfortunately due to this limitation, I just restart ipython when I need to reload extension code. I'd be thrilled to know if there is any easy workaround. It's worth noting that weave is in this regard extremly ipython-friendly: since the name of the extension it loads is a hash of the source, /any/ change to the source results in a newly named extension being built and loaded. So while you get a bit of memory bloat by keeping the old, unused extensions around, the net interactive effect is that 'reloading works' for weave-generated code. One more reason why I think weave is the best thing since toasted bagels. Cheers, f From oliphant at ee.byu.edu Tue Sep 26 15:46:31 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:46:31 -0600 Subject: [Numpy-discussion] typeDict vs sctypeDict In-Reply-To: <200609261835.53563.faltet@carabos.com> References: <200609261835.53563.faltet@carabos.com> Message-ID: <45198397.3050903@ee.byu.edu> Francesc Altet wrote: >Hi, > >I've just noted this: > >In [12]: numpy.typeNA is numpy.sctypeNA >Out[12]: True > >In [13]: numpy.typeDict is numpy.sctypeDict >Out[13]: True > >Why these aliases? Ara sc* replacing its not-sc-prefix counterparts? >I'd like to know this so that I can avoid names that could be deprecated in a >future. > > Yes, The problem is that the type-dictionary is really just a scalar type dictionary. It was created back when scalar-types were synonomous with data-types. When the data-types became a separate object, then the naming was wrong. -Travis From oliphant at ee.byu.edu Tue Sep 26 15:48:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:48:00 -0600 Subject: [Numpy-discussion] Default values in scalar constructors In-Reply-To: <200609261842.46548.faltet@carabos.com> References: <200609261842.46548.faltet@carabos.com> Message-ID: <451983F0.8050904@ee.byu.edu> Francesc Altet wrote: >Hi, > >Is there any reason why the next works for some types: > >In [26]: numpy.int32() >Out[26]: 0 > >In [27]: numpy.float64() >Out[27]: 0.0 > >but don't for the next others? > >In [28]: numpy.int16() >--------------------------------------------------------------------------- > Traceback (most recent call last) > >/home/faltet/python.nobackup/numpy/ in () > >: function takes exactly 1 argument (0 given) > > > I suppose because int() and float() work. I really didn't know that. If that is the case, then we should have all the scalar objects return default values. Please file a ticket. -Travis From oliphant at ee.byu.edu Tue Sep 26 15:49:28 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 26 Sep 2006 13:49:28 -0600 Subject: [Numpy-discussion] reload and f2py In-Reply-To: References: <1d1e6ea70609260710t7c7ed5cbpdf76e917317468b2@mail.gmail.com> Message-ID: <45198448.1010500@ee.byu.edu> Fernando Perez wrote: >On 9/26/06, George Nurser wrote: > > >>I'm running Python 2.3.5 with recent SVN f2py. >> >>Suppose I import an extension I have built with f2py. Then, if I edit >>the fortran and recompile the extension, I cannot use reload to use >>the modified version within the same Python session. >> >>I believe this is an problem with Python, that reload doesn't work >>with externally compiled extensions. >> >> > >As far as I know, you are correct. > > On many systems there is an "unload" command similar to a shared-library load command which Python uses to load shared libraries. But, this is not performed by Python. I've wondered for awhile if a suitable use of such a command in an extension module would allow "reloading" of shared libraries. -Travis From nvf at MIT.EDU Wed Sep 27 00:07:04 2006 From: nvf at MIT.EDU (Nickolas V Fotopoulos) Date: Wed, 27 Sep 2006 00:07:04 -0400 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 Message-ID: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Dear numpy pros, I have a problem and I don't know whether it's from local setup intricacies (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN checkout). I observe the following: >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 31 tests for numpy.core.numerictypes Found 32 tests for numpy.linalg Found 13 tests for numpy.core.umath Found 4 tests for numpy.core.scalarmath Found 9 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 166 tests for numpy.core.multiarray Found 3 tests for numpy.fft.helper Found 36 tests for numpy.core.ma Found 1 tests for numpy.lib.ufunclike Found 12 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 4 tests for numpy.ctypeslib Found 40 tests for numpy.lib.function_base Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ........Segmentation fault In gdb: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 46912496134400 (LWP 25798)] VOID_setitem (op=Variable "op" is not available. ) at numpy/core/src/arraytypes.inc.src:522 522 if (res < 0) break; Version information: [nvf at ldas-pcdev1 nvf]$ uname -m x86_64 [nvf at ldas-pcdev1 nvf]$ python Python 2.5 (r25:51908, Sep 26 2006, 19:06:29) [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> numpy.__version__ '1.0.dev3219' Any idea what might be wrong? One obvious thing to check was the linear algebra, since ACML is suspect. numpy.linalg.eig worked fine for my simple test. I don't know enough to debug further. Thanks, Nick From robert.kern at gmail.com Wed Sep 27 00:31:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 26 Sep 2006 23:31:13 -0500 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> References: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Message-ID: Nickolas V Fotopoulos wrote: > Dear numpy pros, > > I have a problem and I don't know whether it's from local setup intricacies > (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN > checkout). I observe the following: > >>>> import numpy >>>> numpy.test() > ........Segmentation fault Please run numpy.test(10,10). That will print out the name of the test before running it. Thus we will find out which test is segfaulting. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Sep 27 01:03:22 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 26 Sep 2006 23:03:22 -0600 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> References: <20060927000704.opfmd61rwpog08c8@webmail.mit.edu> Message-ID: On 9/26/06, Nickolas V Fotopoulos wrote: > > Dear numpy pros, > > I have a problem and I don't know whether it's from local setup > intricacies > (64-bit Opteron with ACML and Python 2.5) or something in numpy (fresh SVN > checkout). I observe the following: > > > > Any idea what might be wrong? One obvious thing to check was the linear > algebra, since ACML is suspect. numpy.linalg.eig worked fine for my > simple > test. I don't know enough to debug further. Is this your first build, or have you had success before? If the latter what changed? If you have done previous builds it is also a good idea to delete the build directory and, sometimes, the old package in site-packages whenever you have a problem to see if it goes away. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Sep 27 03:48:47 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 09:48:47 +0200 Subject: [Numpy-discussion] More about data types in NumPy Message-ID: <1159343327.3972.5.camel@localhost.localdomain> Hello, Sorry for being insistent, but I recognize that I'm having a bad time with NumPy data type rational. Is there an explanation for this?: >>> numpy.dtype('i4').type >>> numpy.dtype('int32').type >>> numpy.dtype('i4').type == numpy.dtype('int32').type True So far so good, but is the next the intended behaviour? >>> numpy.typeDict['i4'] >>> numpy.typeDict['int32'] >>> numpy.typeDict['i4'] == numpy.typeDict['int32'] False -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From zwqjdf at xnet.hr Wed Sep 27 08:47:42 2006 From: zwqjdf at xnet.hr (ofert) Date: Wed, 27 Sep 2006 10:47:42 -0200 Subject: [Numpy-discussion] Certified Message-ID: <000f01c6e211$9814cfe0$b84c8b53@fugi> expert help with search. rodziny Uniksa dziau spka. zarwno przygod jak bardziej szczegowe powicone wiata. trendy urzdzenia ktre zmieni nasza WiFi nowe standardy Przegld MP dyskiem twardym zobacz Ankieta Jakie zmiany stronie enter.pl uwaasz potrzebne Szata graficzna Nowe dziay adowania strony Wicej artykuw Mniej Nowe dziay adowania strony Wicej artykuw Mniej reklam Kursy online Zmiana nawigacji wyniki Najnowsze wtki forum: Konta nowymi MB poczt WWW spamu wirusw POP SMTP obsuga poczty przez extra Uwagi odnonie prosimy kierowa vbc.pl Copyright by Vogel Burda Sp. o.o. Wszystkie latest messages emailed to Alerts. Terms of Service Privacy Policy copy Magazyn ENTER Konkursy Ksiki Ankiety Redakcja Forum Kcik Reklama Sownik Wirusy BIOS praktyce Linux spka kiosk archiwum aktualny numer enter special pomoc Usugi sieci Rejestruj domen Personal GB Business Server Patronaty medialne wrzenia UOKiK: przeciwko eCard wyjani czy obcianie kosztami tylko wybranych zgodne prawem... wicej gtgt TP bdzie odwoywa Nowa strona Erachunek Plusie UKE: sprzeciw wobec Rozmw rabatem Privacy Policy copy Magazyn ENTER Konkursy Ksiki Ankiety Redakcja Forum Kcik Reklama Sownik Wirusy BIOS praktyce Linux spka kiosk archiwum aktualny numer enter special pomoc Usugi sieci Rejestruj domen Personal GB Business Server Patronaty medialne wrzenia UOKiK: przeciwko eCard wyjani czy obcianie kosztami tylko wybranych zgodne prawem... wicej gtgt TP bdzie odwoywa Nowa strona Erachunek Plusie UKE: sprzeciw wobec Rozmw rabatem tp Gemius SA Jeeli tworzysz swoj wasn lub prostu pragniesz zapozna tematyk pisania aplikacji zajrzyj naszego Kcika HTML oraz podrcznik PHP zbir porad wiele innych. standardy Przegld MP dyskiem twardym zobacz Ankieta Jakie zmiany stronie enter.pl uwaasz potrzebne Szata graficzna Nowe dziay adowania strony Wicej artykuw Mniej reklam Kursy online Zmiana nawigacji wyniki Najnowsze wtki forum: Konta nowymi MB poczt WWW spamu wirusw POP SMTP obsuga poczty przez Try different keywords. general fewer your on can try Answers for expert help with search. Get the -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: zobacz.gif Type: image/gif Size: 11394 bytes Desc: not available URL: From naomi at yahoo.co.jp Wed Sep 27 04:51:00 2006 From: naomi at yahoo.co.jp (=?iso-2022-jp?B?bmFvbWk=?=) Date: Wed, 27 Sep 2006 08:51:00 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/07 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/07 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/07 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/07 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/07 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?07 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/07 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From jg307 at cam.ac.uk Wed Sep 27 05:01:09 2006 From: jg307 at cam.ac.uk (James Graham) Date: Wed, 27 Sep 2006 10:01:09 +0100 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451A3DD5.1020608@cam.ac.uk> Francesc Altet wrote: > So far so good, but is the next the intended behaviour? > >>>> numpy.typeDict['i4'] > >>>> numpy.typeDict['int32'] > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] > False > > By way of comparison, I get (Linux/AMD 64): In [1]: import numpy In [2]: numpy.typeDict['i4'] == numpy.typeDict['int32'] Out[2]: True In [3]: numpy.typeDict['i4'] is numpy.typeDict['int32'] Out[3]: True In [4]: numpy.__version__ Out[4]: '1.0b5' -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead From faltet at carabos.com Wed Sep 27 07:22:37 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 13:22:37 +0200 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <451A3DD5.1020608@cam.ac.uk> References: <1159343327.3972.5.camel@localhost.localdomain> <451A3DD5.1020608@cam.ac.uk> Message-ID: <1159356157.3972.11.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 10:01 +0100, en/na James Graham va escriure: > Francesc Altet wrote: > > So far so good, but is the next the intended behaviour? > > > >>>> numpy.typeDict['i4'] > > > >>>> numpy.typeDict['int32'] > > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] > > False > > > > > By way of comparison, I get (Linux/AMD 64): > > In [1]: import numpy > > In [2]: numpy.typeDict['i4'] == numpy.typeDict['int32'] > Out[2]: True > > In [3]: numpy.typeDict['i4'] is numpy.typeDict['int32'] > Out[3]: True > > In [4]: numpy.__version__ > Out[4]: '1.0b5' Yeah, but I'm on Linux/AMD32. I'd bet that in your machine you will get something like: In [2]: numpy.typeDict['i8'] == numpy.typeDict['int64'] Out[2]: False -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From klemm at phys.ethz.ch Wed Sep 27 08:54:36 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Wed, 27 Sep 2006 14:54:36 +0200 Subject: [Numpy-discussion] numpy 1.0rc1 has problems finding blas In-Reply-To: References: , Message-ID: Thanks for the hint, that solved it! Hanno Charles R Harris said: > ------=_Part_15501_7322175.1159288578594 > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > Content-Transfer-Encoding: 7bit > Content-Disposition: inline > > On 9/26/06, Hanno Klemm wrote: > > > > > > Hello, > > > > I try to build numpy 1.0rc1 with blas and lapack installed in another > > place than /usr/lib. I compiled blas and lapack according to the > > instructions on the scipy webpage and installed them under > > /scratch/python2.4/lib. > > > > I set the environment variables > > BLAS=/scratch/python2.4/lib/libfblas.a and > > LAPACK=/scratch/python2.4/lib/libflapack.a > > > Try putting stuff in the site.cfg file. At one point I needed the following: > > [atlas] > library_dirs = /usr/lib64/atlas > atlas_libs = lapack, blas, cblas, atlas > > Chuck From nvf at MIT.EDU Wed Sep 27 12:33:10 2006 From: nvf at MIT.EDU (Nickolas V Fotopoulos) Date: Wed, 27 Sep 2006 12:33:10 -0400 Subject: [Numpy-discussion] Segfault with 64-bit, ACML, Python 2.5 In-Reply-To: References: Message-ID: <20060927123310.uqivdtr0izkkkowo@webmail.mit.edu> On Tuesday, September 26, Robert Kern wrote: > Please run numpy.test(10,10). That will print out the name of the test before > running it. Thus we will find out which test is segfaulting. Robert, The last line is: Check creation from list of list of tuplesSegmentation fault This matches what gdb said. I wouldn't think there any BLAS or LAPACK calls during array creation, so that would rule out ACML as the culprit. Any quick ideas of where specifically to look? Chuck, I had a rather different setup working. I changed too many variables (python 2.4, no ACML, pre-1.0beta SVN checkout to python 2.5, ACML, yesterday SVN checkout) to be sure what is breaking. I'll regress a bit and see if I can find out what's wrong. I always remove the build directory before reinstalling, but I generally don't remove the site-packages/numpy directory. I'll start doing that. I'm fairly sure I'm not picking up 2.4's site-packages. Oh, another thing is that I had trouble with the gnu (g77) version of the ACML libraries, so I forced gfortran usage with .pydistutils.cfg. Should that break things? Thanks, Nick From oliphant.travis at ieee.org Wed Sep 27 12:35:51 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 10:35:51 -0600 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451AA867.2030709@ieee.org> Francesc Altet wrote: > Hello, > > Sorry for being insistent, but I recognize that I'm having a bad time > with NumPy data type rational. Is there an explanation for this?: > > >>>> numpy.dtype('i4').type >>>> > > >>>> numpy.dtype('int32').type >>>> > > >>>> numpy.dtype('i4').type == numpy.dtype('int32').type >>>> > True > > So far so good, but is the next the intended behaviour? > > >>>> numpy.typeDict['i4'] >>>> > > >>>> numpy.typeDict['int32'] >>>> > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] >>>> No, this isn't correct behavior. This time you've caught an actual problem :-) The typeDict (actually the sctypeDict --- scalar-type-dictionary returns a scalar type given a string not a data-type object) is only used if no other conversion can be found for the object. It used to be much more useful before the data-type objects were formalized last year. -Travis From oliphant.travis at ieee.org Wed Sep 27 13:01:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 11:01:44 -0600 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <1159343327.3972.5.camel@localhost.localdomain> References: <1159343327.3972.5.camel@localhost.localdomain> Message-ID: <451AAE78.8080906@ieee.org> Francesc Altet wrote: > Hello, > > Sorry for being insistent, but I recognize that I'm having a bad time > with NumPy data type rational. Is there an explanation for this?: > > Your actually talking about the array scalar types not the data-type objects. But, more to the point.... >>>> numpy.typeDict['i4'] >>>> > > >>>> numpy.typeDict['int32'] >>>> > > >>>> numpy.typeDict['i4'] == numpy.typeDict['int32'] >>>> > False > > On my 32-bit system I get: numpy.sctypeDict['i4'] is numpy.sctypeDict['int32'] True Hmm..... I don't know why you are getting a different result, but perhaps it has to do with the fact that the character alias ('i4') is not getting set at the same type as 'int32'. I just fixed that. That should fix the problem. -travis From faltet at carabos.com Wed Sep 27 13:29:25 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed, 27 Sep 2006 19:29:25 +0200 Subject: [Numpy-discussion] More about data types in NumPy In-Reply-To: <451AAE78.8080906@ieee.org> References: <1159343327.3972.5.camel@localhost.localdomain> <451AAE78.8080906@ieee.org> Message-ID: <1159378165.3972.14.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 11:01 -0600, en/na Travis Oliphant va escriure: > Francesc Altet wrote: > > Hello, > > > > Sorry for being insistent, but I recognize that I'm having a bad time > > with NumPy data type rational. Is there an explanation for this?: > > > > > Your actually talking about the array scalar types not the data-type > objects. Yes, that's right. It's just that I didn't get used to the correct jargon :-( > On my 32-bit system I get: > > numpy.sctypeDict['i4'] is numpy.sctypeDict['int32'] > > True > > > Hmm..... I don't know why you are getting a different result, but > perhaps it has to do with the fact that the character alias ('i4') is > not getting set at the same type as 'int32'. I just fixed that. That > should fix the problem. Yup. I can confirm that the problem is solved in my machine as well. Thanks! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From haase at msg.ucsf.edu Wed Sep 27 19:32:16 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 27 Sep 2006 16:32:16 -0700 Subject: [Numpy-discussion] should I replace asarray with asanyarray in my code ? Message-ID: <200609271632.16689.haase@msg.ucsf.edu> Hi, This is a vaguely formulated question ... When I work with memmap'ed files/arrays I have a derived class that adds special attributes to the array class (referring to the MRC image file format used in medical / microscopy imaging) What are the pros and cons for asarray() vs. asanyarray() One obvious con for asanyarray is that its longer and asarray is what I have been using for the last few years ;-) Thanks, Sebastian From oliphant.travis at ieee.org Wed Sep 27 21:04:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 19:04:50 -0600 Subject: [Numpy-discussion] should I replace asarray with asanyarray in my code ? In-Reply-To: <200609271632.16689.haase@msg.ucsf.edu> References: <200609271632.16689.haase@msg.ucsf.edu> Message-ID: <451B1FB2.6030504@ieee.org> Sebastian Haase wrote: > Hi, > This is a vaguely formulated question ... > When I work with memmap'ed files/arrays I have a derived class > that adds special attributes to the array class (referring to the MRC image > file format used in medical / microscopy imaging) > > What are the pros and cons for asarray() vs. asanyarray() > > One obvious con for asanyarray is that its longer and asarray is what I have > been using for the last few years ;-) > asarray() guarantees you have a base-class array. Thus, you are not going to be thwarted by an re-definitions of infix operators, or other changed methods or attributes which you might use in your routine. asanyarray() allows a simple way of making sure your function returns any sub-class so that, for example, matrices are passed seamlessly through your function (matrix in and matrix out). However, a big drawback of asanyarray is that you must be sure that the way your function is written will not get confused by how a sub-class may overwrite the array methods and attributes. This significantly limits the application of asanyarray in my mind, as it is pretty difficult to predict what a sub-class *might* do to it's methods (including the special methods implementing the infix operators). A better way to write a function that passes any sub-class is to use asarray() so you are sure of the behavior of all methods and "infix" operators and then use the __array_wrap__ method of the actual input arguments (using __array_priority__ to choose between competing input objects). I expect that a decorator that automates this process will be added to NumPy eventually. Several examples have already been posted on this list. After getting the array result, you call the stored __array_wrap__ function which will take a base-class ndarray and return an object of the right Python-type (without copying data if possible). This is how the ufuncs work and why they can take sub-classes (actually anything with an __array__ method) and the same kind of object. -Travis From oliphant.travis at ieee.org Wed Sep 27 22:24:37 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 20:24:37 -0600 Subject: [Numpy-discussion] Question answered incorrectly at NumPy Tutorial Message-ID: <451B3265.8010902@ieee.org> During my NumPy Tutorial at the SciPy conference last month, somebody asked the question about the memory requirements of index arrays that I gave the wrong impression about. Here is the context and the correct response that should alleviate concerns about large cross-product index arrays. I was noting how copy-based (advanced) indexing using index arrays works in multiple-dimensions by creating an array of the same-shape of the input index arrays constructed by selecting the elements indicated by respective elements of the index arrays. If a is 2-d, then a[[10,12,14],[13, 15, 17]] returns a 1-d array with elements [a[10,13], a[12,15], a[14,17]]. This is *not* the cross-product that some would expect. The cross-product can be generated using the ix_ function a[ix_([10,12,14], [13,15,17])] is equivalent to a[[[10,10,10],[12,12,12],[14,14,14]], [[13,15,17],[13,15,17],[13,15,17]]] which will return [[a[10,13] a[10,15], a[10,17]], [a[12,13] a[12,15], a[12,17]], [a[14,13] a[14,15], a[14,17]]] The concern mentioned at the conference was that the cross-product would generate large intermediate index arrays for large input arrays to ix_. At the time, I think I validated the concern. However, the concern is unfounded. This is because the cross product function does not actually create a large intermediate array, but uses the broad-casting implementation of indexing to generate the 2-d indexing array "on-the-fly" (much like ogrid and other tools in NumPy). Notice: ix_([10,12,14], [13,15,17]) (array([[10], [12], [14]]), array([[13, 15, 17]])) The first indexing array is 3x1, while the second is 1x3. The result array will be 3x3, but the 2-d indexing array is never actually stored. Just to set my mind at ease about possible mis-information I spread during the tutorial, and give a little tutorial on advanced indexing. Best, -Travis From oliphant.travis at ieee.org Wed Sep 27 23:17:53 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 27 Sep 2006 21:17:53 -0600 Subject: [Numpy-discussion] Release candidate 2.0 will come out mid-week next week Message-ID: <451B3EE1.1060501@ieee.org> Hi all, I'd like to release numpy 1.0rc2 on about October 5 of next week. Then, the release of numpy 1.0 official should happen on Monday, October 17. Please try and get all fixes and improvements in before then. Backward-incompatible changes are not acceptable at this point (unless they are minor or actually bug-fixes). I think NumPy has been cooking long enough. Any remaining problems can be fixed with maintenance releases. When 1.0 comes out, we will make a 1.0 release branch where bug-fixes should go as well as on the main trunk (I'd love for a way to do that automatically). There are lots of projects that need to start converting to NumPy 1.0 if we are going to finally have a "merged" community. The actual release of NumPy 1.0 will indicate that we are now committing to stability. Thanks to all that have been contributing so much to the project. -Travis From mg.mailing-list at laposte.net Thu Sep 28 02:55:17 2006 From: mg.mailing-list at laposte.net (mg) Date: Thu, 28 Sep 2006 08:55:17 +0200 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? Message-ID: <451B71D5.9000906@laposte.net> Hello, I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine on Linux-x86 32 and 64bit platforms. Now, I try to compile the both distributions on WindowsXP with VisualStudio2003. No problem to compile Python-2.5, but i have some troubles with Numpy-1.0rc1 and I didn't find any help in the provided setup.py. So, does someone can tell me how to do it? Thanks, Mathieu. From oliphant.travis at ieee.org Thu Sep 28 03:11:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 01:11:04 -0600 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451B71D5.9000906@laposte.net> References: <451B71D5.9000906@laposte.net> Message-ID: <451B7588.4000309@ieee.org> mg wrote: > Hello, > > I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine > on Linux-x86 32 and 64bit platforms. > Now, I try to compile the both distributions on WindowsXP with > VisualStudio2003. No problem to compile Python-2.5, but i have some > troubles with Numpy-1.0rc1 and I didn't find any help in the provided > setup.py. So, does someone can tell me how to do it? > > I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). Tim Hochberg once used a microsoft compiler to compile a previous version of NumPy and some things had to be fixed to make it work. I'm not sure if some in-compatibilities have crept in since then or not. But, I'd sure like to resolve it if they have. So, please post what problems you are having. You may be the first person to try a microsoft compiler with Python 2.5 -Travis From olivetti at itc.it Thu Sep 28 04:18:18 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Thu, 28 Sep 2006 10:18:18 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy Message-ID: <451B854A.5070300@itc.it> Hi, I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and LAPACK from sources too and I need to tune compiler flags. Here is the question: which are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: OPTS = -funroll-all-loops -fno-f2c -O3 instead on scipy.org[0] it's suggested: OPTS = -O2 I assume that using -O3 and unrolling loops should have definitely better performance that just -O2 but I'm wondering if the speedy options can be a problem for current numpy release (numpy-1.0rc1). Is it safe to use '-funroll-all-loops -fno-f2c -O3'? Thanks in advance for answers, Emanuele [0]: http://scipy.org/Installing_SciPy P.S.: my GCC is v3.4.5 From olivetti at itc.it Thu Sep 28 04:20:48 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Thu, 28 Sep 2006 10:20:48 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy In-Reply-To: <451B854A.5070300@itc.it> References: <451B854A.5070300@itc.it> Message-ID: <451B85E0.7030706@itc.it> Ops, sorry for the misleding subject: I wrote ATLAS but I meant LAPACK :) Emanuele Olivetti wrote: > Hi, > I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and > LAPACK from sources too and I need to tune compiler flags. Here is the question: which > are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: > OPTS = -funroll-all-loops -fno-f2c -O3 > instead on scipy.org[0] it's suggested: > OPTS = -O2 ... From faltet at carabos.com Thu Sep 28 05:24:15 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 28 Sep 2006 11:24:15 +0200 Subject: [Numpy-discussion] Release candidate 2.0 will come out mid-week next week In-Reply-To: <451B3EE1.1060501@ieee.org> References: <451B3EE1.1060501@ieee.org> Message-ID: <1159435455.4238.12.camel@localhost.localdomain> El dc 27 de 09 del 2006 a les 21:17 -0600, en/na Travis Oliphant va escriure: > Hi all, > > I'd like to release numpy 1.0rc2 on about October 5 of next week. > Then, the release of numpy 1.0 official should happen on Monday, October > 17. Please try and get all fixes and improvements in before then. > Backward-incompatible changes are not acceptable at this point (unless > they are minor or actually bug-fixes). I think NumPy has been cooking > long enough. Any remaining problems can be fixed with maintenance > releases. When 1.0 comes out, we will make a 1.0 release branch where > bug-fixes should go as well as on the main trunk (I'd love for a way to > do that automatically). In order to reduce the overhead of commiting bug fixes in both trunk and the 1.0 branch, you may want to delay the making of the 1.0 branch as much as possible. Eventually, when you have to start changes that properly belongs to trunk, then it's time to create the branch, but meanwhile you can save yourself quite a few syncronization work. Anyway, it is my pleasure to help finding bugs for NumPy! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From david at ar.media.kyoto-u.ac.jp Thu Sep 28 05:44:21 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 28 Sep 2006 18:44:21 +0900 Subject: [Numpy-discussion] storage order and concatenate Message-ID: <451B9975.9020904@ar.media.kyoto-u.ac.jp> Hi, I noticed a behaviour which I found counter-intuitive at least when using concatenate. I have a function which takes a numpy array of rank 2 as input, let's say foo(in): a = N.randn((10, 2)) foo(a) To test a ctype implementation of foo against the python version, my test has something like X1 = N.linspace(-2, 2, 10)[:, N.newaxis] X2 = N.linspace(-1, 3, 10)[:, N.newaxis] a = N.concatenate(([X1, X2]), 1) which has Fortran storage (column major order), where as creating a as a = N.zeros((10, 2)) a[:,0] = N.linspace(-2, 2, 10) a[:,1] = N.linspace(-1, 3, 10) has C storage (row major order). What are the rules concerning storage with numpy ? I thought it was always C, except if stated explicitly. I can obviously understand why concatenate gives a Fortran order from an implementation point of view, but this looks kind of strange to me, David From isgpn at kaiserkraft.de Thu Sep 28 05:41:41 2006 From: isgpn at kaiserkraft.de (Neddie Cunningham) Date: Thu, 28 Sep 2006 12:41:41 +0300 Subject: [Numpy-discussion] streetcar Message-ID: <001601c6e2e3$1a471f32$e6e8f758@sqabka> Okay, so not quite as much as those people doing shows every night of the week, but it's still been really busy here. And where is their remembrance? The city becomes awash with media-obsessed types, and it's difficult not to get caught up in the hysteria. Certainly, yesterday I felt really quite fragile, and didn't get much done, except for a quick visit to the supermarket for supplies during the afternoon. I was blown away by some truely amazing singing, and some rather fine acting too. Some of their number are modestly decent musicians, and can actually hold a tune. To help track that proces. She's also quite a creative person, who likes using self-made drawing to illustrate her musings. : Drake Gold Resources Inc. I've been watching a bit of TV, reading a bit, and surfing the net for the rest of time. Even with the usual summer lull, there's still been a steady flow of decent jobs over the last few weeks, which indicates a fairly bouyant employment market. But the others obviously carried on without me, and were quite drunk by the time I met up with them again a few hours later. Luckily, I had arrived a good ten minutes early, and had a great spot right at the front. It felt a bit strange to have the bar open for normal business while filming was in progress, but also interesting for us to be in the midst of all this action. But the web-based review sites don't have that limitation, and so there seems little excuse for not publishing the promised reviews. Since then, I've perked up a bit; that's despite having received another job rejection letter. But even without an amorous advance in your direction, you'll leave with a big grin on your face. But even without an amorous advance in your direction, you'll leave with a big grin on your face. And so, by the time we were thrown out at closing time, I was pissed again. Sure, it'll be great to have a rest. : Drake Gold Resources Inc. She's also quite a creative person, who likes using self-made drawing to illustrate her musings. It's an obsessive practice that I think a lot of amateur Fringe performers get dragged into. In all, I've sung in a total of four different concerts and five different church services in the last month. This departing friend had organised his final farewell drinks in one of Edinburgh's most famous public houses. And that's certainly dented my confidence somewhat. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: persistently.gif Type: image/gif Size: 11893 bytes Desc: not available URL: From stefan at sun.ac.za Thu Sep 28 06:29:27 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 28 Sep 2006 12:29:27 +0200 Subject: [Numpy-discussion] return value of negative power Message-ID: <20060928102927.GA16637@mentat.za.net> Hi all, Currently, the power function returns '0' for negative powers of integers: In [1]: N.power(3,-2) Out[1]: 0 (or, more confusingly) In [1]: N.power(a,b) Out[1]: 0 which is almost certainly not the answer you want. Two possible solutions may be to upcast the input to float before calculation, or to return nan. This would be consistent with a function like sqrt: In [10]: N.sqrt(3) Out[10]: 1.7320508075688772 In [11]: N.sqrt(-3) Out[11]: nan Does anyone have an opinion on whether the change is necessary, and if so, on which one would work best? Regards St?fan From klemm at phys.ethz.ch Thu Sep 28 06:47:55 2006 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Thu, 28 Sep 2006 12:47:55 +0200 Subject: [Numpy-discussion] compiling ATLAS for numpy In-Reply-To: <451B85E0.7030706@itc.it> References: <451B854A.5070300@itc.it>, <451B854A.5070300@itc.it> Message-ID: Emanuele, the scipy compiler flags under http://www.scipy.org/Installing_SciPy/BuildingGeneral work well. However, if you happen to have to use gcc 3.2.3 (e.g. often in Redhat Enterprise editions present), you have to turn off optimization, otherwise lapack doesn't build properly. The correct build flags are then OPTS = -m64 -fPIC (at least that's what worked for me) Regards, Hanno Emanuele Olivetti said: > Ops, sorry for the misleding subject: I wrote ATLAS but I meant LAPACK :) > > Emanuele Olivetti wrote: > > Hi, > > I'm installing numpy on a 2 cpus intel pentium 4 Linux box. I'm installing BLAS and > > LAPACK from sources too and I need to tune compiler flags. Here is the question: which > > are the proper flags for compiling LAPACK? Inside lapack.tgz make.inc.LINUX says: > > OPTS = -funroll-all-loops -fno-f2c -O3 > > instead on scipy.org[0] it's suggested: > > OPTS = -O2 > ... > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Hanno Klemm klemm at phys.ethz.ch From oliphant.travis at ieee.org Thu Sep 28 07:15:47 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 05:15:47 -0600 Subject: [Numpy-discussion] storage order and concatenate In-Reply-To: <451B9975.9020904@ar.media.kyoto-u.ac.jp> References: <451B9975.9020904@ar.media.kyoto-u.ac.jp> Message-ID: <451BAEE3.8030809@ieee.org> David Cournapeau wrote: > Hi, > > > What are the rules concerning storage with numpy ? The rule is that a numpy array has "strides" which specify how many bytes to skip to get to the next element in the array. That's the internal model. There are no hard and fast rules about storage order. Internally, C-order is as good as Fortran-order (except the array iterator gives special preference to C-order and all functions for which the order can be specified (like zeros) default to C-order). Thus, the storage order is whatever the strides say it is. Now, there are flags that keep track of whether or not the strides agree with the 2 recognized special cases of "Fortran-order" (first-index varies the fastest) or "C-order" (last-index varies the fastest). But, this is only for convenience. Very few functions actually require a specification of storage order. Those that allow it default to "C-order". You can't think of a NumPy array has having a particular storage order unless you explicitly request it. One of the most common ways that Fortran-order arrays show up, for example is when a C-order array is transposed. A transpose operation does nothing except flip the strides (and therefore the flags) of the array. This is what is happening in concatenate (using axis=1) to give you a Fortran-order array. Bascially, code equivalent to the following is being run: concatenate([X1.T, X2.T]).T In the second example, you explicitly create the array (and therefore the strides) as C-order and then fill it (so it doesn't change on you). The first example used array calculations which don't guarantee the storage order. This is all seamless to the user until you have to interface with extension code. Ideally, you write compiled code that deals with strided arrays. If you can't, then you request an array of the required storage-order. By the way, for interfacing with ctypes, check out the ctypeslib.ndpointer class-creation function for flag checking and the require function for automatic conversion of an array to specific requirements. -Travis From eric at enthought.com Thu Sep 28 08:05:21 2006 From: eric at enthought.com (eric) Date: Thu, 28 Sep 2006 07:05:21 -0500 Subject: [Numpy-discussion] lexsort segfault sorting strings Message-ID: <451BBA81.5050904@enthought.com> I've been using the new record arrays and lexsort from numpy quite a lot lately. Very cool stuff. Using the nightly egg for numpy from here (I believe it is up to date...): http://code.enthought.com/enstaller/eggs/numpy-nightly-py2.4-win32.egg I get segfaults when using lexsort on character arrays. A lot of my columns in record arrays are string based, so sorting the arrays based on these columns would be really handy. Here is an example that crashes for me. C:\wrk\mt\trunk\src\lib\mt\statement\tests>python Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug 2 2006, 12:09:59) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import lexsort >>> lst = [1,2,3] >>> lexsort((lst,)) array([0, 1, 2]) >>> lst = ['abc','cde','fgh'] >>> lexsort((lst,)) Do others see this? I've opened a ticket at: http://projects.scipy.org/scipy/numpy/ticket/298 thanks, eric From tim.hochberg at ieee.org Thu Sep 28 09:26:33 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 06:26:33 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451B7588.4000309@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> Message-ID: <451BCD89.9000902@ieee.org> Travis Oliphant wrote: > mg wrote: > >> Hello, >> >> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine >> on Linux-x86 32 and 64bit platforms. >> Now, I try to compile the both distributions on WindowsXP with >> VisualStudio2003. No problem to compile Python-2.5, but i have some >> troubles with Numpy-1.0rc1 and I didn't find any help in the provided >> setup.py. So, does someone can tell me how to do it? >> >> >> > I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). > Tim Hochberg once used a microsoft compiler to compile a previous > version of NumPy and some things had to be fixed to make it work. I'm > not sure if some in-compatibilities have crept in since then or not. > But, I'd sure like to resolve it if they have. > > So, please post what problems you are having. You may be the first > person to try a microsoft compiler with Python 2.5 > It was VS2003 that I used to compile numpy. However, I switched boxes a while ago and I haven't got around to trying to compile on the new box, I've just been using the released builds. So I'm not much help at the moment. Things are clearing up a little bit here, so maybe I can carve some time out to get things set up to compile in the next couple days. If so I'll let you know what I find. -tim > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From mg.mailing-list at laposte.net Thu Sep 28 10:40:54 2006 From: mg.mailing-list at laposte.net (mg) Date: Thu, 28 Sep 2006 16:40:54 +0200 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BCD89.9000902@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> Message-ID: <451BDEF6.7020307@laposte.net> Unfortunately, no Windows-x86 or Windows-x86-64bit Numpy-1.0rc1 installer are available on SourceForge yet. So the only current solution for us is to compile it. Moreover, our generic C++ framework is compiled with VisualStudio on Windows-native and we compile all additions to it with the same compiler, if compilation is needed. Indeed, we have observed that Visual C++ is still the closest to a "default" or "native" compiler for windows platform, like gcc is for linux....well at least that's our experience when using other commercial (or open-source, think python whose provide visual project files) softwares requiring rebuilding/relinking. Use MinGW for Numpy means risk some un-compatibilities between Numpy and our framework (and even Python) or migrate all our development environment from Visual Studio to MinGW. this is not really an option for the near or mean furture.... If some of you have good experiences for linking (static or dynamic) libraries compiled with mixed compilers (especially mingw and visual), knowing that our libraries contains C++, C and fortran code, we could consider this as a temprary option for numpy, but for convenience we would ultimately prefer to use only 1 compiler to avoid a double maintenance of building tools.... Then, I wonder if the compilation of Numpy with Visual-Studio-2003 (or 2005) is scheduled ? For your information, this is the standard output and the standard error of the compilation of Numpy-1.0rc1 with Visual Studio 2003: >>> command line >>> python setup.py build 1>stdout.txt 2>stderr.txt >>> stdout.txt >>> F2PY Version 2_3198 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in c:\Program Files\python\dist\src\lib libraries mkl,vml,guide not found in C:\ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in c:\Program Files\python\dist\src\lib libraries ptf77blas,ptcblas,atlas not found in C:\ NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in c:\Program Files\python\dist\src\lib libraries f77blas,cblas,atlas not found in C:\ NOT AVAILABLE blas_info: libraries blas not found in c:\Program Files\python\dist\src\lib libraries blas not found in C:\ NOT AVAILABLE blas_src_info: NOT AVAILABLE NOT AVAILABLE lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in c:\Program Files\python\dist\src\lib libraries mkl,vml,guide not found in C:\ NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in c:\Program Files\python\dist\src\lib libraries lapack_atlas not found in c:\Program Files\python\dist\src\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in c:\Program Files\python\dist\src\lib libraries lapack_atlas not found in c:\Program Files\python\dist\src\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: libraries lapack not found in c:\Program Files\python\dist\src\lib libraries lapack not found in C:\ NOT AVAILABLE lapack_src_info: NOT AVAILABLE NOT AVAILABLE running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources Generating build\src.win32-2.5\numpy\core\config.h No module named msvccompiler in numpy.distutils, trying from distutils.. 0 Could not locate executable g77 Could not locate executable f77 Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelVisualFCompiler Could not locate executable ifl Executable ifl does not exist customize AbsoftFCompiler Could not locate executable f90 Executable f90 does not exist customize CompaqVisualFCompiler Could not locate executable DF Executable DF does not exist customize IntelItaniumVisualFCompiler Could not locate executable efl Executable efl does not exist customize Gnu95FCompiler Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist customize G95FCompiler Could not locate executable g95 Executable g95 does not exist customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist customize Gnu95FCompiler Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist Could not locate executable f95 Executable f95 does not exist customize GnuFCompiler Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist Could not locate executable f77 Executable f77 does not exist customize GnuFCompiler using config C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ic:\Program Files\python\dist\src\include -Inumpy\core\src -Inumpy\core\include -Ic:\Program Files\python\dist\src\include -Ic:\Program Files\python\dist\src\PC /Tc_configtest.c /Fo_configtest.obj C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /nologo /INCREMENTAL:NO /LIBPATH:c:\Program Files\python\dist\src\lib /LIBPATH:C:\ _configtest.obj /OUT:_configtest.exe LINK : fatal error LNK1104: cannot open file 'python25.lib' failure. removing: _configtest.c _configtest.obj >>> stderr.txt >>> Running from numpy source directory. C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1296: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1305: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1308: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1205: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1216: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. warnings.warn(LapackNotFoundError.__doc__) C:\Program Files\numpy-1.0rc1\numpy\distutils\system_info.py:1219: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. warnings.warn(LapackSrcNotFoundError.__doc__) Traceback (most recent call last): File "setup.py", line 89, in ? setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\core.py", line 174, in setup return old_setup(**new_attr) File "c:\Program Files\python\dist\src\lib\distutils\core.py", line 149, in setup dist.run_commands() File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 955, in run_commands self.run_command(cmd) File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 975, in run_command cmd_obj.run() File "c:\Program Files\python\dist\src\lib\distutils\command\build.py", line 112, in run self.run_command(cmd_name) File "c:\Program Files\python\dist\src\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "c:\Program Files\python\dist\src\lib\distutils\dist.py", line 975, in run_command cmd_obj.run() File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 87, in run self.build_sources() File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "C:\Program Files\numpy-1.0rc1\numpy\distutils\command\build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 50, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration Thanks a lot for your help. Tim Hochberg wrote: > Travis Oliphant wrote: > >> mg wrote: >> >> >>> Hello, >>> >>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work fine >>> on Linux-x86 32 and 64bit platforms. >>> Now, I try to compile the both distributions on WindowsXP with >>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>> troubles with Numpy-1.0rc1 and I didn't find any help in the provided >>> setup.py. So, does someone can tell me how to do it? >>> >>> >>> >>> >> I don't use VisualStudio2003 on Windows to compile NumPy (I use mingw). >> Tim Hochberg once used a microsoft compiler to compile a previous >> version of NumPy and some things had to be fixed to make it work. I'm >> not sure if some in-compatibilities have crept in since then or not. >> But, I'd sure like to resolve it if they have. >> >> So, please post what problems you are having. You may be the first >> person to try a microsoft compiler with Python 2.5 >> >> > It was VS2003 that I used to compile numpy. However, I switched boxes a > while ago and I haven't got around to trying to compile on the new box, > I've just been using the released builds. So I'm not much help at the > moment. Things are clearing up a little bit here, so maybe I can carve > some time out to get things set up to compile in the next couple days. > If so I'll let you know what I find. > > -tim > > >> -Travis >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From tim.hochberg at ieee.org Thu Sep 28 11:07:43 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 08:07:43 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BCD89.9000902@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> Message-ID: <451BE53F.2060904@ieee.org> Tim Hochberg wrote: > Travis Oliphant wrote: >> mg wrote: >> >>> Hello, >>> >>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work >>> fine on Linux-x86 32 and 64bit platforms. >>> Now, I try to compile the both distributions on WindowsXP with >>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>> troubles with Numpy-1.0rc1 and I didn't find any help in the >>> provided setup.py. So, does someone can tell me how to do it? >>> >>> >> I don't use VisualStudio2003 on Windows to compile NumPy (I use >> mingw). Tim Hochberg once used a microsoft compiler to compile a >> previous version of NumPy and some things had to be fixed to make it >> work. I'm not sure if some in-compatibilities have crept in since >> then or not. But, I'd sure like to resolve it if they have. >> >> So, please post what problems you are having. You may be the first >> person to try a microsoft compiler with Python 2.5 >> > It was VS2003 that I used to compile numpy. However, I switched boxes > a while ago and I haven't got around to trying to compile on the new > box, I've just been using the released builds. So I'm not much help at > the moment. Things are clearing up a little bit here, so maybe I can > carve some time out to get things set up to compile in the next couple > days. If so I'll let you know what I find. FWIW, I just got svn head to compile cleanly against Python *2.4* with VS2003. Later today I will try compiling against 2.5 -tim From tim.hochberg at ieee.org Thu Sep 28 12:32:08 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 09:32:08 -0700 Subject: [Numpy-discussion] how to compile numpy with visual-studio-2003? In-Reply-To: <451BE53F.2060904@ieee.org> References: <451B71D5.9000906@laposte.net> <451B7588.4000309@ieee.org> <451BCD89.9000902@ieee.org> <451BE53F.2060904@ieee.org> Message-ID: <451BF908.8080902@ieee.org> Tim Hochberg wrote: > Tim Hochberg wrote: > >> Travis Oliphant wrote: >> >>> mg wrote: >>> >>> >>>> Hello, >>>> >>>> I just download the newly Python-2.5 and Numpy-1.0rc1 and all work >>>> fine on Linux-x86 32 and 64bit platforms. >>>> Now, I try to compile the both distributions on WindowsXP with >>>> VisualStudio2003. No problem to compile Python-2.5, but i have some >>>> troubles with Numpy-1.0rc1 and I didn't find any help in the >>>> provided setup.py. So, does someone can tell me how to do it? >>>> >>>> >>>> >>> I don't use VisualStudio2003 on Windows to compile NumPy (I use >>> mingw). Tim Hochberg once used a microsoft compiler to compile a >>> previous version of NumPy and some things had to be fixed to make it >>> work. I'm not sure if some in-compatibilities have crept in since >>> then or not. But, I'd sure like to resolve it if they have. >>> >>> So, please post what problems you are having. You may be the first >>> person to try a microsoft compiler with Python 2.5 >>> >>> >> It was VS2003 that I used to compile numpy. However, I switched boxes >> a while ago and I haven't got around to trying to compile on the new >> box, I've just been using the released builds. So I'm not much help at >> the moment. Things are clearing up a little bit here, so maybe I can >> carve some time out to get things set up to compile in the next couple >> days. If so I'll let you know what I find. >> > FWIW, I just got svn head to compile cleanly against Python *2.4* with > VS2003. Later today I will try compiling against 2.5 > > OK. SVN head compiles out of the box and passes all tests for me under both Python2.4 and Python2.5. My stderr output includes all of the same warnings about not being able to find ATLAS, BLAS and LAPACK that mg's does, but not "ERROR: Failed to test configuration". The stdout output also looks similar except for the error at the end. (I can send you the entire stderr/stdout output privately if you like). One thing I did notice is that your Python appears to be in a nonstandard location: \Program Files\python\dist\src instead of the standard \Python25, or at least that's where it's looking for things. Perhaps this is confusing either distutils or some of numpy's addons to distutils and causing it to misplace python25.lib. At least that's my best (and only) guess at the moment. -tim From wfspotz at sandia.gov Thu Sep 28 12:43:26 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 28 Sep 2006 10:43:26 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints Message-ID: I am wrapping code using swig and extending it to use numpy. One class method I wrap (let's call it myElements()) returns an array of ints, and I convert it to a numpy array with PyArray_SimpleNew(1,n,'i'); I obtain the data pointer, fill in the values and return it as the method return argument. In python, it is common to want to loop over this array and treat its elements as integers: for row in map.myElements(): matrix.setElements(row, [row-1,row,row+1], [-1.0,2.0,-1.0]) On a 32-bit machine, this has worked fine, but on a 64-bit machine, I get a type error: TypeError: in method 'setElements', argument 2 of type 'int' because row is a . It would be nice if I could get the integer conversion to work automatically under the covers, but I'm not exactly sure how to make that work. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From tim.hochberg at ieee.org Thu Sep 28 13:08:01 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 28 Sep 2006 10:08:01 -0700 Subject: [Numpy-discussion] return value of negative power In-Reply-To: <20060928102927.GA16637@mentat.za.net> References: <20060928102927.GA16637@mentat.za.net> Message-ID: <451C0171.2070406@ieee.org> Stefan van der Walt wrote: > Hi all, > > Currently, the power function returns '0' for negative powers of > integers: > > In [1]: N.power(3,-2) > Out[1]: 0 > > (or, more confusingly) > > In [1]: N.power(a,b) > Out[1]: 0 > > which is almost certainly not the answer you want. Two possible > solutions may be to upcast the input to float before calculation, or > to return nan. > Returning nan seems silly. There really is a result, or rather two possible results on a tad more sensible than the other. If we were going to punt it makes more sense to raise an exception, but I doubt that's necessary. In addition, nan is really a floating point value; if we're going to return a floating point value, not an integer, we might as well return the actual result. > This would be consistent with a function like sqrt: > > In [10]: N.sqrt(3) > Out[10]: 1.7320508075688772 > > In [11]: N.sqrt(-3) > Out[11]: nan > > Does anyone have an opinion on whether the change is necessary, and if > so, on which one would work best? > This is a complicated tangle of worms. First off there's both "a**b" and "power(a, b)". These don't necessarily need to work the same. In fact they already differ somewhat in that a**b does some optimization when b is a small scalar that power does not (I think anyway -- haven't looked at this a while). However, if having them differ in any significant way is likely to be quite confusing, so it should be only be considered if there's some compelling reason to support multiple behaviors here. There is a solution that is, IMO, simple obvious and not quite right. That is to return floats if the b contains any negative numbers while returning integers otherwise. That sounds appealing at first, but will cause confusion and memory blow ups when one suddenly has all of ones arrays become floats because somewhere or other a negative value crept into an exponent. It's fine for the return type to depend on the types of the arguments, it's not so good for it to depends on the values. This restriction gets a little weaker once future division arrives since ints and floats will be closer to indistinguishable, but it still has some force. On the other hand, this is consistent with the rest of Python's behavior. Another path is to just always return floats from power. One down side of this is that we lose the ability to do true integer powers, which is sometimes useful. A second downside is that it introduces a inconsistency with Python's scalar 'x**y'. One way to finesse both of these issues is to make numpy.power consistent with math.pow; that is, it returns floating point values when passed integers. At the same time, make a**b consistent with the python's '**' operator in that any negative exponents trigger a floating point return values. This isn't perfect, but it's as close to a self consistent solution as I can think of. That's my two cents. -tim From oliphant.travis at ieee.org Thu Sep 28 14:03:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 28 Sep 2006 12:03:56 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: References: Message-ID: <451C0E8C.1040803@ieee.org> Bill Spotz wrote: > I am wrapping code using swig and extending it to use numpy. > > One class method I wrap (let's call it myElements()) returns an array > of ints, and I convert it to a numpy array with > > PyArray_SimpleNew(1,n,'i'); > You should probably use NPY_INT instead of 'i' for the type-code. > I obtain the data pointer, fill in the values and return it as the > method return argument. > > In python, it is common to want to loop over this array and treat its > elements as integers: > > for row in map.myElements(): > matrix.setElements(row, [row-1,row,row+1], [-1.0,2.0,-1.0]) > > On a 32-bit machine, this has worked fine, but on a 64-bit machine, I > get a type error: > > TypeError: in method 'setElements', argument 2 of type 'int' > > because row is a . > > It would be nice if I could get the integer conversion to work > automatically under the covers, but I'm not exactly sure how to make > that work. > Yeah, It can be confusing, at first. You just have to make sure you are matching the right c-data-types. I'm not quite sure what the problem here is given your description, because I don't know what setElements expects. My best guess, is that it is related to the fact that a Python int uses the 'long' c-type. Thus, you should very likely be using PyArray_SimpleNew(1, n, NPY_LONG) instead of int so that your integer array always matches what Python is using as integers. The other option is to improve your converter in setElements so that it can understand any of the array scalar integers and not just the default Python integer. The reason this all worked on 32-bit systems is probably the array scalar corresponding to NPY_INT is a sub-class of the Python integer. It can't be on a 64-bit platform because of binary incompatibility of the layout. Hope that helps. -Travis From wfspotz at sandia.gov Thu Sep 28 20:12:00 2006 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 28 Sep 2006 18:12:00 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: <451C0E8C.1040803@ieee.org> References: <451C0E8C.1040803@ieee.org> Message-ID: <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> On Sep 28, 2006, at 12:03 PM, Travis Oliphant wrote: > The other option is to improve your converter in setElements so > that it > can understand any of the array scalar integers and not just the > default > Python integer. I think this may be the best approach. This may be something worthwhile to put in the numpy.i interface file: a set of typemaps that handle a set of basic conversions for those array scalar types for which it makes sense. I'll look into it. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From oliphant at ee.byu.edu Thu Sep 28 21:22:24 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 28 Sep 2006 19:22:24 -0600 Subject: [Numpy-discussion] 32/64-bit machines, integer arrays and python ints In-Reply-To: <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> References: <451C0E8C.1040803@ieee.org> <649A8AEF-19AE-4191-8708-8CE54E21EA16@sandia.gov> Message-ID: <451C7550.4080009@ee.byu.edu> Bill Spotz wrote: >On Sep 28, 2006, at 12:03 PM, Travis Oliphant wrote: > > > >>The other option is to improve your converter in setElements so >>that it >>can understand any of the array scalar integers and not just the >>default >>Python integer. >> >> > >I think this may be the best approach. > >This may be something worthwhile to put in the numpy.i interface >file: a set of typemaps that handle a set of basic conversions for >those array scalar types for which it makes sense. I'll look into it. > > That's a good idea. Notice that there are some routines for making your life easier here. You should look at the tp_int function for the gentype array (it converts scalars to arrays). You call the "__int__" special method of the scalar to convert it to a Python integer. You should first check to see that it is an integer scalar PyArray_IsScalar(obj, Integer) because the "__int__" method coerces to an integer if it is a float (but maybe you want that behavior). There are other functions in the C-API that return the data directly from the scalar --- check them out. The macros in arrayscalar.h are useful. -Travis From david at ar.media.kyoto-u.ac.jp Fri Sep 29 03:36:54 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 29 Sep 2006 16:36:54 +0900 Subject: [Numpy-discussion] storage order and concatenate In-Reply-To: <451BAEE3.8030809@ieee.org> References: <451B9975.9020904@ar.media.kyoto-u.ac.jp> <451BAEE3.8030809@ieee.org> Message-ID: <451CCD16.7050708@ar.media.kyoto-u.ac.jp> Travis Oliphant wrote: > David Cournapeau wrote: > >> Hi, >> >> >> What are the rules concerning storage with numpy ? >> > The rule is that a numpy array has "strides" which specify how many > bytes to skip to get to the next element in the array. That's the > internal model. There are no hard and fast rules about storage order. > Internally, C-order is as good as Fortran-order (except the array > iterator gives special preference to C-order and all functions for which > the order can be specified (like zeros) default to C-order). > Ok, this is again a bad habit from matlab to think in C or F order... > Thus, the storage order is whatever the strides say it is. Now, there > are flags that keep track of whether or not the strides agree with the 2 > recognized special cases of "Fortran-order" (first-index varies the > fastest) or "C-order" (last-index varies the fastest). But, this is > only for convenience. Very few functions actually require a > specification of storage order. Those that allow it default to "C-order". > > You can't think of a NumPy array has having a particular storage order > unless you explicitly request it. One of the most common ways that > Fortran-order arrays show up, for example is when a C-order array is > transposed. A transpose operation does nothing except flip the strides > (and therefore the flags) of the array. This is what is happening in > concatenate (using axis=1) to give you a Fortran-order array. > Bascially, code equivalent to the following is being run: > concatenate([X1.T, X2.T]).T > > In the second example, you explicitly create the array (and therefore > the strides) as C-order and then fill it (so it doesn't change on you). > The first example used array calculations which don't guarantee the > storage order. > > This is all seamless to the user until you have to interface with > extension code. Ideally, you write compiled code that deals with > strided arrays. If you can't, then you request an array of the required > storage-order. > > By the way, for interfacing with ctypes, check out the > ctypeslib.ndpointer class-creation function for flag checking and the > require function for automatic conversion of an array to specific > requirements. > > I tried to to that at first, but I couldn't make the examples of numpy works; after having tried at home with beta versions, it looks like it is a ctype version problem, as it works with ctype 1.0 + numpy 1.0rc1. Thanks for the explanations, this answers most questions I had in mind for numpy internal layout compared to matlab which I am used to. I think this should be in the wiki somewhere; do you mind if I use your email as a basis for the tentative numpy tutorial (memory layout section, maybe) ? David From faltet at carabos.com Fri Sep 29 07:44:34 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 13:44:34 +0200 Subject: [Numpy-discussion] Negative values with unsigned data types problems Message-ID: <200609291344.35863.faltet@carabos.com> Hi, I'm writing this here because the numpy Trac seems down: {{{ Oops... Trac detected an internal error: The Trac Environment needs to be upgraded. Run trac-admin /home/scipy/trac/numpy upgrade" }}} Well, it seems that there are inconsistencies on creating arrays coming from negative values using unsigned integers: In [41]: numpy.asarray([-1], dtype='uint8') Out[41]: array([255], dtype=uint8) In [42]: numpy.asarray([-1], dtype='uint16') Out[42]: array([65535], dtype=uint16) until here it's fine, but: In [43]: numpy.asarray([-1], dtype='uint32') --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () /usr/local/lib/python2.5/site-packages/numpy/core/numeric.py in asarray(a, dtype, order) 129 are converted to base class ndarray. 130 """ --> 131 return array(a, dtype, copy=False, order=order) 132 133 def asanyarray(a, dtype=None, order=None): : long() argument must be a string or a number, not 'list' and the same happens with 'uint64'. My numpy version is 1.0.dev3216 Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Sep 29 11:19:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 17:19:01 +0200 Subject: [Numpy-discussion] Incorrect removal of NULL char in buffers Message-ID: <200609291719.02380.faltet@carabos.com> Hi, I'm trying to build-up numpy arrays coming from buffers, and I'm getting a somewhat unexpected result. First, for numeric values, everything seems ok (i.e. the NULL character is correctly interpretated), and works equally for both numarray and numpy: In [98]: numarray.array("a\x00b"*4, dtype='Float32',shape=3) Out[98]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], type=Float32) In [99]: numpy.ndarray(buffer="a\x00b"*4, dtype='Float32',shape=3) Out[99]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) However, for string values, numpy seems to work in a strange way. The numarray have an expected behaviour, IMO: In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) Out[100]: CharArray(['a', '', 'ba']) but numpy haven't: In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[101]: array([aba, ba, bab], dtype='|S4') i.e. it seems like numpy is striping-off NULL chars before building the object and I don't think this is correct. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Sep 29 11:23:26 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 17:23:26 +0200 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray Message-ID: <200609291723.26896.faltet@carabos.com> Hello, Is the next a bug a feature? In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) In [103]: f4 Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) In [104]: f4[2] = 2 --------------------------------------------------------------------------- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : array is not writeable In [105]: f4.flags.writeable = True In [106]: f4[2] = 2 In [107]: f4 Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.00000000e+00], dtype=float32) i.e. in an array built from ndarray, the default is that it has to be read-only? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tim.hochberg at ieee.org Fri Sep 29 12:12:09 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 29 Sep 2006 09:12:09 -0700 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <200609291723.26896.faltet@carabos.com> References: <200609291723.26896.faltet@carabos.com> Message-ID: <451D45D9.6050005@ieee.org> Francesc Altet wrote: > Hello, > > Is the next a bug a feature? > > In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) > > In [103]: f4 > Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], > dtype=float32) > > In [104]: f4[2] = 2 > --------------------------------------------------------------------------- > Traceback (most recent call last) > > /home/faltet/python.nobackup/numpy/ in () > > : array is not writeable > > In [105]: f4.flags.writeable = True > > In [106]: f4[2] = 2 > > In [107]: f4 > Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.00000000e+00], > dtype=float32) > > > i.e. in an array built from ndarray, the default is that it has to be > read-only? > It's not that the it's being built from ndarray, it's that the buffer that you are passing it is read only. In fact, I'd argue that allowing the writeable flag to be set to True in this case is actually a bug. Consider this slightly modified example: >>> a = "12345"*4 >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) >>> f4[2] = 99 Traceback (most recent call last): File "", line 1, in ? RuntimeError: array is not writeable >>> f4.flags.writeable = True >>> a '12345123451234512345' >>> f4.flags.writeable = True >>> f4[2] = 99 >>> a '12345123\x00\x00\xc6B34512345' The original, *immutable* string has been mutated. This could get you into real trouble in certain situations. -tim From faltet at carabos.com Fri Sep 29 12:44:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri, 29 Sep 2006 18:44:06 +0200 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D45D9.6050005@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> Message-ID: <200609291844.07928.faltet@carabos.com> A Divendres 29 Setembre 2006 18:12, Tim Hochberg va escriure: > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > > Consider this slightly modified example: > >>> a = "12345"*4 > >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) > >>> f4[2] = 99 > > Traceback (most recent call last): > File "", line 1, in ? > RuntimeError: array is not writeable > > >>> f4.flags.writeable = True > >>> a > > '12345123451234512345' > > >>> f4.flags.writeable = True > >>> f4[2] = 99 > >>> a > > '12345123\x00\x00\xc6B34512345' > > The original, *immutable* string has been mutated. This could get you > into real trouble in certain situations. I see. Thanks for the explanation. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Fri Sep 29 12:55:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 29 Sep 2006 10:55:25 -0600 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D45D9.6050005@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> Message-ID: <451D4FFD.6090202@ieee.org> Tim Hochberg wrote: > Francesc Altet wrote: > > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. This is correct. > In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > It's actually intentional. Strings used as buffers are allowed to be writeable. This is an explicit design decision to allow pickles to load without making 2 copies of the memory. The pickled string that Python creates is used as the actual memory for loaded arrays. Now, I suppose it would be possible to still allow this but be more finnicky about when a string-used-as-the-memory can be set writeable (i.e. we are the only reference to it). But, this would be a fragile solution as well. My inclination is to just warn users not to use strings as buffers unless they know what they are doing. The fact that it is read-only by default is enough of a guard against "accidentally" altering a string you didn't want to alter. -Travis From tim.hochberg at ieee.org Fri Sep 29 12:59:06 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 29 Sep 2006 09:59:06 -0700 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <451D4FFD.6090202@ieee.org> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> <451D4FFD.6090202@ieee.org> Message-ID: <451D50DA.5030901@ieee.org> Travis Oliphant wrote: > Tim Hochberg wrote: > >> Francesc Altet wrote: >> >> It's not that the it's being built from ndarray, it's that the buffer >> that you are passing it is read only. >> > This is correct. > >> In fact, I'd argue that allowing >> the writeable flag to be set to True in this case is actually a bug. >> >> > It's actually intentional. Strings used as buffers are allowed to be > writeable. This is an explicit design decision to allow pickles to load > without making 2 copies of the memory. The pickled string that Python > creates is used as the actual memory for loaded arrays. > > Now, I suppose it would be possible to still allow this but be more > finnicky about when a string-used-as-the-memory can be set writeable > (i.e. we are the only reference to it). But, this would be a fragile > solution as well. > > My inclination is to just warn users not to use strings as buffers > unless they know what they are doing. The fact that it is read-only by > default is enough of a guard against "accidentally" altering a string > you didn't want to alter. > That makes sense. Someday, when string/unicode => bytes/string, we'll be able to use the bytes type for this and it'll be much cleaner. But not for a while I imagine. -tim From robert.kern at gmail.com Fri Sep 29 13:26:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Sep 2006 12:26:35 -0500 Subject: [Numpy-discussion] Negative values with unsigned data types problems In-Reply-To: <200609291344.35863.faltet@carabos.com> References: <200609291344.35863.faltet@carabos.com> Message-ID: Francesc Altet wrote: > Hi, > > I'm writing this here because the numpy Trac seems down: > > {{{ > Oops... > > Trac detected an internal error: The Trac Environment needs to be upgraded. > Run trac-admin /home/scipy/trac/numpy upgrade" > }}} It's back up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Fri Sep 29 14:19:27 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 29 Sep 2006 12:19:27 -0600 Subject: [Numpy-discussion] Incorrect removal of NULL char in buffers In-Reply-To: <200609291719.02380.faltet@carabos.com> References: <200609291719.02380.faltet@carabos.com> Message-ID: <451D63AF.2070106@ieee.org> Francesc Altet wrote: > Hi, > > > However, for string values, numpy seems to work in a strange way. > The numarray have an expected behaviour, IMO: > > In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) > Out[100]: CharArray(['a', '', 'ba']) > > I'm not sure why you think this is "expected." You have non-terminating NULLs in this array and yet they are not printing for you. Just look at the tostring()... > but numpy haven't: > > In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) > Out[101]: > array([aba, ba, bab], > dtype='|S4') > > i.e. it seems like numpy is striping-off NULL chars before building the object > and I don't think this is correct. > Hmmm. I don't see that at all. This is what I get (version of numpy is 1.0.dev3233) In [33]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[33]: array(['a\x00ba', '\x00ba', 'ba\x00b'], dtype='|S4') which to me is very much expected. I.e. only terminating NULLs are stripped off of the strings on printing. I think you are getting different results because string printing used to not include the quotes (which had the side-effect of not printing NULLs in the middle of strings). They are still there, just not showing up in your output. In the end both numarray and numpy have the same data stored internally. It's just a matter of how it is being printed that seems to differ a bit. From my perspective, only NULLs at the end of strings should be stripped off and that is the (current) behavior of NumPy. You are getting different results, because the array-printing for strings was recently updated (to insert the quotes so that it makes more sense). Without these changes, I think the NULLs were being stripped away on printing. In other words, something like print 'a\x00ba' aba used to be happening. -Travis From coatimundi at hotmail.com Fri Sep 29 17:22:21 2006 From: coatimundi at hotmail.com (Kyle Ferrio) Date: Fri, 29 Sep 2006 15:22:21 -0600 Subject: [Numpy-discussion] Q: NumPy (svn) with cygwin build error? Message-ID: Hello. Starting from a freshly installed cygwin, I have built Python 2.5, ATLAS and LAPACK all from source using gcc 3.4.4 (cygming special) for both the C and f77 compilers. Now I am trying to build NumPy from SVN. I have edited site.cfg so that python setup.py config --compiler=mingw32 finds all of the linear algebra packs and completes without error. Then I fire off python setup.py build --compiler=mingw32 and I soon get (while building extension numpy.random.mtrand) _configtest.c:7:2: #error No _WIN32 which is not immediately fatal but definitely a concern. Soon after, building numpy.core.multiarray gives me this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzzled, because /usr/include/sys/select.h does exist. But maybe that's not what pyport.h is actually looking for? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! _________________________________________________________________ Express yourself - download free Windows Live Messenger themes! http://clk.atdmt.com/MSN/go/msnnkwme0020000001msn/direct/01/?href=http://imagine-msn.com/themes/vibe/default.aspx?locale=en-us&source=hmtagline From oliphant at ee.byu.edu Fri Sep 29 18:27:11 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 29 Sep 2006 16:27:11 -0600 Subject: [Numpy-discussion] Non-writeable default for numpy.ndarray In-Reply-To: <200609291844.07928.faltet@carabos.com> References: <200609291723.26896.faltet@carabos.com> <451D45D9.6050005@ieee.org> <200609291844.07928.faltet@carabos.com> Message-ID: <451D9DBF.1030303@ee.byu.edu> Francesc Altet wrote: >I see. Thanks for the explanation. > > You deserve the thanks for the great testing of less-traveled corners of NumPy. It's exactly the kind of thing needed to get NumPy ready for release. -Travis From coatimundi at cox.net Fri Sep 29 20:27:27 2006 From: coatimundi at cox.net (Coatimundi) Date: Fri, 29 Sep 2006 17:27:27 -0700 Subject: [Numpy-discussion] Problem building from svn under cygwin Message-ID: <451DB9EF.6030908@cox.net> I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. Configure works: python setup.py configure --compiler=mingw32 Build fails: python setup.py build --compiler=mingw32 While building numpy.random.mtrand I get this: _configtest.c:7:2: #error No _WIN32 and a little bit later the build aborts completely on multiarray with this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzled, because /usr/include/sys/select.h actually does exist on my cygwin installation. But maybe that's not what pyport.h is actually looking for??? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! From robert.kern at gmail.com Fri Sep 29 20:53:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Sep 2006 19:53:25 -0500 Subject: [Numpy-discussion] Problem building from svn under cygwin In-Reply-To: <451DB9EF.6030908@cox.net> References: <451DB9EF.6030908@cox.net> Message-ID: Coatimundi wrote: > I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. > > Configure works: > > python setup.py configure --compiler=mingw32 > > Build fails: > > python setup.py build --compiler=mingw32 What Python are you using? It looks like you built Python 2.5 using cygwin (that is, the UNIX emulation layer). You cannot build your extensions with mingw32 in that case. Don't use --compiler at all and you will be fine. If you are using the regular Python binaries, then you should be using --compiler=mingw32, but you're getting the wrong Python headers in that case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lars.bittrich at googlemail.com Sat Sep 30 08:16:27 2006 From: lars.bittrich at googlemail.com (Lars Bittrich) Date: Sat, 30 Sep 2006 14:16:27 +0200 Subject: [Numpy-discussion] return type diffences of indexed arrays with Intel C++ compiler (Python 2.5) Message-ID: <200609301416.27222.lars.bittrich@googlemail.com> Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: latest svn(1.0.dev3233) with Intel C compiler (Python 2.5): ----------------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 latest svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. A typical error message from scipy.test(10): ====================================================================== ERROR: check_chebyc (scipy.special.tests.test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/lib/python2.5/site-packages/scipy/special/tests/test_basic.py", line 667, in check_chebyc C1 = chebyc(1) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 461, in chebyc p = orthopoly1d(x,w,hn,kn,wfunc=lambda x: 1.0/sqrt(1-x*x/4.0),limits=(-2,2),monic=monic) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 78, in __init__ equiv_weights = [weights[k] / wfunc(roots[k]) for k in range(len(roots))] TypeError: object of type 'numpy.complex128' has no len() Does anyone know how to resolve the problem? Best regards, Lars From lars.bittrich at googlemail.com Fri Sep 29 08:48:30 2006 From: lars.bittrich at googlemail.com (Lars Bittrich) Date: Fri, 29 Sep 2006 14:48:30 +0200 Subject: [Numpy-discussion] return type diffences of indexed arrays with Intel C++ compiler (Python 2.5) Message-ID: <200609291448.30253.lars.bittrich@googlemail.com> Hi all, recently I tried to compile numpy/scipy with Python 2.5 and Intel C++ compiler 9.1. At first everything was fine, but the scipy test produced a few errors. The reason was a little difference: latest svn(1.0.dev3233) with Intel C compiler (Python 2.5): ----------------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:x[i] Out[4]:1.0 latest svn(1.0.dev3233) with GCC 3.3 (Python 2.3): -------------------------------------------------- In [1]:from numpy import ones, zeros, integer In [2]: In [2]:x = ones(1) In [3]:i = zeros(1, integer) In [4]:print x[i] Out[4]:array([1]) The Intel version gives me a scalar whereas the gcc version an array. Maybe Python 2.5 is causing this problem but my first guess was the compiler. A typical error message from scipy.test(10): ====================================================================== ERROR: check_chebyc (scipy.special.tests.test_basic.test_chebyc) ---------------------------------------------------------------------- Traceback (most recent call last): File "[...]/lib/python2.5/site-packages/scipy/special/tests/test_basic.py", line 667, in check_chebyc C1 = chebyc(1) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 461, in chebyc p = orthopoly1d(x,w,hn,kn,wfunc=lambda x: 1.0/sqrt(1-x*x/4.0),limits=(-2,2),monic=monic) File "[...]/lib/python2.5/site-packages/scipy/special/orthogonal.py", line 78, in __init__ equiv_weights = [weights[k] / wfunc(roots[k]) for k in range(len(roots))] TypeError: object of type 'numpy.complex128' has no len() Does anyone know how to resolve the problem? Best regards, Lars