From matthew.brett at gmail.com Sun Nov 1 01:45:09 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 31 Oct 2009 22:45:09 -0700 Subject: [Numpy-discussion] Recarray comparison and byte order In-Reply-To: References: <1e2af89e0910310938n7aec3ca0w8d5b0f8e6c7b0933@mail.gmail.com> Message-ID: <1e2af89e0910312245g2500085dv127147e834cdaafe@mail.gmail.com> Hi, >> c = a.byteswap().newbyteorder() >> c == a > > In the last two lines, a variable "c" is assigned to a modified "a".? The > next line tests (==) to see if "c" is the same as (==) the unmodified "a". > It isn't, because "c" is the modified "a".? Hence, "False". Sorry, I wasn't very clear - the stuff in brackets is what gets echoed to the terminal, and '==' was deliberate. The field values in 'a' and 'c' are all equal, so I was expecting what I got for comparing - for example 'a' to a copy of itself. Non-record arrays also behave as I expected after byteswapping: In [2]: a = np.arange(3, dtype='u2') In [3]: c = a.byteswap().newbyteorder() In [4]: a == c Out[4]: array([ True, True, True], dtype=bool) Best, Matthew From amenity at enthought.com Sun Nov 1 08:59:42 2009 From: amenity at enthought.com (Amenity Applewhite) Date: Sun, 1 Nov 2009 07:59:42 -0600 Subject: [Numpy-discussion] November 6 EPD Webinar: How do I... use Envisage for GUIs? References: <74384495.1256959480096.JavaMail.root@p2-ws608.ad.prodcc.net> Message-ID: Having trouble viewing this email? Click here Friday, November 6: How do I... use Envisage for GUIs? Dear Leah, Envisage is a Python-based framework for building extensible applications. The Envisage Core and corresponding Envisage Plugins are components of the Enthought Tool Suite. We've found that Envisage grants us a degree of immediate functionality in our custom applications and have come to rely on the framework in much of our development. For November's EPD webinar, Corran Webster will show how you can hook together existing Envisage plugins to quickly create a new GUI. We'll also look at how you can easily turn an existing Traits UI interface into an Envisage plugin. New: Linux-ready webinars! In order to better serve the Linux-users among our subscribers, we've decided to begin hosting our EPD webinars on WebEx instead of GoToMeeting. This means that our original limit of 35 attendees will be scaled back to 30. As usual, EPD subscribers at a Basic level or above will be guaranteed seats for the event while the general public may add their name to the wait list here. EPD Webinar: How do I... use Envisage for GUIs? Friday, November 6 1pm CDT/6pm UTC Wait list We look forward to seeing you Friday! As always, feel free to contact us with questions, concerns, or suggestions for future webinar topics. Thanks, The Enthought Team QUICK LINKS ::: www.enthought.com code.enthought.com Facebook Enthought Blog Forward email This email was sent to leah at enthought.com by amenity at enthought.com. Update Profile/Email Address | Instant removal with SafeUnsubscribe? | Privacy Policy. Enthought, Inc. | 515 Congress Ave. | Suite 2100 | Austin | TX | 78701 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bblinn at gmail.com Sun Nov 1 13:50:51 2009 From: bblinn at gmail.com (Bill Blinn) Date: Sun, 1 Nov 2009 13:50:51 -0500 Subject: [Numpy-discussion] Single view on multiple arrays Message-ID: What is the best way to create a view that is composed of sections of many different arrays? For example, imagine I had a = np.array(range(0, 12)).reshape(3, 4) b = np.array(range(12, 24)).reshape(3, 4) c = np.array(range(24, 36)).reshape(3, 4) v = multiview((3, 4)) #the idea of the following lines is that the 0th row of v is #a view on the first row of a. the same would hold true for #the 1st and 2nd row of v and the 0th rows of b and c, respectively v[0] = a[0] v[1] = b[0] v[2] = c[0] #change the underlying arrays a[0, 0] = 50 b[0, 0] = 51 c[0, 0] = 52 #I would want all these assertions to pass because the view #refers to the rows of a, b and c assert v[0, 0] == 50 assert v[1, 0] == 51 assert v[2, 0] == 52 Is there any way to do this? Thanks, bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sun Nov 1 14:09:13 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 1 Nov 2009 15:09:13 -0400 Subject: [Numpy-discussion] Single view on multiple arrays In-Reply-To: References: Message-ID: 2009/11/1 Bill Blinn : > What is the best way to create a view that is composed of sections of many > different arrays? The short answer is, you can't. Numpy arrays must be located contiguous blocks of memory, and the elements along any dimension must be equally spaced. A view is simply another array that references (some of) the same underlying memory, possibly with different strides. > For example, imagine I had > a = np.array(range(0, 12)).reshape(3, 4) > b = np.array(range(12, 24)).reshape(3, 4) > c = np.array(range(24, 36)).reshape(3, 4) > > v = multiview((3, 4)) > #the idea of the following lines is that the 0th row of v is > #a view on the first row of a. the same would hold true for > #the 1st and 2nd row of v and the 0th rows of b and c, respectively > v[0] = a[0] > v[1] = b[0] > v[2] = c[0] > > #change the underlying arrays > a[0, 0] = 50 > b[0, 0] = 51 > c[0, 0] = 52 > > #I would want all these assertions to pass because the view > #refers to the rows of a, b and c > assert v[0, 0] == 50 > assert v[1, 0] == 51 > assert v[2, 0] == 52 > > Is there any way to do this? If you need to be able to do this, you're going to have to rearrange your code somewhat, so that your original arrays are views of parts of an initial array. It's worth noting that if what you're worried about is the time it takes to copy data, you might well be surprised at how cheap data copying and memory allocation really are. Given that numpy is written in python, only for really enormous arrays will copying data be expensive, and allocating memory is really a very cheap operation (modern malloc()s average something like a few cycles). What's more, since modern CPUs are so heavily cache-bound, using strided views can be quite slow, since you end up loading whole 64-byte cache lines for each 8-byte double you need. In short, you probably need to rethink your design, but while you're doing it, don't worry about copying data. Anne From sturla at molden.no Sun Nov 1 16:38:45 2009 From: sturla at molden.no (Sturla Molden) Date: Sun, 01 Nov 2009 22:38:45 +0100 Subject: [Numpy-discussion] Single view on multiple arrays In-Reply-To: References: Message-ID: <4AEDFFE5.3080209@molden.no> Anne Archibald skrev: > The short answer is, you can't. Not really true. It is possible create an array (sub)class that stores memory addresses (pointers) instead of values. It is doable, but I am not wasting my time implementing it. Sturla From sturla at molden.no Sun Nov 1 16:56:02 2009 From: sturla at molden.no (Sturla Molden) Date: Sun, 01 Nov 2009 22:56:02 +0100 Subject: [Numpy-discussion] Single view on multiple arrays In-Reply-To: References: Message-ID: <4AEE03F2.3070207@molden.no> Bill Blinn skrev: > v = multiview((3, 4)) > #the idea of the following lines is that the 0th row of v is > #a view on the first row of a. the same would hold true for > #the 1st and 2nd row of v and the 0th rows of b and c, respectively > v[0] = a[0] This would not even work, becuase a[0] does not return a view array but a scalar arrays, which is a different type of numpy objects. To get a view, you will need to: v = a[0:1] # view of element 0 in a Also you cannot assign to v[0], as that would trigger a copy as well. > v[1] = b[0] > v[2] = c[0] As I mentioned in the answer to Anne, it would take a completely different array object. It would need to internally store an array with memory addresses. I have not made up my mind if ndarray can be subclassed for this, or if it takes a completely different object (e.g. similar to numpy.memmap). What it would require is __setitem__ to store pointers and __getitem__ to dereference (return an ndarray with values). Good look hacking, it is not even difficult, just tedious. Sturla From thomas.robitaille at gmail.com Sun Nov 1 20:57:17 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sun, 1 Nov 2009 20:57:17 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers Message-ID: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Hi, I'm trying to generate random 64-bit integer values for integers and floats using Numpy, within the entire range of valid values for that type. To generate random 32-bit floats, I can use: np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo (np.float32).max,size=10) which gives for example array([ 1.47351436e+37, 9.93620693e+37, 2.22893053e+38, -3.33828977e+38, 1.08247781e+37, -8.37481260e+37, 2.64176554e+38, -2.72207226e+37, 2.54790459e+38, -2.47883866e+38]) but if I try and use this for 64-bit numbers, i.e. np.random.uniform(low=np.finfo(np.float64).min,high=np.finfo (np.float64).max,size=10) I get array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) Similarly, for integers, I can successfully generate random 32-bit integers: np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo (np.int32).max,size=10) which gives array([-1506183689, 662982379, -1616890435, -1519456789, 1489753527, -604311122, 2034533014, 449680073, -444302414, -1924170329]) but am unsuccessful for 64-bit integers, i.e. np.random.random_integers(np.iinfo(np.int64).min,high=np.iinfo (np.int64).max,size=10) which produces the following error: OverflowError: long int too large to convert to int Is this expected behavior, or are these bugs? Thanks for any help, Thomas From d.l.goldsmith at gmail.com Sun Nov 1 20:58:48 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 1 Nov 2009 17:58:48 -0800 Subject: [Numpy-discussion] Ignorance question Message-ID: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> I Googled scipy brownian and the top hit was the doc for numpy.random.wald, but said doc has a "tone" that suggests there are more "sophisticated" ways to generate a random Brownian signal? Or is wald indeed SotA? Thanks! DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From bdesc100 at gmail.com Sun Nov 1 21:09:01 2009 From: bdesc100 at gmail.com (Benjamin Deschamps) Date: Sun, 1 Nov 2009 21:09:01 -0500 Subject: [Numpy-discussion] odd behaviour with basic operations Message-ID: I am getting strange behaviour with the following code: Pd = ((numpy.sign(C_02) == 1) * Pd_pos) + ((numpy.sign(C_02) == -1) * Pd_neg) Ps = ((numpy.sign(C_02) == 1) * Ps_pos) + ((numpy.sign(C_02) == -1) * Ps_neg) where Pd, Ps, C_02, Pd_pos, Pd_neg, Ps_pos and Ps_neg are all Float32 numpy arrays of the same shape. The problem is that the first line evaluates correctly (Pd is what it should be), but the second line does not. However, if I run the same line of code manually in IDLE, then it evaluates correctly! In other words, Ps as returned by the function does not match the value that I should get and obtain when entering the exact same code in IDLE. Basically, (numpy.sign(C_02) == 1) evaluates to either True or False, and multiplying with another array will give either 0 (when false) or the value of the array. The purpose of this code is to compute Pd and Ps without loops, and to take the value from Pd_pos or Ps_pos when C_02 is positive or of Pd_neg and Ps_neg when C_02 is negative. Using loops, it looks like this: for index in numpy.ndindex(ysize, xsize): if numpy.sign(C_02[index]) == 1: Pd[index] = Pd_pos[index] Ps[index] = Ps_pos[index] elif numpy.sign(C_02[index]) == -1: Pd[index] = Pd_neg[index] Ps[index] = Ps_neg[index] which also works fine, but takes much longer. Python 2.6.3, IDLE 2.6.1, Numpy 1.3.0, Snow Leopard, the script also uses some GDAL, matplotlib and scipy functions... Ideas? Benjamin -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Nov 1 21:19:28 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 1 Nov 2009 21:19:28 -0500 Subject: [Numpy-discussion] odd behaviour with basic operations In-Reply-To: References: Message-ID: <3d375d730911011819v7712798du1f1b72d2fd9923c8@mail.gmail.com> On Sun, Nov 1, 2009 at 21:09, Benjamin Deschamps wrote: > I am getting strange behaviour with the following code: > Pd = ((numpy.sign(C_02) == 1) * Pd_pos) + ((numpy.sign(C_02) == -1) * > Pd_neg) > Ps = ((numpy.sign(C_02) == 1) * Ps_pos) + ((numpy.sign(C_02) == -1) * > Ps_neg) > where Pd, Ps, C_02, Pd_pos, Pd_neg, Ps_pos and Ps_neg are all Float32 numpy > arrays of the same shape. > The problem is that the first line evaluates correctly (Pd is what it should > be), but the second line does not. However, if I run the same line of code > manually in IDLE, then it evaluates correctly! In other words, Ps as > returned by the function does not match the value that I should get and > obtain when entering the exact same code in IDLE. Please provide a self-contained example that demonstrates the problem. Also, be aware that where C_02==0, sign(C_02)==0. You will need to consider what should happen then. A better way to do what you want is to use where(): Pd = numpy.where(C_02 > 0.0, Pd_pos, Pd_neg) Ps = numpy.where(C_02 > 0.0, Ps_pos, Ps_neg) Change the > to >= if you want C_02==0 to use the pos values. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Sun Nov 1 21:26:22 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 1 Nov 2009 22:26:22 -0400 Subject: [Numpy-discussion] Ignorance question In-Reply-To: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> References: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> Message-ID: <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> On Sun, Nov 1, 2009 at 9:58 PM, David Goldsmith wrote: > I Googled scipy brownian and the top hit was the doc for numpy.random.wald, > but said doc has a "tone" that suggests there are more "sophisticated" ways > to generate a random Brownian signal? Or is wald indeed SotA?? Thanks! > > DG Do you mean generating a random sample of a Brownian motion? The standard approach, I have seen, is just cumsum of random normals, with time steps depending on the usage, e.g. http://groups.google.com/group/sympy/browse_thread/thread/65bf82164cae83be?pli=1 However, I never really checked the details of how they generate Brownian Motions or Brownian Bridges in larger Monte Carlo studies. Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Sun Nov 1 21:27:21 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 1 Nov 2009 22:27:21 -0400 Subject: [Numpy-discussion] Ignorance question In-Reply-To: <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> References: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> Message-ID: <1cd32cbb0911011827xafa30a2hd92b472d5b688bce@mail.gmail.com> On Sun, Nov 1, 2009 at 10:26 PM, wrote: > On Sun, Nov 1, 2009 at 9:58 PM, David Goldsmith wrote: >> I Googled scipy brownian and the top hit was the doc for numpy.random.wald, >> but said doc has a "tone" that suggests there are more "sophisticated" ways >> to generate a random Brownian signal? Or is wald indeed SotA?? Thanks! What's a SotA? Josef >> >> DG > > Do you mean generating a random sample of a Brownian motion? The > standard approach, I have seen, is just cumsum of random normals, with > time steps depending on the usage, e.g. > http://groups.google.com/group/sympy/browse_thread/thread/65bf82164cae83be?pli=1 > > However, I never really checked the details of how they generate > Brownian Motions or Brownian Bridges in larger Monte Carlo studies. > > Josef > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From robert.kern at gmail.com Sun Nov 1 21:28:09 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 1 Nov 2009 21:28:09 -0500 Subject: [Numpy-discussion] Ignorance question In-Reply-To: <1cd32cbb0911011827xafa30a2hd92b472d5b688bce@mail.gmail.com> References: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> <1cd32cbb0911011827xafa30a2hd92b472d5b688bce@mail.gmail.com> Message-ID: <3d375d730911011828s740db41ciaab2e7a3eaa49104@mail.gmail.com> On Sun, Nov 1, 2009 at 21:27, wrote: > On Sun, Nov 1, 2009 at 10:26 PM, ? wrote: >> On Sun, Nov 1, 2009 at 9:58 PM, David Goldsmith wrote: >>> I Googled scipy brownian and the top hit was the doc for numpy.random.wald, >>> but said doc has a "tone" that suggests there are more "sophisticated" ways >>> to generate a random Brownian signal? Or is wald indeed SotA?? Thanks! > > What's a SotA? "State of the Art" -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sturla at molden.no Sun Nov 1 21:41:19 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 02 Nov 2009 03:41:19 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: <4AEE46CF.5010009@molden.no> Thomas Robitaille skrev: > np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo > (np.int32).max,size=10) > > which gives > > array([-1506183689, 662982379, -1616890435, -1519456789, 1489753527, > -604311122, 2034533014, 449680073, -444302414, > -1924170329]) > > This fails on my computer (Python 2.6.4, NumPy 1.3.0 on Win32). >>> np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo (np.int32).max,size=10) Traceback (most recent call last): File "", line 2, in (np.int32).max,size=10) File "mtrand.pyx", line 950, in mtrand.RandomState.random_integers File "mtrand.pyx", line 746, in mtrand.RandomState.randint OverflowError: long int too large to convert to int It might have something to do with this: >>> 2**31-1 2147483647L >>> -2**31 -2147483648L In light of this annoying behaviour: def random_int64(size): a0 = np.random.random_integers(0, 0xFFFF, size=size).astype(np.uint64) a1 = np.random.random_integers(0, 0xFFFF, size=size).astype(np.uint64) a2 = np.random.random_integers(0, 0xFFFF, size=size).astype(np.uint64) a3 = np.random.random_integers(0, 0xFFFF, size=size).astype(np.uint64) a = a0 + (a1<<16) + (a2 << 32) + (a3 << 48) return a.view(dtype=np.int64) Sturla From david at ar.media.kyoto-u.ac.jp Sun Nov 1 21:20:13 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 02 Nov 2009 11:20:13 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> Thomas Robitaille wrote: > Hi, > > I'm trying to generate random 64-bit integer values for integers and > floats using Numpy, within the entire range of valid values for that > type. To generate random 32-bit floats, I can use: > > np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo > (np.float32).max,size=10) > > which gives for example > > array([ 1.47351436e+37, 9.93620693e+37, 2.22893053e+38, > -3.33828977e+38, 1.08247781e+37, -8.37481260e+37, > 2.64176554e+38, -2.72207226e+37, 2.54790459e+38, > -2.47883866e+38]) > > but if I try and use this for 64-bit numbers, i.e. > > np.random.uniform(low=np.finfo(np.float64).min,high=np.finfo > (np.float64).max,size=10) > > I get > > array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) > > Similarly, for integers, I can successfully generate random 32-bit > integers: > > np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo > (np.int32).max,size=10) > > which gives > > array([-1506183689, 662982379, -1616890435, -1519456789, 1489753527, > -604311122, 2034533014, 449680073, -444302414, > -1924170329]) > > but am unsuccessful for 64-bit integers, i.e. > > np.random.random_integers(np.iinfo(np.int64).min,high=np.iinfo > (np.int64).max,size=10) > > which produces the following error: > > OverflowError: long int too large to convert to int > > Is this expected behavior, or are these bugs? > I think those are bugs, but it may be difficult to fix. You can check that if you restrict a tiny bit your interval, you get better result: import numpy as np # max/min for double precision is ~ 1.8e308 low, high = -1e308, 1e308 np.random.uniformat(low, high, 100) # bunch of inf low, high = -1e307, 1e307 np.random.uniformat(low, high, 100) # much more reasonable It may be that you are pushing the limits of the random generator. Your min and max may be border cases: if you use the min/max representable numbers, and the random generator needs to do any addition of a positive number, you will 'overflow' your float number (Robert will have a better answer to this). The problem is that it may be difficult to detect this in advance. David From josef.pktd at gmail.com Sun Nov 1 22:03:25 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 1 Nov 2009 23:03:25 -0400 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0911011903i28e1f304w973b48495bf34c3e@mail.gmail.com> On Sun, Nov 1, 2009 at 10:20 PM, David Cournapeau wrote: > Thomas Robitaille wrote: >> Hi, >> >> I'm trying to generate random 64-bit integer values for integers and >> floats using Numpy, within the entire range of valid values for that >> type. To generate random 32-bit floats, I can use: >> >> np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo >> (np.float32).max,size=10) >> >> which gives for example >> >> array([ ?1.47351436e+37, ? 9.93620693e+37, ? 2.22893053e+38, >> ? ? ? ? ?-3.33828977e+38, ? 1.08247781e+37, ?-8.37481260e+37, >> ? ? ? ? ? 2.64176554e+38, ?-2.72207226e+37, ? 2.54790459e+38, >> ? ? ? ? ?-2.47883866e+38]) >> >> but if I try and use this for 64-bit numbers, i.e. >> >> np.random.uniform(low=np.finfo(np.float64).min,high=np.finfo >> (np.float64).max,size=10) >> >> I get >> >> array([ Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf]) >> >> Similarly, for integers, I can successfully generate random 32-bit >> integers: >> >> np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo >> (np.int32).max,size=10) >> >> which gives >> >> array([-1506183689, ? 662982379, -1616890435, -1519456789, ?1489753527, >> ? ? ? ? ?-604311122, ?2034533014, ? 449680073, ?-444302414, >> -1924170329]) >> >> but am unsuccessful for 64-bit integers, i.e. >> >> np.random.random_integers(np.iinfo(np.int64).min,high=np.iinfo >> (np.int64).max,size=10) >> >> which produces the following error: >> >> OverflowError: long int too large to convert to int >> >> Is this expected behavior, or are these bugs? >> > > I think those are bugs, but it may be difficult to fix. > > You can check that if you restrict a tiny bit your interval, you get > better result: > > import numpy as np > # max/min for double precision is ~ 1.8e308 > low, high = -1e308, 1e308 > np.random.uniformat(low, high, 100) # bunch of inf > low, high = -1e307, 1e307 > np.random.uniformat(low, high, 100) # much more reasonable > > It may be that you are pushing the limits of the random generator. Your > min and max may be border cases: if you use the min/max representable > numbers, and the random generator needs to do any addition of a positive > number, you will 'overflow' your float number (Robert will have a better > answer to this). The problem is that it may be difficult to detect this > in advance. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) might actually be the right answer if you want a uniform distribution on the real line. I never realized how many numbers are out there when I saw that most numbers in the example are e+37 or e+38. Josef From robert.kern at gmail.com Sun Nov 1 22:13:51 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 1 Nov 2009 22:13:51 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> On Sun, Nov 1, 2009 at 20:57, Thomas Robitaille wrote: > Hi, > > I'm trying to generate random 64-bit integer values for integers and > floats using Numpy, within the entire range of valid values for that > type. 64-bit and larger integers could be done, but it requires modification. The integer distributions were written to support C longs, not anything larger. You could also use .bytes() and np.fromstring(). > To generate random 32-bit floats, I can use: > np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo > (np.float32).max,size=10) What is the use case here? I know of none. Floating point is a bit weird and will cause you many problems over such an extended range. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Nov 1 21:55:17 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 02 Nov 2009 11:55:17 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911011903i28e1f304w973b48495bf34c3e@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> <1cd32cbb0911011903i28e1f304w973b48495bf34c3e@mail.gmail.com> Message-ID: <4AEE4A15.2080804@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > > array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) > > might actually be the right answer if you want a uniform distribution > on the real line. Does it make sense to define a uniform random variable whose range is the extended real line ? It would not have a distribution w.r.t the Lebesgue measure, right ? I must confess I am quickly lost in the maths in statistics, though, David From sturla at molden.no Sun Nov 1 22:17:05 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 02 Nov 2009 04:17:05 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> Message-ID: <4AEE4F31.1030700@molden.no> Robert Kern skrev: > 64-bit and larger integers could be done, but it requires > modification. The integer distributions were written to support C > longs, not anything larger. You could also use .bytes() and > np.fromstring(). > But as of Python 2.6.4, even 32-bit integers fail, at least on Windows. Sturla From robert.kern at gmail.com Sun Nov 1 22:22:43 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 1 Nov 2009 22:22:43 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE4F31.1030700@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> Message-ID: <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> On Sun, Nov 1, 2009 at 22:17, Sturla Molden wrote: > Robert Kern skrev: >> 64-bit and larger integers could be done, but it requires >> modification. The integer distributions were written to support C >> longs, not anything larger. You could also use .bytes() and >> np.fromstring(). >> > But as of Python 2.6.4, even 32-bit integers fail, at least on Windows. Then let me clarify: it was written to support integer ranges up to sys.maxint. Absolutely, it would be desirable to extend it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Nov 1 22:04:08 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 02 Nov 2009 12:04:08 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE4F31.1030700@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> Message-ID: <4AEE4C28.4040208@ar.media.kyoto-u.ac.jp> Sturla Molden wrote: > Robert Kern skrev: > >> 64-bit and larger integers could be done, but it requires >> modification. The integer distributions were written to support C >> longs, not anything larger. You could also use .bytes() and >> np.fromstring(). >> >> > But as of Python 2.6.4, even 32-bit integers fail, at least on Windows. > It fails on linux as well - I think it is a 32 vs 64 bits issue, not a windows vs linux. I don't know what happens on windows 64, though: we may have issue if we use long. cheers, David From sturla at molden.no Sun Nov 1 22:27:46 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 02 Nov 2009 04:27:46 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> Message-ID: <4AEE51B2.6020507@molden.no> Robert Kern skrev: > Then let me clarify: it was written to support integer ranges up to > sys.maxint. Absolutely, it would be desirable to extend it. > > I know, but look at this: >>> import sys >>> sys.maxint 2147483647 >>> 2**31-1 2147483647L sys.maxint becomes a long, which is what confuses mtrand. From thomas.robitaille at gmail.com Sun Nov 1 22:37:05 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sun, 1 Nov 2009 22:37:05 -0500 Subject: [Numpy-discussion] Formatting uint64 number Message-ID: Hello, I have a question concerning uint64 numbers - let's say I want to format a uint64 number that is > 2**31, at the moment it's necessary to wrap the numpy number inside long before formatting In [3]: "%40i" % np.uint64(2**64-1) Out[3]: ' -1' In [4]: "%40i" % long(np.uint64(2**64-1)) Out[4]: ' 18446744073709551615' Would it be easy to modify numpy such that it automatically converts uint64 numbers to long() instead of int() when implicitly converted to python types? Thanks, Thomas From josef.pktd at gmail.com Sun Nov 1 22:39:49 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 1 Nov 2009 23:39:49 -0400 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE4A15.2080804@ar.media.kyoto-u.ac.jp> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> <1cd32cbb0911011903i28e1f304w973b48495bf34c3e@mail.gmail.com> <4AEE4A15.2080804@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0911011939p578e9d3ck4674eacc75130763@mail.gmail.com> On Sun, Nov 1, 2009 at 10:55 PM, David Cournapeau wrote: > josef.pktd at gmail.com wrote: >> >> array([ Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf]) >> >> might actually be the right answer if you want a uniform distribution >> on the real line. > > Does it make sense to define a uniform random variable whose range is > the extended real line ? It would not have a distribution w.r.t the > Lebesgue measure, right ? > > I must confess I am quickly lost in the maths in statistics, though, No, it wouldn't be a proper distribution. However in Bayesian analysis it is used as an improper (diffuse) prior, which often replicates frequentists results. But it's a theoretical derivation, I don't think anyone tries to simulate this. To simulate huge uniform integers, I think it should be possible to use the floating point random numbers and rescale and round them. Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bdesc100 at gmail.com Sun Nov 1 22:46:15 2009 From: bdesc100 at gmail.com (Benjamin Deschamps) Date: Sun, 1 Nov 2009 22:46:15 -0500 Subject: [Numpy-discussion] odd behaviour with basic operations In-Reply-To: References: Message-ID: <9C685981-E34F-490F-9331-D7392AE60E59@gmail.com> Seems like this was a rookie mistake with code later in the function. Thanks for suggesting the use of numpy.where, that is a much better function for the purpose. Benjamin From sturla at molden.no Sun Nov 1 23:05:47 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 02 Nov 2009 05:05:47 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> Message-ID: <4AEE5A9B.6050509@molden.no> Robert Kern skrev: > Then let me clarify: it was written to support integer ranges up to > sys.maxint. Absolutely, it would be desirable to extend it. > > Actually it only supports integers up to sys.maxint-1, as random_integers call randint. random_integers includes the upper range, but randint excludes the upper range. Thus, this happens on line 1153 in mtrand.pyx: return self.randint(low, high+1, size) The main source of the problem is that number smaller than sys.maxint can become a long. (I have asked why on python-dev, it does not make any sence.) So when random_integers pass "high+1" to randint, it is unneccesarily converted to a long. Then, there is an exception on line 847: hi = high With hi previously declared to long, Cython refuses the conversion. Now, we could try a downcast to int like this: hi = int(high) which would make Cython only raise an exception in case of an integer overflow. >>> int(2**31) 2147483648L >>> int(2**31-1) 2147483647 If there is no overflow, high becomes an int and conversion to C long is allowed. Still, this will only support integer ranges up to sys.maxint - 1. We thus have to swap the order of randint and random_intgers. The one with the inclusive upper interval should call rk_interval. Sturla From sturla at molden.no Sun Nov 1 23:14:41 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 02 Nov 2009 05:14:41 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE5A9B.6050509@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> <4AEE5A9B.6050509@molden.no> Message-ID: <4AEE5CB1.8090306@molden.no> Sturla Molden skrev: > Robert Kern skrev: > >> Then let me clarify: it was written to support integer ranges up to >> sys.maxint. Absolutely, it would be desirable to extend it. >> >> >> > Actually it only supports integers up to sys.maxint-1, as > random_integers call randint. random_integers includes the upper range, > but randint excludes the upper range. Thus, this happens on line 1153 in > mtrand.pyx: > > return self.randint(low, high+1, size) > > inclusive upper interval should call rk_interval > I love this one: cdef long lo, hi, diff [...] diff = hi - lo - 1 which silently overflows, and is the reason for this strange exception: >>> np.random.random_integers(-2147483648,high=2147483646,size=10) Traceback (most recent call last): File "", line 1, in np.random.random_integers(-2147483648,high=2147483646,size=10) File "mtrand.pyx", line 950, in mtrand.RandomState.random_integers File "mtrand.pyx", line 750, in mtrand.RandomState.randint ValueError: low >= high I'll call this a bug. Sturla From robert.kern at gmail.com Sun Nov 1 23:19:41 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 1 Nov 2009 23:19:41 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE5CB1.8090306@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> <4AEE5A9B.6050509@molden.no> <4AEE5CB1.8090306@molden.no> Message-ID: <3d375d730911012019k14e6167dlf6e5fc2029476a5@mail.gmail.com> On Sun, Nov 1, 2009 at 23:14, Sturla Molden wrote: > I'll call this a bug. Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Nov 1 23:04:08 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 02 Nov 2009 13:04:08 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911011939p578e9d3ck4674eacc75130763@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <4AEE41DD.9060206@ar.media.kyoto-u.ac.jp> <1cd32cbb0911011903i28e1f304w973b48495bf34c3e@mail.gmail.com> <4AEE4A15.2080804@ar.media.kyoto-u.ac.jp> <1cd32cbb0911011939p578e9d3ck4674eacc75130763@mail.gmail.com> Message-ID: <4AEE5A38.9070803@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > > No, it wouldn't be a proper distribution. However in Bayesian analysis > it is used as an improper (diffuse) prior Ah, right - I wonder how this is handled rigorously, though. I know some basics of Bayesian statistics, but I don't much about Bayesian statistics from a theoretical POV (i.e. a rigorous mathematical development). > To simulate huge uniform integers, I think it should be possible to use > the floating point random numbers and rescale and round them. > Rescaling and especially rounding may bias the distribution, no ? The best (but long term) strategy would be to support arbitrary precision integer, as mentioned by Robert. David From david at ar.media.kyoto-u.ac.jp Sun Nov 1 23:07:00 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 02 Nov 2009 13:07:00 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AEE5CB1.8090306@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <3d375d730911011913r7378f20bidb57db393b448a2a@mail.gmail.com> <4AEE4F31.1030700@molden.no> <3d375d730911011922o7d9caa70pfeff878911bde6a8@mail.gmail.com> <4AEE5A9B.6050509@molden.no> <4AEE5CB1.8090306@molden.no> Message-ID: <4AEE5AE4.4010607@ar.media.kyoto-u.ac.jp> Sturla Molden wrote: > Sturla Molden skrev: > >> Robert Kern skrev: >> >> >>> Then let me clarify: it was written to support integer ranges up to >>> sys.maxint. Absolutely, it would be desirable to extend it. >>> >>> >>> >>> >> Actually it only supports integers up to sys.maxint-1, as >> random_integers call randint. random_integers includes the upper range, >> but randint excludes the upper range. Thus, this happens on line 1153 in >> mtrand.pyx: >> >> return self.randint(low, high+1, size) >> >> inclusive upper interval should call rk_interval >> >> > > I love this one: > > cdef long lo, hi, diff > [...] > diff = hi - lo - 1 > > which silently overflows, and is the reason for this strange exception: > > >>> np.random.random_integers(-2147483648,high=2147483646,size=10) > > Traceback (most recent call last): > File "", line 1, in > np.random.random_integers(-2147483648,high=2147483646,size=10) > File "mtrand.pyx", line 950, in mtrand.RandomState.random_integers > File "mtrand.pyx", line 750, in mtrand.RandomState.randint > ValueError: low >= high > > I'll call this a bug. > Yep, I was bitten by it as well: http://projects.scipy.org/numpy/ticket/965 David From charlesr.harris at gmail.com Sun Nov 1 23:49:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 1 Nov 2009 21:49:29 -0700 Subject: [Numpy-discussion] Formatting uint64 number In-Reply-To: References: Message-ID: On Sun, Nov 1, 2009 at 8:37 PM, Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > Hello, > > I have a question concerning uint64 numbers - let's say I want to > format a uint64 number that is > 2**31, at the moment it's necessary > to wrap the numpy number inside long before formatting > > In [3]: "%40i" % np.uint64(2**64-1) > Out[3]: ' -1' > > In [4]: "%40i" % long(np.uint64(2**64-1)) > Out[4]: ' 18446744073709551615' > > Would it be easy to modify numpy such that it automatically converts > uint64 numbers to long() instead of int() when implicitly converted to > python types? > > Hmm, I suspect this is a bug whose source is uint64 having an integer conversion function as part of the type whereas it should be undefined. A quick look at the source leaves me befuddled, so tracking down just how this happens might be a bit of work. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Nov 1 23:57:27 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 1 Nov 2009 21:57:27 -0700 Subject: [Numpy-discussion] Ignorance question In-Reply-To: <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> References: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> Message-ID: On Sun, Nov 1, 2009 at 7:26 PM, wrote: > On Sun, Nov 1, 2009 at 9:58 PM, David Goldsmith > wrote: > > I Googled scipy brownian and the top hit was the doc for > numpy.random.wald, > > but said doc has a "tone" that suggests there are more "sophisticated" > ways > > to generate a random Brownian signal? Or is wald indeed SotA? Thanks! > > > > DG > > Do you mean generating a random sample of a Brownian motion? The > standard approach, I have seen, is just cumsum of random normals, with > time steps depending on the usage, e.g. > Oddly enough, if you divide an interval into an infinite integer number of samples this also works for the theory side ;) Euler would understand, but such odd constructions with extended number systems have fallen out of favour... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 2 03:29:18 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 2 Nov 2009 17:29:18 +0900 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. Message-ID: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Hi, I think it is about time to release 1.4.0. Instead of proposing a release date, I am setting a firm date for 1st December, and 16th november to freeze the trunk. If someone wants a different date, you have to speak now. There are a few issues I would like to clear up: - Documentation for datetime, in particular for the public C API - Snow Leopard issues, if any Otherwise, I think there has been quite a lot of new features. If people want to add new functionalities or features, please do it soon, cheers, David From dsdale24 at gmail.com Mon Nov 2 07:32:19 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 2 Nov 2009 07:32:19 -0500 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Mon, Nov 2, 2009 at 3:29 AM, David Cournapeau wrote: > Hi, > > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > ?- Documentation for datetime, in particular for the public C API > ?- Snow Leopard issues, if any > > Otherwise, I think there has been quite a lot of new features. If > people want to add new functionalities or features, please do it soon, I wanted to get __input_prepare__ in for the 1.4 release, but I don't think I can get it in and tested by November 16. Darren From pav+sp at iki.fi Mon Nov 2 07:38:44 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Mon, 2 Nov 2009 12:38:44 +0000 (UTC) Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: Mon, 02 Nov 2009 17:29:18 +0900, David Cournapeau wrote: > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > - Documentation for datetime, in particular for the public C API > - Snow Leopard issues, if any > > Otherwise, I think there has been quite a lot of new features. If people > want to add new functionalities or features, please do it soon, Can we get the complex functions to npy_math for 1.4.0: could be useful for the next Scipy? This is pretty quick to do, I can just write up some more tests one evening and commit. -- Pauli Virtanen From peridot.faceted at gmail.com Mon Nov 2 10:09:36 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 2 Nov 2009 11:09:36 -0400 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: 2009/11/1 Thomas Robitaille : > Hi, > > I'm trying to generate random 64-bit integer values for integers and > floats using Numpy, within the entire range of valid values for that > type. To generate random 32-bit floats, I can use: Others have addressed why this is giving bogus results. But it's worth thinking about what you're actually trying to do. If it's "fuzz tests", that is, producing unrestricted random floats to feed to a function, then even if this worked it wouldn't produce what you want: it will never produce floats of order unity, for example. If you want do what you described, you could produce floats uniformly from -1 to 1 and then multiply the results by the largest representable float. If you want "random floats", I suggest generating an array of random bytes and reinterpreting them as floats. You'll get a fair number of NaNs and infinities, so you may want to take only the finite values and regenerate the rest. This will give you some numbers from all over the range of floats, including tiny values (and denormalized values, which are a potentially important special case). Anne > np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo > (np.float32).max,size=10) > > which gives for example > > array([ ?1.47351436e+37, ? 9.93620693e+37, ? 2.22893053e+38, > ? ? ? ? -3.33828977e+38, ? 1.08247781e+37, ?-8.37481260e+37, > ? ? ? ? ?2.64176554e+38, ?-2.72207226e+37, ? 2.54790459e+38, > ? ? ? ? -2.47883866e+38]) > > but if I try and use this for 64-bit numbers, i.e. > > np.random.uniform(low=np.finfo(np.float64).min,high=np.finfo > (np.float64).max,size=10) > > I get > > array([ Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf, ?Inf]) > > Similarly, for integers, I can successfully generate random 32-bit > integers: > > np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo > (np.int32).max,size=10) > > which gives > > array([-1506183689, ? 662982379, -1616890435, -1519456789, ?1489753527, > ? ? ? ? -604311122, ?2034533014, ? 449680073, ?-444302414, > -1924170329]) > > but am unsuccessful for 64-bit integers, i.e. > > np.random.random_integers(np.iinfo(np.int64).min,high=np.iinfo > (np.int64).max,size=10) > > which produces the following error: > > OverflowError: long int too large to convert to int > > Is this expected behavior, or are these bugs? > > Thanks for any help, > > Thomas > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Nov 2 11:04:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 2 Nov 2009 09:04:39 -0700 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Mon, Nov 2, 2009 at 1:29 AM, David Cournapeau wrote: > Hi, > > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > - Documentation for datetime, in particular for the public C API > - Snow Leopard issues, if any > > It would be nice to track down the reduceat bug. Not that I've managed to get myself working on the problem. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 2 11:50:53 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 3 Nov 2009 01:50:53 +0900 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: <5b8d13220911020850x2935b46ana1d5846342f0c0ca@mail.gmail.com> On Mon, Nov 2, 2009 at 9:38 PM, Pauli Virtanen wrote: > > Can we get the complex functions to npy_math for 1.4.0: could be useful > for the next Scipy? This is pretty quick to do, I can just write up some > more tests one evening and commit. The idea was partially to set a date to force me putting my code in shape soon. The whole point of setting release dates is to organize ones own time around it, after all :) David From tsyu80 at gmail.com Mon Nov 2 12:02:04 2009 From: tsyu80 at gmail.com (Tony S Yu) Date: Mon, 2 Nov 2009 12:02:04 -0500 Subject: [Numpy-discussion] Setting a firm release date for 1st december. In-Reply-To: References: Message-ID: <6B62015D-A65D-4EC7-8CBB-119320840749@gmail.com> On Nov 2, 2009, at 11:09 AM, numpy-discussion-request at scipy.org wrote: > From: David Cournapeau > Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st > december. > To: Discussion of Numerical Python > Message-ID: > <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653 at mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hi, > > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > - Documentation for datetime, in particular for the public C API > - Snow Leopard issues, if any > > Otherwise, I think there has been quite a lot of new features. If > people want to add new functionalities or features, please do it soon, > > cheers, > > David This seems like an appropriate opportunity to ping an issue I posted on Trac: http://projects.scipy.org/numpy/ticket/1177 Basically, interp doesn't currently play nice with numpy scalars. Patch included with ticket (see second comment). Cheers, -Tony -------------- next part -------------- An HTML attachment was scrubbed... URL: From reckoner at gmail.com Mon Nov 2 15:42:25 2009 From: reckoner at gmail.com (Reckoner) Date: Mon, 2 Nov 2009 12:42:25 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: Anybody have any ideas here? Otherwise, I'm thinking this should be posted to the numpy bugs list. What's the best way to report a bug of this kind? Thanks! On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >> Robert Kern wrote: >> You can import numpy.core.multiarray on both machines? > > Yes. For each machine separately, you can cPickle files with numpy > arrays without problems loading/dumping. The problem comes from > transferring the win32 cPickle'd files to Linux 64 bit and then trying > to load them. Transferring cPickle'd files that do *not* have numpy > arrays work as expected. In other words, cPICKLE'd lists transfer fine > back and forth between the two machines. In fact, we currently get > around this problem by converting the numpy arrays to lists, > transferring them, and then re-numpy-ing them on the respective hosts > > thanks. > > > On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >> Hi, >> >> % python -c 'import numpy.core.multiarray' >> >> works just fine, but when I try to load a file that I have transferred >> from another machine running Windows to one running Linux, I get: >> >> % python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >> >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named multiarray >> >> otherwise, cPickle works normally when transferring files that *do* >> not contain numpy arrays. >> >> I am using version 1.2 on both machines. It's not so easy for me to >> change versions, by the way, since this is the version that my working >> group has decided on to standardize on for this effort. >> >> >> Any help appreciated. >> > From robert.kern at gmail.com Mon Nov 2 15:44:51 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Nov 2009 15:44:51 -0500 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: <3d375d730911021244u75d1540j8020274d52aec334@mail.gmail.com> On Mon, Nov 2, 2009 at 15:42, Reckoner wrote: > Anybody have any ideas here? Nope! > Otherwise, I'm thinking this should be posted to the numpy bugs list. > What's the best way to report a bug of this kind? Create a new ticket on the Trac: http://projects.scipy.org/numpy/ If you can attach the offending pickle files, that would be ideal. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bsouthey at gmail.com Mon Nov 2 17:43:56 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 2 Nov 2009 16:43:56 -0600 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: > Anybody have any ideas here? > > Otherwise, I'm thinking this should be posted to the numpy bugs list. > What's the best way to report a bug of this kind? > > Thanks! > > On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>> Robert Kern wrote: >>> You can import numpy.core.multiarray on both machines? >> >> Yes. For each machine separately, you can cPickle files with numpy >> arrays without problems loading/dumping. The problem comes from >> transferring the win32 cPickle'd files to Linux 64 bit and then trying >> to load them. Transferring cPickle'd files that do *not* have numpy >> arrays work as expected. In other words, cPICKLE'd lists transfer fine >> back and forth between the two machines. In fact, we currently get >> around this problem by converting the numpy arrays to lists, >> transferring them, and then re-numpy-ing them on the respective hosts >> >> thanks. >> >> >> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>> Hi, >>> >>> % python -c 'import numpy.core.multiarray' >>> >>> works just fine, but when I try to load a file that I have transferred >>> from another machine running Windows to one running Linux, I get: >>> >>> % ?python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>> >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ImportError: No module named multiarray >>> >>> otherwise, cPickle works normally when transferring files that *do* >>> not contain numpy arrays. >>> >>> I am using version 1.2 on both machines. It's not so easy for me to >>> change versions, by the way, since this is the version that my working >>> group has decided on to standardize on for this effort. >>> >>> >>> Any help appreciated. >>> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Have you have tried the other Cookbook approaches: http://www.scipy.org/Cookbook/InputOutput Like using numpy's own array io functions - load/save(z)? (seems to work between 64-bit Windows 7 and 64-bit Linux - each has different numpy versions) Bruce From reckoner at gmail.com Mon Nov 2 19:31:38 2009 From: reckoner at gmail.com (Reckoner) Date: Mon, 2 Nov 2009 16:31:38 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: thanks for the suggestion! I will look into it. The other thing is that the numpy arrays in question are actually embedded in another object. When I convert the numpy arrays into plain lists, and then cPickle them, there is no problem with any of the larger objects. That is the way we are currently working around this issue. Thanks again. On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: > On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >> Anybody have any ideas here? >> >> Otherwise, I'm thinking this should be posted to the numpy bugs list. >> What's the best way to report a bug of this kind? >> >> Thanks! >> >> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>> Robert Kern wrote: >>>> You can import numpy.core.multiarray on both machines? >>> >>> Yes. For each machine separately, you can cPickle files with numpy >>> arrays without problems loading/dumping. The problem comes from >>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>> to load them. Transferring cPickle'd files that do *not* have numpy >>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>> back and forth between the two machines. In fact, we currently get >>> around this problem by converting the numpy arrays to lists, >>> transferring them, and then re-numpy-ing them on the respective hosts >>> >>> thanks. >>> >>> >>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>> Hi, >>>> >>>> % python -c 'import numpy.core.multiarray' >>>> >>>> works just fine, but when I try to load a file that I have transferred >>>> from another machine running Windows to one running Linux, I get: >>>> >>>> % python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>> >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> ImportError: No module named multiarray >>>> >>>> otherwise, cPickle works normally when transferring files that *do* >>>> not contain numpy arrays. >>>> >>>> I am using version 1.2 on both machines. It's not so easy for me to >>>> change versions, by the way, since this is the version that my working >>>> group has decided on to standardize on for this effort. >>>> >>>> >>>> Any help appreciated. >>>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > Have you have tried the other Cookbook approaches: > http://www.scipy.org/Cookbook/InputOutput > Like using numpy's own array io functions - load/save(z)? > (seems to work between 64-bit Windows 7 and 64-bit Linux - each has > different numpy versions) > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From thomas.robitaille at gmail.com Mon Nov 2 23:35:19 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Mon, 2 Nov 2009 23:35:19 -0500 Subject: [Numpy-discussion] Automatic string length in recarray Message-ID: Hi, I'm having trouble with creating np.string_ fields in recarrays. If I create a recarray using np.rec.fromrecords([(1,'hello'),(2,'world')],names=['a','b']) the result looks fine: rec.array([(1, 'hello'), (2, 'world')], dtype=[('a', ' Hi, I am using numpy distutils to build the extension modules of a project, which have been so far written in C, and wrapped by SWIG. Now I would like to try cython (as everynone!), but still be able to use the numpy distutils. I have found the thread [1], which offers some solution, but it does not mention using the Configuration class from numpy.distutils.misc_util which I use. How can I tell Configuration to use Cython instead of Pyrex? thanks! r. [1] http://old.nabble.com/problem-with-numpy.distutils-and-Cython-td25100957.html From bsouthey at gmail.com Tue Nov 3 07:55:27 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 3 Nov 2009 06:55:27 -0600 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: > thanks for the suggestion! I will look into it. The other thing is > that the numpy arrays in question are actually embedded in another > object. When I convert the numpy arrays into plain lists, and then > cPickle them, there is no problem with any of the larger objects. That > is the way we are currently working around this issue. > > Thanks again. > > On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>> Anybody have any ideas here? >>> >>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>> What's the best way to report a bug of this kind? >>> >>> Thanks! >>> >>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>> Robert Kern wrote: >>>>> You can import numpy.core.multiarray on both machines? >>>> >>>> Yes. For each machine separately, you can cPickle files with numpy >>>> arrays without problems loading/dumping. The problem comes from >>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>> back and forth between the two machines. In fact, we currently get >>>> around this problem by converting the numpy arrays to lists, >>>> transferring them, and then re-numpy-ing them on the respective hosts >>>> >>>> thanks. >>>> >>>> >>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>> Hi, >>>>> >>>>> % python -c 'import numpy.core.multiarray' >>>>> >>>>> works just fine, but when I try to load a file that I have transferred >>>>> from another machine running Windows to one running Linux, I get: >>>>> >>>>> % ?python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>> >>>>> Traceback (most recent call last): >>>>> ?File "", line 1, in >>>>> ImportError: No module named multiarray >>>>> >>>>> otherwise, cPickle works normally when transferring files that *do* >>>>> not contain numpy arrays. >>>>> >>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>> change versions, by the way, since this is the version that my working >>>>> group has decided on to standardize on for this effort. >>>>> >>>>> >>>>> Any help appreciated. >>>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> Have you have tried the other Cookbook approaches: >> http://www.scipy.org/Cookbook/InputOutput >> Like using numpy's own array io functions - load/save(z)? >> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >> different numpy versions) >> >> Bruce >> _______________________________________________ Can you provide you provide a small self-contained example of the problem including object creation especially as your example does not import numpy? Really you have to start at the beginning (like pickling and transferring numpy arrays) and then increase the complexity to include the object. Bruce From gbsuar at gmail.com Tue Nov 3 08:16:39 2009 From: gbsuar at gmail.com (Gabriel) Date: Tue, 3 Nov 2009 10:16:39 -0300 Subject: [Numpy-discussion] fft trouble In-Reply-To: <91906ddf0911030454u44c65bf7g3b1b56675f004a12@mail.gmail.com> References: <91906ddf0911030454u44c65bf7g3b1b56675f004a12@mail.gmail.com> Message-ID: <91906ddf0911030516i4328f8bep47382e5614022810@mail.gmail.com> Hi it`s my first time. I have a trouble whit fft. For example: s = sin(wt) f = abs(fft(s)) In f I have the module the fft(s) which should be equal to 1 for w but it's 1000 for w. All values is multiplied by 1000. Someone know why? Thank's -- Gabriel Antes de imprimir, pens? en tu responsabilidad y compromiso con el MEDIO AMBIENTE. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.raspaud at smhi.se Tue Nov 3 08:39:42 2009 From: martin.raspaud at smhi.se (Raspaud Martin) Date: Tue, 3 Nov 2009 14:39:42 +0100 Subject: [Numpy-discussion] fft trouble In-Reply-To: <91906ddf0911030516i4328f8bep47382e5614022810@mail.gmail.com> References: <91906ddf0911030454u44c65bf7g3b1b56675f004a12@mail.gmail.com> <91906ddf0911030516i4328f8bep47382e5614022810@mail.gmail.com> Message-ID: <783F32138ED65D4A9CF016980481B6BF01CB57A7@CORRE.ad.smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gabriel wrote: > Hi it`s my first time. > > I have a trouble whit fft. > For example: > s = sin(wt) > > f = abs(fft(s)) > > In f I have the module the fft(s) which should be equal to 1 for w but > it's 1000 for w. All values is multiplied by 1000. > > Someone know why? Hi, I don?t know if this is numpy related, but I know a little bit about fft in general, and it looks like you did not normalize the result. The documentation does not specify that the resulting array is normalized, so you should do : s = sin(wt) f = abs(fft(s) / len(s)) I guess your wt has a length of 1000 ? Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJK8DKdAAoJEBdvyODiyJI4zCkH/Ahe8dh+zfVK6nTWeaKqjcPy fqpn2PSvb3l0z+dfOxtmkv9DB24ntJadQ+H26t1VeK5HcZxSunncb8vfGsrWtIZN Z773aE7ZtZZpIYl+wbIQKCiAQ6gZFUbuK4qJxFl0iLGlQXcGjM4EjqO8nDhKHuOr hpjhtfp56s35e3rXJFT7Za4XNt17pGnMsk/91/X3/ACCjAJNH1F1KZl7yULZqEUj KBY62UMWA136xOi2bFKacMW7Ir4qhSVEZnkA0LN2VArAZ4RYV7290dARFZ+N78+V W4sX4DBPECMZ3vGl7y+mjfMqExk9GDC6hpV8MiVfidMeFhYV5+PgNodE5Ectekg= =9PY4 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: martin_raspaud.vcf Type: text/x-vcard Size: 260 bytes Desc: martin_raspaud.vcf URL: From bsouthey at gmail.com Tue Nov 3 09:15:31 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 3 Nov 2009 08:15:31 -0600 Subject: [Numpy-discussion] [OT] Python-URL! QOTW Message-ID: Way to go Robert! QOTW: "I consider "import *" the first error to be fixed ..." - Robert Kern, author of PyFlakes, a potential replacement for Pylint and Pychecker, on his personal style http://groups.google.com/group/comp.lang.python/msg/5bf77b21b3b0caf2 Python 2.6.4 is out; it fixes some small but important bugs in previous 2.6.3 release: http://groups.google.com/group/comp.lang.python/t/44f4c166de5e6703/ PEP391: A new way to configure Python logging, using dictionaries: http://groups.google.com/group/comp.lang.python/t/3fc3138c22a50678/ Why do you use Python? http://groups.google.com/group/comp.lang.python/t/66ccb10d2da5b740/ Is there any value in forcing one class/function per module? (long thread): http://groups.google.com/group/comp.lang.python/t/1e0c1e2091539512/ A review of complex solutions to an almost inexistent problem shows constructive criticism in action: http://groups.google.com/group/comp.lang.python/t/d218212cabe9670b/ A user module may shadow a standard module of the same name - is it avoidable? http://groups.google.com/group/comp.lang.python/t/b31f74b6145c9d94/ copy.deepcopy() is not as "deep" as its name implies: http://groups.google.com/group/comp.lang.python/t/d827fd118189fe01/ A long thread about Web development: why to use templates, and why frameworks actually do help developers: http://groups.google.com/group/comp.lang.python/t/367025d4d9a2e15d/ Using a module as a place to store application global settings: http://groups.google.com/group/comp.lang.python/t/23e21edf1b232b32/ A simple list comprehension question leads to discuss mutability, identity, and Cousin Chuy's Super-Burritos: http://groups.google.com/group/comp.lang.python/t/17cfd91ece6ed54f/ Combining map, zip and filter into an equivalent list comprehension: http://groups.google.com/group/comp.lang.python/t/259976305282b0c0?q=map timeit: make sure you actually measure what you want, and not other code: http://groups.google.com/group/comp.lang.python/t/7ecae76e11f720ee/ Why do compiled C extensions on Windows use '.pyd' in their name instead of '.dll'? http://groups.google.com/group/comp.lang.python/t/c8d93871cc5f31dc/ Pyfora, a web forum about Python, was lauched recently. Will this fragment the community? http://groups.google.com/group/comp.lang.python/t/e6e0d99c995da697/ A guy's future book on programming Python doesn't fit standard practice (a long thread!): http://groups.google.com/group/comp.lang.python/t/6918784f36d147d2/ ======================================================================== Everything Python-related you want is probably one or two clicks away in these pages: Python.org's Python Language Website is the traditional center of Pythonia http://www.python.org Notice especially the master FAQ http://www.python.org/doc/FAQ.html PythonWare complements the digest you're reading with the marvelous daily python url http://www.pythonware.com/daily Just beginning with Python? This page is a great place to start: http://wiki.python.org/moin/BeginnersGuide/Programmers The Python Papers aims to publish "the efforts of Python enthusiasts": http://pythonpapers.org/ The Python Magazine is a technical monthly devoted to Python: http://pythonmagazine.com Readers have recommended the "Planet" site: http://planet.python.org comp.lang.python.announce announces new Python software. Be sure to scan this newsgroup weekly. http://groups.google.com/group/comp.lang.python.announce/topics Python411 indexes "podcasts ... to help people learn Python ..." Updates appear more-than-weekly: http://www.awaretek.com/python/index.html The Python Package Index catalogues packages. http://www.python.org/pypi/ Much of Python's real work takes place on Special-Interest Group mailing lists http://www.python.org/sigs/ Python Success Stories--from air-traffic control to on-line match-making--can inspire you or decision-makers to whom you're subject with a vision of what the language makes practical. http://www.pythonology.com/success The Python Software Foundation (PSF) has replaced the Python Consortium as an independent nexus of activity. It has official responsibility for Python's development and maintenance. http://www.python.org/psf/ Among the ways you can support PSF is with a donation. http://www.python.org/psf/donations/ The Summary of Python Tracker Issues is an automatically generated report summarizing new bugs, closed ones, and patch submissions. http://search.gmane.org/?author=status%40bugs.python.org&group=gmane.comp.python.devel&sort=date Although unmaintained since 2002, the Cetus collection of Python hyperlinks retains a few gems. http://www.cetus-links.org/oo_python.html Python FAQTS http://python.faqts.com/ The Cookbook is a collaborative effort to capture useful and interesting recipes. http://code.activestate.com/recipes/langs/python/ Many Python conferences around the world are in preparation. Watch this space for links to them. Among several Python-oriented RSS/RDF feeds available, see: http://www.python.org/channews.rdf For more, see: http://www.syndic8.com/feedlist.php?ShowMatch=python&ShowStatus=all The old Python "To-Do List" now lives principally in a SourceForge reincarnation. http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse http://www.python.org/dev/peps/pep-0042/ del.icio.us presents an intriguing approach to reference commentary. It already aggregates quite a bit of Python intelligence. http://del.icio.us/tag/python Enjoy the *Python Magazine*. http://pymag.phparch.com/ *Py: the Journal of the Python Language* http://www.pyzine.com Dr.Dobb's Portal is another source of Python news and articles: http://www.ddj.com/TechSearch/searchResults.jhtml?queryText=python and Python articles regularly appear at IBM DeveloperWorks: http://www.ibm.com/developerworks/search/searchResults.jsp?searchSite=dW&searchScope=dW&encodedQuery=python&rankprofile=8 Previous - (U)se the (R)esource, (L)uke! - messages are listed here: http://search.gmane.org/?query=python+URL+weekly+news+links&group=gmane.comp.python.general&sort=date http://groups.google.com/groups/search?q=Python-URL!+group%3Acomp.lang.python&start=0&scoring=d& http://lwn.net/Search/DoSearch?words=python-url&ctype3=yes&cat_25=yes There is *not* an RSS for "Python-URL!"--at least not yet. Arguments for and against are occasionally entertained. Suggestions/corrections for next week's posting are always welcome. E-mail to should get through. To receive a new issue of this posting in e-mail each Monday morning (approximately), ask to subscribe. Mention "Python-URL!". Write to the same address to unsubscribe. -- The Python-URL! Team-- Phaseit, Inc. (http://phaseit.net) is pleased to participate in and sponsor the "Python-URL!" project. Watch this space for upcoming news about posting archives. From robince at gmail.com Tue Nov 3 11:29:49 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 16:29:49 +0000 Subject: [Numpy-discussion] strange performance on mac 2.5/2.6 32/64 bit Message-ID: <2d5132a50911030829o3766f48t239c77173c41f4c7@mail.gmail.com> Hi, I'm not sure if this is of much interest but it's been really puzzling me so I thought I'd ask. In an earlier post I described how I was surprised a simple f2py wrapped fortran bincount was 4x faster than np.bincount - but that differential only seemed to be on my mac; on moving to linux they both took more or less the same time. I'm trying to work out if it is worth moving some of my bottlenecks to fortran (most of which are np builtins). So far it looks like it is - but only on my mac and only 32bit (see below). Well the only explanation I thought was that the gcc-4.0 used to build numpy on a mac didn't perform so well, so after upgrading to snow leopard I've been trying to look at this again. I was hoping I could get the equivalent performance on my mac, like on linux, which would result in the np c stuff being a couple of times faster. So far, with Python 2.6.3 in 64 bit - numpy seems to be significantly slower and my fortran code _much_ slower - even from the same compiler. Can anyone help me understand what is going on? I have only been able to build 32 bit numpy against 2.5.4 with apple gcc-4.0 and 64 bit numpy against 2.6.3 universal with gcc-4.2. I haven't been able to get a numpy I can import on 2.6.3 in 32 bit mode ( http://projects.scipy.org/numpy/ticket/1221 ). Here are the results for python.org 32 bit 2.5.4, numpy compiled with apple gcc 4.0, f2py using att gfortran 4.2: In [2]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) 1 loops, best of 3: 2.86 s per loop In [3]: x = np.random.random_integers(0,1023,100000000).astype(int) In [4]: timeit np.bincount(x) 1 loops, best of 3: 435 ms per loop In [6]: timeit gf42.bincount(x,1024) 10 loops, best of 3: 129 ms per loop In [7]: np.__version__ Out[7]: '1.4.0.dev7618' And for self-built (apple gcc 4.2) 64 bit 2.6.3, numpy compiled with apple gcc 4.2, f2py using the same att gfortran 4.2: In [3]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) 1 loops, best of 3: 3.91 s per loop # 37% slower than 32bit In [4]: x = np.random.random_integers(0,1023,100000000).astype(int) In [5]: timeit np.bincount(x) 1 loops, best of 3: 582 ms per loop # 34 % slower than 32 bit In [8]: timeit gf42_64.bincount(x,1024) 1 loops, best of 3: 803 ms per loop # 522% slower than 32 bit So why is there this big difference in performance? I'd really like to know why the fortran compiled with the same compiler is so much slower in 64 bit mode. As far as I can tell the flags used are the same. Also why is numpy slower. I was surprised the I was able to import the 64 bit universal module built with f2py from 2.6 inside 32 bit 3.5 and there it was quick again - so it seems the x64_64 code generated by the fortran compiler is much slower (rather than any wrappers or such). I tried using some more recent gfortrans from macports - but could only use them to build modules against the 64 bit python/numpy since I couldn't find a way to get f2py to force 32 bit output. But the performance was more or less the same (always several times slower the 32 bit att gfortran). Any advice appreciated. Cheers Robin -------- subroutine bincount (x,c,n,m) implicit none integer, intent(in) :: n,m integer, dimension(0:n-1), intent(in) :: x integer, dimension(0:m-1), intent(out) :: c integer :: i c = 0 do i = 0, n-1 c(x(i)) = c(x(i)) + 1 end do end From d.l.goldsmith at gmail.com Tue Nov 3 11:41:34 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Tue, 3 Nov 2009 08:41:34 -0800 Subject: [Numpy-discussion] Ignorance question In-Reply-To: References: <45d1ab480911011758h674008ccp9b264951d2cff37d@mail.gmail.com> <1cd32cbb0911011826i4df0d240x7b3dcb202e1be49c@mail.gmail.com> Message-ID: <45d1ab480911030841v7b4878d7kab23e0dbeabd7ec4@mail.gmail.com> Thanks, that's what I wanted to know! On 11/1/09, Charles R Harris wrote: > On Sun, Nov 1, 2009 at 7:26 PM, wrote: > >> On Sun, Nov 1, 2009 at 9:58 PM, David Goldsmith >> wrote: >> > I Googled scipy brownian and the top hit was the doc for >> numpy.random.wald, >> > but said doc has a "tone" that suggests there are more "sophisticated" >> ways >> > to generate a random Brownian signal? Or is wald indeed SotA? Thanks! >> > >> > DG >> >> Do you mean generating a random sample of a Brownian motion? The >> standard approach, I have seen, is just cumsum of random normals, with >> time steps depending on the usage, e.g. >> > > Oddly enough, if you divide an interval into an infinite integer number of > samples this also works for the theory side ;) Euler would understand, but > such odd constructions with extended number systems have fallen out of > favour... > > Chuck > From dwf at cs.toronto.edu Tue Nov 3 11:43:25 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Tue, 3 Nov 2009 11:43:25 -0500 Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: References: Message-ID: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> On 2-Nov-09, at 11:35 PM, Thomas Robitaille wrote: > But if I want to specify the data types: > > np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8), > ('b',np.str)]) > > the string field is set to a length of zero: > > rec.array([(1, ''), (2, '')], dtype=[('a', '|i1'), ('b', '|S0')]) > > I need to specify datatypes for all numerical types since I care about > int8/16/32, etc, but I would like to benefit from the auto string > length detection that works if I don't specify datatypes. I tried > replacing np.str by None but no luck. I know I can specify '|S5' for > example, but I don't know in advance what the string length should be > set to. This is a limitation of the way the dtype code works, and AFAIK there's no easy fix. In some code I wrote recently I had to loop through the entire list of records i.e. max(len(foo[2]) for foo in records). David From pgmdevlist at gmail.com Tue Nov 3 12:40:00 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 3 Nov 2009 12:40:00 -0500 Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> References: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> Message-ID: <2DB988E6-1851-44BA-9C9D-29AEE0A283D3@gmail.com> On Nov 3, 2009, at 11:43 AM, David Warde-Farley wrote: > On 2-Nov-09, at 11:35 PM, Thomas Robitaille wrote: > >> But if I want to specify the data types: >> >> np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8), >> ('b',np.str)]) >> >> the string field is set to a length of zero: >> >> rec.array([(1, ''), (2, '')], dtype=[('a', '|i1'), ('b', '|S0')]) >> >> I need to specify datatypes for all numerical types since I care >> about >> int8/16/32, etc, but I would like to benefit from the auto string >> length detection that works if I don't specify datatypes. I tried >> replacing np.str by None but no luck. I know I can specify '|S5' for >> example, but I don't know in advance what the string length should be >> set to. > > This is a limitation of the way the dtype code works, and AFAIK > there's no easy fix. In some code I wrote recently I had to loop > through the entire list of records i.e. max(len(foo[2]) for foo in > records). As a workwaround, perhaps you could use np.object instead of np.str while defining your array. You can then get the maximum string length by looping, as David suggested, and then use .astype to transform your array... From robince at gmail.com Tue Nov 3 13:14:26 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 18:14:26 +0000 Subject: [Numpy-discussion] strange performance on mac 2.5/2.6 32/64 bit In-Reply-To: <2d5132a50911030829o3766f48t239c77173c41f4c7@mail.gmail.com> References: <2d5132a50911030829o3766f48t239c77173c41f4c7@mail.gmail.com> Message-ID: <2d5132a50911031014y173630e1m5f4bc13ad3320727@mail.gmail.com> After some more pootling about I figured out a lot of the performance loss comes from using 32 bit integers by default when compiles 64 bit. I asked this question on stackoverflow: http://stackoverflow.com/questions/1668899/fortran-32-bit-64-bit-performance-portability is there any way to use fortran with f2py from python in a way that doesn't require the code to be changed depending on platform? Or should I just pack it all in and use weave? Robin On Tue, Nov 3, 2009 at 4:29 PM, Robin wrote: > Hi, > > I'm not sure if this is of much interest but it's been really puzzling > me so I thought I'd ask. > > In an earlier post I described how I was surprised a simple f2py > wrapped fortran bincount was 4x faster than np.bincount - but that > differential only seemed to be on my mac; on moving to linux they both > took more or less the same time. I'm trying to work out if it is worth > moving some of my bottlenecks to fortran (most of which are np > builtins). So far it looks like it is - but only on my mac and only > 32bit (see below). > Well the only explanation I thought was that the gcc-4.0 used to build > numpy on a mac didn't perform so well, so after upgrading to snow > leopard I've been trying to look at this again. I was hoping I could > get the equivalent performance on my mac, like on linux, which would > result in the np c stuff being a couple of times faster. > > So far, with Python 2.6.3 in 64 bit - numpy seems to be significantly > slower and my fortran code _much_ slower - even from the same > compiler. Can anyone help me understand what is going on? > > I have only been able to build 32 bit numpy against 2.5.4 with apple > gcc-4.0 and 64 bit numpy against 2.6.3 universal with gcc-4.2. I > haven't been able to get a numpy I can import on 2.6.3 in 32 bit mode > ( http://projects.scipy.org/numpy/ticket/1221 ). > > Here are the results for python.org 32 bit 2.5.4, numpy compiled with > apple gcc 4.0, f2py using att gfortran 4.2: > In [2]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) > 1 loops, best of 3: 2.86 s per loop > In [3]: x = np.random.random_integers(0,1023,100000000).astype(int) > In [4]: timeit np.bincount(x) > 1 loops, best of 3: 435 ms per loop > In [6]: timeit gf42.bincount(x,1024) > 10 loops, best of 3: 129 ms per loop > In [7]: np.__version__ > Out[7]: '1.4.0.dev7618' > > And for self-built (apple gcc 4.2) 64 bit 2.6.3, numpy compiled with > apple gcc 4.2, f2py using the same att gfortran 4.2: > In [3]: timeit x = np.random.random_integers(0,1023,100000000).astype(int) > 1 loops, best of 3: 3.91 s per loop ?# 37% slower than 32bit > In [4]: x = np.random.random_integers(0,1023,100000000).astype(int) > In [5]: timeit np.bincount(x) > 1 loops, best of 3: 582 ms per loop # 34 % slower than 32 bit > In [8]: timeit gf42_64.bincount(x,1024) > 1 loops, best of 3: 803 ms per loop # 522% slower than 32 bit > > > So why is there this big difference in performance? I'd really like to > know why the fortran compiled with the same compiler is so much slower > in 64 bit mode. As far as I can tell the flags used are the same. Also > why is numpy slower. I was surprised the I was able to import the 64 > bit universal module built with f2py from 2.6 inside 32 bit 3.5 and > there it was quick again - so it seems the x64_64 code generated by > the fortran compiler is much slower (rather than any wrappers or > such). > > I tried using some more recent gfortrans from macports - but could > only use them to build modules against the 64 bit python/numpy since I > couldn't find a way to get f2py to force 32 bit output. But the > performance was more or less the same (always several times slower the > 32 bit att gfortran). > > Any advice appreciated. > > Cheers > > Robin > > -------- > subroutine bincount (x,c,n,m) > ? ?implicit none > ? ?integer, intent(in) :: n,m > ? ?integer, dimension(0:n-1), intent(in) :: x > ? ?integer, dimension(0:m-1), intent(out) :: c > ? ?integer :: i > > ? ?c = 0 > ? ?do i = 0, n-1 > ? ? ? ?c(x(i)) = c(x(i)) + 1 > ? ?end do > end > From robince at gmail.com Tue Nov 3 13:47:00 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 18:47:00 +0000 Subject: [Numpy-discussion] f2py error with -DF2PY_REPORT_ATEXIT Message-ID: <2d5132a50911031047p2be5f50ci9a231038cb67b61@mail.gmail.com> Hi, When I try to build a fortran module with f2py from '1.4.0.dev7618' with gfortran 4.2.3 from att.com and apple gcc 4.2 on snow leopard I get the following error when I try to use -DF2PY_REPORT_ATEXIT: In [1]: import gfint --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /Users/robince/fortran/ in () ImportError: dlopen(./gfint.so, 2): Symbol not found: _on_exit Referenced from: /Users/robince/fortran/gfint.so Expected in: dynamic lookup Is numpy trac the place for f2py tickets? Or is there an obvious fix. Inspecting gfint.so shows the same symbols for both architectures, and _on_exit is listed there with a U which I guess means undefined. Cheers Robin From robince at gmail.com Tue Nov 3 13:52:50 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 18:52:50 +0000 Subject: [Numpy-discussion] f2py error with -DF2PY_REPORT_ATEXIT In-Reply-To: <2d5132a50911031047p2be5f50ci9a231038cb67b61@mail.gmail.com> References: <2d5132a50911031047p2be5f50ci9a231038cb67b61@mail.gmail.com> Message-ID: <2d5132a50911031052w6ec89b41i1acce35e240a2d36@mail.gmail.com> Just noticed this is only supported on linux - sorry for the noise (not having a very good day today!) Robin On Tue, Nov 3, 2009 at 6:47 PM, Robin wrote: > Hi, > > When I try to build a fortran module with f2py from '1.4.0.dev7618' > with gfortran 4.2.3 from att.com and apple gcc 4.2 on snow leopard I > get the following error when I try to use -DF2PY_REPORT_ATEXIT: > > In [1]: import gfint > --------------------------------------------------------------------------- > ImportError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call last) > > /Users/robince/fortran/ in () > > ImportError: dlopen(./gfint.so, 2): Symbol not found: _on_exit > ?Referenced from: /Users/robince/fortran/gfint.so > ?Expected in: dynamic lookup > > Is numpy trac the place for f2py tickets? Or is there an obvious fix. > Inspecting gfint.so shows the same symbols for both architectures, and > _on_exit is listed there with a U which I guess means undefined. > > Cheers > > Robin > From robince at gmail.com Tue Nov 3 14:12:02 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 19:12:02 +0000 Subject: [Numpy-discussion] strange performance on mac 2.5/2.6 32/64 bit In-Reply-To: <2d5132a50911031014y173630e1m5f4bc13ad3320727@mail.gmail.com> References: <2d5132a50911030829o3766f48t239c77173c41f4c7@mail.gmail.com> <2d5132a50911031014y173630e1m5f4bc13ad3320727@mail.gmail.com> Message-ID: <2d5132a50911031112m30a575bxaf14c4ceefbc7bfc@mail.gmail.com> On Tue, Nov 3, 2009 at 6:14 PM, Robin wrote: > After some more pootling about I figured out a lot of the performance > loss comes from using 32 bit integers by default when compiles 64 bit. > I asked this question on stackoverflow: > http://stackoverflow.com/questions/1668899/fortran-32-bit-64-bit-performance-portability > > is there any way to use fortran with f2py from python in a way that > doesn't require the code to be changed depending on platform? Including the -DF2PY_REPORT_ON_ARRAY_COPY option showed that the big performance hit was from f2py copying the arrays to cast from 64 bit to 32 bit. Is there a recommended way to easily write fortran extensions that work on both 32bit and 64bit machines (something like using -fdefault-int-8 and f2py not casting on a 64 bit platform, and not using the option and not casting on a 32 bit platform)? Cheers Robin From fperez.net at gmail.com Tue Nov 3 14:28:55 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 3 Nov 2009 11:28:55 -0800 Subject: [Numpy-discussion] [ANN] For SF Bay Area residents: a discussion with Guido at the Berkeley Py4Science seminar Message-ID: Hi folks, if you reside in the San Francisco Bay Area, you may be interested in a meeting we'll be having tomorrow November 4 (2-4 pm), as part of our regular py4science meeting series. Guido van Rossum, the creator of the Python language, will visit for a session where we will first do a very rapid overview of a number of scientific projects that use Python (in a lightning talk format) and then we will have an open discussion with Guido with hopefully interesting questions going in both directions. The meeting is open to all, bring your questions! More details on this seminar series (including location) can be found here: https://cirl.berkeley.edu/view/Py4Science Cheers, f From robince at gmail.com Tue Nov 3 16:42:58 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 21:42:58 +0000 Subject: [Numpy-discussion] f2py-users list not working Message-ID: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> I just tried to send the message below to f2py-users - f2py-users at cens.ioc.ee, but delivery failed. Not sure where else to report this so hopefully here is ok. Cheers Robin ---------- Forwarded message ---------- From: Mail Delivery Subsystem Date: Tue, Nov 3, 2009 at 9:40 PM Subject: Delivery Status Notification (Failure) To: robince at gmail.com Delivery to the following recipient failed permanently: ? ? f2py-users at cens.ioc.ee Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 Unrouteable address (state 14). ----- Original message ----- Subject: writing module for 32/64 bit From: Robin To: f2py-users at cens.ioc.ee Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, I would like to write some subroutines in fortran involving integers and distribute them in a small package. ?Users might be on 32 bit or 64 bit. Is there an easy or recommended way to approach this? Ideally I would like to work with the native integer type - but if I use 'integer' in fortran it is always 32 bit and f2py converts input aways when numpy is 64 bit. If I use integer*8 in the code then its fine for 64 bit, but on 32 bit platforms, both f2py has to convert and its not the native integer size. What I (think) I'd like to do is be able to use the native platform integer type, like numpy does, and then not worry about. I found there are options like -fdefault-int-8 for gfortran, but when I add that stuff breaks (bus error) since I guess f2py doesn't know about it and is converting and passing 32 bits anyway. Is there any way around this, or do I need to maintain 2 different versions with different fortran code for 32bit/64bit? Or is it possible to acheive something like this with preprossor #ifdefs? (not sure how this works with fortran, or if f2py would be aware of it). Cheers Robin From charlesr.harris at gmail.com Tue Nov 3 17:05:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Nov 2009 15:05:29 -0700 Subject: [Numpy-discussion] f2py-users list not working In-Reply-To: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> References: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> Message-ID: On Tue, Nov 3, 2009 at 2:42 PM, Robin wrote: > I just tried to send the message below to f2py-users - > f2py-users at cens.ioc.ee, but delivery failed. > > Not sure where else to report this so hopefully here is ok. > I believe f2py development has moved elsewhere, the new mailing list seems to be at http://groups.google.com/group/f2py-dev?pli=1 . It doesn't look to be very active. Perhaps the summer additions to cython adding some fortran support might be useful to you. I don't know much about that but perhaps Dag will offer some advice. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Tue Nov 3 17:34:17 2009 From: robince at gmail.com (Robin) Date: Tue, 3 Nov 2009 22:34:17 +0000 Subject: [Numpy-discussion] f2py-users list not working In-Reply-To: References: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> Message-ID: <2d5132a50911031434l59aad34do1dcac81ceb6ab686@mail.gmail.com> On Tue, Nov 3, 2009 at 10:05 PM, Charles R Harris wrote: > I believe f2py development has moved elsewhere, the new mailing list seems > to be at http://groups.google.com/group/f2py-dev?pli=1 . It doesn't look to > be very active. Perhaps the summer additions to cython adding some fortran > support might be useful to you. I don't know much about that but perhaps Dag > will offer some advice. Thanks - from the archives of f2py-users http://cens.ioc.ee/pipermail/f2py-users/ it looks like there was activity as recently as October (15th) which is why I assumed it was still supposed to be working. Certainly more traffic more recently than the google group. Cheers Robin From brennan.williams at visualreservoir.com Tue Nov 3 20:50:20 2009 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Wed, 04 Nov 2009 14:50:20 +1300 Subject: [Numpy-discussion] using FortranFile to read data from a binary Fortran file Message-ID: <4AF0DDDC.3080405@visualreservoir.com> Hi I'm using FortranFile on 32 bit XP. The first record in the file has both 32 bit integer values and double precision float values. I've used readInts('i') to read the data into what I presume is a 32-bit integer numpy array. Items 0 and 1 of the array are the first double precision value. What's the best way to convert/cast these to get the double precision float value? I assume I need to do some sort of dpval=ival.astype('float64') so far.... f= FortranFile(fname,endian='<') ihdr=f.readInts('i') Brennan From brennan.williams at visualreservoir.com Tue Nov 3 21:18:13 2009 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Wed, 04 Nov 2009 15:18:13 +1300 Subject: [Numpy-discussion] using FortranFile to read data from a binary Fortran file In-Reply-To: <4AF0DDDC.3080405@visualreservoir.com> References: <4AF0DDDC.3080405@visualreservoir.com> Message-ID: <4AF0E465.3000607@visualreservoir.com> Brennan Williams wrote: > Hi > > I'm using FortranFile on 32 bit XP. > > The first record in the file has both 32 bit integer values and double > precision float values. > > I've used readInts('i') to read the data into what I presume is a 32-bit > integer numpy array. > > Items 0 and 1 of the array are the first double precision value. > > What's the best way to convert/cast these to get the double precision > float value? > > I assume I need to do some sort of dpval=ival.astype('float64') > > so far.... > > f= FortranFile(fname,endian='<') > ihdr=f.readInts('i') > > ok I took a closer look at FortranFile and I'm now doing the following. Note that the first line in the file I'm reading has two double precision reals/floats followed by 8 32 bit integers. f=FortranFile(fullfilename,endian='<') if f: hdr=f.readString() print 'hdr=',hdr print 'len=',len(hdr) t=struct.unpack('<2d',hdr[0:16]) print 't=',t i=struct.unpack('<8i',hdr[16:]) print 'i=',i This gives me... len=48 t=(0.0,2000.0) i=(0,0,0,5,0,0,1,213) which is correct. So is that the best way to do it, i.e. if I have a line of mixed data types, use readString and then do my own unpacking? Brennan From gbsuar at gmail.com Tue Nov 3 22:29:32 2009 From: gbsuar at gmail.com (Gabriel) Date: Wed, 4 Nov 2009 00:29:32 -0300 Subject: [Numpy-discussion] fft trouble In-Reply-To: <783F32138ED65D4A9CF016980481B6BF01CB57A7@CORRE.ad.smhi.se> References: <91906ddf0911030454u44c65bf7g3b1b56675f004a12@mail.gmail.com> <91906ddf0911030516i4328f8bep47382e5614022810@mail.gmail.com> <783F32138ED65D4A9CF016980481B6BF01CB57A7@CORRE.ad.smhi.se> Message-ID: <91906ddf0911031929o48ed0fd7p79837fde9fe7eea3@mail.gmail.com> Yes!!! it was... I didn't realize. Thank you very much 2009/11/3 Raspaud Martin > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Gabriel wrote: > > Hi it`s my first time. > > > > I have a trouble whit fft. > > For example: > > s = sin(wt) > > > > f = abs(fft(s)) > > > > In f I have the module the fft(s) which should be equal to 1 for w but > > it's 1000 for w. All values is multiplied by 1000. > > > > Someone know why? > > Hi, > > I don?t know if this is numpy related, but I know a little bit about fft > in general, and it looks like you did not normalize the result. > > The documentation does not specify that the resulting array is > normalized, so you should do : > > s = sin(wt) > f = abs(fft(s) / len(s)) > > I guess your wt has a length of 1000 ? > > Regards, > > Martin > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJK8DKdAAoJEBdvyODiyJI4zCkH/Ahe8dh+zfVK6nTWeaKqjcPy > fqpn2PSvb3l0z+dfOxtmkv9DB24ntJadQ+H26t1VeK5HcZxSunncb8vfGsrWtIZN > Z773aE7ZtZZpIYl+wbIQKCiAQ6gZFUbuK4qJxFl0iLGlQXcGjM4EjqO8nDhKHuOr > hpjhtfp56s35e3rXJFT7Za4XNt17pGnMsk/91/X3/ACCjAJNH1F1KZl7yULZqEUj > KBY62UMWA136xOi2bFKacMW7Ir4qhSVEZnkA0LN2VArAZ4RYV7290dARFZ+N78+V > W4sX4DBPECMZ3vGl7y+mjfMqExk9GDC6hpV8MiVfidMeFhYV5+PgNodE5Ectekg= > =9PY4 > -----END PGP SIGNATURE----- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Gabriel Antes de imprimir, pens? en tu responsabilidad y compromiso con el MEDIO AMBIENTE. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Wed Nov 4 00:55:33 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Nov 2009 00:55:33 -0500 Subject: [Numpy-discussion] (apparent) infinite loop in LAPACK/ATLAS Message-ID: Hi all (mostly David C. since he probably knows all this horrible stuff), I noticed on my new laptop (with an Atom N280 in it) that when I run numpy.test() about the 34th test would loop, seemingly forever. Finding this a little odd, I tried an SVD on a small matrix and observed the same behaviour, narrowed it down with gdb to what seems to be an infinite loop involving dlamc1_ and dlamc3_, which I gather are routines for converting from row-major to column-major. I should mention that I built ATLAS (3.9.7, because that's the tarball I had around) and LAPACK (3.1.1) from source. Oh, and this is on Ubuntu 9.10, Karmic Koala, using gcc 4.4.1 (with the exception of the ATLAS kernels which are using gcc 4.2.4 for the reasons stated in the install document). I came upon this mention of a similar problem ( https://projects.coin-or.org/BuildTools/ticket/66 ), so far. I'm going to try using the flag they suggest but it'll be a while before I have a verdict on whether it's fixed or not, so I'm soliciting any other ideas as to the cause of/solution to such an issue. Thanks in advance, David From cournape at gmail.com Wed Nov 4 02:50:42 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Nov 2009 16:50:42 +0900 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: <5b8d13220911032350w7df4248bj2059e8423d124f54@mail.gmail.com> On Mon, Nov 2, 2009 at 9:38 PM, Pauli Virtanen wrote: > > Can we get the complex functions to npy_math for 1.4.0: could be useful > for the next Scipy? Ok, the complex math stuff is in the trunk now. I have not thoroughly tested it again, but since I tested the former branch on most platforms, any remaining issues should be quick to fix. I hope you will be able to put your own tests in the trunk easily - I mostly need to test this on windows, as that's the major platform without any C99 complex support and which will exercise the code in the branch. cheers, David From cournape at gmail.com Wed Nov 4 04:23:22 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Nov 2009 18:23:22 +0900 Subject: [Numpy-discussion] (apparent) infinite loop in LAPACK/ATLAS In-Reply-To: References: Message-ID: <5b8d13220911040123l234d7662x4eb2a3194d22df8b@mail.gmail.com> On Wed, Nov 4, 2009 at 2:55 PM, David Warde-Farley wrote: > Hi all (mostly David C. since he probably knows all this horrible > stuff), > > I noticed on my new laptop (with an Atom N280 in it) that when I run > numpy.test() about the 34th test would loop, seemingly forever. > > Finding this a little odd, I tried an SVD on a small matrix and > observed the same behaviour, narrowed it down with gdb to what seems > to be an infinite loop involving dlamc1_ and dlamc3_, Did you compile them without any optimized as suggested in the makefiles ? NOOPT should not contain -O option David From ralf.gommers at googlemail.com Wed Nov 4 04:37:26 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 Nov 2009 10:37:26 +0100 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Mon, Nov 2, 2009 at 9:29 AM, David Cournapeau wrote: > Hi, > > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > - Documentation for datetime, in particular for the public C API > - Snow Leopard issues, if any > > Otherwise, I think there has been quite a lot of new features. If > people want to add new functionalities or features, please do it soon, > > It would be good if we could also have one more merge of the work in the doc editor (close to 300 new/changed docstrings now). I can have it all reviewed by the 13th. Unless you object, I'd also like to include the distutils docs. Complete docs with some possible minor inaccuracies is better than no docs. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Wed Nov 4 05:05:41 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 4 Nov 2009 02:05:41 -0800 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 1:37 AM, Ralf Gommers wrote: > It would be good if we could also have one more merge of the work in the doc > editor (close to 300 new/changed docstrings now). I can have it all reviewed > by the 13th. That would be great. Thanks for taking care of that. > Unless you object, I'd also like to include the distutils docs. Complete > docs with some possible minor inaccuracies is better than no docs. +1 -- Jarrod Millman Helen Wills Neuroscience Institute 10 Giannini Hall, UC Berkeley http://cirl.berkeley.edu/ From dwf at cs.toronto.edu Wed Nov 4 05:09:22 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Nov 2009 05:09:22 -0500 Subject: [Numpy-discussion] (apparent) infinite loop in LAPACK/ATLAS In-Reply-To: <5b8d13220911040123l234d7662x4eb2a3194d22df8b@mail.gmail.com> References: <5b8d13220911040123l234d7662x4eb2a3194d22df8b@mail.gmail.com> Message-ID: <556CC63C-1F61-44BF-AD75-28D5C84E2EA0@cs.toronto.edu> Hi David, On 4-Nov-09, at 4:23 AM, David Cournapeau wrote: > Did you compile them without any optimized as suggested in the > makefiles ? NOOPT should not contain -O option Yup, it contained -O0 -fPIC (-O0 I think is in fact more strict than having no -O option?). Have you seen this problem with broken NOOPT? I'm not certain yet, but it looks like some combination of gcc 4.4 and the Atom microarchitecture reintroduces this old bug, as "-ffloat- store" seems to have fixed it. numpy.test() runs through just fine once I recompiled LAPACK and then ATLAS as well (just rebuilding the lapack_LINUX.a and then running 'make' in the ATLAS obj dir didn't seem to do it, so I had to wait through several hours of tuning again). For anyone who might stumble upon this problem again, just to be safe I added -ffloat-store to both the flag lists in LAPACK's make.inc, and also used -Fa fi '-ffloat-store' when running configure with ATLAS. If this is indeed either a gcc 4.4 regression or an Atom-specific quirk it might start popping up more and more around here. David From gnurser at googlemail.com Wed Nov 4 05:59:10 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 4 Nov 2009 10:59:10 +0000 Subject: [Numpy-discussion] strange performance on mac 2.5/2.6 32/64 bit In-Reply-To: <2d5132a50911031112m30a575bxaf14c4ceefbc7bfc@mail.gmail.com> References: <2d5132a50911030829o3766f48t239c77173c41f4c7@mail.gmail.com> <2d5132a50911031014y173630e1m5f4bc13ad3320727@mail.gmail.com> <2d5132a50911031112m30a575bxaf14c4ceefbc7bfc@mail.gmail.com> Message-ID: <1d1e6ea70911040259s16ad102eo68dade249b23277d@mail.gmail.com> 2009/11/3 Robin : > On Tue, Nov 3, 2009 at 6:14 PM, Robin wrote: >> After some more pootling about I figured out a lot of the performance >> loss comes from using 32 bit integers by default when compiles 64 bit. >> I asked this question on stackoverflow: >> http://stackoverflow.com/questions/1668899/fortran-32-bit-64-bit-performance-portability This seems surprising -- our HPC fortran codes use 32 bit integers on 64 bit linux. Do you get a performance hit in a pure fortran program? Is it a problem with the gfortran compiler perhaps? >> is there any way to use fortran with f2py from python in a way that >> doesn't require the code to be changed depending on platform? > > Including the -DF2PY_REPORT_ON_ARRAY_COPY option showed that the big > performance hit was from f2py copying the arrays to cast from 64 bit > to 32 bit. Fortran 90 introduced the INTERFACE block, which allows you to use different variable types as arguments to what appears externally to be the same routine. It then feeds the arguments to the appropriate version of the routine. I don't think f2py supports this, but it would be really useful if it could. Regards, George Nurser. From reckoner at gmail.com Wed Nov 4 09:06:12 2009 From: reckoner at gmail.com (Reckoner) Date: Wed, 4 Nov 2009 06:06:12 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: Here's an example: On winxp 64-bit: Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import cPickle >>> a = numpy.eye(10) >>> cPickle.dump(a,open('from32bitxp.pkl','w')) >>> import numpy.core.multiarray >>> numpy.__version__ '1.0.4' >>> On linux 64 bit: Python 2.5.4 (r254:67916, Feb 5 2009, 19:52:35) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import cPickle >>> cPickle.load(open('from32bitxp.pkl')) Traceback (most recent call last): File "", line 1, in ImportError: No module named multiarray >>> numpy.__version__ '1.2.1' >>> import numpy.core.multiarray >>> Note that I transfer the from32bitxp file from the winxp32 machine to the linux host. Also, I've tried this with version 1.3 on winxp and get the same problem on the linux host. Here's more interesting info: On linux: >>> a = numpy.eye(10) >>> cPickle.dump(a,open('from64bitLinux.pkl','w')) upon transferring the file to winxp 32 and on winxp32: >>> cPickle.load(open('from64bitLinux.pkl')) See? No problem going from linux to winxp32; but problems going the other way. Please let me know if you need more info on this. Any help appreciated. On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey wrote: > On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: >> thanks for the suggestion! I will look into it. The other thing is >> that the numpy arrays in question are actually embedded in another >> object. When I convert the numpy arrays into plain lists, and then >> cPickle them, there is no problem with any of the larger objects. That >> is the way we are currently working around this issue. >> >> Thanks again. >> >> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>>> Anybody have any ideas here? >>>> >>>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>>> What's the best way to report a bug of this kind? >>>> >>>> Thanks! >>>> >>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>>> Robert Kern wrote: >>>>>> You can import numpy.core.multiarray on both machines? >>>>> >>>>> Yes. For each machine separately, you can cPickle files with numpy >>>>> arrays without problems loading/dumping. The problem comes from >>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>>> back and forth between the two machines. In fact, we currently get >>>>> around this problem by converting the numpy arrays to lists, >>>>> transferring them, and then re-numpy-ing them on the respective hosts >>>>> >>>>> thanks. >>>>> >>>>> >>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>>> Hi, >>>>>> >>>>>> % python -c 'import numpy.core.multiarray' >>>>>> >>>>>> works just fine, but when I try to load a file that I have transferred >>>>>> from another machine running Windows to one running Linux, I get: >>>>>> >>>>>> % python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>>> >>>>>> Traceback (most recent call last): >>>>>> File "", line 1, in >>>>>> ImportError: No module named multiarray >>>>>> >>>>>> otherwise, cPickle works normally when transferring files that *do* >>>>>> not contain numpy arrays. >>>>>> >>>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>>> change versions, by the way, since this is the version that my working >>>>>> group has decided on to standardize on for this effort. >>>>>> >>>>>> >>>>>> Any help appreciated. >>>>>> >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >>> Have you have tried the other Cookbook approaches: >>> http://www.scipy.org/Cookbook/InputOutput >>> Like using numpy's own array io functions - load/save(z)? >>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >>> different numpy versions) >>> >>> Bruce >>> _______________________________________________ > > Can you provide you provide a small self-contained example of the > problem including object creation especially as your example does not > import numpy? > > Really you have to start at the beginning (like pickling and > transferring numpy arrays) and then increase the complexity to include > the object. > > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gnurser at googlemail.com Wed Nov 4 09:38:40 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 4 Nov 2009 14:38:40 +0000 Subject: [Numpy-discussion] f2py-users list not working In-Reply-To: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> References: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> Message-ID: <1d1e6ea70911040638k75f9d21el9094d31236b7fa82@mail.gmail.com> Fortran can accept preprocessor directives, but f2py cannot. You first need to preprocess a .F (or .F90) file to create a .f (or .f90) file which you then pass to f2py The way I preprocess the .F file is to have statements like int int*INTSIZE :: i,j,k So preprocess file.F e.g. in gfortran with gfortran -E -DINTSIZE=8 file.F -o outdir/file.f The outdir is necessary in a case-insensitive file system (like default mac OSX) to prevent the .f files overwriting the .F file. Alternatively, it may be possible to use some other suffix than .f, but I've not tried that. Then f2py file.f George Nurser. > ----- Original message ----- > Subject: writing module for 32/64 bit > From: Robin > To: f2py-users at cens.ioc.ee > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > > Hi, > > I would like to write some subroutines in fortran involving integers > and distribute them in a small package. ?Users might be on 32 bit or > 64 bit. > > Is there an easy or recommended way to approach this? Ideally I would > like to work with the native integer type - but if I use 'integer' in > fortran it is always 32 bit and f2py converts input aways when numpy > is 64 bit. If I use integer*8 in the code then its fine for 64 bit, > but on 32 bit platforms, both f2py has to convert and its not the > native integer size. > > What I (think) I'd like to do is be able to use the native platform > integer type, like numpy does, and then not worry about. I found there > are options like -fdefault-int-8 for gfortran, but when I add that > stuff breaks (bus error) since I guess f2py doesn't know about it and > is converting and passing 32 bits anyway. > > Is there any way around this, or do I need to maintain 2 different > versions with different fortran code for 32bit/64bit? Or is it > possible to acheive something like this with preprossor #ifdefs? (not > sure how this works with fortran, or if f2py would be aware of it). > > Cheers > > Robin > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jeanluc.menut at free.fr Wed Nov 4 09:52:15 2009 From: jeanluc.menut at free.fr (Jean-Luc Menut) Date: Wed, 04 Nov 2009 15:52:15 +0100 Subject: [Numpy-discussion] arrays comparison issue Message-ID: <4AF1951F.9030208@free.fr> Hello all, If if define 2 variables a and b by doing the following : on [5]: a Out[5]: array([ 1.7]) In [6]: b=array([0.8])+array([0.9]) In [7]: b Out[7]: array([ 1.7]) if I test the equality of a and b, instead to obatin True, I have : In [8]: a==b Out[8]: array([False], dtype=bool) I know it not related only to numpy but also to python. How do you tackle this problem ? (I also tried on IDL and it works well : I guess it truncate the number). Thanks , Jean-Luc From lev at columbia.edu Wed Nov 4 09:59:53 2009 From: lev at columbia.edu (Lev Givon) Date: Wed, 4 Nov 2009 09:59:53 -0500 Subject: [Numpy-discussion] arrays comparison issue In-Reply-To: <4AF1951F.9030208@free.fr> References: <4AF1951F.9030208@free.fr> Message-ID: <20091104145953.GI17673@localhost.columbia.edu> Received from Jean-Luc Menut on Wed, Nov 04, 2009 at 09:52:15AM EST: > Hello all, > > > > If if define 2 variables a and b by doing the following : > > on [5]: a > Out[5]: array([ 1.7]) > > In [6]: b=array([0.8])+array([0.9]) > > In [7]: b > Out[7]: array([ 1.7]) > > > if I test the equality of a and b, instead to obatin True, I have : > In [8]: a==b > Out[8]: array([False], dtype=bool) > > I know it not related only to numpy but also to python. > How do you tackle this problem ? > > (I also tried on IDL and it works well : I guess it truncate the number). > > Thanks , Try using the numpy.allclose function (with a suitable tolerance) to compare the arrays. L.G. From cournape at gmail.com Wed Nov 4 10:07:16 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 00:07:16 +0900 Subject: [Numpy-discussion] arrays comparison issue In-Reply-To: <4AF1951F.9030208@free.fr> References: <4AF1951F.9030208@free.fr> Message-ID: <5b8d13220911040707j60b5cd49y9bc744e23ef79563@mail.gmail.com> On Wed, Nov 4, 2009 at 11:52 PM, Jean-Luc Menut wrote: > Hello all, > > > > If if define 2 variables a and b by doing the following : > > on [5]: a > Out[5]: array([ 1.7]) > > In [6]: b=array([0.8])+array([0.9]) > > In [7]: b > Out[7]: array([ 1.7]) > > > if I test the equality of a and b, instead to obatin True, I have : You should never test for strict equality with floating point numbers except for particular situations. You should instead test whether they are close in some sense (up to N decimal, for example). David From charlesr.harris at gmail.com Wed Nov 4 10:17:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 08:17:36 -0700 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: On Wed, Nov 4, 2009 at 7:06 AM, Reckoner wrote: > Here's an example: > > On winxp 64-bit: > > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> import cPickle > >>> a = numpy.eye(10) > >>> cPickle.dump(a,open('from32bitxp.pkl','w')) > >>> import numpy.core.multiarray > >>> numpy.__version__ > '1.0.4' > >>> > > On linux 64 bit: > > Python 2.5.4 (r254:67916, Feb 5 2009, 19:52:35) > [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> import cPickle > >>> cPickle.load(open('from32bitxp.pkl')) > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named multiarray > >>> numpy.__version__ > '1.2.1' > >>> import numpy.core.multiarray > >>> > > I wonder if this is a numpy version problem. Do you have a windows machine with a more recent version of numpy on it? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robince at gmail.com Wed Nov 4 10:53:01 2009 From: robince at gmail.com (Robin) Date: Wed, 4 Nov 2009 15:53:01 +0000 Subject: [Numpy-discussion] f2py-users list not working In-Reply-To: <1d1e6ea70911040638k75f9d21el9094d31236b7fa82@mail.gmail.com> References: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> <1d1e6ea70911040638k75f9d21el9094d31236b7fa82@mail.gmail.com> Message-ID: <2d5132a50911040753x69032c35r33fcb962290c8a88@mail.gmail.com> On Wed, Nov 4, 2009 at 2:38 PM, George Nurser wrote: > Fortran can accept preprocessor directives, but f2py cannot. > You first need to preprocess a .F (or .F90) file to create a .f (or > .f90) file which you then pass to f2py > The way I preprocess the .F file is to have statements like > int int*INTSIZE :: i,j,k > > So preprocess file.F e.g. in gfortran with > gfortran -E -DINTSIZE=8 file.F ?-o outdir/file.f > > The outdir is necessary in a case-insensitive file system (like > default mac OSX) to prevent the .f files overwriting the .F file. > Alternatively, it may be possible to use some other suffix than .f, > but I've not tried that. > > Then f2py file.f That's great thanks very much! It's more or less exactly what I was hoping for. I wonder if it's possible to get distutils to do the preprocess step from a setup.py script? Cheers Robin From reckoner at gmail.com Wed Nov 4 11:14:49 2009 From: reckoner at gmail.com (Reckoner) Date: Wed, 4 Nov 2009 08:14:49 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: Thanks for your reply. No. I just tried it with the latest Windows XP 32-bit version Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.3.0' Same result on the Linux side: ImportError: No module named multiarray. Thanks! On Wed, Nov 4, 2009 at 7:17 AM, Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 7:06 AM, Reckoner wrote: >> >> Here's an example: >> >> On winxp 64-bit: >> >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit >> (Intel)] on >> win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numpy >> >>> import cPickle >> >>> a = numpy.eye(10) >> >>> cPickle.dump(a,open('from32bitxp.pkl','w')) >> >>> import numpy.core.multiarray >> >>> numpy.__version__ >> '1.0.4' >> >>> >> >> On linux 64 bit: >> >> Python 2.5.4 (r254:67916, Feb 5 2009, 19:52:35) >> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numpy >> >>> import cPickle >> >>> cPickle.load(open('from32bitxp.pkl')) >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named multiarray >> >>> numpy.__version__ >> '1.2.1' >> >>> import numpy.core.multiarray >> >>> >> > > I wonder if this is a numpy version problem. Do you have a windows machine > with a more recent version of numpy on it? > > > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From rlw at stsci.edu Wed Nov 4 11:33:22 2009 From: rlw at stsci.edu (Rick White) Date: Wed, 4 Nov 2009 11:33:22 -0500 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 11 In-Reply-To: References: Message-ID: <1E766C81-0888-4200-832C-58E43B610D90@stsci.edu> The difference between IDL and numpy is that IDL uses single precision floats by default while numpy uses doubles. If you try it with doubles in IDL, you will see that it also returns false. As David Cournapeau said, you should not expect different floating point arithmetic operations to give exactly the same result. It's just luck if they do. E.g., in IDL you'll find that 0.1+0.6 is not equal to 0.7. That's because 0.1 and 0.6 are not exactly representable as floats. On Nov 4, 2009, Jean-Luc Menut wrote: > Hello all, > > > > If if define 2 variables a and b by doing the following : > > on [5]: a > Out[5]: array([ 1.7]) > > In [6]: b=array([0.8])+array([0.9]) > > In [7]: b > Out[7]: array([ 1.7]) > > > if I test the equality of a and b, instead to obatin True, I have : > In [8]: a==b > Out[8]: array([False], dtype=bool) > > I know it not related only to numpy but also to python. > How do you tackle this problem ? > > (I also tried on IDL and it works well : I guess it truncate the > number). > > Thanks , > > Jean-Luc From thomas.robitaille at gmail.com Wed Nov 4 11:35:25 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Wed, 4 Nov 2009 08:35:25 -0800 (PST) Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: <2DB988E6-1851-44BA-9C9D-29AEE0A283D3@gmail.com> References: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> <2DB988E6-1851-44BA-9C9D-29AEE0A283D3@gmail.com> Message-ID: <26199762.post@talk.nabble.com> Pierre GM-2 wrote: > > As a workwaround, perhaps you could use np.object instead of np.str > while defining your array. You can then get the maximum string length > by looping, as David suggested, and then use .astype to transform your > array... > I tried this: np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8),('b',np.object_)]) but I get a TypeError: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /Users/tom/ in () /Users/tom/Library/Python/2.6/site-packages/numpy/core/records.pyc in fromrecords(recList, dtype, shape, formats, names, titles, aligned, byteorder) 625 res = retval.view(recarray) 626 --> 627 res.dtype = sb.dtype((record, res.dtype)) 628 return res 629 /Users/tom/Library/Python/2.6/site-packages/numpy/core/records.pyc in __setattr__(self, attr, val) 432 if attr not in fielddict: 433 exctype, value = sys.exc_info()[:2] --> 434 raise exctype, value 435 else: 436 fielddict = ndarray.__getattribute__(self,'dtype').fields or {} TypeError: Cannot change data-type for object array. Is this a bug? Thanks, Thomas -- View this message in context: http://old.nabble.com/Automatic-string-length-in-recarray-tp26174810p26199762.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From jeanluc.menut at free.fr Wed Nov 4 12:13:51 2009 From: jeanluc.menut at free.fr (Jean-Luc Menut) Date: Wed, 04 Nov 2009 18:13:51 +0100 Subject: [Numpy-discussion] arrays comparison issue In-Reply-To: <4AF1951F.9030208@free.fr> References: <4AF1951F.9030208@free.fr> Message-ID: <4AF1B64F.60101@free.fr> thanks to everybody From mdroe at stsci.edu Wed Nov 4 12:15:54 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 04 Nov 2009 12:15:54 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken Message-ID: <4AF1B6CA.1060608@stsci.edu> I'm getting the following from r7603 on Solaris Sparc -- somehow related to not having a long double version of next after available. I realise not everyone has access to (or is dependent on) this platform, so I'm willing to help in whatever way I can, I'm just not sure I understand the change yet. compile options: '-Inumpy/core/include -Ibuild/src.solaris-2.8-sun4u-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/stsci/pyssg/Python-2.5.1/include/python2.5 -Ibuild/src.solaris-2.8-sun4u-2.5/numpy/core/src/multiarray -Ibuild/src.solaris-2.8-sun4u-2.5/numpy/core/src/umath -c' cc: numpy/core/src/npymath/ieee754.c "numpy/core/src/npymath/ieee754.c", line 172: #error: Needs nextafterl implementation for this platform cc: acomp failed for numpy/core/src/npymath/ieee754.c "numpy/core/src/npymath/ieee754.c", line 172: #error: Needs nextafterl implementation for this platform cc: acomp failed for numpy/core/src/npymath/ieee754.c Cheers, Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From dyamins at gmail.com Wed Nov 4 12:38:17 2009 From: dyamins at gmail.com (Dan Yamins) Date: Wed, 4 Nov 2009 12:38:17 -0500 Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> References: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> Message-ID: <15e4667e0911040938scff5bd9s5041b03262d4f1c4@mail.gmail.com> On Tue, Nov 3, 2009 at 11:43 AM, David Warde-Farley wrote: > On 2-Nov-09, at 11:35 PM, Thomas Robitaille wrote: > > > But if I want to specify the data types: > > > > np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8), > > ('b',np.str)]) > > > > the string field is set to a length of zero: > > > > rec.array([(1, ''), (2, '')], dtype=[('a', '|i1'), ('b', '|S0')]) > > > > I need to specify datatypes for all numerical types since I care about > > int8/16/32, etc, but I would like to benefit from the auto string > > length detection that works if I don't specify datatypes. I tried > > replacing np.str by None but no luck. I know I can specify '|S5' for > > example, but I don't know in advance what the string length should be > > set to. > > This is a limitation of the way the dtype code works, and AFAIK > there's no easy fix. In some code I wrote recently I had to loop > through the entire list of records i.e. max(len(foo[2]) for foo in > records). > > Not to shamelessly plug my own project ... but more robust string type detection is one of the features of Tabular ( http://bitbucket.org/elaine/tabular/), and is one of the (kinds of) reasons we wrote the package. Perhaps using Tabular could be useful to you? Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From nmb at wartburg.edu Wed Nov 4 12:43:33 2009 From: nmb at wartburg.edu (Neil Martinsen-Burrell) Date: Wed, 04 Nov 2009 11:43:33 -0600 Subject: [Numpy-discussion] using FortranFile to read data from a binary Fortran file In-Reply-To: <4AF0E465.3000607@visualreservoir.com> References: <4AF0DDDC.3080405@visualreservoir.com> <4AF0E465.3000607@visualreservoir.com> Message-ID: <4AF1BD45.4050608@wartburg.edu> On 2009-11-03 20:18 , Brennan Williams wrote: > ok I took a closer look at FortranFile and I'm now doing the following. > Note that the first line in the file I'm reading > has two double precision reals/floats followed by 8 32 bit integers. > > f=FortranFile(fullfilename,endian='<') > if f: > hdr=f.readString() > print 'hdr=',hdr > print 'len=',len(hdr) > t=struct.unpack('<2d',hdr[0:16]) > print 't=',t > i=struct.unpack('<8i',hdr[16:]) > print 'i=',i > > This gives me... > > len=48 > t=(0.0,2000.0) > i=(0,0,0,5,0,0,1,213) > > which is correct. > > So is that the best way to do it, i.e. if I have a line of mixed data > types, use readString and then do my own unpacking? That's correct. FortranFile works most readily with records (equivalent to individual write statements) that are of uniform types and precisions. This is a leftover from the way that my own Fortran codes were doing I/O. To solve the problem correctly in FortranFile requires a way to specify the sequence of types to expect in a single record. This could then give the equivalent of what you have done above, which could also be written in a single unpack call ans = struct.unpack('<2d8i',hdr); t=ans[:2];i=ans[2:]) The readString method just takes care of stripping off and error-checking the record length information that fortran unformatted I/O often uses. I don't have much opportunity to work on Fortran unformatted I/O these days, but I would gladly accept any contributions. -Neil From gnurser at googlemail.com Wed Nov 4 12:49:32 2009 From: gnurser at googlemail.com (George Nurser) Date: Wed, 4 Nov 2009 17:49:32 +0000 Subject: [Numpy-discussion] f2py-users list not working In-Reply-To: <2d5132a50911040753x69032c35r33fcb962290c8a88@mail.gmail.com> References: <2d5132a50911031342r648d9778n6da01ec485c28f5e@mail.gmail.com> <1d1e6ea70911040638k75f9d21el9094d31236b7fa82@mail.gmail.com> <2d5132a50911040753x69032c35r33fcb962290c8a88@mail.gmail.com> Message-ID: <1d1e6ea70911040949n196dc34i8d0fff2453b6f44b@mail.gmail.com> 2009/11/4 Robin : > On Wed, Nov 4, 2009 at 2:38 PM, George Nurser wrote: >> Fortran can accept preprocessor directives, but f2py cannot. >> You first need to preprocess a .F (or .F90) file to create a .f (or >> .f90) file which you then pass to f2py >> The way I preprocess the .F file is to have statements like >> int int*INTSIZE :: i,j,k >> >> So preprocess file.F e.g. in gfortran with >> gfortran -E -DINTSIZE=8 file.F ?-o outdir/file.f >> >> The outdir is necessary in a case-insensitive file system (like >> default mac OSX) to prevent the .f files overwriting the .F file. >> Alternatively, it may be possible to use some other suffix than .f, >> but I've not tried that. >> >> Then f2py file.f > > That's great thanks very much! It's more or less exactly what I was hoping for. > > I wonder if it's possible to get distutils to do the preprocess step > from a setup.py script? If it's just search-and-replace, of course python can do it directly.... --George From pgmdevlist at gmail.com Wed Nov 4 13:20:19 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 4 Nov 2009 13:20:19 -0500 Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: <26199762.post@talk.nabble.com> References: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> <2DB988E6-1851-44BA-9C9D-29AEE0A283D3@gmail.com> <26199762.post@talk.nabble.com> Message-ID: <4DC56369-6792-4649-97FE-1C56665EEE34@gmail.com> On Nov 4, 2009, at 11:35 AM, Thomas Robitaille wrote: > > > Pierre GM-2 wrote: >> >> As a workwaround, perhaps you could use np.object instead of np.str >> while defining your array. You can then get the maximum string length >> by looping, as David suggested, and then use .astype to transform >> your >> array... >> > > I tried this: > > np.rec.fromrecords([(1,'hello'),(2,'world')],dtype=[('a',np.int8), > ('b',np.object_)]) > > but I get a TypeError: Confirmed, it's a bug all right. Would you mind opening a ticket ? I'll try to take care of that in the next few days. From cournape at gmail.com Wed Nov 4 14:11:18 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 04:11:18 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF1B6CA.1060608@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> Message-ID: <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom wrote: > I'm getting the following from r7603 on Solaris Sparc -- somehow related > to not having a long double version of next after available. ?I realise > not everyone has access to (or is dependent on) this platform, so I'm > willing to help in whatever way I can, I'm just not sure I understand > the change yet. The only way to implement nextafter that I know of requires to know the exact representation of the floating point number, and long double is unfortunately platform dependent. What is the long double format on solaris sparc ? (big endian I suppose, but how many bits for the mantissa and exponent ? Does it follow IEER754 ?) cheers, David From thomas.robitaille at gmail.com Wed Nov 4 14:31:48 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Wed, 4 Nov 2009 11:31:48 -0800 (PST) Subject: [Numpy-discussion] Automatic string length in recarray In-Reply-To: <4DC56369-6792-4649-97FE-1C56665EEE34@gmail.com> References: <9EBF952F-FE32-4DBF-BAA9-1C768EDAB367@cs.toronto.edu> <2DB988E6-1851-44BA-9C9D-29AEE0A283D3@gmail.com> <26199762.post@talk.nabble.com> <4DC56369-6792-4649-97FE-1C56665EEE34@gmail.com> Message-ID: <26203110.post@talk.nabble.com> Pierre GM-2 wrote: > > Confirmed, it's a bug all right. Would you mind opening a ticket ? > I'll try to take care of that in the next few days. > Done - http://projects.scipy.org/numpy/ticket/1283 Thanks! Thomas -- View this message in context: http://old.nabble.com/Automatic-string-length-in-recarray-tp26174810p26203110.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From charlesr.harris at gmail.com Wed Nov 4 14:35:12 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 12:35:12 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 12:11 PM, David Cournapeau wrote: > On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom > wrote: > > I'm getting the following from r7603 on Solaris Sparc -- somehow related > > to not having a long double version of next after available. I realise > > not everyone has access to (or is dependent on) this platform, so I'm > > willing to help in whatever way I can, I'm just not sure I understand > > the change yet. > > The only way to implement nextafter that I know of requires to know > the exact representation of the floating point number, and long double > is unfortunately platform dependent. > > What is the long double format on solaris sparc ? (big endian I > suppose, but how many bits for the mantissa and exponent ? Does it > follow IEER754 ?) > > Long double on SPARC is quad precision, which I believe *is* in one of the ieee specs. In any case, it has a lot more precision than the extended precision found in Intel derived architectures. Hmm, I wonder what ia64 does? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 4 14:38:59 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 12:38:59 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 12:35 PM, Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 12:11 PM, David Cournapeau wrote: > >> On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom >> wrote: >> > I'm getting the following from r7603 on Solaris Sparc -- somehow related >> > to not having a long double version of next after available. I realise >> > not everyone has access to (or is dependent on) this platform, so I'm >> > willing to help in whatever way I can, I'm just not sure I understand >> > the change yet. >> >> The only way to implement nextafter that I know of requires to know >> the exact representation of the floating point number, and long double >> is unfortunately platform dependent. >> >> What is the long double format on solaris sparc ? (big endian I >> suppose, but how many bits for the mantissa and exponent ? Does it >> follow IEER754 ?) >> >> > Long double on SPARC is quad precision, which I believe *is* in one of the > ieee specs. In any case, it has a lot more precision than the extended > precision found in Intel derived architectures. Hmm, I wonder what ia64 > does? > > HP9000 also has quad precision: http://docs.hp.com/en/B3906-90006/ch02s02.html Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 4 14:47:24 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 12:47:24 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 12:38 PM, Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 12:35 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Wed, Nov 4, 2009 at 12:11 PM, David Cournapeau wrote: >> >>> On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom >>> wrote: >>> > I'm getting the following from r7603 on Solaris Sparc -- somehow >>> related >>> > to not having a long double version of next after available. I realise >>> > not everyone has access to (or is dependent on) this platform, so I'm >>> > willing to help in whatever way I can, I'm just not sure I understand >>> > the change yet. >>> >>> The only way to implement nextafter that I know of requires to know >>> the exact representation of the floating point number, and long double >>> is unfortunately platform dependent. >>> >>> What is the long double format on solaris sparc ? (big endian I >>> suppose, but how many bits for the mantissa and exponent ? Does it >>> follow IEER754 ?) >>> >>> >> Long double on SPARC is quad precision, which I believe *is* in one of the >> ieee specs. In any case, it has a lot more precision than the extended >> precision found in Intel derived architectures. Hmm, I wonder what ia64 >> does? >> >> > HP9000 also has quad precision: > http://docs.hp.com/en/B3906-90006/ch02s02.html > > And it looks like extended precision has disappeared from the latest version or the ieee_754-2008 standard, being replaced by quad precision. I know Intel is also working on quad precision FPU's, so I think that is where things are heading. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Wed Nov 4 14:55:20 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 04 Nov 2009 14:55:20 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: <4AF1DC28.6030403@stsci.edu> David Cournapeau wrote: > On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom wrote: > >> I'm getting the following from r7603 on Solaris Sparc -- somehow related >> to not having a long double version of next after available. I realise >> not everyone has access to (or is dependent on) this platform, so I'm >> willing to help in whatever way I can, I'm just not sure I understand >> the change yet. >> > > The only way to implement nextafter that I know of requires to know > the exact representation of the floating point number, and long double > is unfortunately platform dependent. > > What is the long double format on solaris sparc ? (big endian I > suppose, but how many bits for the mantissa and exponent ? Does it > follow IEER754 ?) > I honestly don't know -- I've never had to use them. It would be great to solve this properly but it's difficult to find definitive information about these things. Assuming we can't solve this the "right" way before the next release, would it be possible for this to raise a runtime "NotImplemented" error (by not defining the LONGDOUBLE_nextafter ufunc) rather than raising a compiler error which prevents the build from completing? Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From dwf at cs.toronto.edu Wed Nov 4 15:09:45 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Nov 2009 15:09:45 -0500 Subject: [Numpy-discussion] converting discrete data to unique integers Message-ID: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> Hi, Suppose I have an array 'd' In [75]: d Out[75]: array(['parrot', 'parrot', 'dog', 'cat', 'parrot', 'dog', 'parrot', 'cat', 'dog', 'dog', 'dog', 'cat', 'cat', 'dog', 'cat', 'parrot', 'cat', 'cat', 'dog', 'parrot', 'parrot', 'parrot', 'cat', 'dog', 'parrot', 'dog', 'dog', 'dog', 'dog', 'parrot', 'parrot', 'cat', 'dog', 'parrot', 'cat', 'parrot', 'cat', 'dog', 'parrot', 'cat', 'parrot', 'cat', 'parrot', 'parrot', 'parrot', 'parrot', 'dog', 'cat', 'parrot', 'cat'], dtype='|S6') I'd like to map every unique element (these could be strings, objects, or already ints) to a unique integer between 0 and len(unique(d)) - 1. The solution I've come up with is In [76]: uniqueind, vectorind = (d == unique(d)[:, newaxis]).nonzero() In [77]: myints = uniqueind[argsort(vectorind)] But I wonder if there's a better way to do this. Anyone ever run into this problem before? David From aisaac at american.edu Wed Nov 4 15:21:03 2009 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 04 Nov 2009 15:21:03 -0500 Subject: [Numpy-discussion] converting discrete data to unique integers In-Reply-To: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> Message-ID: <4AF1E22F.2020905@american.edu> On 11/4/2009 3:09 PM, David Warde-Farley wrote: > I'd like to map every unique element (these could be strings, objects, > or already ints) to a unique integer between 0 and len(unique(d)) - 1. mymap = dict((k,v) for v,k in enumerate(set(a))) fwiw, Alan Isaac From dwf at cs.toronto.edu Wed Nov 4 15:22:32 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Nov 2009 15:22:32 -0500 Subject: [Numpy-discussion] converting discrete data to unique integers In-Reply-To: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> Message-ID: <4EAD315A-D54F-4B3A-A918-9AFD20915629@cs.toronto.edu> On 4-Nov-09, at 3:09 PM, David Warde-Farley wrote: > But I wonder if there's a better way to do this. Anyone ever run into > this problem before? Obviously I find the answer right after I hit "send". unique(d, return_inverse=True). Sorry for the noise. David From robert.kern at gmail.com Wed Nov 4 15:22:19 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Nov 2009 14:22:19 -0600 Subject: [Numpy-discussion] converting discrete data to unique integers In-Reply-To: <4AF1E22F.2020905@american.edu> References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> <4AF1E22F.2020905@american.edu> Message-ID: <3d375d730911041222g5b064b23p97171db6df5fa900@mail.gmail.com> On Wed, Nov 4, 2009 at 14:21, Alan G Isaac wrote: > On 11/4/2009 3:09 PM, David Warde-Farley wrote: >> I'd like to map every unique element (these could be strings, objects, >> or already ints) to a unique integer between 0 and len(unique(d)) - 1. > > mymap = dict((k,v) for v,k in enumerate(set(a))) I'd toss in a sorted() just to keep the mapping stable. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dwf at cs.toronto.edu Wed Nov 4 15:57:19 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 4 Nov 2009 15:57:19 -0500 Subject: [Numpy-discussion] converting discrete data to unique integers In-Reply-To: <3d375d730911041222g5b064b23p97171db6df5fa900@mail.gmail.com> References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> <4AF1E22F.2020905@american.edu> <3d375d730911041222g5b064b23p97171db6df5fa900@mail.gmail.com> Message-ID: <268E605A-EB1F-4E3C-8854-282B0332B73F@cs.toronto.edu> Thanks Alan and Robert, I probably should have mentioned that I was interested in obtaining the corresponding integer for each value in the array d, in which case the dictionary bit works but would require a further loop to expand. On 4-Nov-09, at 3:22 PM, Robert Kern wrote: > I'd toss in a sorted() just to keep the mapping stable. Good point. With the return_inverse solution, is unique() guaranteed to give back the same array of unique values in the same (presumably sorted) order? That is, for two arrays A and B which have elements only drawn from a set S, is all(unique(A) == unique(B)) guaranteed? The code is a quite clever and a bit hard to follow, but it *looks* like it will provide a stable mapping since it's using a sort. David From josef.pktd at gmail.com Wed Nov 4 16:36:39 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 4 Nov 2009 16:36:39 -0500 Subject: [Numpy-discussion] converting discrete data to unique integers In-Reply-To: <268E605A-EB1F-4E3C-8854-282B0332B73F@cs.toronto.edu> References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> <4AF1E22F.2020905@american.edu> <3d375d730911041222g5b064b23p97171db6df5fa900@mail.gmail.com> <268E605A-EB1F-4E3C-8854-282B0332B73F@cs.toronto.edu> Message-ID: <1cd32cbb0911041336u6d70f202jc3959fe3f4feba68@mail.gmail.com> On Wed, Nov 4, 2009 at 3:57 PM, David Warde-Farley wrote: > Thanks Alan and Robert, I probably should have mentioned that I was > interested in obtaining the corresponding integer for each value in > the array d, in which case the dictionary bit works but would require > a further loop to expand. > > On 4-Nov-09, at 3:22 PM, Robert Kern wrote: > >> I'd toss in a sorted() just to keep the mapping stable. > > Good point. With the return_inverse solution, is unique() guaranteed > to give back the same array of unique values in the same (presumably > sorted) order? That is, for two arrays A and B which have elements > only drawn from a set S, is all(unique(A) == unique(B)) guaranteed? > The code is a quite clever and a bit hard to follow, but it *looks* > like it will provide a stable mapping since it's using a sort. I looked at it some time ago, and from what I remember, the sort is done if return_inverse=True but for some codepath it uses set. So, you would need to check which version includes the sort. Josef > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From neilcrighton at gmail.com Wed Nov 4 17:23:05 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Wed, 4 Nov 2009 22:23:05 +0000 (UTC) Subject: [Numpy-discussion] converting discrete data to unique integers References: <8C42CDA8-A3B6-4CB3-A1C0-2FA8ED417615@cs.toronto.edu> <4AF1E22F.2020905@american.edu> <3d375d730911041222g5b064b23p97171db6df5fa900@mail.gmail.com> <268E605A-EB1F-4E3C-8854-282B0332B73F@cs.toronto.edu> <1cd32cbb0911041336u6d70f202jc3959fe3f4feba68@mail.gmail.com> Message-ID: gmail.com> writes: > > Good point. With the return_inverse solution, is unique() guaranteed > > to give back the same array of unique values in the same (presumably > > sorted) order? That is, for two arrays A and B which have elements > > only drawn from a set S, is all(unique(A) == unique(B)) guaranteed? > > The code is a quite clever and a bit hard to follow, but it *looks* > > like it will provide a stable mapping since it's using a sort. > > I looked at it some time ago, and from what I remember, the sort > is done if return_inverse=True but for some codepath it uses > set. > unique always sorts, even if it uses set. So I'm pretty sure all(unique(A) == unique(B)) is guaranteed. Neil From bsouthey at gmail.com Wed Nov 4 18:56:03 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 4 Nov 2009 17:56:03 -0600 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: On Wed, Nov 4, 2009 at 8:06 AM, Reckoner wrote: > Here's an example: > > On winxp 64-bit: > > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy >>>> import cPickle >>>> a = numpy.eye(10) >>>> cPickle.dump(a,open('from32bitxp.pkl','w')) >>>> import numpy.core.multiarray >>>> numpy.__version__ > '1.0.4' >>>> > > On linux 64 bit: > > Python 2.5.4 (r254:67916, Feb ?5 2009, 19:52:35) > [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy >>>> import cPickle >>>> cPickle.load(open('from32bitxp.pkl')) > Traceback (most recent call last): > ?File "", line 1, in > ImportError: No module named multiarray >>>> numpy.__version__ > '1.2.1' >>>> import numpy.core.multiarray >>>> > > Note that I transfer the from32bitxp file from the winxp32 machine to > the linux host. Also, I've tried this with version 1.3 on winxp and > get the same problem on the linux host. > > Here's more interesting info: > > On linux: > >>>> a = numpy.eye(10) >>>> cPickle.dump(a,open('from64bitLinux.pkl','w')) > > upon transferring the file to winxp 32 and on winxp32: > >>>> cPickle.load(open('from64bitLinux.pkl')) > > See? No problem going from linux to winxp32; but problems going the other way. > > Please let me know if you need more info on this. > > Any help appreciated. > > On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey wrote: >> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: >>> thanks for the suggestion! I will look into it. The other thing is >>> that the numpy arrays in question are actually embedded in another >>> object. When I convert the numpy arrays into plain lists, and then >>> cPickle them, there is no problem with any of the larger objects. That >>> is the way we are currently working around this issue. >>> >>> Thanks again. >>> >>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>>>> Anybody have any ideas here? >>>>> >>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>>>> What's the best way to report a bug of this kind? >>>>> >>>>> Thanks! >>>>> >>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>>>> Robert Kern wrote: >>>>>>> You can import numpy.core.multiarray on both machines? >>>>>> >>>>>> Yes. For each machine separately, you can cPickle files with numpy >>>>>> arrays without problems loading/dumping. The problem comes from >>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>>>> back and forth between the two machines. In fact, we currently get >>>>>> around this problem by converting the numpy arrays to lists, >>>>>> transferring them, and then re-numpy-ing them on the respective hosts >>>>>> >>>>>> thanks. >>>>>> >>>>>> >>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>>>> Hi, >>>>>>> >>>>>>> % python -c 'import numpy.core.multiarray' >>>>>>> >>>>>>> works just fine, but when I try to load a file that I have transferred >>>>>>> from another machine running Windows to one running Linux, I get: >>>>>>> >>>>>>> % ?python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> ?File "", line 1, in >>>>>>> ImportError: No module named multiarray >>>>>>> >>>>>>> otherwise, cPickle works normally when transferring files that *do* >>>>>>> not contain numpy arrays. >>>>>>> >>>>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>>>> change versions, by the way, since this is the version that my working >>>>>>> group has decided on to standardize on for this effort. >>>>>>> >>>>>>> >>>>>>> Any help appreciated. >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>> >>>> Have you have tried the other Cookbook approaches: >>>> http://www.scipy.org/Cookbook/InputOutput >>>> Like using numpy's own array io functions - load/save(z)? >>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >>>> different numpy versions) >>>> >>>> Bruce >>>> _______________________________________________ >> >> Can you provide you provide a small self-contained example of the >> problem including object creation especially as your example does not >> import numpy? >> >> Really you have to start at the beginning (like pickling and >> transferring numpy arrays) and then increase the complexity to include >> the object. >> >> >> Bruce >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Hi, I did not see the file 'from32bitxp.pkl'. It would really help if you can provide the full example that includes the creation of the complete object so at least people could try doing the same process with the object that you are using. Bruce From cournape at gmail.com Wed Nov 4 20:57:36 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 10:57:36 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF1DC28.6030403@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> Message-ID: <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> On Thu, Nov 5, 2009 at 4:55 AM, Michael Droettboom wrote: > David Cournapeau wrote: >> On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom wrote: >> >>> I'm getting the following from r7603 on Solaris Sparc -- somehow related >>> to not having a long double version of next after available. ?I realise >>> not everyone has access to (or is dependent on) this platform, so I'm >>> willing to help in whatever way I can, I'm just not sure I understand >>> the change yet. >>> >> >> The only way to implement nextafter that I know of requires to know >> the exact representation of the floating point number, and long double >> is unfortunately platform dependent. >> >> What is the long double format on solaris sparc ? (big endian I >> suppose, but how many bits for the mantissa and ?exponent ? Does it >> follow IEER754 ?) >> > I honestly don't know -- I've never had to use them. ?It would be great > to solve this properly but it's difficult to find definitive information > about these things. > > Assuming we can't solve this the "right" way before the next release, > would it be possible for this to raise a runtime "NotImplemented" error > (by not defining the LONGDOUBLE_nextafter ufunc) rather than raising a > compiler error which prevents the build from completing? To be honest, I thought this condition would never arise (I am quite surprised that solaris does not have nextafter - both BSD and GNU libc use the Sun implementation for this function). If this is not fixed within the code freeze (16th november), the feature will be removed altogether for 1.4.0. I don't want to go down the road of different feature set for different platforms. David From reckoner at gmail.com Wed Nov 4 22:21:57 2009 From: reckoner at gmail.com (Reckoner) Date: Wed, 4 Nov 2009 19:21:57 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: Bruce : The file in question was created as shown in the prior e-mail. Here it is again: >> cPickle.dump(a,open('from32bitxp.pkl','w')) Thanks! On Wed, Nov 4, 2009 at 3:56 PM, Bruce Southey wrote: > On Wed, Nov 4, 2009 at 8:06 AM, Reckoner wrote: >> Here's an example: >> >> On winxp 64-bit: >> >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on >> win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> import cPickle >>>>> a = numpy.eye(10) >>>>> cPickle.dump(a,open('from32bitxp.pkl','w')) >>>>> import numpy.core.multiarray >>>>> numpy.__version__ >> '1.0.4' >>>>> >> >> On linux 64 bit: >> >> Python 2.5.4 (r254:67916, Feb 5 2009, 19:52:35) >> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> import cPickle >>>>> cPickle.load(open('from32bitxp.pkl')) >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named multiarray >>>>> numpy.__version__ >> '1.2.1' >>>>> import numpy.core.multiarray >>>>> >> >> Note that I transfer the from32bitxp file from the winxp32 machine to >> the linux host. Also, I've tried this with version 1.3 on winxp and >> get the same problem on the linux host. >> >> Here's more interesting info: >> >> On linux: >> >>>>> a = numpy.eye(10) >>>>> cPickle.dump(a,open('from64bitLinux.pkl','w')) >> >> upon transferring the file to winxp 32 and on winxp32: >> >>>>> cPickle.load(open('from64bitLinux.pkl')) >> >> See? No problem going from linux to winxp32; but problems going the other way. >> >> Please let me know if you need more info on this. >> >> Any help appreciated. >> >> On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey wrote: >>> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: >>>> thanks for the suggestion! I will look into it. The other thing is >>>> that the numpy arrays in question are actually embedded in another >>>> object. When I convert the numpy arrays into plain lists, and then >>>> cPickle them, there is no problem with any of the larger objects. That >>>> is the way we are currently working around this issue. >>>> >>>> Thanks again. >>>> >>>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >>>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>>>>> Anybody have any ideas here? >>>>>> >>>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>>>>> What's the best way to report a bug of this kind? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>>>>> Robert Kern wrote: >>>>>>>> You can import numpy.core.multiarray on both machines? >>>>>>> >>>>>>> Yes. For each machine separately, you can cPickle files with numpy >>>>>>> arrays without problems loading/dumping. The problem comes from >>>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>>>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>>>>> back and forth between the two machines. In fact, we currently get >>>>>>> around this problem by converting the numpy arrays to lists, >>>>>>> transferring them, and then re-numpy-ing them on the respective hosts >>>>>>> >>>>>>> thanks. >>>>>>> >>>>>>> >>>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> % python -c 'import numpy.core.multiarray' >>>>>>>> >>>>>>>> works just fine, but when I try to load a file that I have transferred >>>>>>>> from another machine running Windows to one running Linux, I get: >>>>>>>> >>>>>>>> % python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "", line 1, in >>>>>>>> ImportError: No module named multiarray >>>>>>>> >>>>>>>> otherwise, cPickle works normally when transferring files that *do* >>>>>>>> not contain numpy arrays. >>>>>>>> >>>>>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>>>>> change versions, by the way, since this is the version that my working >>>>>>>> group has decided on to standardize on for this effort. >>>>>>>> >>>>>>>> >>>>>>>> Any help appreciated. >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>> >>>>> Have you have tried the other Cookbook approaches: >>>>> http://www.scipy.org/Cookbook/InputOutput >>>>> Like using numpy's own array io functions - load/save(z)? >>>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >>>>> different numpy versions) >>>>> >>>>> Bruce >>>>> _______________________________________________ >>> >>> Can you provide you provide a small self-contained example of the >>> problem including object creation especially as your example does not >>> import numpy? >>> >>> Really you have to start at the beginning (like pickling and >>> transferring numpy arrays) and then increase the complexity to include >>> the object. >>> >>> >>> Bruce >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > Hi, > I did not see the file 'from32bitxp.pkl'. It would really help if you > can provide the full example that includes the creation of the > complete object so at least people could try doing the same process > with the object that you are using. > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From reckoner at gmail.com Wed Nov 4 22:31:18 2009 From: reckoner at gmail.com (Reckoner) Date: Wed, 4 Nov 2009 19:31:18 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: FYI, I uploaded the two files in question to the numpy ticket http://projects.scipy.org/numpy/ticket/1284 Thanks! On Wed, Nov 4, 2009 at 3:56 PM, Bruce Southey wrote: > On Wed, Nov 4, 2009 at 8:06 AM, Reckoner wrote: >> Here's an example: >> >> On winxp 64-bit: >> >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on >> win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> import cPickle >>>>> a = numpy.eye(10) >>>>> cPickle.dump(a,open('from32bitxp.pkl','w')) >>>>> import numpy.core.multiarray >>>>> numpy.__version__ >> '1.0.4' >>>>> >> >> On linux 64 bit: >> >> Python 2.5.4 (r254:67916, Feb 5 2009, 19:52:35) >> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> import cPickle >>>>> cPickle.load(open('from32bitxp.pkl')) >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: No module named multiarray >>>>> numpy.__version__ >> '1.2.1' >>>>> import numpy.core.multiarray >>>>> >> >> Note that I transfer the from32bitxp file from the winxp32 machine to >> the linux host. Also, I've tried this with version 1.3 on winxp and >> get the same problem on the linux host. >> >> Here's more interesting info: >> >> On linux: >> >>>>> a = numpy.eye(10) >>>>> cPickle.dump(a,open('from64bitLinux.pkl','w')) >> >> upon transferring the file to winxp 32 and on winxp32: >> >>>>> cPickle.load(open('from64bitLinux.pkl')) >> >> See? No problem going from linux to winxp32; but problems going the other way. >> >> Please let me know if you need more info on this. >> >> Any help appreciated. >> >> On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey wrote: >>> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: >>>> thanks for the suggestion! I will look into it. The other thing is >>>> that the numpy arrays in question are actually embedded in another >>>> object. When I convert the numpy arrays into plain lists, and then >>>> cPickle them, there is no problem with any of the larger objects. That >>>> is the way we are currently working around this issue. >>>> >>>> Thanks again. >>>> >>>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >>>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>>>>> Anybody have any ideas here? >>>>>> >>>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>>>>> What's the best way to report a bug of this kind? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>>>>> Robert Kern wrote: >>>>>>>> You can import numpy.core.multiarray on both machines? >>>>>>> >>>>>>> Yes. For each machine separately, you can cPickle files with numpy >>>>>>> arrays without problems loading/dumping. The problem comes from >>>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>>>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>>>>> back and forth between the two machines. In fact, we currently get >>>>>>> around this problem by converting the numpy arrays to lists, >>>>>>> transferring them, and then re-numpy-ing them on the respective hosts >>>>>>> >>>>>>> thanks. >>>>>>> >>>>>>> >>>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> % python -c 'import numpy.core.multiarray' >>>>>>>> >>>>>>>> works just fine, but when I try to load a file that I have transferred >>>>>>>> from another machine running Windows to one running Linux, I get: >>>>>>>> >>>>>>>> % python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "", line 1, in >>>>>>>> ImportError: No module named multiarray >>>>>>>> >>>>>>>> otherwise, cPickle works normally when transferring files that *do* >>>>>>>> not contain numpy arrays. >>>>>>>> >>>>>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>>>>> change versions, by the way, since this is the version that my working >>>>>>>> group has decided on to standardize on for this effort. >>>>>>>> >>>>>>>> >>>>>>>> Any help appreciated. >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> NumPy-Discussion mailing list >>>>>> NumPy-Discussion at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>> >>>>> >>>>> Have you have tried the other Cookbook approaches: >>>>> http://www.scipy.org/Cookbook/InputOutput >>>>> Like using numpy's own array io functions - load/save(z)? >>>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >>>>> different numpy versions) >>>>> >>>>> Bruce >>>>> _______________________________________________ >>> >>> Can you provide you provide a small self-contained example of the >>> problem including object creation especially as your example does not >>> import numpy? >>> >>> Really you have to start at the beginning (like pickling and >>> transferring numpy arrays) and then increase the complexity to include >>> the object. >>> >>> >>> Bruce >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > Hi, > I did not see the file 'from32bitxp.pkl'. It would really help if you > can provide the full example that includes the creation of the > complete object so at least people could try doing the same process > with the object that you are using. > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Thu Nov 5 00:56:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 14:56:39 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> On Thu, Nov 5, 2009 at 4:47 AM, Charles R Harris wrote: > > > And it looks like extended precision has disappeared from the latest version > or the ieee_754-2008 standard, being replaced by quad precision. I know > Intel is also working on quad precision FPU's, so I think that is where > things are heading. The problem is not so much the format itself, but that it is different for almost every arch, and that it needs to be distinguished between OS and compilers (long double on ia32 does not mean the same thing on mac os x and windows, and does not mean the same thing on windows depending on the compiler and even the compiler option). This sounds like a maintenance nightmare, with tens of configurations to implement (and test). I can't see a simple solution, short of dropping support for long double for any feature which needs knowledge of the exact representation. cheers, David From cournape at gmail.com Thu Nov 5 00:58:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 14:58:39 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> Message-ID: <5b8d13220911042158o3f632f0xc9d96c6dfad72eee@mail.gmail.com> On Thu, Nov 5, 2009 at 4:35 AM, Charles R Harris wrote: > > Long double on SPARC is quad precision, which I believe *is* in one of the > ieee specs. In any case, it has a lot more precision than the extended > precision found in Intel derived architectures. Hmm, I wonder what ia64 > does? If I read the FreeBSD libc rights, it is the same as x86: http://svn.freebsd.org/base/release/7.2.0/lib/libc/ia64/_fpmath.h cheers, David From charlesr.harris at gmail.com Thu Nov 5 01:37:23 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 23:37:23 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 10:56 PM, David Cournapeau wrote: > On Thu, Nov 5, 2009 at 4:47 AM, Charles R Harris > wrote: > > > > > > And it looks like extended precision has disappeared from the latest > version > > or the ieee_754-2008 standard, being replaced by quad precision. I know > > Intel is also working on quad precision FPU's, so I think that is where > > things are heading. > > The problem is not so much the format itself, but that it is different > for almost every arch, and that it needs to be distinguished between > OS and compilers (long double on ia32 does not mean the same thing on > mac os x and windows, and does not mean the same thing on windows > depending on the compiler and even the compiler option). > > This sounds like a maintenance nightmare, with tens of configurations > to implement (and test). I can't see a simple solution, short of > dropping support for long double for any feature which needs knowledge > of the exact representation. > > I don't think it's that bad. Leaving out the ppc and sticking to ieee, there is only double precision, extended precision and quad precision versions of long double and they are easily determined at run time. The ppc is the odd one out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 01:30:21 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 15:30:21 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> Message-ID: <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I don't think it's that bad. Leaving out the ppc and sticking to ieee, > there is only double precision, extended precision and quad precision > versions of long double and they are easily determined at run time. How would you determine this at runtime ? cheers, David From charlesr.harris at gmail.com Thu Nov 5 01:52:43 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 23:52:43 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 11:37 PM, Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 10:56 PM, David Cournapeau wrote: > >> On Thu, Nov 5, 2009 at 4:47 AM, Charles R Harris >> wrote: >> > >> > >> > And it looks like extended precision has disappeared from the latest >> version >> > or the ieee_754-2008 standard, being replaced by quad precision. I know >> > Intel is also working on quad precision FPU's, so I think that is where >> > things are heading. >> >> The problem is not so much the format itself, but that it is different >> for almost every arch, and that it needs to be distinguished between >> OS and compilers (long double on ia32 does not mean the same thing on >> mac os x and windows, and does not mean the same thing on windows >> depending on the compiler and even the compiler option). >> >> This sounds like a maintenance nightmare, with tens of configurations >> to implement (and test). I can't see a simple solution, short of >> dropping support for long double for any feature which needs knowledge >> of the exact representation. >> >> > I don't think it's that bad. Leaving out the ppc and sticking to ieee, > there is only double precision, extended precision and quad precision > versions of long double and they are easily determined at run time. The ppc > is the odd one out. > > A good place to start is to rip out the routine used by finfo. It's a bit of crap taken, IIRC, from Numerical Recipes and is way more complicated than needed to deal with ieee numbers. An exception for PPC can be hardwired in and the needed data for that is already part of cpu_info.h Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 5 01:57:54 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Nov 2009 23:57:54 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > I don't think it's that bad. Leaving out the ppc and sticking to ieee, > > there is only double precision, extended precision and quad precision > > versions of long double and they are easily determined at run time. > > How would you determine this at runtime ? > > Excepting the PPC, just loop adding a number to one, dividing it by two at each iteration, and stop when the result is equal to one. There has been a version of that in the scipy zeros module since forever. PyMODINIT_FUNC init_zeros(void) { double tol; /* Determine relative precision of doubles, assumes binary */ for(tol = 1; tol + 1 != 1; tol /= 2); scipy_zeros_rtol = 2*tol; Py_InitModule("_zeros", Zerosmethods); } Or you could just do byte comparisons against known numbers. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 01:39:19 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 15:39:19 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> Message-ID: <4AF27317.1080404@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > > > > I don't think it's that bad. Leaving out the ppc and sticking to > ieee, > > there is only double precision, extended precision and quad > precision > > versions of long double and they are easily determined at run time. > > How would you determine this at runtime ? > > > Excepting the PPC, just loop adding a number to one, dividing it by > two at each iteration, and stop when the result is equal to one. But that's not what I need. I need to know exactly the binary representation: how many bits in the mantissa/exponent and where, the exponent, where does subnormals start, the range of NAN representations, etc... David From charlesr.harris at gmail.com Thu Nov 5 02:07:02 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 00:07:02 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF27317.1080404@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > > > wrote: > > > > Charles R Harris wrote: > > > > > > > > I don't think it's that bad. Leaving out the ppc and sticking to > > ieee, > > > there is only double precision, extended precision and quad > > precision > > > versions of long double and they are easily determined at run > time. > > > > How would you determine this at runtime ? > > > > > > Excepting the PPC, just loop adding a number to one, dividing it by > > two at each iteration, and stop when the result is equal to one. > > But that's not what I need. I need to know exactly the binary > representation: how many bits in the mantissa/exponent and where, the > exponent, where does subnormals start, the range of NAN representations, > etc... > > It tells you how many bits are in the mantissa, and given ieee the rest follows. We only support ieee anyway. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 02:09:25 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 16:09:25 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> Message-ID: <4AF27A25.90604@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > > > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > >> > > wrote: > > > > Charles R Harris wrote: > > > > > > > > I don't think it's that bad. Leaving out the ppc and > sticking to > > ieee, > > > there is only double precision, extended precision and quad > > precision > > > versions of long double and they are easily determined at > run time. > > > > How would you determine this at runtime ? > > > > > > Excepting the PPC, just loop adding a number to one, dividing it by > > two at each iteration, and stop when the result is equal to one. > > But that's not what I need. I need to know exactly the binary > representation: how many bits in the mantissa/exponent and where, the > exponent, where does subnormals start, the range of NAN > representations, > etc... > > > It tells you how many bits are in the mantissa, and given ieee the > rest follows. We only support ieee anyway. But is this reliable ? It does not seem to work for long double in intel, for example (but seems to work for sparc64, at least using qemu). David From david at ar.media.kyoto-u.ac.jp Thu Nov 5 02:11:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 16:11:23 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF27A25.90604@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> Message-ID: <4AF27A9B.7040304@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Charles R Harris wrote: > >> On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau >> > >> wrote: >> >> Charles R Harris wrote: >> > >> > >> > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau >> > > >> > >> >> > wrote: >> > >> > Charles R Harris wrote: >> > >> > > >> > > I don't think it's that bad. Leaving out the ppc and >> sticking to >> > ieee, >> > > there is only double precision, extended precision and quad >> > precision >> > > versions of long double and they are easily determined at >> run time. >> > >> > How would you determine this at runtime ? >> > >> > >> > Excepting the PPC, just loop adding a number to one, dividing it by >> > two at each iteration, and stop when the result is equal to one. >> >> But that's not what I need. I need to know exactly the binary >> representation: how many bits in the mantissa/exponent and where, the >> exponent, where does subnormals start, the range of NAN >> representations, >> etc... >> >> >> It tells you how many bits are in the mantissa, and given ieee the >> rest follows. We only support ieee anyway. >> > > But is this reliable ? It does not seem to work for long double in > intel, for example (but seems to work for sparc64, at least using qemu). > Sorry - I forgot 80 bits extended precision format has no implicit bit, so the relationship between ibeta and number of bits of mantissa is different in that case. David From charlesr.harris at gmail.com Thu Nov 5 02:47:26 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 00:47:26 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF27A25.90604@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 12:09 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau > > > > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > > > > > >> > > > wrote: > > > > > > Charles R Harris wrote: > > > > > > > > > > > I don't think it's that bad. Leaving out the ppc and > > sticking to > > > ieee, > > > > there is only double precision, extended precision and quad > > > precision > > > > versions of long double and they are easily determined at > > run time. > > > > > > How would you determine this at runtime ? > > > > > > > > > Excepting the PPC, just loop adding a number to one, dividing it by > > > two at each iteration, and stop when the result is equal to one. > > > > But that's not what I need. I need to know exactly the binary > > representation: how many bits in the mantissa/exponent and where, the > > exponent, where does subnormals start, the range of NAN > > representations, > > etc... > > > > > > It tells you how many bits are in the mantissa, and given ieee the > > rest follows. We only support ieee anyway. > > But is this reliable ? It does not seem to work for long double in > intel, for example (but seems to work for sparc64, at least using qemu). > > Works fine here: #include int main(int argc, char **args) { long double tol; int i; for (i = 0, tol = 1; 1 + tol != 1; tol /=2, i++); printf("count: %d\n", i - 1); return 0; } $[charris at ubuntu ~]$ gcc precision.c $[charris at ubuntu ~]$ ./a.out count: 63 That's 63+1 for the mantissa, which is what intel extended precision is. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 5 02:55:04 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 00:55:04 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 12:47 AM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 12:09 AM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> Charles R Harris wrote: >> > >> > >> > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau >> > > >> > wrote: >> > >> > Charles R Harris wrote: >> > > >> > > >> > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau >> > > > > >> > > > >> >> > > wrote: >> > > >> > > Charles R Harris wrote: >> > > >> > > > >> > > > I don't think it's that bad. Leaving out the ppc and >> > sticking to >> > > ieee, >> > > > there is only double precision, extended precision and quad >> > > precision >> > > > versions of long double and they are easily determined at >> > run time. >> > > >> > > How would you determine this at runtime ? >> > > >> > > >> > > Excepting the PPC, just loop adding a number to one, dividing it >> by >> > > two at each iteration, and stop when the result is equal to one. >> > >> > But that's not what I need. I need to know exactly the binary >> > representation: how many bits in the mantissa/exponent and where, >> the >> > exponent, where does subnormals start, the range of NAN >> > representations, >> > etc... >> > >> > >> > It tells you how many bits are in the mantissa, and given ieee the >> > rest follows. We only support ieee anyway. >> >> But is this reliable ? It does not seem to work for long double in >> intel, for example (but seems to work for sparc64, at least using qemu). >> >> > Works fine here: > > #include > > int main(int argc, char **args) > { > long double tol; > int i; > > for (i = 0, tol = 1; 1 + tol != 1; tol /=2, i++); > printf("count: %d\n", i - 1); > return 0; > } > > > $[charris at ubuntu ~]$ gcc precision.c > $[charris at ubuntu ~]$ ./a.out > count: 63 > > That's 63+1 for the mantissa, which is what intel extended precision is. > > Googling around, it seems that the SUN quad precision is done in software, not hardware and is available but not used by the compilers in the current Intel based SUN systems, but will be in the next OS version. So it looks dependent on the compiler, meaning we probably need a run time check. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 02:39:35 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 16:39:35 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911042156r38a8b638i1bdb1af3d44ff92c@mail.gmail.com> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> Message-ID: <4AF28137.5050002@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 12:47 AM, Charles R Harris > > wrote: > > > > On Thu, Nov 5, 2009 at 12:09 AM, David Cournapeau > > wrote: > > Charles R Harris wrote: > > > > > > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau > > > >> > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > > > > > > > > >>> > > > wrote: > > > > > > Charles R Harris wrote: > > > > > > > > > > > I don't think it's that bad. Leaving out the ppc and > > sticking to > > > ieee, > > > > there is only double precision, extended > precision and quad > > > precision > > > > versions of long double and they are easily > determined at > > run time. > > > > > > How would you determine this at runtime ? > > > > > > > > > Excepting the PPC, just loop adding a number to one, > dividing it by > > > two at each iteration, and stop when the result is > equal to one. > > > > But that's not what I need. I need to know exactly the > binary > > representation: how many bits in the mantissa/exponent > and where, the > > exponent, where does subnormals start, the range of NAN > > representations, > > etc... > > > > > > It tells you how many bits are in the mantissa, and given > ieee the > > rest follows. We only support ieee anyway. > > But is this reliable ? It does not seem to work for long double in > intel, for example (but seems to work for sparc64, at least > using qemu). > > > Works fine here: > > #include > > int main(int argc, char **args) > { > long double tol; > int i; > > for (i = 0, tol = 1; 1 + tol != 1; tol /=2, i++); > printf("count: %d\n", i - 1); > return 0; > } > > > $[charris at ubuntu ~]$ gcc precision.c > $[charris at ubuntu ~]$ ./a.out > count: 63 > > That's 63+1 for the mantissa, which is what intel extended > precision is. > > > Googling around, it seems that the SUN quad precision is done in > software, not hardware and is available but not used by the compilers > in the current Intel based SUN systems, but will be in the next OS > version. So it looks dependent on the compiler, meaning we probably > need a run time check. Now that I think about it, if we only support quad precision, double == long double and 80 bits Intel format, we could just check for the size and be done with it. I much prefer compile time check which do not need to run on the target machine :) cheers, David From charlesr.harris at gmail.com Thu Nov 5 03:15:07 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 01:15:07 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF28137.5050002@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 12:39 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Thu, Nov 5, 2009 at 12:47 AM, Charles R Harris > > > wrote: > > > > > > > > On Thu, Nov 5, 2009 at 12:09 AM, David Cournapeau > > > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau > > > > > > > >> > > > wrote: > > > > > > Charles R Harris wrote: > > > > > > > > > > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau > > > > > > > > > > > > > > > > > > >>> > > > > wrote: > > > > > > > > Charles R Harris wrote: > > > > > > > > > > > > > > I don't think it's that bad. Leaving out the ppc > and > > > sticking to > > > > ieee, > > > > > there is only double precision, extended > > precision and quad > > > > precision > > > > > versions of long double and they are easily > > determined at > > > run time. > > > > > > > > How would you determine this at runtime ? > > > > > > > > > > > > Excepting the PPC, just loop adding a number to one, > > dividing it by > > > > two at each iteration, and stop when the result is > > equal to one. > > > > > > But that's not what I need. I need to know exactly the > > binary > > > representation: how many bits in the mantissa/exponent > > and where, the > > > exponent, where does subnormals start, the range of NAN > > > representations, > > > etc... > > > > > > > > > It tells you how many bits are in the mantissa, and given > > ieee the > > > rest follows. We only support ieee anyway. > > > > But is this reliable ? It does not seem to work for long double > in > > intel, for example (but seems to work for sparc64, at least > > using qemu). > > > > > > Works fine here: > > > > #include > > > > int main(int argc, char **args) > > { > > long double tol; > > int i; > > > > for (i = 0, tol = 1; 1 + tol != 1; tol /=2, i++); > > printf("count: %d\n", i - 1); > > return 0; > > } > > > > > > $[charris at ubuntu ~]$ gcc precision.c > > $[charris at ubuntu ~]$ ./a.out > > count: 63 > > > > That's 63+1 for the mantissa, which is what intel extended > > precision is. > > > > > > Googling around, it seems that the SUN quad precision is done in > > software, not hardware and is available but not used by the compilers > > in the current Intel based SUN systems, but will be in the next OS > > version. So it looks dependent on the compiler, meaning we probably > > need a run time check. > > Now that I think about it, if we only support quad precision, double == > long double and 80 bits Intel format, we could just check for the size > and be done with it. > > Won't work. For alignment, 80 bits is stored in 128 bits on 64 bit machines, the same as quad precision. I've been complaining about not having a decent distinction between those two types for several years ;) But the runtime determination could just set some function pointers in a jump table if you want to simplify the interface. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 5 03:24:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 01:24:36 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 1:15 AM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 12:39 AM, David Cournapeau < > david at ar.media.kyoto-u.ac.jp> wrote: > >> Charles R Harris wrote: >> > >> > >> > On Thu, Nov 5, 2009 at 12:47 AM, Charles R Harris >> > > wrote: >> > >> > >> > >> > On Thu, Nov 5, 2009 at 12:09 AM, David Cournapeau >> > > > > wrote: >> > >> > Charles R Harris wrote: >> > > >> > > >> > > On Wed, Nov 4, 2009 at 11:39 PM, David Cournapeau >> > > > > >> > > > >> >> > > wrote: >> > > >> > > Charles R Harris wrote: >> > > > >> > > > >> > > > On Wed, Nov 4, 2009 at 11:30 PM, David Cournapeau >> > > > > > >> > > > > > >> > > > > >> > > > > >>> >> > > > wrote: >> > > > >> > > > Charles R Harris wrote: >> > > > >> > > > > >> > > > > I don't think it's that bad. Leaving out the ppc >> and >> > > sticking to >> > > > ieee, >> > > > > there is only double precision, extended >> > precision and quad >> > > > precision >> > > > > versions of long double and they are easily >> > determined at >> > > run time. >> > > > >> > > > How would you determine this at runtime ? >> > > > >> > > > >> > > > Excepting the PPC, just loop adding a number to one, >> > dividing it by >> > > > two at each iteration, and stop when the result is >> > equal to one. >> > > >> > > But that's not what I need. I need to know exactly the >> > binary >> > > representation: how many bits in the mantissa/exponent >> > and where, the >> > > exponent, where does subnormals start, the range of NAN >> > > representations, >> > > etc... >> > > >> > > >> > > It tells you how many bits are in the mantissa, and given >> > ieee the >> > > rest follows. We only support ieee anyway. >> > >> > But is this reliable ? It does not seem to work for long double >> in >> > intel, for example (but seems to work for sparc64, at least >> > using qemu). >> > >> > >> > Works fine here: >> > >> > #include >> > >> > int main(int argc, char **args) >> > { >> > long double tol; >> > int i; >> > >> > for (i = 0, tol = 1; 1 + tol != 1; tol /=2, i++); >> > printf("count: %d\n", i - 1); >> > return 0; >> > } >> > >> > >> > $[charris at ubuntu ~]$ gcc precision.c >> > $[charris at ubuntu ~]$ ./a.out >> > count: 63 >> > >> > That's 63+1 for the mantissa, which is what intel extended >> > precision is. >> > >> > >> > Googling around, it seems that the SUN quad precision is done in >> > software, not hardware and is available but not used by the compilers >> > in the current Intel based SUN systems, but will be in the next OS >> > version. So it looks dependent on the compiler, meaning we probably >> > need a run time check. >> >> Now that I think about it, if we only support quad precision, double == >> long double and 80 bits Intel format, we could just check for the size >> and be done with it. >> >> > Won't work. For alignment, 80 bits is stored in 128 bits on 64 bit > machines, the same as quad precision. I've been complaining about not having > a decent distinction between those two types for several years ;) But the > runtime determination could just set some function pointers in a jump table > if you want to simplify the interface. > > Or I suppose you could initialize a structure with the relevant info and have a function reference it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 03:06:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 17:06:06 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF270FD.8020405@ar.media.kyoto-u.ac.jp> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> Message-ID: <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > Won't work. For alignment, 80 bits is stored in 128 bits on 64 bit > machines, the same as quad precision. But AFAIK, the only arch with 80 bits support is Intel, and it does not support quad precision, so we only really need to support sizeof(long double) != sizeof(double). David From charlesr.harris at gmail.com Thu Nov 5 03:39:03 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 01:39:03 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 1:06 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > Won't work. For alignment, 80 bits is stored in 128 bits on 64 bit > > machines, the same as quad precision. > > But AFAIK, the only arch with 80 bits support is Intel, and it does not > support quad precision, so we only really need to support sizeof(long > double) != sizeof(double). > > Hmm, you mean use the architecture information along with the size? How does that work for open solaris, which at some point will be implementing quads on intel, but doesn't now? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Nov 5 03:39:26 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 05 Nov 2009 17:39:26 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF27317.1080404@ar.media.kyoto-u.ac.jp> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> Message-ID: <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > Hmm, you mean use the architecture information along with the size? > How does that work for open solaris, which at some point will be > implementing quads on intel, but doesn't now? I would say let's focus on existing platforms which cause problem before worrying about non existent platforms :) Setting the functions at runtime is too inconvenient to warrant it unless strictly necessary (slow, complicated, etc...). David From charlesr.harris at gmail.com Thu Nov 5 04:15:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 02:15:29 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 1:39 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > Hmm, you mean use the architecture information along with the size? > > How does that work for open solaris, which at some point will be > > implementing quads on intel, but doesn't now? > > I would say let's focus on existing platforms which cause problem before > worrying about non existent platforms :) > > I don't think it is that far off. The quad precision isn't done in hardware, not on SPARC or Intel, it is done in software and that software is already present on Intel, just not turned on. Someone else's quick and dirty fix. But folks need to port software from one platform to the other, so I expect things will change. > Setting the functions at runtime is too inconvenient to warrant it > unless strictly necessary (slow, complicated, etc...). > > Well, I don't agree that it is slow or complicated ;) Hey, if we needed code that went zoom, zoom, zoom, we wouldn't be using python, now would we. We would write in assembler like real programmers do. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Nov 5 05:37:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Nov 2009 19:37:27 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF27A25.90604@ar.media.kyoto-u.ac.jp> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> On Thu, Nov 5, 2009 at 6:15 PM, Charles R Harris wrote: > > Well, I don't agree that it is slow or complicated ;) Hey, if we needed code > that went zoom, zoom, zoom, we wouldn't be using python, now would we. If we do not care about speed at all, there is no point implementing those in C :) The functionalities which depend on the representation are in macro/inline, making them functions would considerably slow them down I am afraid. But I found a way to do it at compile time - it would just need special-case for windows and universal build to work properly, David From millman at berkeley.edu Thu Nov 5 05:51:44 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 5 Nov 2009 02:51:44 -0800 Subject: [Numpy-discussion] ANN: SciPy India 2009 Call for Presentations Message-ID: Call for Presentations ====================== The SciPy India 2009 Program Committee is currently developing the conference program. ?We are seeking presentations from industry as well as the academic world. We look forward to hearing your recent breakthroughs using Python! Please read the full `call for papers `_. SciPy India 2009 Conference --------------------------- The first `SciPy India Conference `_ will be held from December 12th to 17th, 2009 at the `Technopark in Trivandrum `_, Kerala, India. The theme of the conference will be "Scientific Python in Action" with respect to application and teaching. ?We are pleased to have Travis Oliphant, the creator and lead developer of `numpy `_ as the keynote speaker. Please register `here `_. Important Dates --------------- * Friday, Nov. 20: Abstracts Due * Friday, Nov. 27: Announce accepted talks, post schedule * Saturday-Sunday, Dec. 12-13 ? ? ? ? ? ? ? Conference * Monday-Tuesday, Dec. 14-15 ? ? ? ?Tutorials * Wednesday-Thursday, Dec. 16-17 ? ? Sprints Organizers ---------- * Jarrod Millman, Neuroscience Institute, UC Berkeley, USA (Conference Co-Chair) * Prabhu Ramachandran, Department of Aerospace Engineering, ?IIT Bombay, India (Conference Co-Chair) * FOSSEE Team Sponsors -------- * National Mission On Education through ICT - Ministry of Human Resource Development, Government of India * SPACE-Kerala (India) * Kerala State IT Mission(KSITM) * SIG-FOSS Of CSI From bsouthey at gmail.com Thu Nov 5 08:43:48 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 5 Nov 2009 07:43:48 -0600 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: On Wed, Nov 4, 2009 at 9:31 PM, Reckoner wrote: > FYI, I uploaded the two files in question to the numpy ticket > > ? ?http://projects.scipy.org/numpy/ticket/1284 > > Thanks! > > > On Wed, Nov 4, 2009 at 3:56 PM, Bruce Southey wrote: >> On Wed, Nov 4, 2009 at 8:06 AM, Reckoner wrote: >>> Here's an example: >>> >>> On winxp 64-bit: >>> >>> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on >>> win32 >>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> import numpy >>>>>> import cPickle >>>>>> a = numpy.eye(10) >>>>>> cPickle.dump(a,open('from32bitxp.pkl','w')) >>>>>> import numpy.core.multiarray >>>>>> numpy.__version__ >>> '1.0.4' >>>>>> >>> >>> On linux 64 bit: >>> >>> Python 2.5.4 (r254:67916, Feb ?5 2009, 19:52:35) >>> [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> import numpy >>>>>> import cPickle >>>>>> cPickle.load(open('from32bitxp.pkl')) >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ImportError: No module named multiarray >>>>>> numpy.__version__ >>> '1.2.1' >>>>>> import numpy.core.multiarray >>>>>> >>> >>> Note that I transfer the from32bitxp file from the winxp32 machine to >>> the linux host. Also, I've tried this with version 1.3 on winxp and >>> get the same problem on the linux host. >>> >>> Here's more interesting info: >>> >>> On linux: >>> >>>>>> a = numpy.eye(10) >>>>>> cPickle.dump(a,open('from64bitLinux.pkl','w')) >>> >>> upon transferring the file to winxp 32 and on winxp32: >>> >>>>>> cPickle.load(open('from64bitLinux.pkl')) >>> >>> See? No problem going from linux to winxp32; but problems going the other way. >>> >>> Please let me know if you need more info on this. >>> >>> Any help appreciated. >>> >>> On Tue, Nov 3, 2009 at 4:55 AM, Bruce Southey wrote: >>>> On Mon, Nov 2, 2009 at 6:31 PM, Reckoner wrote: >>>>> thanks for the suggestion! I will look into it. The other thing is >>>>> that the numpy arrays in question are actually embedded in another >>>>> object. When I convert the numpy arrays into plain lists, and then >>>>> cPickle them, there is no problem with any of the larger objects. That >>>>> is the way we are currently working around this issue. >>>>> >>>>> Thanks again. >>>>> >>>>> On Mon, Nov 2, 2009 at 2:43 PM, Bruce Southey wrote: >>>>>> On Mon, Nov 2, 2009 at 2:42 PM, Reckoner wrote: >>>>>>> Anybody have any ideas here? >>>>>>> >>>>>>> Otherwise, I'm thinking this should be posted to the numpy bugs list. >>>>>>> What's the best way to report a bug of this kind? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> On Fri, Oct 30, 2009 at 5:48 PM, Reckoner wrote: >>>>>>>>> Robert Kern wrote: >>>>>>>>> You can import numpy.core.multiarray on both machines? >>>>>>>> >>>>>>>> Yes. For each machine separately, you can cPickle files with numpy >>>>>>>> arrays without problems loading/dumping. The problem comes from >>>>>>>> transferring the win32 cPickle'd files to Linux 64 bit and then trying >>>>>>>> to load them. Transferring cPickle'd files that do *not* have numpy >>>>>>>> arrays work as expected. In other words, cPICKLE'd lists transfer fine >>>>>>>> back and forth between the two machines. In fact, we currently get >>>>>>>> around this problem by converting the numpy arrays to lists, >>>>>>>> transferring them, and then re-numpy-ing them on the respective hosts >>>>>>>> >>>>>>>> thanks. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Oct 30, 2009 at 11:13 AM, Reckoner wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> % python -c 'import numpy.core.multiarray' >>>>>>>>> >>>>>>>>> works just fine, but when I try to load a file that I have transferred >>>>>>>>> from another machine running Windows to one running Linux, I get: >>>>>>>>> >>>>>>>>> % ?python -c 'import cPickle;a=cPickle.load(open("matrices.pkl"))' >>>>>>>>> >>>>>>>>> Traceback (most recent call last): >>>>>>>>> ?File "", line 1, in >>>>>>>>> ImportError: No module named multiarray >>>>>>>>> >>>>>>>>> otherwise, cPickle works normally when transferring files that *do* >>>>>>>>> not contain numpy arrays. >>>>>>>>> >>>>>>>>> I am using version 1.2 on both machines. It's not so easy for me to >>>>>>>>> change versions, by the way, since this is the version that my working >>>>>>>>> group has decided on to standardize on for this effort. >>>>>>>>> >>>>>>>>> >>>>>>>>> Any help appreciated. >>>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> NumPy-Discussion mailing list >>>>>>> NumPy-Discussion at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>>>> >>>>>> >>>>>> Have you have tried the other Cookbook approaches: >>>>>> http://www.scipy.org/Cookbook/InputOutput >>>>>> Like using numpy's own array io functions - load/save(z)? >>>>>> (seems to work between 64-bit Windows 7 and 64-bit Linux - each has >>>>>> different numpy versions) >>>>>> >>>>>> Bruce >>>>>> _______________________________________________ >>>> >>>> Can you provide you provide a small self-contained example of the >>>> problem including object creation especially as your example does not >>>> import numpy? >>>> >>>> Really you have to start at the beginning (like pickling and >>>> transferring numpy arrays) and then increase the complexity to include >>>> the object. >>>> >>>> >>>> Bruce >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> Hi, >> I did not see the file 'from32bitxp.pkl'. It would really help if you >> can provide the full example that includes the creation of the >> complete object so at least people could try doing the same process >> with the object that you are using. >> >> Bruce >> _______________________________________________ Hi I read both of those files on a 32bit Windows vista system running Python 2.6.4 and numpy 1.3. Perhaps no surprise as it appears that both files are identical. Bruce From pav+sp at iki.fi Thu Nov 5 08:47:51 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Thu, 5 Nov 2009 13:47:51 +0000 (UTC) Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines References: Message-ID: Wed, 04 Nov 2009 19:21:57 -0800, Reckoner wrote: > Bruce : > The file in question was created as shown in the prior e-mail. Here it > is again: > >>> cPickle.dump(a,open('from32bitxp.pkl','w')) Your pickle files produced on Windows are contaminated by \r\n line feeds. AFAIK, pickle is supposed to be a binary stream. You need to open the files in binary 'wb' mode, not in text 'w' mode. -- Pauli Virtanen From charlesr.harris at gmail.com Thu Nov 5 11:12:26 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 09:12:26 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 3:37 AM, David Cournapeau wrote: > On Thu, Nov 5, 2009 at 6:15 PM, Charles R Harris > wrote: > > > > > Well, I don't agree that it is slow or complicated ;) Hey, if we needed > code > > that went zoom, zoom, zoom, we wouldn't be using python, now would we. > > If we do not care about speed at all, there is no point implementing > those in C :) The functionalities which depend on the representation > are in macro/inline, making them functions would considerably slow > them down I am afraid. > > But I found a way to do it at compile time - it would just need > special-case for windows and universal build to work properly, > > So you are going to leave us all hanging here in curiosity? What is your solution? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaston.fiore at gmail.com Thu Nov 5 12:24:55 2009 From: gaston.fiore at gmail.com (Gaston Fiore) Date: Thu, 5 Nov 2009 12:24:55 -0500 Subject: [Numpy-discussion] http://www.scipy.org/Installing_SciPy down Message-ID: <51dfb45c0911050924w1ee5494cg2cb9a9f19c046587@mail.gmail.com> Hello, It seems that http://www.scipy.org/Installing_SciPy among other URLs with links on http://numpy.scipy.org/ is down. Does anyone know an estimate for when they'll be back up? Thanks, -Gaston From robert.kern at gmail.com Thu Nov 5 12:32:33 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 5 Nov 2009 11:32:33 -0600 Subject: [Numpy-discussion] http://www.scipy.org/Installing_SciPy down In-Reply-To: <51dfb45c0911050924w1ee5494cg2cb9a9f19c046587@mail.gmail.com> References: <51dfb45c0911050924w1ee5494cg2cb9a9f19c046587@mail.gmail.com> Message-ID: <3d375d730911050932g2e4e95f2pdbe5cd74e99353e0@mail.gmail.com> On Thu, Nov 5, 2009 at 11:24, Gaston Fiore wrote: > Hello, > > It seems that http://www.scipy.org/Installing_SciPy among other URLs > with links on http://numpy.scipy.org/ is down. Does anyone know an > estimate for when they'll be back up? We're on it. Thanks! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Nov 5 12:37:19 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 5 Nov 2009 11:37:19 -0600 Subject: [Numpy-discussion] http://www.scipy.org/Installing_SciPy down In-Reply-To: <51dfb45c0911050924w1ee5494cg2cb9a9f19c046587@mail.gmail.com> References: <51dfb45c0911050924w1ee5494cg2cb9a9f19c046587@mail.gmail.com> Message-ID: <3d375d730911050937v6f17ab73y3dc2218675658f38@mail.gmail.com> On Thu, Nov 5, 2009 at 11:24, Gaston Fiore wrote: > Hello, > > It seems that http://www.scipy.org/Installing_SciPy among other URLs > with links on http://numpy.scipy.org/ is down. Does anyone know an > estimate for when they'll be back up? It's back up now. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From reckoner at gmail.com Thu Nov 5 15:11:32 2009 From: reckoner at gmail.com (Reckoner) Date: Thu, 5 Nov 2009 12:11:32 -0800 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: Pauli : Thanks for your reply. Using 'wb' instead of 'w' for the file mode evidently makes this problem go away. The mystery is that why does this work with plain lists *and* 'w' and not with numpy arrays and 'w'. In other words, why is it that numpy.array needs the binary mode while list does not? Also, for my own edification, how did you know that my "pickle files produced on Windows were contaminated by \r\n line feeds". Thanks again! On Thu, Nov 5, 2009 at 5:47 AM, Pauli Virtanen wrote: > Wed, 04 Nov 2009 19:21:57 -0800, Reckoner wrote: >> Bruce : >> The file in question was created as shown in the prior e-mail. Here it >> is again: >> >>>> cPickle.dump(a,open('from32bitxp.pkl','w')) > > Your pickle files produced on Windows are contaminated by \r\n line > feeds. AFAIK, pickle is supposed to be a binary stream. > > You need to open the files in binary 'wb' mode, not in text 'w' mode. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Nov 5 15:17:07 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 5 Nov 2009 14:17:07 -0600 Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines In-Reply-To: References: Message-ID: <3d375d730911051217m2ac22e74sf12d2f15925133b7@mail.gmail.com> On Thu, Nov 5, 2009 at 14:11, Reckoner wrote: > Pauli : > > Thanks for your reply. Using 'wb' instead of 'w' for the file mode > evidently makes this problem go away. > > The mystery is that why does this work with plain lists *and* 'w' and > not with numpy arrays and 'w'. In other words, why is it that > numpy.array needs the binary mode while list does not? I'm not sure what to say except "because it does". Pickles don't necessarily contain binary data, particularly using the default protocol=0 and only builtin types, but they may. Always use 'wb'. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav+sp at iki.fi Thu Nov 5 15:29:03 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Thu, 5 Nov 2009 20:29:03 +0000 (UTC) Subject: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines References: Message-ID: Thu, 05 Nov 2009 12:11:32 -0800, Reckoner wrote: > Pauli : > > Thanks for your reply. Using 'wb' instead of 'w' for the file mode > evidently makes this problem go away. > > The mystery is that why does this work with plain lists *and* 'w' and > not with numpy arrays and 'w'. In other words, why is it that > numpy.array needs the binary mode while list does not? Requiring binary is not specific to numpy, but just probably the fact that when you have \r\n linefeeds, pickle ends up trying to load a module called "numpy.core.multiarray\r" which does not exists. Same thing happens with any other types -- it's just that lists are builtin types and don't need any modules loaded to unpickle properly. > Also, for my own edification, how did you know that my "pickle files > produced on Windows were contaminated by \r\n line feeds". I looked in the two files you attached on Numpy trac, and noticed that they were different (sha1sum). -- Pauli Virtanen From d.l.goldsmith at gmail.com Thu Nov 5 16:35:31 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Nov 2009 13:35:31 -0800 Subject: [Numpy-discussion] Formatting uint64 number In-Reply-To: References: Message-ID: <45d1ab480911051335x142dee15kbcae53ba2bc12123@mail.gmail.com> On Sun, Nov 1, 2009 at 8:49 PM, Charles R Harris wrote: > > On Sun, Nov 1, 2009 at 8:37 PM, Thomas Robitaille < > thomas.robitaille at gmail.com> wrote: > >> Hello, >> >> I have a question concerning uint64 numbers - let's say I want to >> format a uint64 number that is > 2**31, at the moment it's necessary >> to wrap the numpy number inside long before formatting >> >> In [3]: "%40i" % np.uint64(2**64-1) >> Out[3]: ' -1' >> >> In [4]: "%40i" % long(np.uint64(2**64-1)) >> Out[4]: ' 18446744073709551615' >> >> Would it be easy to modify numpy such that it automatically converts >> uint64 numbers to long() instead of int() when implicitly converted to >> python types? >> >> Hmm, I suspect this is a bug whose source is uint64 having an integer > conversion function as part of the type whereas it should be undefined. A > quick look at the source leaves me befuddled, so tracking down just how this > happens might be a bit of work. > > Chuck > Based on this, unless someone has other ideas/input, Thomas, please file a bug ticket at: http://projects.scipy.org/numpy (Note: you'll have to register first - please see http://scipy.org/BugReportfor more on that, as well as bug report submission guidelines.) Thanks, DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Thu Nov 5 16:54:19 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Nov 2009 13:54:19 -0800 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> Interesting thread, which leaves me wondering two things: is it documented somewhere (e.g., at the IEEE site) precisely how many *decimal* mantissae are representable using the 64-bit IEEE standard for float representation (if that makes sense); and are such decimal mantissae uniformly distributed, i.e., is the distance between each the same, or does the incommensurability of two and five (and thus binary and decimal) make these irregularly, or bi-periodically, i.e., every other mantissae are regularly spaced? DG On Mon, Nov 2, 2009 at 7:09 AM, Anne Archibald wrote: > 2009/11/1 Thomas Robitaille : > > Hi, > > > > I'm trying to generate random 64-bit integer values for integers and > > floats using Numpy, within the entire range of valid values for that > > type. To generate random 32-bit floats, I can use: > > Others have addressed why this is giving bogus results. But it's worth > thinking about what you're actually trying to do. If it's "fuzz > tests", that is, producing unrestricted random floats to feed to a > function, then even if this worked it wouldn't produce what you want: > it will never produce floats of order unity, for example. > > If you want do what you described, you could produce floats uniformly > from -1 to 1 and then multiply the results by the largest > representable float. > > If you want "random floats", I suggest generating an array of random > bytes and reinterpreting them as floats. You'll get a fair number of > NaNs and infinities, so you may want to take only the finite values > and regenerate the rest. This will give you some numbers from all over > the range of floats, including tiny values (and denormalized values, > which are a potentially important special case). > > Anne > > > np.random.uniform(low=np.finfo(np.float32).min,high=np.finfo > > (np.float32).max,size=10) > > > > which gives for example > > > > array([ 1.47351436e+37, 9.93620693e+37, 2.22893053e+38, > > -3.33828977e+38, 1.08247781e+37, -8.37481260e+37, > > 2.64176554e+38, -2.72207226e+37, 2.54790459e+38, > > -2.47883866e+38]) > > > > but if I try and use this for 64-bit numbers, i.e. > > > > np.random.uniform(low=np.finfo(np.float64).min,high=np.finfo > > (np.float64).max,size=10) > > > > I get > > > > array([ Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf, Inf]) > > > > Similarly, for integers, I can successfully generate random 32-bit > > integers: > > > > np.random.random_integers(np.iinfo(np.int32).min,high=np.iinfo > > (np.int32).max,size=10) > > > > which gives > > > > array([-1506183689, 662982379, -1616890435, -1519456789, 1489753527, > > -604311122, 2034533014, 449680073, -444302414, > > -1924170329]) > > > > but am unsuccessful for 64-bit integers, i.e. > > > > np.random.random_integers(np.iinfo(np.int64).min,high=np.iinfo > > (np.int64).max,size=10) > > > > which produces the following error: > > > > OverflowError: long int too large to convert to int > > > > Is this expected behavior, or are these bugs? > > > > Thanks for any help, > > > > Thomas > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Nov 5 18:26:53 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 5 Nov 2009 18:26:53 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> Message-ID: <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > Interesting thread, which leaves me wondering two things: is it > documented > somewhere (e.g., at the IEEE site) precisely how many *decimal* > mantissae > are representable using the 64-bit IEEE standard for float > representation > (if that makes sense); IEEE-754 says nothing about decimal representations aside from how to round when converting to and from strings. You have to provide/accept *at least* 9 decimal digits in the significand for single-precision and 17 for double-precision (section 5.6). AFAIK implementations will vary in how they handle cases where a binary significand would yield more digits than that. David From charlesr.harris at gmail.com Thu Nov 5 18:36:40 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 16:36:40 -0700 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> Message-ID: On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley wrote: > On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > > > Interesting thread, which leaves me wondering two things: is it > > documented > > somewhere (e.g., at the IEEE site) precisely how many *decimal* > > mantissae > > are representable using the 64-bit IEEE standard for float > > representation > > (if that makes sense); > > IEEE-754 says nothing about decimal representations aside from how to > round when converting to and from strings. You have to provide/accept > *at least* 9 decimal digits in the significand for single-precision > and 17 for double-precision (section 5.6). AFAIK implementations will > vary in how they handle cases where a binary significand would yield > more digits than that. > > I believe that was the argument for the extended precision formats. The givien number of decimal digits is sufficient to recover the same float that produced them if a slightly higher precision is used in the conversion. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Nov 5 21:04:38 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Nov 2009 21:04:38 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> Message-ID: <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> On Thu, Nov 5, 2009 at 6:36 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley > wrote: >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >> >> > Interesting thread, which leaves me wondering two things: is it >> > documented >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >> > mantissae >> > are representable using the 64-bit IEEE standard for float >> > representation >> > (if that makes sense); >> >> IEEE-754 says nothing about decimal representations aside from how to >> round when converting to and from strings. You have to provide/accept >> *at least* 9 decimal digits in the significand for single-precision >> and 17 for double-precision (section 5.6). AFAIK implementations will >> vary in how they handle cases where a binary significand would yield >> more digits than that. >> > > I believe that was the argument for the extended precision formats. The > givien number of decimal digits is sufficient to recover the same float that > produced them if a slightly higher precision is used in the conversion. > > Chuck >From the discussion for the floating point representation, it seems that a uniform random number generator would have a very coarse grid in the range for example -1e30 to +1e30 compared to interval -0.5,0.5. How many points can be represented by a float in [-0.5,0.5] compared to [1e30, 1e30+1.]? If I interpret this correctly, then there are as many floating point numbers in [0,1] as in [1,inf), or am I misinterpreting this. So how does a PRNG handle a huge interval of uniform numbers? Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From gokhansever at gmail.com Thu Nov 5 21:21:15 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 5 Nov 2009 20:21:15 -0600 Subject: [Numpy-discussion] Comparing variable time-shifted two measurements Message-ID: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> Hello, I have two aircraft based aerosol measurements. The first one is dccnConSTP (blue), and the latter is CPCConc (red) as shown in this screen capture. ( http://img513.imageshack.us/img513/7498/ccncpclag.png). My goal is to compare these two measurements. It is expected to see that they must have a positive correlation throughout the flight. However, the instrument that gives CPCConc was experiencing a sampling issue and therefore making a varying time-shifted measurements with respect to the first instrument. (From the first box it is about 20 seconds, 24 from the seconds before the dccnConSTP measurements shows up.) In other words in different altitude levels, I have varying time differences in between these two measurements in terms of their shapes. So, my goal turns to addressing this variable shifting issue before I start doing the comparisons. Is there a known automated approach to correct this mentioned varying-lag issue? If so, how? Thank you. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 5 21:23:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 19:23:01 -0700 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 7:04 PM, wrote: > On Thu, Nov 5, 2009 at 6:36 PM, Charles R Harris > wrote: > > > > > > On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley > > wrote: > >> > >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > >> > >> > Interesting thread, which leaves me wondering two things: is it > >> > documented > >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* > >> > mantissae > >> > are representable using the 64-bit IEEE standard for float > >> > representation > >> > (if that makes sense); > >> > >> IEEE-754 says nothing about decimal representations aside from how to > >> round when converting to and from strings. You have to provide/accept > >> *at least* 9 decimal digits in the significand for single-precision > >> and 17 for double-precision (section 5.6). AFAIK implementations will > >> vary in how they handle cases where a binary significand would yield > >> more digits than that. > >> > > > > I believe that was the argument for the extended precision formats. The > > givien number of decimal digits is sufficient to recover the same float > that > > produced them if a slightly higher precision is used in the conversion. > > > > Chuck > > >From the discussion for the floating point representation, it seems that > a uniform random number generator would have a very coarse grid > in the range for example -1e30 to +1e30 compared to interval -0.5,0.5. > > How many points can be represented by a float in [-0.5,0.5] compared > to [1e30, 1e30+1.]? > If I interpret this correctly, then there are as many floating point > numbers > in [0,1] as in [1,inf), or am I misinterpreting this. > > So how does a PRNG handle a huge interval of uniform numbers? > > There are several implementations, but the ones I'm familiar with reduce to scaling. If the rng produces random unsigned integers, then the range of integers is scaled to the interval [0,1). The variations involve explicit scaling (portable) or bit twiddling of the IEEE formats. In straight forward scaling some ranges of the random integers may not map 1-1, so the unused bits are masked off first; if you want doubles you only need 52 bits, etc. For bit twiddling there is an implicit 1 in the mantissa, so the basic range works out to [1,2), but that can be fixed by subtracting 1 from the result. Handling larger ranges than [0,1) just involves another scaling. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Nov 5 21:53:41 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Nov 2009 21:53:41 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> Message-ID: <1cd32cbb0911051853w680c9c25yffc9f1521e6e2d97@mail.gmail.com> On Thu, Nov 5, 2009 at 9:23 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 7:04 PM, wrote: >> >> On Thu, Nov 5, 2009 at 6:36 PM, Charles R Harris >> wrote: >> > >> > >> > On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley >> > wrote: >> >> >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >> >> >> >> > Interesting thread, which leaves me wondering two things: is it >> >> > documented >> >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >> >> > mantissae >> >> > are representable using the 64-bit IEEE standard for float >> >> > representation >> >> > (if that makes sense); >> >> >> >> IEEE-754 says nothing about decimal representations aside from how to >> >> round when converting to and from strings. You have to provide/accept >> >> *at least* 9 decimal digits in the significand for single-precision >> >> and 17 for double-precision (section 5.6). AFAIK implementations will >> >> vary in how they handle cases where a binary significand would yield >> >> more digits than that. >> >> >> > >> > I believe that was the argument for the extended precision formats. The >> > givien number of decimal digits is sufficient to recover the same float >> > that >> > produced them if a slightly higher precision is used in the conversion. >> > >> > Chuck >> >> >From the discussion for the floating point representation, it seems that >> a uniform random number generator would have a very coarse grid >> in the range for example -1e30 to +1e30 compared to interval -0.5,0.5. >> >> How many points can be represented by a float in [-0.5,0.5] compared >> to [1e30, 1e30+1.]? >> If I interpret this correctly, then there are as many floating point >> numbers >> in [0,1] as in [1,inf), or am I misinterpreting this. >> >> So how does a PRNG handle a huge interval of uniform numbers? >> > > There are several implementations, but the ones I'm familiar with reduce to > scaling. If the rng produces random unsigned integers, then the range of > integers is scaled to the interval [0,1). The variations involve explicit > scaling (portable) or bit twiddling of the IEEE formats. In straight forward > scaling some ranges of the random integers may not map 1-1, so the unused > bits are masked off first; if you want doubles you only need 52 bits, etc. > For bit twiddling there is an implicit 1 in the mantissa, so the basic range > works out to [1,2), but that can be fixed by subtracting 1 from the result. > Handling larger ranges than [0,1) just involves another scaling. So, since this is then a discrete distribution, what is the number of points in the support? (My guess would be 2**52, but I don't know much about numerical representations.) This would be the largest set of integers that could be generated without gaps in the distribution, and would determine the grid size for floating point random variables. (?) for Davids example: low, high = -1e307, 1e307 np.random.uniform(low, high, 100) # much more reasonable this would imply a grid size of >>> 2*1e307/2.**52 4.4408920985006261e+291 or something similar. (floating points are not very dense in the real line.) Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From cournape at gmail.com Thu Nov 5 22:02:30 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Nov 2009 12:02:30 +0900 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> Message-ID: <5b8d13220911051902p50cdeeaqbfeea6ae52aef4d2@mail.gmail.com> On Fri, Nov 6, 2009 at 6:54 AM, David Goldsmith wrote: > Interesting thread, which leaves me wondering two things: is it documented > somewhere (e.g., at the IEEE site) precisely how many *decimal* mantissae > are representable using the 64-bit IEEE standard for float representation > (if that makes sense); and are such decimal mantissae uniformly distributed They are definitely not uniformly distributed: that's why two numbers are close around 1 when they have only a few EPS difference, but around 1e100, you have to add quite a few EPS to even get a different number at all. That may be my audio processing background, but I like to think about float as numbers which have the same relative precision at any "level" - a kind of dB scale. If you want numbers with a fixed number of decimals, you need a fixed point representation. David From charlesr.harris at gmail.com Thu Nov 5 22:28:04 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 20:28:04 -0700 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911051853w680c9c25yffc9f1521e6e2d97@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> <1cd32cbb0911051853w680c9c25yffc9f1521e6e2d97@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 7:53 PM, wrote: > On Thu, Nov 5, 2009 at 9:23 PM, Charles R Harris > wrote: > > > > > > On Thu, Nov 5, 2009 at 7:04 PM, wrote: > >> > >> On Thu, Nov 5, 2009 at 6:36 PM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley < > dwf at cs.toronto.edu> > >> > wrote: > >> >> > >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > >> >> > >> >> > Interesting thread, which leaves me wondering two things: is it > >> >> > documented > >> >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* > >> >> > mantissae > >> >> > are representable using the 64-bit IEEE standard for float > >> >> > representation > >> >> > (if that makes sense); > >> >> > >> >> IEEE-754 says nothing about decimal representations aside from how to > >> >> round when converting to and from strings. You have to provide/accept > >> >> *at least* 9 decimal digits in the significand for single-precision > >> >> and 17 for double-precision (section 5.6). AFAIK implementations will > >> >> vary in how they handle cases where a binary significand would yield > >> >> more digits than that. > >> >> > >> > > >> > I believe that was the argument for the extended precision formats. > The > >> > givien number of decimal digits is sufficient to recover the same > float > >> > that > >> > produced them if a slightly higher precision is used in the > conversion. > >> > > >> > Chuck > >> > >> >From the discussion for the floating point representation, it seems > that > >> a uniform random number generator would have a very coarse grid > >> in the range for example -1e30 to +1e30 compared to interval -0.5,0.5. > >> > >> How many points can be represented by a float in [-0.5,0.5] compared > >> to [1e30, 1e30+1.]? > >> If I interpret this correctly, then there are as many floating point > >> numbers > >> in [0,1] as in [1,inf), or am I misinterpreting this. > >> > >> So how does a PRNG handle a huge interval of uniform numbers? > >> > > > > There are several implementations, but the ones I'm familiar with reduce > to > > scaling. If the rng produces random unsigned integers, then the range of > > integers is scaled to the interval [0,1). The variations involve explicit > > scaling (portable) or bit twiddling of the IEEE formats. In straight > forward > > scaling some ranges of the random integers may not map 1-1, so the unused > > bits are masked off first; if you want doubles you only need 52 bits, > etc. > > For bit twiddling there is an implicit 1 in the mantissa, so the basic > range > > works out to [1,2), but that can be fixed by subtracting 1 from the > result. > > Handling larger ranges than [0,1) just involves another scaling. > > So, since this is then a discrete distribution, what is the number of > points > in the support? (My guess would be 2**52, but I don't know much about > numerical representations.) > > This would be the largest set of integers that could be generated > without gaps in the distribution, and would determine the grid size > for floating point random variables. (?) > > Yes. > for Davids example: > low, high = -1e307, 1e307 > np.random.uniform(low, high, 100) # much more reasonable > > this would imply a grid size of > >>> 2*1e307/2.**52 > 4.4408920985006261e+291 > > or something similar. (floating points are not very dense in the real > line.) > > Yes, or rather, they are more or less logarithmically distributed. Floats are basically logarithms base 2 with a mantissa of fixed precision. Actual logarithm code uses the exponent and does a correction to the mantissa, i.e., takes the logarithm base two of numbers in the range [1,2). Random integers are treated differently. To get random integers in a given range, say [0,100], the bitstream produced by the rng would be broken up into 7 bit chunks [0, 127] that are used in a sampling algorithm that loops until a value in the range is produced. So any range of integers can be produced. Code for that comes with the Mersenne Twister, but I don't know if numpy uses it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Thu Nov 5 22:42:51 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Nov 2009 19:42:51 -0800 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> Message-ID: <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley wrote: > On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > > > Interesting thread, which leaves me wondering two things: is it > > documented > > somewhere (e.g., at the IEEE site) precisely how many *decimal* > > mantissae > > are representable using the 64-bit IEEE standard for float > > representation > > (if that makes sense); > > IEEE-754 says nothing about decimal representations aside from how to > round when converting to and from strings. You have to provide/accept > *at least* 9 decimal digits in the significand for single-precision > and 17 for double-precision (section 5.6). AFAIK implementations will > vary in how they handle cases where a binary significand would yield > more digits than that. > I was actually more interested in the opposite situation, where the decimal representation (which is what a user would most likely provide) doesn't have a finite binary expansion: what happens then, something analogous to the decimal "rule of fives"? DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Nov 5 23:07:41 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 5 Nov 2009 23:07:41 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> Message-ID: 2009/11/5 David Goldsmith : > On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley > wrote: >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >> >> > Interesting thread, which leaves me wondering two things: is it >> > documented >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >> > mantissae >> > are representable using the 64-bit IEEE standard for float >> > representation >> > (if that makes sense); >> >> IEEE-754 says nothing about decimal representations aside from how to >> round when converting to and from strings. You have to provide/accept >> *at least* 9 decimal digits in the significand for single-precision >> and 17 for double-precision (section 5.6). AFAIK implementations will >> vary in how they handle cases where a binary significand would yield >> more digits than that. > > I was actually more interested in the opposite situation, where the decimal > representation (which is what a user would most likely provide) doesn't have > a finite binary expansion: what happens then, something analogous to the > decimal "rule of fives"? If you interpret "0.1" as 1/10, then this is a very general floating-point issue: how you you round off numbers you can't represent exactly? The usual answer (leaving aside internal extended-precision shenanigans) is to round, with the rule that when you're exactly between two floating-point numbers you round to the one that's even, rather than always up or always down (the numerical analysis wizards tell us that this is more numerically stable). Anne > DG > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Thu Nov 5 23:14:39 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Nov 2009 23:14:39 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> Message-ID: <1cd32cbb0911052014u34218b54k61d00ec9d5cdf8f7@mail.gmail.com> On Thu, Nov 5, 2009 at 10:42 PM, David Goldsmith wrote: > On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley > wrote: >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >> >> > Interesting thread, which leaves me wondering two things: is it >> > documented >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >> > mantissae >> > are representable using the 64-bit IEEE standard for float >> > representation >> > (if that makes sense); >> >> IEEE-754 says nothing about decimal representations aside from how to >> round when converting to and from strings. You have to provide/accept >> *at least* 9 decimal digits in the significand for single-precision >> and 17 for double-precision (section 5.6). AFAIK implementations will >> vary in how they handle cases where a binary significand would yield >> more digits than that. > > I was actually more interested in the opposite situation, where the decimal > representation (which is what a user would most likely provide) doesn't have > a finite binary expansion: what happens then, something analogous to the > decimal "rule of fives"? Since according to my calculations there are only about >>> 4* 10**17 * 308 123200000000000000000L double-precision floats, there are huge gaps in the floating point representation of the real line. Any user input or calculation result just gets converted to the closest float. Josef > > DG > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Thu Nov 5 23:17:33 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Nov 2009 23:17:33 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <1cd32cbb0911051804t5fd4d383o8b224870a51b8e43@mail.gmail.com> <1cd32cbb0911051853w680c9c25yffc9f1521e6e2d97@mail.gmail.com> Message-ID: <1cd32cbb0911052017j2e6b8121pc55a252f2d9e6d54@mail.gmail.com> On Thu, Nov 5, 2009 at 10:28 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 7:53 PM, wrote: >> >> On Thu, Nov 5, 2009 at 9:23 PM, Charles R Harris >> wrote: >> > >> > >> > On Thu, Nov 5, 2009 at 7:04 PM, wrote: >> >> >> >> On Thu, Nov 5, 2009 at 6:36 PM, Charles R Harris >> >> wrote: >> >> > >> >> > >> >> > On Thu, Nov 5, 2009 at 4:26 PM, David Warde-Farley >> >> > >> >> > wrote: >> >> >> >> >> >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >> >> >> >> >> >> > Interesting thread, which leaves me wondering two things: is it >> >> >> > documented >> >> >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >> >> >> > mantissae >> >> >> > are representable using the 64-bit IEEE standard for float >> >> >> > representation >> >> >> > (if that makes sense); >> >> >> >> >> >> IEEE-754 says nothing about decimal representations aside from how >> >> >> to >> >> >> round when converting to and from strings. You have to >> >> >> provide/accept >> >> >> *at least* 9 decimal digits in the significand for single-precision >> >> >> and 17 for double-precision (section 5.6). AFAIK implementations >> >> >> will >> >> >> vary in how they handle cases where a binary significand would yield >> >> >> more digits than that. >> >> >> >> >> > >> >> > I believe that was the argument for the extended precision formats. >> >> > The >> >> > givien number of decimal digits is sufficient to recover the same >> >> > float >> >> > that >> >> > produced them if a slightly higher precision is used in the >> >> > conversion. >> >> > >> >> > Chuck >> >> >> >> >From the discussion for the floating point representation, it seems >> >> that >> >> a uniform random number generator would have a very coarse grid >> >> in the range for example -1e30 to +1e30 compared to interval -0.5,0.5. >> >> >> >> How many points can be represented by a float in [-0.5,0.5] compared >> >> to [1e30, 1e30+1.]? >> >> If I interpret this correctly, then there are as many floating point >> >> numbers >> >> in [0,1] as in [1,inf), or am I misinterpreting this. >> >> >> >> So how does a PRNG handle a huge interval of uniform numbers? >> >> >> > >> > There are several implementations, but the ones I'm familiar with reduce >> > to >> > scaling. If the rng produces random unsigned integers, then the range of >> > integers is scaled to the interval [0,1). The variations involve >> > explicit >> > scaling (portable) or bit twiddling of the IEEE formats. In straight >> > forward >> > scaling some ranges of the random integers may not map 1-1, so the >> > unused >> > bits are masked off first; if you want doubles you only need 52 bits, >> > etc. >> > For bit twiddling there is an implicit 1 in the mantissa, so the basic >> > range >> > works out to [1,2), but that can be fixed by subtracting 1 from the >> > result. >> > Handling larger ranges than [0,1) just involves another scaling. >> >> So, since this is then a discrete distribution, what is the number of >> points >> in the support? (My guess would be 2**52, but I don't know much about >> numerical representations.) >> >> This would be the largest set of integers that could be generated >> without gaps in the distribution, and would determine the grid size >> for floating point random variables. (?) >> > > Yes. > >> >> for Davids example: >> low, high = -1e307, 1e307 >> np.random.uniform(low, high, 100) # much more reasonable >> >> this would imply a grid size of >> >>> 2*1e307/2.**52 >> 4.4408920985006261e+291 >> >> or something similar. (floating points are not very dense in the real >> line.) >> > > Yes, or rather, they are more or less logarithmically distributed. Floats > are basically logarithms base 2 with a mantissa of fixed precision. Actual > logarithm code uses the exponent and does a correction to the mantissa, > i.e., takes the logarithm base two of numbers in the range [1,2). > > Random integers are treated differently. To get random integers in a given > range, say [0,100], the bitstream produced by the rng would be broken up > into 7 bit chunks [0, 127] that are used in a sampling algorithm that loops > until a value in the range is produced. So any range of integers can be > produced. Code for that comes with the Mersenne Twister, but I don't know if > numpy uses it. > > Chuck Thanks for the explanation. It's good to understand some of the limitations, but I don't think in any Monte Carlo, I will have a distribution with tails that are fat enough that the discretization becomes a problem. Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From peridot.faceted at gmail.com Thu Nov 5 23:32:37 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 5 Nov 2009 23:32:37 -0500 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911052014u34218b54k61d00ec9d5cdf8f7@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> <1cd32cbb0911052014u34218b54k61d00ec9d5cdf8f7@mail.gmail.com> Message-ID: 2009/11/5 : > On Thu, Nov 5, 2009 at 10:42 PM, David Goldsmith > wrote: >> On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley >> wrote: >>> >>> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: >>> >>> > Interesting thread, which leaves me wondering two things: is it >>> > documented >>> > somewhere (e.g., at the IEEE site) precisely how many *decimal* >>> > mantissae >>> > are representable using the 64-bit IEEE standard for float >>> > representation >>> > (if that makes sense); >>> >>> IEEE-754 says nothing about decimal representations aside from how to >>> round when converting to and from strings. You have to provide/accept >>> *at least* 9 decimal digits in the significand for single-precision >>> and 17 for double-precision (section 5.6). AFAIK implementations will >>> vary in how they handle cases where a binary significand would yield >>> more digits than that. >> >> I was actually more interested in the opposite situation, where the decimal >> representation (which is what a user would most likely provide) doesn't have >> a finite binary expansion: what happens then, something analogous to the >> decimal "rule of fives"? > > Since according to my calculations there are only about > >>>> 4* 10**17 * 308 > 123200000000000000000L More straightforwardly, it's not too far below 2**64. > double-precision floats, there are huge gaps in the floating point > representation of the real line. > Any user input or calculation result just gets converted to the > closest float. Yes. But the "huge" gaps are only huge in absolute size; in fractional error they're always about the same size. They are usually only an issue if you're representing numbers in a small range with a huge offset. To take a not-so-random example, you could be representing times in days since November 17 1858, when what you care about are the microsecond-scale differences in photon arrival times. Even then you're probably okay as long as you compute directly (t[i] = i*dt/86400 + start_t) rather than having some sort of running accumulator (t[i]=t; t += dt/86400). Professor Kahan's reasoning for using doubles for most everything is that they generally have so much more precision than you actually need that you can get away with being sloppy. Anne > Josef > > >> >> DG >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Nov 5 23:46:20 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 21:46:20 -0700 Subject: [Numpy-discussion] Comparing variable time-shifted two measurements In-Reply-To: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> References: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 7:21 PM, G?khan Sever wrote: > Hello, > > I have two aircraft based aerosol measurements. The first one is dccnConSTP > (blue), and the latter is CPCConc (red) as shown in this screen capture. ( > http://img513.imageshack.us/img513/7498/ccncpclag.png). My goal is to > compare these two measurements. It is expected to see that they must have a > positive correlation throughout the flight. However, the instrument that > gives CPCConc was experiencing a sampling issue and therefore making a > varying time-shifted measurements with respect to the first instrument. > (From the first box it is about 20 seconds, 24 from the seconds before the > dccnConSTP measurements shows up.) In other words in different altitude > levels, I have varying time differences in between these two measurements in > terms of their shapes. So, my goal turns to addressing this variable > shifting issue before I start doing the comparisons. > > Is there a known automated approach to correct this mentioned varying-lag > issue? If so, how? > > Can you be more explicit about the varying lags? Do you know what the lags are? If not, how much information about them, such as range, probability, etc., can you supply? Are there dropouts, or do the lags varying sort of continuously? Can you parameterise the lags, so on and so forth. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 5 23:49:12 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 21:49:12 -0700 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <1cd32cbb0911052014u34218b54k61d00ec9d5cdf8f7@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> <1cd32cbb0911052014u34218b54k61d00ec9d5cdf8f7@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 9:14 PM, wrote: > On Thu, Nov 5, 2009 at 10:42 PM, David Goldsmith > wrote: > > On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley > > wrote: > >> > >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > >> > >> > Interesting thread, which leaves me wondering two things: is it > >> > documented > >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* > >> > mantissae > >> > are representable using the 64-bit IEEE standard for float > >> > representation > >> > (if that makes sense); > >> > >> IEEE-754 says nothing about decimal representations aside from how to > >> round when converting to and from strings. You have to provide/accept > >> *at least* 9 decimal digits in the significand for single-precision > >> and 17 for double-precision (section 5.6). AFAIK implementations will > >> vary in how they handle cases where a binary significand would yield > >> more digits than that. > > > > I was actually more interested in the opposite situation, where the > decimal > > representation (which is what a user would most likely provide) doesn't > have > > a finite binary expansion: what happens then, something analogous to the > > decimal "rule of fives"? > > Since according to my calculations there are only about > > >>> 4* 10**17 * 308 > 123200000000000000000L > > double-precision floats, there are huge gaps in the floating point > representation of the real line. > 2**64 minus some flags for nans and such. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 6 00:00:03 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 22:00:03 -0700 Subject: [Numpy-discussion] Comparing variable time-shifted two measurements In-Reply-To: References: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> Message-ID: On Thu, Nov 5, 2009 at 9:46 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 7:21 PM, G?khan Sever wrote: > >> Hello, >> >> I have two aircraft based aerosol measurements. The first one is >> dccnConSTP (blue), and the latter is CPCConc (red) as shown in this screen >> capture. (http://img513.imageshack.us/img513/7498/ccncpclag.png). My goal >> is to compare these two measurements. It is expected to see that they must >> have a positive correlation throughout the flight. However, the instrument >> that gives CPCConc was experiencing a sampling issue and therefore making a >> varying time-shifted measurements with respect to the first instrument. >> (From the first box it is about 20 seconds, 24 from the seconds before the >> dccnConSTP measurements shows up.) In other words in different altitude >> levels, I have varying time differences in between these two measurements in >> terms of their shapes. So, my goal turns to addressing this variable >> shifting issue before I start doing the comparisons. >> >> Is there a known automated approach to correct this mentioned varying-lag >> issue? If so, how? >> >> > Can you be more explicit about the varying lags? Do you know what the lags > are? If not, how much information about them, such as range, probability, > etc., can you supply? Are there dropouts, or do the lags varying sort of > continuously? Can you parameterise the lags, so on and so forth. > > What does a straight forward scatter plot look like? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Nov 6 00:35:50 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 06 Nov 2009 14:35:50 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> Message-ID: <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > So you are going to leave us all hanging here in curiosity? What is > your solution? I had to sleep :) The solution is based on parsing the generated binary code - that's how MPFR is doing it, so it has been tested in the wild. The code to compile is something like: /* "before" is 16 bytes to ensure there's no padding between it and "x". * We're not expecting any "long double" bigger than 16 bytes or with * alignment requirements stricter than 16 bytes. */ typedef long double test_type; struct { char before[16]; test_type x; char after[8]; } foo = { { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\001', '\043', '\105', '\147', '\211', '\253', '\315', '\357' }, -123456789.0, { '\376', '\334', '\272', '\230', '\166', '\124', '\062', '\020' } }; They compile it, parse with awk from the dump od -f. I have the python code to replace the awk script and of -f, and added support for Intel quad precision (Intel compiler can generate it). The only problems is universal build. I don't think the latter has any solution which is not runtime based, so we will have to special case for it, as we did for endianess detection. I have tested it with sparc64, sparc32, ppc, amd64 cross compilers, work on all of them (but that only include linux OS). David From d.l.goldsmith at gmail.com Fri Nov 6 01:04:51 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 5 Nov 2009 22:04:51 -0800 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <7B7442EE-6097-4C1E-8DC3-D8A7DD41ABC4@cs.toronto.edu> <45d1ab480911051942k1c7fc59eyffdaefa2b1200d7a@mail.gmail.com> Message-ID: <45d1ab480911052204m1a98038bk102420405d200f77@mail.gmail.com> On Thu, Nov 5, 2009 at 8:07 PM, Anne Archibald wrote: > 2009/11/5 David Goldsmith : > > On Thu, Nov 5, 2009 at 3:26 PM, David Warde-Farley > > wrote: > >> > >> On 5-Nov-09, at 4:54 PM, David Goldsmith wrote: > >> > >> > Interesting thread, which leaves me wondering two things: is it > >> > documented > >> > somewhere (e.g., at the IEEE site) precisely how many *decimal* > >> > mantissae > >> > are representable using the 64-bit IEEE standard for float > >> > representation > >> > (if that makes sense); > >> > >> IEEE-754 says nothing about decimal representations aside from how to > >> round when converting to and from strings. You have to provide/accept > >> *at least* 9 decimal digits in the significand for single-precision > >> and 17 for double-precision (section 5.6). AFAIK implementations will > >> vary in how they handle cases where a binary significand would yield > >> more digits than that. > > > > I was actually more interested in the opposite situation, where the > decimal > > representation (which is what a user would most likely provide) doesn't > have > > a finite binary expansion: what happens then, something analogous to the > > decimal "rule of fives"? > > If you interpret "0.1" as 1/10, then this is a very general > floating-point issue: how you you round off numbers you can't > represent exactly? The usual answer (leaving aside internal > extended-precision shenanigans) is to round, with the rule that when > you're exactly between two floating-point numbers you round to the one > that's even, rather than always up or always down (the numerical > analysis wizards tell us that this is more numerically stable). > More numerically stable, or simply unbiased, since half the time your rounding up, and the other half rounding down (that's the rationale as it was explained to me). DG > > Anne > > > DG > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 6 01:44:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Nov 2009 23:44:36 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > So you are going to leave us all hanging here in curiosity? What is > > your solution? > > I had to sleep :) > > The solution is based on parsing the generated binary code - that's how > MPFR is doing it, so it has been tested in the wild. The code to compile > is something like: > > /* "before" is 16 bytes to ensure there's no padding between it and "x". > * We're not expecting any "long double" bigger than 16 bytes or with > * alignment requirements stricter than 16 bytes. */ > typedef long double test_type; > > struct { > char before[16]; > test_type x; > char after[8]; > } foo = { > { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', > '\001', '\043', '\105', '\147', '\211', '\253', '\315', '\357' }, > -123456789.0, > { '\376', '\334', '\272', '\230', '\166', '\124', '\062', '\020' } > }; > > They compile it, parse with awk from the dump od -f. > So the before/after bits are tags that mark the beginning/end of the type for the parse? Any particular reason not to use a string? "David Cournapeau" should work :-) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Nov 6 02:10:58 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 06 Nov 2009 16:10:58 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF28137.5050002@ar.media.kyoto-u.ac.jp> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> Message-ID: <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau > > > wrote: > > Charles R Harris wrote: > > So you are going to leave us all hanging here in curiosity? What is > > your solution? > > I had to sleep :) > > The solution is based on parsing the generated binary code - > that's how > MPFR is doing it, so it has been tested in the wild. The code to > compile > is something like: > > /* "before" is 16 bytes to ensure there's no padding between it > and "x". > * We're not expecting any "long double" bigger than 16 bytes > or with > * alignment requirements stricter than 16 bytes. */ > typedef long double test_type; > > struct { > char before[16]; > test_type x; > char after[8]; > } foo = { > { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', > '\001', '\043', '\105', '\147', '\211', '\253', '\315', > '\357' }, > -123456789.0, > { '\376', '\334', '\272', '\230', '\166', '\124', '\062', > '\020' } > }; > > They compile it, parse with awk from the dump od -f. > > > So the before/after bits are tags that mark the beginning/end of the > type for the parse? Any particular reason not to use a string? As for why not using my name, I am not that megalomaniac, at least not yet :) od prints a dump in octal form, so a string is not much more readable. I am not sure why they use od instead of another dump format - but there is value if keeping the same format as them for testing, all other things being equal. Anyway, the detection code is checked in in svn now, I just need to fix universal build issue, but this should not be difficult, David From fperez.net at gmail.com Fri Nov 6 07:18:53 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 6 Nov 2009 04:18:53 -0800 Subject: [Numpy-discussion] [ANN] For SF Bay Area residents: a discussion with Guido at the Berkeley Py4Science seminar In-Reply-To: References: Message-ID: On Tue, Nov 3, 2009 at 11:28 AM, Fernando Perez wrote: > if you reside in the San Francisco Bay Area, you may be interested in > a meeting we'll be having tomorrow November 4 (2-4 pm), as part of our > regular py4science meeting series. ?Guido van Rossum, the creator of > the Python language, will visit for a session where we will first do a > very rapid overview of a number of scientific projects that use Python > (in a lightning talk format) and then we will have an open discussion > with Guido with hopefully interesting questions going in both > directions. ?The meeting is open to all, bring your questions! Video of the event: http://www.archive.org/details/ucb_py4science_2009_11_04_Guido_van_Rossum Slides: http://fperez.org/py4science/2009_guido_ucb/index.html A few blog posts about it: - Guido: http://neopythonic.blogspot.com/2009/11/python-in-scientific-world.html - Jarrod: http://jarrodmillman.blogspot.com/2009/11/visit-from-guido-van-rossum.html - Matthew: http://nipyworld.blogspot.com/2009/11/guido-van-rossum-talks-about-python-3.html - Me: http://fdoperez.blogspot.com/2009/11/guido-van-rossum-at-uc-berkeleys.html Attendance was excellent (standing room only, and I saw some people leave because it was too full). Many thanks to all the presenters! Cheers, f From trevor at notcows.com Fri Nov 6 07:57:18 2009 From: trevor at notcows.com (Trevor Clarke) Date: Fri, 6 Nov 2009 07:57:18 -0500 Subject: [Numpy-discussion] ctypes and numpy Message-ID: <7bde5d400911060457u7d8b220csb83dd725ae2a9fbd@mail.gmail.com> I've found some info on ctypes and numpy but it mostly tells me how to pass numpy objects to C funcs. I've got a C func which returns a c_void_p to a buffer and I'd like to access the data using numpy without creating a copy of the data. Could someone point me in the right direction? -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 6 10:05:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 08:05:39 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <4AF2876E.4080406@ar.media.kyoto-u.ac.jp> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Nov 6, 2009 at 12:10 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > > > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau > > > > > wrote: > > > > Charles R Harris wrote: > > > So you are going to leave us all hanging here in curiosity? What is > > > your solution? > > > > I had to sleep :) > > > > The solution is based on parsing the generated binary code - > > that's how > > MPFR is doing it, so it has been tested in the wild. The code to > > compile > > is something like: > > > > /* "before" is 16 bytes to ensure there's no padding between it > > and "x". > > * We're not expecting any "long double" bigger than 16 bytes > > or with > > * alignment requirements stricter than 16 bytes. */ > > typedef long double test_type; > > > > struct { > > char before[16]; > > test_type x; > > char after[8]; > > } foo = { > > { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', > > '\001', '\043', '\105', '\147', '\211', '\253', '\315', > > '\357' }, > > -123456789.0, > > { '\376', '\334', '\272', '\230', '\166', '\124', '\062', > > '\020' } > > }; > > > > They compile it, parse with awk from the dump od -f. > > > > > > So the before/after bits are tags that mark the beginning/end of the > > type for the parse? Any particular reason not to use a string? > As for why not using my name, I am not that megalomaniac, at least not > yet :) > > od prints a dump in octal form, so a string is not much more readable. I > am not sure why they use od instead of another dump format - but there > is value if keeping the same format as them for testing, all other > things being equal. > > Is od available on all platforms? Couldn't you just read the file using Python? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Nov 6 12:10:49 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 02:10:49 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> On Sat, Nov 7, 2009 at 12:05 AM, Charles R Harris wrote: > > > On Fri, Nov 6, 2009 at 12:10 AM, David Cournapeau > wrote: >> >> Charles R Harris wrote: >> > >> > >> > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau >> > > >> > wrote: >> > >> > ? ? Charles R Harris wrote: >> > ? ? > So you are going to leave us all hanging here in curiosity? What >> > is >> > ? ? > your solution? >> > >> > ? ? I had to sleep :) >> > >> > ? ? The solution is based on parsing the generated binary code - >> > ? ? that's how >> > ? ? MPFR is doing it, so it has been tested in the wild. The code to >> > ? ? compile >> > ? ? is something like: >> > >> > ? ? /* "before" is 16 bytes to ensure there's no padding between it >> > ? ? and "x". >> > ? ? ?* ? ?We're not expecting any "long double" bigger than 16 bytes >> > ? ? or with >> > ? ? ?* ? ? ? alignment requirements stricter than 16 bytes. ?*/ >> > ? ? typedef long double test_type; >> > >> > ? ? struct { >> > ? ? ? ? ? ?char ? ? ? ? before[16]; >> > ? ? ? ? ? ?test_type ? ?x; >> > ? ? ? ? ? ?char ? ? ? ? after[8]; >> > ? ? } foo = { >> > ? ? ? ? ? ?{ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', >> > ? ? ? ? ? ? ?'\001', '\043', '\105', '\147', '\211', '\253', '\315', >> > ? ? '\357' }, >> > ? ? ? ? ? ?-123456789.0, >> > ? ? ? ? ? ?{ '\376', '\334', '\272', '\230', '\166', '\124', '\062', >> > ? ? '\020' } >> > ? ? }; >> > >> > ? ? They compile it, parse with awk from the dump od -f. >> > >> > >> > So the before/after bits are tags that mark the beginning/end of the >> > type for the parse? Any particular reason not to use a string? >> As for why not using my name, I am not that megalomaniac, at least not >> yet :) >> >> od prints a dump in octal form, so a string is not much more readable. I >> am not sure why they use od instead of another dump format - but there >> is value if keeping the same format as them for testing, all other >> things being equal. >> > > Is od available on all platforms? Couldn't you just read the file using > Python? That's actually what I have done. I am pretty sure windows does not had od, for once :) It seems to work pretty well on every platform I have tested. Once I take the time to fix the configuration for universal build, I will use those for low level long double handling, to implement nextafterl - this WE hopefully David From cournape at gmail.com Fri Nov 6 12:16:40 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 02:16:40 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF28F3E.2060306@ar.media.kyoto-u.ac.jp> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> Message-ID: <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> On Sat, Nov 7, 2009 at 2:10 AM, David Cournapeau wrote: > On Sat, Nov 7, 2009 at 12:05 AM, Charles R Harris > wrote: >> >> >> On Fri, Nov 6, 2009 at 12:10 AM, David Cournapeau >> wrote: >>> >>> Charles R Harris wrote: >>> > >>> > >>> > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau >>> > > >>> > wrote: >>> > >>> > ? ? Charles R Harris wrote: >>> > ? ? > So you are going to leave us all hanging here in curiosity? What >>> > is >>> > ? ? > your solution? >>> > >>> > ? ? I had to sleep :) >>> > >>> > ? ? The solution is based on parsing the generated binary code - >>> > ? ? that's how >>> > ? ? MPFR is doing it, so it has been tested in the wild. The code to >>> > ? ? compile >>> > ? ? is something like: >>> > >>> > ? ? /* "before" is 16 bytes to ensure there's no padding between it >>> > ? ? and "x". >>> > ? ? ?* ? ?We're not expecting any "long double" bigger than 16 bytes >>> > ? ? or with >>> > ? ? ?* ? ? ? alignment requirements stricter than 16 bytes. ?*/ >>> > ? ? typedef long double test_type; >>> > >>> > ? ? struct { >>> > ? ? ? ? ? ?char ? ? ? ? before[16]; >>> > ? ? ? ? ? ?test_type ? ?x; >>> > ? ? ? ? ? ?char ? ? ? ? after[8]; >>> > ? ? } foo = { >>> > ? ? ? ? ? ?{ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', >>> > ? ? ? ? ? ? ?'\001', '\043', '\105', '\147', '\211', '\253', '\315', >>> > ? ? '\357' }, >>> > ? ? ? ? ? ?-123456789.0, >>> > ? ? ? ? ? ?{ '\376', '\334', '\272', '\230', '\166', '\124', '\062', >>> > ? ? '\020' } >>> > ? ? }; >>> > >>> > ? ? They compile it, parse with awk from the dump od -f. >>> > >>> > >>> > So the before/after bits are tags that mark the beginning/end of the >>> > type for the parse? Any particular reason not to use a string? >>> As for why not using my name, I am not that megalomaniac, at least not >>> yet :) >>> >>> od prints a dump in octal form, so a string is not much more readable. I >>> am not sure why they use od instead of another dump format - but there >>> is value if keeping the same format as them for testing, all other >>> things being equal. >>> >> >> Is od available on all platforms? Couldn't you just read the file using >> Python? > > That's actually what I have done. I am pretty sure windows does not > had od, for once :) the implementation is here if you want to look at it: http://svn.scipy.org/svn/numpy/trunk/numpy/core/setup_common.py The code for pyod is ugly, there are wasteful back and forth conversions, but it works, so.. David From charlesr.harris at gmail.com Fri Nov 6 12:28:49 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 10:28:49 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> Message-ID: On Fri, Nov 6, 2009 at 10:16 AM, David Cournapeau wrote: > On Sat, Nov 7, 2009 at 2:10 AM, David Cournapeau > wrote: > > On Sat, Nov 7, 2009 at 12:05 AM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Nov 6, 2009 at 12:10 AM, David Cournapeau > >> wrote: > >>> > >>> Charles R Harris wrote: > >>> > > >>> > > >>> > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau > >>> > > > >>> > wrote: > >>> > > >>> > Charles R Harris wrote: > >>> > > So you are going to leave us all hanging here in curiosity? > What > >>> > is > >>> > > your solution? > >>> > > >>> > I had to sleep :) > >>> > > >>> > The solution is based on parsing the generated binary code - > >>> > that's how > >>> > MPFR is doing it, so it has been tested in the wild. The code to > >>> > compile > >>> > is something like: > >>> > > >>> > /* "before" is 16 bytes to ensure there's no padding between it > >>> > and "x". > >>> > * We're not expecting any "long double" bigger than 16 bytes > >>> > or with > >>> > * alignment requirements stricter than 16 bytes. */ > >>> > typedef long double test_type; > >>> > > >>> > struct { > >>> > char before[16]; > >>> > test_type x; > >>> > char after[8]; > >>> > } foo = { > >>> > { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', > >>> > '\001', '\043', '\105', '\147', '\211', '\253', '\315', > >>> > '\357' }, > >>> > -123456789.0, > >>> > { '\376', '\334', '\272', '\230', '\166', '\124', '\062', > >>> > '\020' } > >>> > }; > >>> > > >>> > They compile it, parse with awk from the dump od -f. > >>> > > >>> > > >>> > So the before/after bits are tags that mark the beginning/end of the > >>> > type for the parse? Any particular reason not to use a string? > >>> As for why not using my name, I am not that megalomaniac, at least not > >>> yet :) > >>> > >>> od prints a dump in octal form, so a string is not much more readable. > I > >>> am not sure why they use od instead of another dump format - but there > >>> is value if keeping the same format as them for testing, all other > >>> things being equal. > >>> > >> > >> Is od available on all platforms? Couldn't you just read the file using > >> Python? > > > > That's actually what I have done. I am pretty sure windows does not > > had od, for once :) > > the implementation is here if you want to look at it: > > http://svn.scipy.org/svn/numpy/trunk/numpy/core/setup_common.py > > The code for pyod is ugly, there are wasteful back and forth > conversions, but it works, so.. > > I already looked at that commit. That's kind of a neat hack. I take it that there is no platform out there at the moment that has big endian ieee extended doubles? And if not now, probably never. I expect the same trick would detect PPC also, although that probably isn't necessary. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 6 12:40:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 10:40:13 -0700 Subject: [Numpy-discussion] Deprecate np.max/np.min ? Message-ID: People import these functions -- yes, they shouldn't do that -- and the python builtin versions are overloaded, causing hard to locate errors. It would be easy enough to put a small wrapper around the current definition in core/__init__.py and raise a deprecation warning. The round function might be another candidate for deprecation. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Nov 6 12:57:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Nov 2009 12:57:03 -0500 Subject: [Numpy-discussion] Deprecate np.max/np.min ? In-Reply-To: References: Message-ID: <1cd32cbb0911060957p732c6472jad79b1867bbf04b4@mail.gmail.com> On Fri, Nov 6, 2009 at 12:40 PM, Charles R Harris wrote: > People import these functions -- yes, they shouldn't do that -- and the > python builtin versions are overloaded, causing hard to locate errors. It > would be easy enough to put a small wrapper around the current definition in > core/__init__.py and raise a deprecation warning. The round function might > be another candidate for deprecation. > > Thoughts? I like np.min, np.max (corresponding to min, max methods of arrays), and I don't like the `a` in front of the names. Users need to learn to get their namespaces correct, because there are a lot of functions that have different meanings and syntax across packages. It follows the same pattern for missing brackets as many of the other functions, where I often have typos. >>> np.min(3,2) 3 >>> np.sum(3,2) 3 >>> np.ones(3,2) Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\core\numeric.py", line 1567, in ones a = empty(shape, dtype, order) TypeError: data type not understood Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Fri Nov 6 13:09:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 11:09:29 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> Message-ID: On Fri, Nov 6, 2009 at 10:16 AM, David Cournapeau wrote: > On Sat, Nov 7, 2009 at 2:10 AM, David Cournapeau > wrote: > > On Sat, Nov 7, 2009 at 12:05 AM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Nov 6, 2009 at 12:10 AM, David Cournapeau > >> wrote: > >>> > >>> Charles R Harris wrote: > >>> > > >>> > > >>> > On Thu, Nov 5, 2009 at 10:35 PM, David Cournapeau > >>> > > > >>> > wrote: > >>> > > >>> > Charles R Harris wrote: > >>> > > So you are going to leave us all hanging here in curiosity? > What > >>> > is > >>> > > your solution? > >>> > > >>> > I had to sleep :) > >>> > > >>> > The solution is based on parsing the generated binary code - > >>> > that's how > >>> > MPFR is doing it, so it has been tested in the wild. The code to > >>> > compile > >>> > is something like: > >>> > > >>> > /* "before" is 16 bytes to ensure there's no padding between it > >>> > and "x". > >>> > * We're not expecting any "long double" bigger than 16 bytes > >>> > or with > >>> > * alignment requirements stricter than 16 bytes. */ > >>> > typedef long double test_type; > >>> > > >>> > struct { > >>> > char before[16]; > >>> > test_type x; > >>> > char after[8]; > >>> > } foo = { > >>> > { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', > >>> > '\001', '\043', '\105', '\147', '\211', '\253', '\315', > >>> > '\357' }, > >>> > -123456789.0, > >>> > { '\376', '\334', '\272', '\230', '\166', '\124', '\062', > >>> > '\020' } > >>> > }; > >>> > > >>> > They compile it, parse with awk from the dump od -f. > >>> > > >>> > > >>> > So the before/after bits are tags that mark the beginning/end of the > >>> > type for the parse? Any particular reason not to use a string? > >>> As for why not using my name, I am not that megalomaniac, at least not > >>> yet :) > >>> > >>> od prints a dump in octal form, so a string is not much more readable. > I > >>> am not sure why they use od instead of another dump format - but there > >>> is value if keeping the same format as them for testing, all other > >>> things being equal. > >>> > >> > >> Is od available on all platforms? Couldn't you just read the file using > >> Python? > > > > That's actually what I have done. I am pretty sure windows does not > > had od, for once :) > > the implementation is here if you want to look at it: > > http://svn.scipy.org/svn/numpy/trunk/numpy/core/setup_common.py > > The code for pyod is ugly, there are wasteful back and forth > conversions, but it works, so.. > > I'm a bit concerned about the MPFR license. It is LGPL, but I'm not sure what that means when it comes to using bits of the source code, as opposed to linking. I always found the LGPL a bit confusing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Nov 6 13:22:59 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 06 Nov 2009 20:22:59 +0200 Subject: [Numpy-discussion] Deprecate np.max/np.min ? In-Reply-To: <1cd32cbb0911060957p732c6472jad79b1867bbf04b4@mail.gmail.com> References: <1cd32cbb0911060957p732c6472jad79b1867bbf04b4@mail.gmail.com> Message-ID: <1257531778.5931.7.camel@idol> pe, 2009-11-06 kello 12:57 -0500, josef.pktd at gmail.com kirjoitti: [clip] > It follows the same pattern for missing brackets as many of the other > functions, where I often have typos. > > >>> np.min(3,2) > 3 > >>> np.sum(3,2) > 3 > >>> np.ones(3,2) [clip] I think you are thinking about Matlab now... The issue here is that Numpy's `min` and `max`, if used like their Python counterparts, give surprising results. `sum`, `ones`, `round`, etc. don't have this problem. The prepended `a` in the beginning is IMO not a big price to pay for reduced overloading. OTOH, one can ask, why is np.min(3, 2) allowed when np.min([3], 2) gives "ValueError: axis(=2) out of bounds". It seems to me that 0-dimensional objects should accept only None as the axis? (Fixing this would also make misuse of np.min and np.max more difficult.) -- Pauli Virtanen From cournape at gmail.com Fri Nov 6 13:30:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 03:30:09 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911050237taef803fu553ae4345069ee5d@mail.gmail.com> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> Message-ID: <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> On Sat, Nov 7, 2009 at 3:09 AM, Charles R Harris wrote: > > I'm a bit concerned about the MPFR license. I used the same technique, but the implementation is from scratch, in a different language. Strictly speaking, I guess this could be considered as derivative work, but given that we are open source, I doubt mpfr copyright holders would be bothered with this. FWIW, all build tools with configure checks use techniques pioneered by autoconf (cmake, waf, scons). I can check with mpfr developers if you are really worried, though, cheers, David From charlesr.harris at gmail.com Fri Nov 6 13:37:45 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 11:37:45 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> Message-ID: On Fri, Nov 6, 2009 at 11:30 AM, David Cournapeau wrote: > On Sat, Nov 7, 2009 at 3:09 AM, Charles R Harris > wrote: > > > > > I'm a bit concerned about the MPFR license. > > I used the same technique, but the implementation is from scratch, in > a different language. Strictly speaking, I guess this could be > considered as derivative work, but given that we are open source, I > doubt mpfr copyright holders would be bothered with this. FWIW, all > build tools with configure checks use techniques pioneered by autoconf > (cmake, waf, scons). > > I can check with mpfr developers if you are really worried, though, > > IANAL, plus I have always found the LPGL a bit confusing as opposed to the other licenses. So I don't know what it all means. It just seemed something to check. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Nov 6 13:41:28 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 03:41:28 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <4AF3B5B6.5020104@ar.media.kyoto-u.ac.jp> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> Message-ID: <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> On Sat, Nov 7, 2009 at 3:37 AM, Charles R Harris wrote: > IANAL, plus I have always found the LPGL a bit confusing as opposed to the > other licenses. I was not so concerned about LGPL itself, but rather whether my code constitutes derivative work. I don't think LGPL has much meaning for python code, especially pure python code (and m4 for that matter) - I think it should be considered the same as GPL for all practical purpose. David From cournape at gmail.com Fri Nov 6 13:46:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 03:46:05 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> Message-ID: <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> On Sat, Nov 7, 2009 at 3:41 AM, David Cournapeau wrote: > I don't think LGPL has much meaning for > python code, especially pure python code (and m4 for that matter) This is funny - besides pyqt, the only LGPL reference with python I found was twisted, and they claim that: "... It would be a halfway accurate statement that I selected the LGPL exactly because it doesn't make any sense." http://twistedmatrix.com/pipermail/twisted-python/2004-May/007946.html David From josef.pktd at gmail.com Fri Nov 6 14:01:02 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Nov 2009 14:01:02 -0500 Subject: [Numpy-discussion] Deprecate np.max/np.min ? In-Reply-To: <1257531778.5931.7.camel@idol> References: <1cd32cbb0911060957p732c6472jad79b1867bbf04b4@mail.gmail.com> <1257531778.5931.7.camel@idol> Message-ID: <1cd32cbb0911061101j54f4f3fbm1d3abb18be43b417@mail.gmail.com> On Fri, Nov 6, 2009 at 1:22 PM, Pauli Virtanen wrote: > pe, 2009-11-06 kello 12:57 -0500, josef.pktd at gmail.com kirjoitti: > [clip] >> It follows the same pattern for missing brackets as many of the other >> functions, where I often have typos. >> >> >>> np.min(3,2) >> 3 >> >>> np.sum(3,2) >> 3 >> >>> np.ones(3,2) > [clip] > > I think you are thinking about Matlab now... I'm using (almost) as much matlab as python, so some confusion and typos show up very regularly. > > The issue here is that Numpy's `min` and `max`, if used like their > Python counterparts, give surprising results. `sum`, `ones`, `round`, > etc. don't have this problem. The prepended `a` in the beginning is IMO > not a big price to pay for reduced overloading. > > OTOH, one can ask, why is > > ? ? ? ?np.min(3, 2) > > allowed when > > ? ? ? ?np.min([3], 2) > > gives "ValueError: axis(=2) out of bounds". It seems to me that > 0-dimensional objects should accept only None as the axis? (Fixing this > would also make misuse of np.min and np.max more difficult.) That was also my first reaction, I think this axis check would be a good improvement. I wanted to say we could improve the documentation for numpy.min and add a warning, but I don't find the function min in the docs (only amin and the method) In general, I think shadowing python's built-in functions is a bad idea and the missing namespaces make some code difficult to read and makes it difficult to find problems. I could always do import numpy.amin as npmin which would be only a change by one dot. But still breaking a lot of code, just for some cosmetic changes ? What about np.ma.min, np.recarray.min (raises exception)? (matrix.min is only available as method) Josef > > -- > Pauli Virtanen > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From thomas.robitaille at gmail.com Fri Nov 6 14:02:16 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 6 Nov 2009 11:02:16 -0800 (PST) Subject: [Numpy-discussion] Formatting uint64 number In-Reply-To: <45d1ab480911051335x142dee15kbcae53ba2bc12123@mail.gmail.com> References: <45d1ab480911051335x142dee15kbcae53ba2bc12123@mail.gmail.com> Message-ID: <26230859.post@talk.nabble.com> David Goldsmith-5 wrote: > > Thomas, please file a bug ticket > Done - http://projects.scipy.org/numpy/ticket/1287 Cheers, Thomas -- View this message in context: http://old.nabble.com/Formatting-uint64-number-tp26158021p26230859.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From thomas.robitaille at gmail.com Fri Nov 6 14:10:10 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 6 Nov 2009 11:10:10 -0800 (PST) Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> Message-ID: <26230862.post@talk.nabble.com> Anne Archibald wrote: > > If you want "random floats", I suggest generating an array of random > bytes and reinterpreting them as floats. You'll get a fair number of > NaNs and infinities, so you may want to take only the finite values > and regenerate the rest. This will give you some numbers from all over > the range of floats, including tiny values (and denormalized values, > which are a potentially important special case). > This is a very good point - I've decided to do this. For the record, here is the function I wrote for ints: def random_int_array(dtype, shape): n = np.product(shape) n = n * np.iinfo(dtype).bits / 8 s = "".join(chr(random.randrange(0, 256)) for i in xrange(n)) return np.fromstring(s, dtype=dtype).reshape(shape) And I wrote something similar for floats. Cheers, Thomas -- View this message in context: http://old.nabble.com/Random-int64-and-float64-numbers-tp26157235p26230862.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From thomas.robitaille at gmail.com Fri Nov 6 14:11:55 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 6 Nov 2009 14:11:55 -0500 Subject: [Numpy-discussion] Bits info in np.finfo Message-ID: <47600B70-69BF-46EA-AAFF-869184778C42@gmail.com> Hi, Is there a reason that np.finfo does not have a bits attribute? In [6]: np.iinfo(np.int16).bits Out[6]: 16 In [7]: np.finfo(np.float32).bits --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /Users/tom/ in () AttributeError: 'finfo' object has no attribute 'bits' Should I submit a bug report? Thanks, Thomas From charlesr.harris at gmail.com Fri Nov 6 14:23:26 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 12:23:26 -0700 Subject: [Numpy-discussion] Bits info in np.finfo In-Reply-To: <47600B70-69BF-46EA-AAFF-869184778C42@gmail.com> References: <47600B70-69BF-46EA-AAFF-869184778C42@gmail.com> Message-ID: On Fri, Nov 6, 2009 at 12:11 PM, Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > Hi, > > Is there a reason that np.finfo does not have a bits attribute? > > In [6]: np.iinfo(np.int16).bits > Out[6]: 16 > > In [7]: np.finfo(np.float32).bits > --------------------------------------------------------------------------- > AttributeError Traceback (most recent call > last) > > /Users/tom/ in () > > AttributeError: 'finfo' object has no attribute 'bits' > > Should I submit a bug report? > > Maybe a request for enhancement. There are actually two possible interpretations here: actual bits and storage bits. Extended doubles are 80 bits, but stored as 96 or 128 bits for alignment purposes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.robitaille at gmail.com Fri Nov 6 14:41:47 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 6 Nov 2009 14:41:47 -0500 Subject: [Numpy-discussion] masked record arrays Message-ID: <43B4D32C-5C83-4C59-B059-401E8DCDB6E1@gmail.com> Hello, What is the status on masked record arrays? It looks from previous discussions as though there are two implementations but a lot of these discussions seem outdated, so just wondering what the best way to create masked record arrays is at the moment. I need to be able to mask individual fields in individual records. Is this possible? Thanks, Thomas From thomas.robitaille at gmail.com Fri Nov 6 15:39:01 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 6 Nov 2009 15:39:01 -0500 Subject: [Numpy-discussion] object to string for numpy 1.3.0 Message-ID: <28D5BE0D-6174-4BBF-B481-3848273C7346@gmail.com> Hello, Until recently, object to string conversion was buggy: In [1]: import numpy as np In [2]: a = np.array(['4TxismCut','nXe4f0sAs'],dtype=np.object_) In [3]: np.array(a,dtype='|S9') Out[3]: array(['4TxismCun', 'Xe4f0sAs'], dtype='|S9') (notice the strings are not quite the same) This is now fixed in svn, but is there a way to write the conversion differently such that it does work with numpy 1.3.0? (so as not to have to force users of a package to upgrade to numpy 1.4.0) Thanks, Thomas From darryl.wallace at prosensus.ca Fri Nov 6 15:56:57 2009 From: darryl.wallace at prosensus.ca (Darryl Wallace) Date: Fri, 6 Nov 2009 15:56:57 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question Message-ID: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> What I'm doing is importing some data from excel and sometimes there are strings in the worksheet. Often times a user will use an empty cell or a string to represent data that is missing. e.g. from numpy import * mixedArray=array([1, 2, '', 3, 4, 'String'], dtype=object) Two questions: 1) Is there a quick way to find the elements in the array that are the strings without iterating over each element in the array? or 2) Could I quickly turn it into a masked array of type float where all string elements are set as missing points? I've been struggling with this for a while and can't come across a method that will all me to do it without iterating over each element. Any help or pointers in the right direction would be greatly appreciated. Thanks, Darryl -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Nov 6 16:21:26 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 6 Nov 2009 16:21:26 -0500 Subject: [Numpy-discussion] masked record arrays In-Reply-To: <43B4D32C-5C83-4C59-B059-401E8DCDB6E1@gmail.com> References: <43B4D32C-5C83-4C59-B059-401E8DCDB6E1@gmail.com> Message-ID: On Nov 6, 2009, at 2:41 PM, Thomas Robitaille wrote: > Hello, > > What is the status on masked record arrays? It looks from previous > discussions as though there are two implementations but a lot of these > discussions seem outdated, so just wondering what the best way to > create masked record arrays is at the moment. I need to be able to > mask individual fields in individual records. Is this possible? Mmh. With a recent (1.3) version of numpy, you should already be able to mask individual fields of a structured array without problems. If you need fields to be accessed as attributes the np.recarray way, you can give numpy.ma.mrecords.MaskedRecords a try. It's been a while I haven't touched it, so you may run into the occasional bug. FYI, I'm not a big fan of record arrays and tend to prefer structured ones... What two implementations were you talking about ? In any case, feel free to try and please, report any issue you run into with MaskedRecords. Cheers P. From d.l.goldsmith at gmail.com Fri Nov 6 16:24:40 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 6 Nov 2009 13:24:40 -0800 Subject: [Numpy-discussion] Formatting uint64 number In-Reply-To: <26230859.post@talk.nabble.com> References: <45d1ab480911051335x142dee15kbcae53ba2bc12123@mail.gmail.com> <26230859.post@talk.nabble.com> Message-ID: <45d1ab480911061324q489b756cm2d8a20f1d8a1f069@mail.gmail.com> Thanks!!! DG On Fri, Nov 6, 2009 at 11:02 AM, Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > > > David Goldsmith-5 wrote: > > > > Thomas, please file a bug ticket > > > > Done - http://projects.scipy.org/numpy/ticket/1287 > > Cheers, > > Thomas > -- > View this message in context: > http://old.nabble.com/Formatting-uint64-number-tp26158021p26230859.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 6 17:12:52 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Nov 2009 15:12:52 -0700 Subject: [Numpy-discussion] Backout complex changes for next release? Message-ID: There seems to be a rat's nest of problems showing up in scipy due to the recent changes in numpy complex support. The problems are of two basic sorts: 1) reserved name conflicts and 2) file conflicts. The first could be dealt with, but the second might be trickier, both c/c++ defined complex.h but the files are different. C++ has moved on, but not all platforms seem to have followed. Given the proposed schedule for the next numpy release and the developing problems, it seems to me we should back out the changes to complex support and concentrate on more pressing matters. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Nov 6 17:20:39 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 6 Nov 2009 17:20:39 -0500 Subject: [Numpy-discussion] Vector interpolation on a 2D grid Message-ID: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> All, I have a vector of observations (latitude,longitude,value) that I need to interpolate over a given area. So far, I used Jeff W's basemap to transform the lat/lon in arbitrary x/y and construct a regular grid, Robert K's delaunay triangulation routine to project the points on the grid and its nearest-neighbor interpolator to get a 2D array of estimates. Have a look for yourself at http://grab.by/gY3 Some colleagues complained that the result looked a bit too choppy, meaning that too much weight was given to the actual observations. What are my other options ? Thx in advance for any idea P. From jdh2358 at gmail.com Fri Nov 6 17:26:48 2009 From: jdh2358 at gmail.com (John Hunter) Date: Fri, 6 Nov 2009 16:26:48 -0600 Subject: [Numpy-discussion] Fwd: [Distutils] People want CPAN :-) In-Reply-To: <4AF49FCC.7050403@nextday.fi> References: <4AF49FCC.7050403@nextday.fi> Message-ID: <88e473830911061426n683a6c1ak6ead164ee8425b4c@mail.gmail.com> I assume many of you follow distutils, but FYI in case you want to contribute to the discussion ---------- Forwarded message ---------- From: Alex Gr?nholm Date: 2009/11/6 Subject: Re: [Distutils] People want CPAN :-) To: distutils-sig at python.org Guido van Rossum kirjoitti: > > I just found this comment on my blog. People have told me this in > person too, so I believe it is real pain (even if the solution may be > elusive and the suggested solutions may not work). But I don't know > how to improve the world. Is the work on distutils-sig going to be > enough? Or do we need some other kind of work in addition? Do we need > more than PyPI? > > --Guido > > ---------- Forwarded message ---------- > From: dalloliogm > Date: Fri, Nov 6, 2009 at 8:01 AM > Subject: [Neopythonic] New comment on Python in the Scientific World. > To: gvanrossum at gmail.com > > > dalloliogm has left a new comment on your post "Python in the Scientific World": > > Python is suffering a lot in the scientific word, because it has not a > CPAN-like repository. > > PyPI is fine, but it is still far from the level of CPAN, CRAN, > Bioconductor, etc.. > > Scientists who use programming usually have a lot of different > interests and approaches, therefore it is really difficult to write a > package that can be useful to everyone. > Other programming language like Perl and R have repository-like > structure which enable people to download packages easily, and to > upload new ones and organize them withouth having to worry about > having to integrate them into existing packages. > > This is what is happening to biopython now: it is a monolitic package > that it is supposed to work for any bioinformatic problem; but this is > so general that to accomplish that you would need to add a lot of > dependencies, to numpy, networkx, suds, any kind of library. > However, since easy_install is not as ready yet as the counterparts in > other languages, if the biopython developers add too many > dependencies, nobody will be able to install it properly, and nobody > will use it. > I for one did not understand the problem. What does CPAN have that PyPI doesn't? It is natural for packages (distributions, in distutils terms) to have dependencies on each other. Why is this a problem? > > > Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM > > > _______________________________________________ Distutils-SIG maillist ?- ?Distutils-SIG at python.org http://mail.python.org/mailman/listinfo/distutils-sig From rmay31 at gmail.com Fri Nov 6 17:33:23 2009 From: rmay31 at gmail.com (Ryan May) Date: Fri, 6 Nov 2009 16:33:23 -0600 Subject: [Numpy-discussion] Vector interpolation on a 2D grid In-Reply-To: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> Message-ID: On Fri, Nov 6, 2009 at 4:20 PM, Pierre GM wrote: > All, > I have a vector of observations (latitude,longitude,value) that I need > to interpolate over a given area. > So far, I used Jeff W's basemap to transform the lat/lon in arbitrary > x/y and construct a regular grid, Robert K's delaunay triangulation > routine to project the points on the grid and its nearest-neighbor > interpolator to get a 2D array of estimates. > Have a look for yourself at http://grab.by/gY3 > Some colleagues complained that the result looked a bit too choppy, > meaning that too much weight was given to the actual observations. > What are my other options ? > Thx in advance for any idea You could try smoothing the grid with some kind of weighted average (like a gaussian kernel). Out of curiosity, what values are you plotting there? I Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From pav at iki.fi Fri Nov 6 17:45:22 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 07 Nov 2009 00:45:22 +0200 Subject: [Numpy-discussion] Vector interpolation on a 2D grid In-Reply-To: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> Message-ID: <1257547522.5931.24.camel@idol> pe, 2009-11-06 kello 17:20 -0500, Pierre GM kirjoitti: > All, > I have a vector of observations (latitude,longitude,value) that I need > to interpolate over a given area. > So far, I used Jeff W's basemap to transform the lat/lon in > arbitrary x/y and construct a regular grid, Robert K's delaunay > triangulation > routine to project the points on the grid and its nearest-neighbor > interpolator to get a 2D array of estimates. > Have a look for yourself at http://grab.by/gY3 > Some colleagues complained that the result looked a bit too choppy, > meaning that too much weight was given to the actual observations. > What are my other options ? You could try to use linear interpolation from the delaynay package (only supports slicing, so is a bit tricky to use). I may be slightly smoother than the natural neighbour one. For scattered data, when I wasn't happy with delaunay, I've also used the radial basis functions (Rbf) from scipy.interpolate -- something like interpolator = Rbf(x, y, z, function='thin-plate', smooth=1e-9) should give an interpolator that is reasonably smooth between the data points. (I'd guess you can give it leeway in trading interpolation to smoothness by increasing the smooth parameter.) Other Rbf functions than thin-plate splines might also work. I don't think RBF have any monotonicity guarantees, though -- if you have some data points much closer to each other than others, some spurious swings may develop, especially if the closely packed points have large gradients between them. Perhaps another fallback could be SmoothBivariateSpline. But with it, I've often had serious problems: with sparse data I tend to get very nonmonotonic behavior between the data points. -- Pauli Virtanen From gokhansever at gmail.com Fri Nov 6 18:20:15 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 6 Nov 2009 17:20:15 -0600 Subject: [Numpy-discussion] Comparing variable time-shifted two measurements In-Reply-To: References: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> Message-ID: <49d6b3500911061520m2727f0d6j8f8ac28b9cc0153c@mail.gmail.com> On Thu, Nov 5, 2009 at 10:46 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 7:21 PM, G?khan Sever wrote: > >> Hello, >> >> I have two aircraft based aerosol measurements. The first one is >> dccnConSTP (blue), and the latter is CPCConc (red) as shown in this screen >> capture. (http://img513.imageshack.us/img513/7498/ccncpclag.png). My goal >> is to compare these two measurements. It is expected to see that they must >> have a positive correlation throughout the flight. However, the instrument >> that gives CPCConc was experiencing a sampling issue and therefore making a >> varying time-shifted measurements with respect to the first instrument. >> (From the first box it is about 20 seconds, 24 from the seconds before the >> dccnConSTP measurements shows up.) In other words in different altitude >> levels, I have varying time differences in between these two measurements in >> terms of their shapes. So, my goal turns to addressing this variable >> shifting issue before I start doing the comparisons. >> >> Is there a known automated approach to correct this mentioned varying-lag >> issue? If so, how? >> >> > Sorry for the late response. I was working on a lab report and took more than I thought to finish. > Can you be more explicit about the varying lags? Do you know what the lags > are? > I can identify the lag in between the two instrument by going over the time-series data. I know exactly what causes the lag, that is the CPC device cannot maintain 1L/min sample flow rate at higher altitudes. This goes my varying lag complain. The flow rate of the sample air changes depending on the ambient pressure. Unfortunately we couldn't record the sample flow rate from the instrument during the experimentation period. However we have pressure recordings throughout the flights, where I can somehow relate this to correct the variable lag issue. However this requires me to do manual observations on the CPCConc data. (i.e, get some sample points to determine the lag manualy by panning and zooming in the data and looking the pressure levels at these points and coming up with a fit and use that for the correction. > If not, how much information about them, such as range, probability, etc., > can you supply? > I think, I clarified these points by mentioning the pressure dependence. > Are there dropouts, or do the lags varying sort of continuously? Can you > parameterise the lags, so on and so forth. > There are no drop-outs. I don't know how would I parameterize the lags besides using the external pressure information. > > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Fri Nov 6 18:42:10 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 6 Nov 2009 17:42:10 -0600 Subject: [Numpy-discussion] Comparing variable time-shifted two measurements In-Reply-To: References: <49d6b3500911051821l77618452tc3229af345b5d685@mail.gmail.com> Message-ID: <49d6b3500911061542o30dfec5fo6609f61b1c5a50fc@mail.gmail.com> On Thu, Nov 5, 2009 at 11:00 PM, Charles R Harris wrote: > > > On Thu, Nov 5, 2009 at 9:46 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Thu, Nov 5, 2009 at 7:21 PM, G?khan Sever wrote: >> >>> Hello, >>> >>> I have two aircraft based aerosol measurements. The first one is >>> dccnConSTP (blue), and the latter is CPCConc (red) as shown in this screen >>> capture. (http://img513.imageshack.us/img513/7498/ccncpclag.png). My >>> goal is to compare these two measurements. It is expected to see that they >>> must have a positive correlation throughout the flight. However, the >>> instrument that gives CPCConc was experiencing a sampling issue and >>> therefore making a varying time-shifted measurements with respect to the >>> first instrument. (From the first box it is about 20 seconds, 24 from the >>> seconds before the dccnConSTP measurements shows up.) In other words in >>> different altitude levels, I have varying time differences in between these >>> two measurements in terms of their shapes. So, my goal turns to addressing >>> this variable shifting issue before I start doing the comparisons. >>> >>> Is there a known automated approach to correct this mentioned varying-lag >>> issue? If so, how? >>> >>> >> Can you be more explicit about the varying lags? Do you know what the lags >> are? If not, how much information about them, such as range, probability, >> etc., can you supply? Are there dropouts, or do the lags varying sort of >> continuously? Can you parameterise the lags, so on and so forth. >> >> > What does a straight forward scatter plot look like? > > Chuck > > Here it goes for you http://img44.imageshack.us/img44/1038/scatter.png Both axes is in log-scale. X-axes being CPCConc, and the y is dccnConSTP. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Fri Nov 6 19:10:17 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 6 Nov 2009 19:10:17 -0500 Subject: [Numpy-discussion] dtype 'groups' Message-ID: <7ED5213D-AC6E-4FD7-A91B-8349CC897859@cs.toronto.edu> Just to confirm my suspicions, what's the preferred way to check if a dtype is... a) an integer type vs. a floating point type vs. a string type? I'm assuming dt.kind. b) a compound type vs. a builtin dtype? I'm assuming dt.isbuiltin. Can these properties be relied upon as a stable part of the API that is unlikely to change? David From pav at iki.fi Fri Nov 6 20:20:32 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 07 Nov 2009 03:20:32 +0200 Subject: [Numpy-discussion] dtype 'groups' In-Reply-To: <7ED5213D-AC6E-4FD7-A91B-8349CC897859@cs.toronto.edu> References: <7ED5213D-AC6E-4FD7-A91B-8349CC897859@cs.toronto.edu> Message-ID: <1257556832.21764.94.camel@idol> pe, 2009-11-06 kello 19:10 -0500, David Warde-Farley kirjoitti: > Just to confirm my suspicions, what's the preferred way to check if a > dtype is... > > a) an integer type vs. a floating point type vs. a string type? I'm > assuming dt.kind. > b) a compound type vs. a builtin dtype? I'm assuming dt.isbuiltin. The scalar types form a hierarchy [1], so isinstance on dtype.type might work for a)? No idea about canonicality here. .. [1] http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html -- Pauli Virtanen From sccolbert at gmail.com Fri Nov 6 22:18:50 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 7 Nov 2009 04:18:50 +0100 Subject: [Numpy-discussion] ctypes and numpy In-Reply-To: <7bde5d400911060457u7d8b220csb83dd725ae2a9fbd@mail.gmail.com> References: <7bde5d400911060457u7d8b220csb83dd725ae2a9fbd@mail.gmail.com> Message-ID: <7f014ea60911061918l3110fae7meaf0e2d62065a43e@mail.gmail.com> Are you bound to using ctypes as the only option? This could be done quite easily in Cython... On Fri, Nov 6, 2009 at 1:57 PM, Trevor Clarke wrote: > I've found some info on ctypes and numpy but it mostly tells me how to pass > numpy objects to C funcs. I've got a C func which returns a c_void_p to a > buffer and I'd like to access the data using numpy without creating a copy > of the data. Could someone point me in the right direction? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From cournape at gmail.com Sat Nov 7 03:43:58 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 7 Nov 2009 17:43:58 +0900 Subject: [Numpy-discussion] Backout complex changes for next release? In-Reply-To: References: Message-ID: <5b8d13220911070043m460e5c46laeffe129f09c62b0@mail.gmail.com> On Sat, Nov 7, 2009 at 7:12 AM, Charles R Harris wrote: > There seems to be a rat's nest of problems showing up in scipy due to the > recent changes in numpy complex support. The problems are of two basic > sorts: 1) reserved name conflicts and 2) file conflicts. The first could be > dealt with, but the second might be trickier, both c/c++ defined complex.h > but the files are different. C++ has moved on, but not all platforms seem to > have followed. The solution for 2 is easy: I should not have included complex.h in the public headers. I don't think you can include complex.h in C++ without causing trouble. The scipy C++ code will need to be fixed to avoid using reserved names, but that should not be difficult, David From stefan at sun.ac.za Sat Nov 7 04:32:39 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 7 Nov 2009 11:32:39 +0200 Subject: [Numpy-discussion] dtype 'groups' In-Reply-To: <7ED5213D-AC6E-4FD7-A91B-8349CC897859@cs.toronto.edu> References: <7ED5213D-AC6E-4FD7-A91B-8349CC897859@cs.toronto.edu> Message-ID: <9457e7c80911070132oc7164d1g47054b1365b0eb71@mail.gmail.com> 2009/11/7 David Warde-Farley : > Just to confirm my suspicions, what's the preferred way to check if a > dtype is... Not sure what the "official" way is, but this method works well: > a) an integer type vs. a floating point type vs. a string type? I'm > assuming dt.kind. np.issubdtype(dt, float) Regards St?fan From Chris.Barker at noaa.gov Sat Nov 7 11:04:07 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sat, 07 Nov 2009 08:04:07 -0800 Subject: [Numpy-discussion] Vector interpolation on a 2D grid In-Reply-To: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> Message-ID: <4AF59A77.40606@noaa.gov> Pierre GM wrote: > I have a vector of observations (latitude,longitude,value) that I need > to interpolate over a given area. > Some colleagues complained that the result looked a bit too choppy, > meaning that too much weight was given to the actual observations. > What are my other options ? Kriging might work well for this. I don't know of any python accessible libs, but I'd like to hear about it if anyone else does. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From zachary.pincus at yale.edu Sat Nov 7 12:36:13 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sat, 7 Nov 2009 12:36:13 -0500 Subject: [Numpy-discussion] ctypes and numpy In-Reply-To: <7f014ea60911061918l3110fae7meaf0e2d62065a43e@mail.gmail.com> References: <7bde5d400911060457u7d8b220csb83dd725ae2a9fbd@mail.gmail.com> <7f014ea60911061918l3110fae7meaf0e2d62065a43e@mail.gmail.com> Message-ID: <27ED0E69-A165-4A4A-BE35-F8350672F9FD@yale.edu> Check out this thread: http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg01154.html In shot, it can be done, but it can be tricky to make sure you don't leak memory. A better option if possible is to pre-allocate the array with numpy and pass that buffer into the C code -- but that's not always possible... Zach On Nov 6, 2009, at 10:18 PM, Chris Colbert wrote: > Are you bound to using ctypes as the only option? > > This could be done quite easily in Cython... > > On Fri, Nov 6, 2009 at 1:57 PM, Trevor Clarke > wrote: >> I've found some info on ctypes and numpy but it mostly tells me how >> to pass >> numpy objects to C funcs. I've got a C func which returns a >> c_void_p to a >> buffer and I'd like to access the data using numpy without creating >> a copy >> of the data. Could someone point me in the right direction? >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From neilcrighton at gmail.com Sat Nov 7 13:27:22 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Sat, 7 Nov 2009 18:27:22 +0000 (UTC) Subject: [Numpy-discussion] Deprecate np.max/np.min ? References: Message-ID: Charles R Harris gmail.com> writes: > People import these functions -- yes, they shouldn't do that -- and the python builtin versions are overloaded, causing hard to locate errors. While I would love less duplication in the numpy namespace, I don't think the small gain here is worth the pain of deprecation. > OTOH, one can ask, why is > > np.min(3, 2) > > allowed when > > np.min([3], 2) > > gives "ValueError: axis(=2) out of bounds". It seems to me that > 0-dimensional objects should accept only None as the axis? (Fixing this > would also make misuse of np.min and np.max more difficult.) I think it would be better to fix this issue. np.min(3,2) should also give "ValueError: axis(=2) out of bounds". Fixing this also removes any possibility of generating hard-to-find errors by overwriting the builtin min/max. (Unless there's some corner case I'm missing). Neil From stanconn at gmail.com Sat Nov 7 13:51:15 2009 From: stanconn at gmail.com (Stas K) Date: Sat, 7 Nov 2009 21:51:15 +0300 Subject: [Numpy-discussion] How to get rid of the loop? Message-ID: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> Can I get rid of the loop in this example? And what is the fastest way to get v in the example? ar = array([1,2,3]) for a in ar: for b in ar: v = a**2+b**2 From josef.pktd at gmail.com Sat Nov 7 13:57:14 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 7 Nov 2009 13:57:14 -0500 Subject: [Numpy-discussion] How to get rid of the loop? In-Reply-To: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> References: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> Message-ID: <1cd32cbb0911071057v72e60ffeof9db949925e9bc0f@mail.gmail.com> On Sat, Nov 7, 2009 at 1:51 PM, Stas K wrote: > Can I get rid of the loop in this example? And what is the fastest way > to get v in the example? > > ar = array([1,2,3]) > for a in ar: > ? ?for b in ar: > ? ? ? ?v = a**2+b**2 >>> ar[:,None]**2 + ar**2 array([[ 2, 5, 10], [ 5, 8, 13], [10, 13, 18]]) I think, for this case there is also directly a function in numpy hypot which should also work. Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From aisaac at american.edu Sat Nov 7 14:04:28 2009 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 07 Nov 2009 14:04:28 -0500 Subject: [Numpy-discussion] How to get rid of the loop? In-Reply-To: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> References: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> Message-ID: <4AF5C4BC.8040203@american.edu> On 11/7/2009 1:51 PM, Stas K wrote: > Can I get rid of the loop in this example? And what is the fastest way > to get v in the example? > > ar = array([1,2,3]) > for a in ar: > for b in ar: > v = a**2+b**2 >>> a2 = a*a >>> np.add.outer(a2,a2) array([[ 2, 5, 10], [ 5, 8, 13], [10, 13, 18]]) hth, Alan Isaac From stanconn at gmail.com Sat Nov 7 14:16:51 2009 From: stanconn at gmail.com (Stas K) Date: Sat, 7 Nov 2009 22:16:51 +0300 Subject: [Numpy-discussion] How to get rid of the loop? In-Reply-To: <1cd32cbb0911071057v72e60ffeof9db949925e9bc0f@mail.gmail.com> References: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> <1cd32cbb0911071057v72e60ffeof9db949925e9bc0f@mail.gmail.com> Message-ID: <9E1A0A6C-13B2-4FAC-B749-429E00848178@gmail.com> Thank you, Josef It is exactly what I want: >>>> ar[:,None]**2 + ar**2 Do you know something about performance of this? In my real program ar have ~ 10k elements, and expression for v more complicated (it has some trigonometric functions) On 07.11.2009, at 21:57, josef.pktd at gmail.com wrote: > On Sat, Nov 7, 2009 at 1:51 PM, Stas K wrote: >> Can I get rid of the loop in this example? And what is the fastest >> way >> to get v in the example? >> >> ar = array([1,2,3]) >> for a in ar: >> for b in ar: >> v = a**2+b**2 > >>>> ar[:,None]**2 + ar**2 > array([[ 2, 5, 10], > [ 5, 8, 13], > [10, 13, 18]]) > > I think, for this case there is also directly a function in numpy > hypot which should also work. > > Josef >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebzur at codelab.pl Sat Nov 7 14:23:07 2009 From: sebzur at codelab.pl (Sebastian =?UTF-8?B?xbt1cmVr?=) Date: Sat, 07 Nov 2009 20:23:07 +0100 Subject: [Numpy-discussion] PGI vs. Intel Compiler Message-ID: Hi! I'm not sure, if it's the right group to ask, but it's a kind of a numeric question involving Python, so... My faculty is going to spend some money on commercial compiler. The two choices are: - PGI C++ compiler - Intel C++ compiler I wonder, if anyone as some experience with Python numerical packages (and Python itself) compiled with the compiler listed above? Can anyone provide some pro and cons? How about the basic tools like BLAS/LAPACK behavior compiled with the PGI/Intel C++ tools? Any suggestion is welcome! Thanks in advance! Sebastian From thomas.robitaille at gmail.com Sat Nov 7 14:26:54 2009 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Sat, 7 Nov 2009 11:26:54 -0800 (PST) Subject: [Numpy-discussion] masked record arrays In-Reply-To: References: <43B4D32C-5C83-4C59-B059-401E8DCDB6E1@gmail.com> Message-ID: <26247808.post@talk.nabble.com> Pierre GM-2 wrote: > > Mmh. With a recent (1.3) version of numpy, you should already be able > to mask individual fields of a structured array without problems. If > you need fields to be accessed as attributes the np.recarray way, you > can give numpy.ma.mrecords.MaskedRecords a try. It's been a while I > haven't touched it, so you may run into the occasional bug. FYI, I'm > not a big fan of record arrays and tend to prefer structured ones... > What two implementations were you talking about ? > In any case, feel free to try and please, report any issue you run > into with MaskedRecords. > Cheers > Thanks for the advice! I'm somewhat confused by the difference between structured and record arrays. My understanding is that record arrays allow you to access fields by attribute (e.g. r.field_name), but I imagine that there are much more fundamental differences for the two to be treated separately in numpy. I find the numpy documentation somewhat confusing in that respect - if you have a look at this page http://docs.scipy.org/doc/numpy/user/basics.rec.html I think the 'aka record arrays' is especially confusing as this would suggest the two are the same. So is there good information anywhere about what exactly are the differences between the two? This page is also confusing: http://docs.scipy.org/doc/numpy/reference/generated/numpy.recarray.html as to me "Construct an ndarray that allows field access using attributes" suggests that all a recarray is is an ndarray/structured array with overloaded __getattr__/__setattr__ methods. Is that all recarrays are? If so, why was a completely separate package developed for masked record arrays - can one not just use masked structured arrays and overload getattr/setattr? Cheers, Thomas -- View this message in context: http://old.nabble.com/masked-record-arrays-tp26237612p26247808.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From peridot.faceted at gmail.com Sat Nov 7 14:33:12 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 7 Nov 2009 14:33:12 -0500 Subject: [Numpy-discussion] How to get rid of the loop? In-Reply-To: <9E1A0A6C-13B2-4FAC-B749-429E00848178@gmail.com> References: <42E3B1BF-6158-469D-B2FA-F3E9D39FC5C5@gmail.com> <1cd32cbb0911071057v72e60ffeof9db949925e9bc0f@mail.gmail.com> <9E1A0A6C-13B2-4FAC-B749-429E00848178@gmail.com> Message-ID: 2009/11/7 Stas K : > Thank you, Josef > It is exactly what I want: > > ar[:,None]**2 + ar**2 > > Do you know something about performance of this? In my real program ar ?have > ~ 10k elements, and expression for v more complicated (it has some > trigonometric functions) The construction of ar[:,None] (which I prefer to spell ar[:,np.newaxis]) is cheap, since it just constructs a "view" into ar. Computing ar**2 and ar[:,None] require squaring each element of ar, but this is done in a C loop. (It's done twice in this expression, so this isn't as efficient as it might be). Only when it comes time to do the addition do you do n**2 operations. For very large n, this can be a crushing memory burden, and if you only need these values for an intermediate calculation it sometimes turns out that it's better to loop over one of the dimensions. But this expression is probably about as efficient as you can hope for, given what it does. If you need to do some more complicated calculation, the main thing to be careful of is that you do as much calculation as possible on the separate arrays, while they're still only n elements and not n**2. Anne > On 07.11.2009, at 21:57, josef.pktd at gmail.com wrote: > > On Sat, Nov 7, 2009 at 1:51 PM, Stas K wrote: > > Can I get rid of the loop in this example? And what is the fastest way > > to get v in the example? > > ar = array([1,2,3]) > > for a in ar: > > ? ?for b in ar: > > ? ? ? ?v = a**2+b**2 > > ar[:,None]**2 + ar**2 > > array([[ 2, ?5, 10], > ??????[ 5, ?8, 13], > ??????[10, 13, 18]]) > > I think, for this case there is also directly a function in numpy > hypot which should also work. > > Josef > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From sturla at molden.no Sat Nov 7 14:40:07 2009 From: sturla at molden.no (Sturla Molden) Date: Sat, 07 Nov 2009 20:40:07 +0100 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <5b8d13220911051902p50cdeeaqbfeea6ae52aef4d2@mail.gmail.com> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <5b8d13220911051902p50cdeeaqbfeea6ae52aef4d2@mail.gmail.com> Message-ID: <4AF5CD17.1070108@molden.no> David Cournapeau wrote: > On Fri, Nov 6, 2009 at 6:54 AM, David Goldsmith wrote: > >> Interesting thread, which leaves me wondering two things: is it documented >> somewhere (e.g., at the IEEE site) precisely how many *decimal* mantissae >> are representable using the 64-bit IEEE standard for float representation >> (if that makes sense); and are such decimal mantissae uniformly distributed >> > > They are definitely not uniformly distributed: that's why two numbers > are close around 1 when they have only a few EPS difference, but > around 1e100, you have to add quite a few EPS to even get a different > number at all. > > That may be my audio processing background, but I like to think about > float as numbers which have the same relative precision at any "level" > - a kind of dB scale. If you want numbers with a fixed number of > decimals, you need a fixed point representation. > David Godsmith was asking about the mantissae. For a double, that is a 53 bit signed integer. I.e. you have 52 bit fractional part (bit 0-51), 11 bit exponent (bit 52-62), and one sign bit (bit 63). The mantissae is uniformly distributed like any signed integer. The mantissae of a double have 2**53 different integer values: -2**52 to 2**52-1. But the value of a floating point number is value = (-1)**signbit * 2**(exponent - bias) * (1 - fraction) with bias = 1023 for a double. Thus, floating point numbers are not uniformly distributed, but the mantissae is. For numerical illiterates this might come as a surprise. But in numerical mathematics, the resolution is in the number of "significant digits", not in "the number of decimals". 101 and .00201 have the same numerical precision. A decimal, on the other hand, can be thought of as a floating point number using base-10 instead of base-2 for the exponent: value = (-1)**signbit * 10**(exponent - bias) * (1 - fraction) Decimals and floats are not fundamentally different. There are number exactly representable with a decimal that cannot be exactly represented with a float. But numerical computation do not become more precise with a decimal than a float. From jakevdp at gmail.com Sat Nov 7 15:12:35 2009 From: jakevdp at gmail.com (Jake VanderPlas) Date: Sat, 7 Nov 2009 12:12:35 -0800 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API Message-ID: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> Hello, I'm working on wrapping a set of C++ routines for manifold learning (LLE, Isomap, LTSA, etc) in python. In the LLE routine, it is necessary to loop through the input points and perform an svd of each local covariance matrix. Presently I have my own C-LAPACK wrapper that I call within a C loop, but this seems non-ideal because numpy already wraps LAPACK/ATLAS for linear algebra. Does anybody know a way to directly access the numpy.linalg routines from a C extension, without the overhead of a python callback? Thanks for the help. -Jake From sturla at molden.no Sat Nov 7 15:41:16 2009 From: sturla at molden.no (Sturla Molden) Date: Sat, 07 Nov 2009 21:41:16 +0100 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> References: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> Message-ID: <4AF5DB6C.1070506@molden.no> Jake VanderPlas wrote: > Does anybody know a > way to directly access the numpy.linalg routines from a C extension, > without the overhead of a python callback? Thanks for the help. > You find a C function pointer wrapped in a CObject in the ._cpointer attribute. From d.l.goldsmith at gmail.com Sat Nov 7 16:04:35 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 7 Nov 2009 13:04:35 -0800 Subject: [Numpy-discussion] Use-case for np.choose Message-ID: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> Hi, all! I'm working to clarify the docstring for np.choose (Stefan pointed out to me that it is pretty unclear, and I agreed), and, now that (I'm pretty sure that) I've figured out what it does in its full generality (e.g., when the 'choices' array is greater than 2-D), I'm curious as to use-cases. (Not that I'm suggesting that we deprecate it, but I am curious as to whether or not anyone is presently using it and if so, how they're using it, esp. if anyone *is* using it with 'choices' arrays 3-D or greater.) Also, my experimenting suggests that the index array ('a', the first argument in the func. sig.) *must* have shape (choices.shape[-1],) - someone please let me know ASAP if this is not the case, and please furnish me w/ a counterexample because I was unable to generate one myself. Thanks, DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sat Nov 7 16:32:20 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 7 Nov 2009 16:32:20 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> Message-ID: 2009/11/7 David Goldsmith : > Hi, all!? I'm working to clarify the docstring for np.choose (Stefan pointed > out to me that it is pretty unclear, and I agreed), and, now that (I'm > pretty sure that) I've figured out what it does in its full generality > (e.g., when the 'choices' array is greater than 2-D), I'm curious as to > use-cases.? (Not that I'm suggesting that we deprecate it, but I am curious > as to whether or not anyone is presently using it and if so, how they're > using it, esp. if anyone *is* using it with 'choices' arrays 3-D or > greater.) It's a generalized np.where(), allowing more than two options: In [10]: a = np.arange(2*3*5).reshape(2,3,5) % 3 In [11]: o = np.ones((2,3,5)) In [12]: np.choose(a,(o,0.5*o,0.1*o)) Out[12]: array([[[ 1. , 0.5, 0.1, 1. , 0.5], [ 0.1, 1. , 0.5, 0.1, 1. ], [ 0.5, 0.1, 1. , 0.5, 0.1]], [[ 1. , 0.5, 0.1, 1. , 0.5], [ 0.1, 1. , 0.5, 0.1, 1. ], [ 0.5, 0.1, 1. , 0.5, 0.1]]]) > Also, my experimenting suggests that the index array ('a', the first > argument in the func. sig.) *must* have shape (choices.shape[-1],) - someone > please let me know ASAP if this is not the case, and please furnish me w/ a > counterexample because I was unable to generate one myself. It seems like a and each of the choices must have the same shape (with the exception that choices acn be scalars), but I would consider this a bug. Really, a and all the choices should be broadcast to the same shape. Or maybe it doesn't make sense to broadcast a - it could be valuable to know that the result is always exactly the same shape as a - but broadcasting all the choice arrays presents an important improvement of choose over fancy indexing. There's a reason choose accepts a sequence of arrays as its second argument, rather than a higher-dimensional array. Anne > Thanks, > > DG > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From pav at iki.fi Sat Nov 7 16:41:20 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 07 Nov 2009 23:41:20 +0200 Subject: [Numpy-discussion] Deprecate np.max/np.min ? In-Reply-To: References: Message-ID: <1257630079.2461.9.camel@idol> la, 2009-11-07 kello 18:27 +0000, Neil Crighton kirjoitti: [clip] > I think it would be better to fix this issue. np.min(3,2) should also give > "ValueError: axis(=2) out of bounds". Fixing this also removes any possibility > of generating hard-to-find errors by overwriting the builtin min/max. (Unless > there's some corner case I'm missing). Fixed in r7697, r7698, which change the behavior of scalars in many functions using axis=xxx. I don't think this breaks anyone's code -- using out-of-bounds values of axis is almost certainly an error. I left axis=-1 and axis=0 allowed, in addition to axis=None. These seemed to be required by at least the masked arrays unit tests... -- Pauli Virtanen From trevor at notcows.com Sat Nov 7 17:07:47 2009 From: trevor at notcows.com (Trevor Clarke) Date: Sat, 7 Nov 2009 17:07:47 -0500 Subject: [Numpy-discussion] ctypes and numpy In-Reply-To: <27ED0E69-A165-4A4A-BE35-F8350672F9FD@yale.edu> References: <7bde5d400911060457u7d8b220csb83dd725ae2a9fbd@mail.gmail.com> <7f014ea60911061918l3110fae7meaf0e2d62065a43e@mail.gmail.com> <27ED0E69-A165-4A4A-BE35-F8350672F9FD@yale.edu> Message-ID: <7bde5d400911071407o3360fc2cwb5df4f94fc7703db@mail.gmail.com> this looks like what I need...I'm not concerned with leaking memory as it's a borrowed pointer which will be cleaned up in C code later. Thanks for the pointer. On Sat, Nov 7, 2009 at 12:36 PM, Zachary Pincus wrote: > Check out this thread: > > http://www.mail-archive.com/numpy-discussion at lists.sourceforge.net/msg01154.html > > In shot, it can be done, but it can be tricky to make sure you don't > leak memory. A better option if possible is to pre-allocate the array > with numpy and pass that buffer into the C code -- but that's not > always possible... > > Zach > > > On Nov 6, 2009, at 10:18 PM, Chris Colbert wrote: > > > Are you bound to using ctypes as the only option? > > > > This could be done quite easily in Cython... > > > > On Fri, Nov 6, 2009 at 1:57 PM, Trevor Clarke > > wrote: > >> I've found some info on ctypes and numpy but it mostly tells me how > >> to pass > >> numpy objects to C funcs. I've got a C func which returns a > >> c_void_p to a > >> buffer and I'd like to access the data using numpy without creating > >> a copy > >> of the data. Could someone point me in the right direction? > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Sat Nov 7 17:31:37 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 7 Nov 2009 17:31:37 -0500 Subject: [Numpy-discussion] masked record arrays In-Reply-To: <26247808.post@talk.nabble.com> References: <43B4D32C-5C83-4C59-B059-401E8DCDB6E1@gmail.com> <26247808.post@talk.nabble.com> Message-ID: <9BE05A8F-0F98-4F9D-9C1C-55E7E9A64D1C@gmail.com> On Nov 7, 2009, at 2:26 PM, Thomas Robitaille wrote: > > Thanks for the advice! I'm somewhat confused by the difference between > structured and record arrays. My understanding is that record arrays > allow > you to access fields by attribute (e.g. r.field_name), but I imagine > that > there are much more fundamental differences for the two to be treated > separately in numpy. Actually, no. recarray is just ndarray w/ a special __getattribute__/ __setattr__ . They bring the convenience of exposing fields as properties, but they come to the cost of overloading __getattribute__ > I find the numpy documentation somewhat confusing in > that respect - if you have a look at this page > > http://docs.scipy.org/doc/numpy/user/basics.rec.html > > I think the 'aka record arrays' is especially confusing as this would > suggest the two are the same. Not the most fortunate formulation, true... > So is there good information anywhere about > what exactly are the differences between the two? This page is also > confusing: > > http://docs.scipy.org/doc/numpy/reference/generated/ > numpy.recarray.html > > as to me "Construct an ndarray that allows field access using > attributes" > suggests that all a recarray is is an ndarray/structured array with > overloaded __getattr__/__setattr__ methods. Is that all recarrays are? Yep. > If > so, why was a completely separate package developed for masked > record arrays > - can one not just use masked structured arrays and overload > getattr/setattr? Mostly historical reasons. Initially, there was only limited support for structured masked arrays and masked records filled the gap (albeit experimentally). With the 1.3 release, MaskedArray fully supports structured type, giving the possibility to mask individual fields and masked records became less useful. Still, it's cleaner to have a specific module where to store functions like fromrecords, fromarrays and so forth. Note that the doc of numpy.ma.mrecords is a tad outdated, any help to this regard would be welcome. From pgmdevlist at gmail.com Sat Nov 7 18:38:33 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 7 Nov 2009 18:38:33 -0500 Subject: [Numpy-discussion] Vector interpolation on a 2D grid (with realistic results) In-Reply-To: <1257547522.5931.24.camel@idol> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> <1257547522.5931.24.camel@idol> Message-ID: <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> On Nov 6, 2009, at 5:45 PM, Pauli Virtanen wrote: > pe, 2009-11-06 kello 17:20 -0500, Pierre GM kirjoitti: >> All, >> I have a vector of observations (latitude,longitude,value) that I > need >> to interpolate over a given area. > You could try to use linear interpolation from the delaynay package > (only supports slicing, so is a bit tricky to use). I may be slightly > smoother than the natural neighbour one. > > For scattered data, when I wasn't happy with delaunay, I've also used > the radial basis functions (Rbf) from scipy.interpolate -- something > like > > interpolator = Rbf(x, y, z, function='thin-plate', smooth=1e-9) > > should give an interpolator that is reasonably smooth between the data > points. (I'd guess you can give it leeway in trading interpolation to > smoothness by increasing the smooth parameter.) Other Rbf functions > than > thin-plate splines might also work. Linear interpolation with the delaunay package doesn't work great for my data. I played with the radial basis functions, but I'm afraid they're leading me down the dark, dark path of parameter fiddling. In particular, I'm not sure how to prevent my interpolated values to be bounded by the min and max of my actual observations. Ralf' suggestion of smoothing the values afterwards is tempting, but sounds a bit too ad-hoc (btw, Ralf, those were relative differences of monthly average precipitation between El Ni?o and Neutral phases for November). Chris, I gonna poke around and try to find some kriging algorithms. I'll report in a few. In the meantime, if anybody has anythng already implemented, please just let us know. From d.l.goldsmith at gmail.com Sat Nov 7 19:53:13 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 7 Nov 2009 16:53:13 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> Message-ID: <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> Thanks, Anne. On Sat, Nov 7, 2009 at 1:32 PM, Anne Archibald wrote: > 2009/11/7 David Goldsmith : > > Also, my experimenting suggests that the index array ('a', the first > > argument in the func. sig.) *must* have shape (choices.shape[-1],) - > someone > > please let me know ASAP if this is not the case, and please furnish me w/ > a > > counterexample because I was unable to generate one myself. > > It seems like a and each of the choices must have the same shape So in essence, at least as it presently functions, the shape of 'a' *defines* what the individual choices are within 'choices`, and if 'choices' can't be parsed into an integer number of such individual choices, that's when an exception is raised? > (with > the exception that choices acn be scalars), but I would consider this > a bug. OK, then we definitely need more people to opine on this, because, if the the two don't match, our established policy is to document *desired* behavior, not extant behavior (and file a bug ticket). > Really, a and all the choices should be broadcast to the same > shape. Or maybe it doesn't make sense to broadcast a - it could be > Thus begging the question: does anyone actually have an extant, specific use-case? > valuable to know that the result is always exactly the same shape as a > - but broadcasting all the choice arrays presents an important > improvement of choose over fancy indexing. Then perhaps we need either another function, or a flag specifying which behavior this one should exhibit. > There's a reason choose > accepts a sequence of arrays as its second argument, rather than a > higher-dimensional array. > And that reason is probably supposed to be transparent above, but I've confused it by this point, so can you please reiterate it here, in so many words. :-) Thanks again, DG > > Anne > > > Thanks, > > > > DG > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Sat Nov 7 21:11:00 2009 From: rmay31 at gmail.com (Ryan May) Date: Sat, 7 Nov 2009 20:11:00 -0600 Subject: [Numpy-discussion] Vector interpolation on a 2D grid (with realistic results) In-Reply-To: <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> <1257547522.5931.24.camel@idol> <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> Message-ID: On Sat, Nov 7, 2009 at 5:38 PM, Pierre GM wrote: > Linear interpolation with the delaunay package doesn't work great for > my data. I played with the radial basis functions, but I'm afraid > they're leading me down the dark, dark path of parameter fiddling. In > particular, I'm not sure how to prevent my interpolated values to be > bounded by the min and max of my actual observations. > Ralf' suggestion of smoothing the values afterwards is tempting, but > sounds a bit too ad-hoc (btw, Ralf, those were relative differences of > monthly average precipitation between El Ni?o and Neutral phases for > November). That was me, not Ralf. :) And I agree, I the interpolated field does look a bit noisy for such data. I've been doing the smoothing on top of natural neighbor for doing some of my own meteorological analysis. Using the Gaussian kernel isn't really *that* ad hoc considering the prevalence of Barnes/Cressman weighting for spatial averaging typically used in meteorology. And if you have no idea what I'm talking about, Google them, and you'll see. :) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From rmay31 at gmail.com Sat Nov 7 21:14:27 2009 From: rmay31 at gmail.com (Ryan May) Date: Sat, 7 Nov 2009 20:14:27 -0600 Subject: [Numpy-discussion] Vector interpolation on a 2D grid (with realistic results) In-Reply-To: <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> <1257547522.5931.24.camel@idol> <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> Message-ID: On Sat, Nov 7, 2009 at 5:38 PM, Pierre GM wrote: > Linear interpolation with the delaunay package doesn't work great for > my data. I played with the radial basis functions, but I'm afraid > they're leading me down the dark, dark path of parameter fiddling. In > particular, I'm not sure how to prevent my interpolated values to be > bounded by the min and max of my actual observations. > Ralf' suggestion of smoothing the values afterwards is tempting, but > sounds a bit too ad-hoc (btw, Ralf, those were relative differences of > monthly average precipitation between El Ni?o and Neutral phases for > November). That was me, not Ralf. :) And I agree, I the interpolated field does look a bit noisy for such data. I've been doing the smoothing on top of natural neighbor for doing some of my own meteorological analysis. Using the Gaussian kernel isn't really *that* ad hoc considering the prevalence of Barnes/Cressman weighting for spatial averaging typically used in meteorology. And if you have no idea what I'm talking about, Google them, and you'll see. :) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From alan at ajackson.org Sat Nov 7 22:56:29 2009 From: alan at ajackson.org (alan at ajackson.org) Date: Sat, 7 Nov 2009 21:56:29 -0600 Subject: [Numpy-discussion] initializing an array of lists Message-ID: <20091107215629.6009523e@ajackson.org> I want to build a 2D array of lists, and so I need to initialize the array with empty lists : myarray = array([[[],[],[]] ,[[],[],[]]]) Is there a clever way to do this? I could define the array myarray = zeros( (xdim,ydim), dtype=object) and then iterate through the elements initializing then to empty lists, but surely there is a better way. -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From aisaac at american.edu Sat Nov 7 23:26:35 2009 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 07 Nov 2009 23:26:35 -0500 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <20091107215629.6009523e@ajackson.org> References: <20091107215629.6009523e@ajackson.org> Message-ID: <4AF6487B.9050808@american.edu> On 11/7/2009 10:56 PM, alan at ajackson.org wrote: > I want to build a 2D array of lists, and so I need to initialize the > array with empty lists : > > myarray = array([[[],[],[]] ,[[],[],[]]]) [[[] for i in range(3)] for j in range(2) ] fwiw, Alan Isaac From josef.pktd at gmail.com Sun Nov 8 00:06:35 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Nov 2009 00:06:35 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> Message-ID: <1cd32cbb0911072106q2096d5b7o77e68ed33dbd1d68@mail.gmail.com> On Sat, Nov 7, 2009 at 7:53 PM, David Goldsmith wrote: > Thanks, Anne. > > On Sat, Nov 7, 2009 at 1:32 PM, Anne Archibald > wrote: >> >> 2009/11/7 David Goldsmith : > > > >> >> > Also, my experimenting suggests that the index array ('a', the first >> > argument in the func. sig.) *must* have shape (choices.shape[-1],) - >> > someone >> > please let me know ASAP if this is not the case, and please furnish me >> > w/ a >> > counterexample because I was unable to generate one myself. >> >> It seems like a and each of the choices must have the same shape > > So in essence, at least as it presently functions, the shape of 'a' > *defines* what the individual choices are within 'choices`, and if 'choices' > can't be parsed into an integer number of such individual choices, that's > when an exception is raised? > >> >> (with >> >> the exception that choices acn be scalars), but I would consider this >> a bug. > > OK, then we definitely need more people to opine on this, because, if the > the two don't match, our established policy is to document *desired* > behavior, not extant behavior (and file a bug ticket). > >> >> Really, a and all the choices should be broadcast to the same >> shape. Or maybe it doesn't make sense to broadcast a - it could be > > Thus begging the question: does anyone actually have an extant, specific > use-case? > >> >> valuable to know that the result is always exactly the same shape as a >> - but broadcasting all the choice arrays presents an important >> improvement of choose over fancy indexing. > > Then perhaps we need either another function, or a flag specifying which > behavior this one should exhibit. > >> >> There's a reason choose >> accepts a sequence of arrays as its second argument, rather than a >> higher-dimensional array. > > And that reason is probably supposed to be transparent above, but I've > confused it by this point, so can you please reiterate it here, in so many > words. :-) >From looking at a few special cases, I think that full broadcasting rules apply. First a and all choice array are broadcast to the same shape, then the selection is done according to the elements of (the broadcasted) a. For broadcasting it doesn't matter whether they are scalars or 1d or 2d or a 2d single column array. (I haven't tried more than 2 dimensions) The examples look a bit messy, but broadcasting is relatively straightforward. (I think, np.where is a bit easier to use because `a` is just a condition and doesn't require an index array) Josef >>> np.choose(1, (3,4)) 4 >>> np.choose(0, (3,4)) 3 >>> np.choose(0, (np.arange(3)[:,None],np.arange(4),0)) array([[0, 0, 0, 0], [1, 1, 1, 1], [2, 2, 2, 2]]) >>> np.choose(2, (np.arange(3)[:,None],np.arange(4),0)) array([[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]) >>> np.choose(1, (np.arange(3)[:,None],np.arange(4),0)) array([[0, 1, 2, 3], [0, 1, 2, 3], [0, 1, 2, 3]]) >>> np.choose([1,2,0,0], (np.arange(3)[:,None],np.arange(4),0)) array([[0, 0, 0, 0], [0, 0, 1, 1], [0, 0, 2, 2]]) >>> np.choose(np.array([[1,2,0,0]]), (np.arange(3)[:,None],np.arange(4),0)) array([[0, 0, 0, 0], [0, 0, 1, 1], [0, 0, 2, 2]]) >>> np.choose(np.array([[1,2,0]]).T, (np.arange(3)[:,None],np.arange(4),0)) array([[0, 1, 2, 3], [0, 0, 0, 0], [2, 2, 2, 2]]) > > Thanks again, > > DG >> >> Anne >> >> > Thanks, >> > >> > DG >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From peridot.faceted at gmail.com Sun Nov 8 02:59:16 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 8 Nov 2009 02:59:16 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> Message-ID: 2009/11/7 David Goldsmith : > Thanks, Anne. > > On Sat, Nov 7, 2009 at 1:32 PM, Anne Archibald > wrote: >> >> 2009/11/7 David Goldsmith : > > > >> >> > Also, my experimenting suggests that the index array ('a', the first >> > argument in the func. sig.) *must* have shape (choices.shape[-1],) - >> > someone >> > please let me know ASAP if this is not the case, and please furnish me >> > w/ a >> > counterexample because I was unable to generate one myself. >> >> It seems like a and each of the choices must have the same shape > > So in essence, at least as it presently functions, the shape of 'a' > *defines* what the individual choices are within 'choices`, and if 'choices' > can't be parsed into an integer number of such individual choices, that's > when an exception is raised? Um, I don't think so. Think of it this way: you provide np.choose with a selector array, a, and a list (not array!) [c0, c1, ..., cM] of choices. You construct an output array, say r, the same shape as a (no matter how many dimensions it has). The (i0, i1, ..., iN) element of the output array is obtained by looking at the (i0, i1, ..., iN) element of a, which should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] = cj[i0, i1, ..., iN]. That is, each element of the selector array determines which of the choice arrays to pull the corresponding element from. For example, suppose that you are processing an array C, and have constructed a selector array A the same shape as C in which a value is 0, 1, or 2 depending on whether the C value is too small, okay, or too big respectively. Then you might do something like: C = np.choose(A, [-inf, C, inf]) This is something you might want to do no matter what shape A and C have. It's important not to require that the choices be an array of choices, because they often have quite different shapes (here, two are scalars) and it would be wasteful to broadcast them up to the same shape as C, just to stack them. Anne From peridot.faceted at gmail.com Sun Nov 8 03:13:46 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 8 Nov 2009 03:13:46 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <1cd32cbb0911072106q2096d5b7o77e68ed33dbd1d68@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <1cd32cbb0911072106q2096d5b7o77e68ed33dbd1d68@mail.gmail.com> Message-ID: 2009/11/8 : > On Sat, Nov 7, 2009 at 7:53 PM, David Goldsmith wrote: >> Thanks, Anne. >> >> On Sat, Nov 7, 2009 at 1:32 PM, Anne Archibald >> wrote: >>> >>> 2009/11/7 David Goldsmith : >> >> >> >>> >>> > Also, my experimenting suggests that the index array ('a', the first >>> > argument in the func. sig.) *must* have shape (choices.shape[-1],) - >>> > someone >>> > please let me know ASAP if this is not the case, and please furnish me >>> > w/ a >>> > counterexample because I was unable to generate one myself. >>> >>> It seems like a and each of the choices must have the same shape >> >> So in essence, at least as it presently functions, the shape of 'a' >> *defines* what the individual choices are within 'choices`, and if 'choices' >> can't be parsed into an integer number of such individual choices, that's >> when an exception is raised? >> >>> >>> (with >>> >>> the exception that choices acn be scalars), but I would consider this >>> a bug. >> >> OK, then we definitely need more people to opine on this, because, if the >> the two don't match, our established policy is to document *desired* >> behavior, not extant behavior (and file a bug ticket). >> >>> >>> Really, a and all the choices should be broadcast to the same >>> shape. Or maybe it doesn't make sense to broadcast a - it could be >> >> Thus begging the question: does anyone actually have an extant, specific >> use-case? >> >>> >>> valuable to know that the result is always exactly the same shape as a >>> - but broadcasting all the choice arrays presents an important >>> improvement of choose over fancy indexing. >> >> Then perhaps we need either another function, or a flag specifying which >> behavior this one should exhibit. >> >>> >>> There's a reason choose >>> accepts a sequence of arrays as its second argument, rather than a >>> higher-dimensional array. >> >> And that reason is probably supposed to be transparent above, but I've >> confused it by this point, so can you please reiterate it here, in so many >> words. :-) > > >From looking at a few special cases, I think that full broadcasting rules apply. > First a and all choice array are broadcast to the same shape, > then the selection is done according to the elements of (the broadcasted) a. > > For broadcasting it doesn't matter whether they are scalars or 1d or 2d or > a 2d single column array. (I haven't tried more than 2 dimensions) This must have changed since 1.2.1, since I get ValueError: too many dimensions for all those examples. (Though the 1.2.1 docs claim broadcasting.) > The examples look a bit messy, but broadcasting is relatively straightforward. > > (I think, np.where is a bit easier to use because `a` is just a > condition and doesn't > require an index array) Well, a condition *is* an index array, it is just restricted to the values 0 and 1. But I agree it's not so natural to construct an index array with more values. I guess you could do something like: A = (C>-1) + (C>0) + (C>1) B = np.choose(A, (1, -C, C, 1)) But that's one of those magical numpy expressions that make the reader stop and ask "how on earth does that work?" For further bewilderment you'd make it computed: A = np.sum(C[...,None]>transition_values, axis=-1) B = np.choose(A, [piece(C) for piece in piecewise_pieces]) Anne > Josef > >>>> np.choose(1, (3,4)) > 4 >>>> np.choose(0, (3,4)) > 3 >>>> np.choose(0, (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 0, 0, 0], > ? ? ? [1, 1, 1, 1], > ? ? ? [2, 2, 2, 2]]) >>>> np.choose(2, (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 0, 0, 0], > ? ? ? [0, 0, 0, 0], > ? ? ? [0, 0, 0, 0]]) >>>> np.choose(1, (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 1, 2, 3], > ? ? ? [0, 1, 2, 3], > ? ? ? [0, 1, 2, 3]]) >>>> np.choose([1,2,0,0], (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 0, 0, 0], > ? ? ? [0, 0, 1, 1], > ? ? ? [0, 0, 2, 2]]) >>>> np.choose(np.array([[1,2,0,0]]), (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 0, 0, 0], > ? ? ? [0, 0, 1, 1], > ? ? ? [0, 0, 2, 2]]) >>>> np.choose(np.array([[1,2,0]]).T, (np.arange(3)[:,None],np.arange(4),0)) > array([[0, 1, 2, 3], > ? ? ? [0, 0, 0, 0], > ? ? ? [2, 2, 2, 2]]) > > > > >> >> Thanks again, >> >> DG >>> >>> Anne >>> >>> > Thanks, >>> > >>> > DG >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From d.l.goldsmith at gmail.com Sun Nov 8 03:33:16 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 00:33:16 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> Message-ID: <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald wrote: > 2009/11/7 David Goldsmith : > > So in essence, at least as it presently functions, the shape of 'a' > > *defines* what the individual choices are within 'choices`, and if > 'choices' > > can't be parsed into an integer number of such individual choices, that's > > when an exception is raised? > > Um, I don't think so. > > Think of it this way: you provide np.choose with a selector array, a, > and a list (not array!) [c0, c1, ..., cM] of choices. You construct an > output array, say r, the same shape as a (no matter how many > dimensions it has). Except that I haven't yet seen a working example with 'a' greater than 1-D, Josef's last two examples notwithstanding; or is that what you're saying is the bug. The (i0, i1, ..., iN) element of the output array > is obtained by looking at the (i0, i1, ..., iN) element of a, which > should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] > = cj[i0, i1, ..., iN]. That is, each element of the selector array > determines which of the choice arrays to pull the corresponding > element from. > That's pretty clear (thanks for doing my work for me). ;-), Yet, see above. For example, suppose that you are processing an array C, and have > constructed a selector array A the same shape as C in which a value is > 0, 1, or 2 depending on whether the C value is too small, okay, or too > big respectively. Then you might do something like: > > C = np.choose(A, [-inf, C, inf]) > > This is something you might want to do no matter what shape A and C > have. It's important not to require that the choices be an array of > choices, because they often have quite different shapes (here, two are > scalars) and it would be wasteful to broadcast them up to the same > shape as C, just to stack them. > OK, that's a pretty generic use-case, thanks; let me see if I understand it correctly: A is some how created independently with a 0 everywhere C is too small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere C is too small, inf everywhere C is too large, and C otherwise (and since -inf and inf are scalars, this implies broadcasting of these is taking place). This is what you're asserting *should* be the behavior. So, unless there is disagreement about this (you yourself said the opposite viewpoint might rationally be held) np.choose definitely presently has a bug, namely, the index array can't be of arbitrary shape. DG > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sun Nov 8 03:57:32 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 8 Nov 2009 03:57:32 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> Message-ID: 2009/11/8 David Goldsmith : > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald > wrote: >> >> 2009/11/7 David Goldsmith : >> > So in essence, at least as it presently functions, the shape of 'a' >> > *defines* what the individual choices are within 'choices`, and if >> > 'choices' >> > can't be parsed into an integer number of such individual choices, >> > that's >> > when an exception is raised? >> >> Um, I don't think so. >> >> Think of it this way: you provide np.choose with a selector array, a, >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct an >> output array, say r, the same shape as a (no matter how many >> dimensions it has). > > Except that I haven't yet seen a working example with 'a' greater than 1-D, > Josef's last two examples notwithstanding; or is that what you're saying is > the bug. There's nothing magic about A being one-dimensional. C = np.random.randn(2,3,5) A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) R = np.choose(A, (-1, -C, C, 1)) Requv = np.minimum(np.abs(C),1) or: def wedge(*functions): """Return a function whose value is the minimum of those of functions""" def wedgef(X): fXs = [f(X) for f in functions] A = np.argmin(fXs, axis=0) return np.choose(A,fXs) return wedgef so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) This works no matter what shape of X the user supplies - so a wedged function can be somewhat ufunclike - by making A the same shape. >> The (i0, i1, ..., iN) element of the output array >> is obtained by looking at the (i0, i1, ..., iN) element of a, which >> should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] >> = cj[i0, i1, ..., iN]. That is, each element of the selector array >> determines which of the choice arrays to pull the corresponding >> element from. > > That's pretty clear (thanks for doing my work for me). ;-), Yet, see above. > >> For example, suppose that you are processing an array C, and have >> constructed a selector array A the same shape as C in which a value is >> 0, 1, or 2 depending on whether the C value is too small, okay, or too >> big respectively. Then you might do something like: >> >> C = np.choose(A, [-inf, C, inf]) >> >> This is something you might want to do no matter what shape A and C >> have. It's important not to require that the choices be an array of >> choices, because they often have quite different shapes (here, two are >> scalars) and it would be wasteful to broadcast them up to the same >> shape as C, just to stack them. > > OK, that's a pretty generic use-case, thanks; let me see if I understand it > correctly: A is some how created independently with a 0 everywhere C is too > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere C is > too small, inf everywhere C is too large, and C otherwise (and since -inf > and inf are scalars, this implies broadcasting of these is taking place). > This is what you're asserting *should* be the behavior.? So, unless there is > disagreement about this (you yourself said the opposite viewpoint might > rationally be held) np.choose definitely presently has a bug, namely, the > index array can't be of arbitrary shape. There seems to be some disagreement between versions, but both Josef and I find that the index array *can* be arbitrary shape. In numpy 1.2.1 I find that all the choose items must be the same shape as it, which I think is a bug. What I suggested might be okay was if the index array was not broadcasted, so that the outputs always had exactly the same shape as the index array. But upon reflection it's useful to be able to use a 1-d array to select rows from a set of matrices, so I now think that all of A and the elements of choose should be broadcast to the same shape. This seems to be what Josef observes in his version of numpy, so maybe there's nothing to do. Anne > DG > >> >> Anne >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From d.l.goldsmith at gmail.com Sun Nov 8 05:00:27 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 02:00:27 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> Message-ID: <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald wrote: > 2009/11/8 David Goldsmith : > > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> 2009/11/7 David Goldsmith : > >> > So in essence, at least as it presently functions, the shape of 'a' > >> > *defines* what the individual choices are within 'choices`, and if > >> > 'choices' > >> > can't be parsed into an integer number of such individual choices, > >> > that's > >> > when an exception is raised? > >> > >> Um, I don't think so. > >> > >> Think of it this way: you provide np.choose with a selector array, a, > >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct an > >> output array, say r, the same shape as a (no matter how many > >> dimensions it has). > > > > Except that I haven't yet seen a working example with 'a' greater than > 1-D, > > Josef's last two examples notwithstanding; or is that what you're saying > is > > the bug. > > There's nothing magic about A being one-dimensional. > > C = np.random.randn(2,3,5) > A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) > > R = np.choose(A, (-1, -C, C, 1)) > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for choose's arguments is that both can be broadcast to a common shape (as you state below), but choose won't reshape the arguments for you to make this possible, you have to do so yourself first, if necessary. That does appear to be what's happening now; but do we want choose to be smarter than that (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user doesn't need to include the .reshape((3,1)))? DG > Requv = np.minimum(np.abs(C),1) > > or: > > def wedge(*functions): > """Return a function whose value is the minimum of those of > functions""" > def wedgef(X): > fXs = [f(X) for f in functions] > A = np.argmin(fXs, axis=0) > return np.choose(A,fXs) > return wedgef > > so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) > > This works no matter what shape of X the user supplies - so a wedged > function can be somewhat ufunclike - by making A the same shape. > > >> The (i0, i1, ..., iN) element of the output array > >> is obtained by looking at the (i0, i1, ..., iN) element of a, which > >> should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] > >> = cj[i0, i1, ..., iN]. That is, each element of the selector array > >> determines which of the choice arrays to pull the corresponding > >> element from. > > > > That's pretty clear (thanks for doing my work for me). ;-), Yet, see > above. > > > >> For example, suppose that you are processing an array C, and have > >> constructed a selector array A the same shape as C in which a value is > >> 0, 1, or 2 depending on whether the C value is too small, okay, or too > >> big respectively. Then you might do something like: > >> > >> C = np.choose(A, [-inf, C, inf]) > >> > >> This is something you might want to do no matter what shape A and C > >> have. It's important not to require that the choices be an array of > >> choices, because they often have quite different shapes (here, two are > >> scalars) and it would be wasteful to broadcast them up to the same > >> shape as C, just to stack them. > > > > OK, that's a pretty generic use-case, thanks; let me see if I understand > it > > correctly: A is some how created independently with a 0 everywhere C is > too > > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then > > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere C > is > > too small, inf everywhere C is too large, and C otherwise (and since -inf > > and inf are scalars, this implies broadcasting of these is taking place). > > This is what you're asserting *should* be the behavior. So, unless there > is > > disagreement about this (you yourself said the opposite viewpoint might > > rationally be held) np.choose definitely presently has a bug, namely, the > > index array can't be of arbitrary shape. > > There seems to be some disagreement between versions, but both Josef > and I find that the index array *can* be arbitrary shape. In numpy > 1.2.1 I find that all the choose items must be the same shape as it, > which I think is a bug. > > What I suggested might be okay was if the index array was not > broadcasted, so that the outputs always had exactly the same shape as > the index array. But upon reflection it's useful to be able to use a > 1-d array to select rows from a set of matrices, so I now think that > all of A and the elements of choose should be broadcast to the same > shape. This seems to be what Josef observes in his version of numpy, > so maybe there's nothing to do. > > Anne > > > DG > > > >> > >> Anne > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Sun Nov 8 05:07:55 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 02:07:55 -0800 Subject: [Numpy-discussion] Random int64 and float64 numbers In-Reply-To: <4AF5CD17.1070108@molden.no> References: <0E41C9AC-5972-492A-B3F2-34DD97085112@gmail.com> <45d1ab480911051354y445b970dq395b0e4b280baae0@mail.gmail.com> <5b8d13220911051902p50cdeeaqbfeea6ae52aef4d2@mail.gmail.com> <4AF5CD17.1070108@molden.no> Message-ID: <45d1ab480911080207k18ddf609x5a25bc3fbfdb22be@mail.gmail.com> Thanks, Sturla, you've confirmed what I thought was the case (and explained more thoroughly the answer others gave more succinctly, but also more opaquely). :-) DG On Sat, Nov 7, 2009 at 11:40 AM, Sturla Molden wrote: > David Cournapeau wrote: > > On Fri, Nov 6, 2009 at 6:54 AM, David Goldsmith > wrote: > > > >> Interesting thread, which leaves me wondering two things: is it > documented > >> somewhere (e.g., at the IEEE site) precisely how many *decimal* > mantissae > >> are representable using the 64-bit IEEE standard for float > representation > >> (if that makes sense); and are such decimal mantissae uniformly > distributed > >> > > > > They are definitely not uniformly distributed: that's why two numbers > > are close around 1 when they have only a few EPS difference, but > > around 1e100, you have to add quite a few EPS to even get a different > > number at all. > > > > That may be my audio processing background, but I like to think about > > float as numbers which have the same relative precision at any "level" > > - a kind of dB scale. If you want numbers with a fixed number of > > decimals, you need a fixed point representation. > > > > David Godsmith was asking about the mantissae. For a double, that is a > 53 bit signed integer. I.e. you have 52 bit fractional part (bit 0-51), > 11 bit exponent (bit 52-62), and one sign bit (bit 63). The mantissae is > uniformly distributed like any signed integer. The mantissae of a double > have 2**53 different integer values: -2**52 to 2**52-1. > > But the value of a floating point number is > > value = (-1)**signbit * 2**(exponent - bias) * (1 - fraction) > > with bias = 1023 for a double. Thus, floating point numbers are not > uniformly distributed, but the mantissae is. > > For numerical illiterates this might come as a surprise. But in > numerical mathematics, the resolution is in the number of "significant > digits", not in "the number of decimals". 101 and .00201 have the same > numerical precision. > > A decimal, on the other hand, can be thought of as a floating point > number using base-10 instead of base-2 for the exponent: > > value = (-1)**signbit * 10**(exponent - bias) * (1 - fraction) > > Decimals and floats are not fundamentally different. There are number > exactly representable with a decimal that cannot be exactly represented > with a float. But numerical computation do not become more precise with > a decimal than a float. > > > > > > > > > > > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Sun Nov 8 08:17:05 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 8 Nov 2009 15:17:05 +0200 Subject: [Numpy-discussion] distutils, fcompiler and Mayavi Message-ID: <710F2847B0018641891D9A21602763605AD201@ex3.envision.co.il> Hi all, Recentrly I tried to install mayavi-3,3,0, and it failed with a long chain of error messages ended with . . . File "/usr/lib64/python2.6/site-packages/numpy/distutils/command/config_compiler.py", line 66, in finalize_options v = getattr(c,a) File "/usr/lib64/python2.6/distutils/cmd.py", line 112, in __getattr__ raise AttributeError, attr AttributeError: fcompiler I solved (following Robert Kern's post) by (temporary) commenting out the loop that starts by: for a in ['fcompiler']: Is there an elegant solution? I an using numpy 1.4.0.dev7539 on python-2.6.4 Nadav. From yves.frederix at gmail.com Sun Nov 8 08:25:11 2009 From: yves.frederix at gmail.com (Yves Frederix) Date: Sun, 8 Nov 2009 14:25:11 +0100 Subject: [Numpy-discussion] f2py function callback: error while using repeated arguments in function call Message-ID: <62e6eafb0911080525x10d31a30le5d67f1500c9ee3e@mail.gmail.com> Hi, I am doing a simple function callback from fortran to python for which the actual function call in fortran has repeated arguments. ! callback_error.f90: subroutine testfun(x) ? ?double precision, intent(in) :: x ? ?double precision :: y !f2py intent(callback) foo !f2py double precision :: arg1 !f2py double precision :: arg2 !f2py double precision :: y !f2py external y = foo(arg1, arg2) ? ?external foo ? ?y = foo(x, x) ? ? ! ?<-- this causes problems ? ?print *, 'y:', y end subroutine testfun Running f2py gives the following error: compile options: '-I/tmp/tmpTVsLUT/src.linux-i686-2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/usr/include/python2.6 -c' gcc: /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:256: error: redefinition of parameter ?x_cb_capi? /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:256: note: previous definition of ?x_cb_capi? was here /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c: In function ?foo_?: /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:265: error: redefinition of ?x? /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:264: note: previous definition of ?x? was here /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:256: error: redefinition of parameter ?x_cb_capi? /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:256: note: previous definition of ?x_cb_capi? was here /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c: In function ?foo_?: /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:265: error: redefinition of ?x? /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c:264: note: previous definition of ?x? was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/tmp/tmpTVsLUT/src.linux-i686-2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/usr/include/python2.6 -c /tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.c -o /tmp/tmpTVsLUT/tmp/tmpTVsLUT/src.linux-i686-2.6/callbackmodule.o" failed with exit status 1 If I change the fortran code slightly, making a copy of the input argument and using this copy instead of the second 'x' in the function call, the compilation succeeds. ! callback_ok.f90 subroutine testfun(x) ? ?double precision, intent(in) :: x ? ?double precision :: y ? ?double precision :: xtmp !f2py intent(callback) foo !f2py double precision :: arg1 !f2py double precision :: arg2 !f2py double precision :: y !f2py external y = foo(arg1, arg2) ? ?external foo ? ?xtmp = x ? ?y = foo(x, xtmp) ? ?! <-- this works ? ?print *, 'y:', y end subroutine testfun Is this expected behavior? I am using f2py that comes with the python-numpy package (version 1:1.3.0-3) from Ubuntu Karmic. Regards, YVES From josef.pktd at gmail.com Sun Nov 8 08:36:00 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Nov 2009 08:36:00 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> Message-ID: <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith wrote: > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald > wrote: >> >> 2009/11/8 David Goldsmith : >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >> > >> > wrote: >> >> >> >> 2009/11/7 David Goldsmith : >> >> > So in essence, at least as it presently functions, the shape of 'a' >> >> > *defines* what the individual choices are within 'choices`, and if >> >> > 'choices' >> >> > can't be parsed into an integer number of such individual choices, >> >> > that's >> >> > when an exception is raised? >> >> >> >> Um, I don't think so. >> >> >> >> Think of it this way: you provide np.choose with a selector array, a, >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct an >> >> output array, say r, the same shape as a (no matter how many >> >> dimensions it has). >> > >> > Except that I haven't yet seen a working example with 'a' greater than >> > 1-D, >> > Josef's last two examples notwithstanding; or is that what you're saying >> > is >> > the bug. >> >> There's nothing magic about A being one-dimensional. >> >> C = np.random.randn(2,3,5) >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >> >> R = np.choose(A, (-1, -C, C, 1)) > > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for choose's > arguments is that both can be broadcast to a common shape (as you state > below), but choose won't reshape the arguments for you to make this > possible, you have to do so yourself first, if necessary.? That does appear > to be what's happening now; but do we want choose to be smarter than that > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user > doesn't need to include the .reshape((3,1)))? No, I don't think we want to be that smart. If standard broadcasting rules apply, as I think they do, then I wouldn't want any special newaxis or reshapes done automatically. It will be confusing, the function wouldn't know what to do if there are, e.g., as many rows as columns, and this looks like a big source of errors. Standard broadcasting is pretty nice (once I got the hang of it), and adding a lot of np.newaxis (or some reshapes) to the code is only a small price to pay. Josef > > DG > >> >> Requv = np.minimum(np.abs(C),1) >> >> or: >> >> def wedge(*functions): >> ? ? """Return a function whose value is the minimum of those of >> functions""" >> ? ? def wedgef(X): >> ? ? ? ? ?fXs = [f(X) for f in functions] >> ? ? ? ? ?A = np.argmin(fXs, axis=0) >> ? ? ? ? ?return np.choose(A,fXs) >> ? ? return wedgef >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >> >> This works no matter what shape of X the user supplies - so a wedged >> function can be somewhat ufunclike - by making A the same shape. >> >> >> The (i0, i1, ..., iN) element of the output array >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, which >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array >> >> determines which of the choice arrays to pull the corresponding >> >> element from. >> > >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, see >> > above. >> > >> >> For example, suppose that you are processing an array C, and have >> >> constructed a selector array A the same shape as C in which a value is >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or too >> >> big respectively. Then you might do something like: >> >> >> >> C = np.choose(A, [-inf, C, inf]) >> >> >> >> This is something you might want to do no matter what shape A and C >> >> have. It's important not to require that the choices be an array of >> >> choices, because they often have quite different shapes (here, two are >> >> scalars) and it would be wasteful to broadcast them up to the same >> >> shape as C, just to stack them. >> > >> > OK, that's a pretty generic use-case, thanks; let me see if I understand >> > it >> > correctly: A is some how created independently with a 0 everywhere C is >> > too >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere C >> > is >> > too small, inf everywhere C is too large, and C otherwise (and since >> > -inf >> > and inf are scalars, this implies broadcasting of these is taking >> > place). >> > This is what you're asserting *should* be the behavior.? So, unless >> > there is >> > disagreement about this (you yourself said the opposite viewpoint might >> > rationally be held) np.choose definitely presently has a bug, namely, >> > the >> > index array can't be of arbitrary shape. >> >> There seems to be some disagreement between versions, but both Josef >> and I find that the index array *can* be arbitrary shape. In numpy >> 1.2.1 I find that all the choose items must be the same shape as it, >> which I think is a bug. >> >> What I suggested might be okay was if the index array was not >> broadcasted, so that the outputs always had exactly the same shape as >> the index array. But upon reflection it's useful to be able to use a >> 1-d array to select rows from a set of matrices, so I now think that >> all of A and the elements of choose should be broadcast to the same >> shape. This seems to be what Josef observes in his version of numpy, >> so maybe there's nothing to do. >> >> Anne >> >> > DG >> > >> >> >> >> Anne >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From brennan.williams at visualreservoir.com Sun Nov 8 18:33:47 2009 From: brennan.williams at visualreservoir.com (Brennan Williams) Date: Mon, 09 Nov 2009 12:33:47 +1300 Subject: [Numpy-discussion] problem with FortranFile In-Reply-To: <4AF1BD45.4050608@wartburg.edu> References: <4AF0DDDC.3080405@visualreservoir.com> <4AF0E465.3000607@visualreservoir.com> <4AF1BD45.4050608@wartburg.edu> Message-ID: <4AF7555B.2010500@visualreservoir.com> I'm using FortranFile to read a binary Fortran file. It has a bit of header data at the top of the file which I'm reading with a combination of readString and struct.unpack This is then followed by a number of lines/records, each of which has 20 double precision reals/floats. For some reason it reads the first 19 ok and then things start going wrong. To read the double precision data I'm using.... for il in range(0,nlines): try: darray=f.readReals('d') except: print 'problem reading well data line',il I've added print statements to my code and to fortranfile.py and the print output I get is.... line 19 readRecord:l= 160 readRecord:len(data_str)= 160 readRecord:check_size= 160 readReals:len(data_str)= 160 calcsizeprec= 8 num= 20 numbers (4.2843303680419922, 0.0, 0.0, 0.0, 4955.73974609375, 0.0, -1000.0, -1000.0, 0.0, 0.0, 0.0, 21.22749137878418, 0.0, 0.0, 0.0, 0.0, 0.2054678201675415, 0.0, 6386.78271484375, 6356.27001953125) il= 19 dsize= 20 darray= [ 4.28433037e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 4.95573975e+03 0.00000000e+00 -1.00000000e+03 -1.00000000e+03 0.00000000e+00 0.00000000e+00 0.00000000e+00 2.12274914e+01 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 2.05467820e-01 0.00000000e+00 6.38678271e+03 6.35627002e+03] --------------------- line 20 readRecord:l= 160 readRecord:len(data_str)= 132 problem reading well data line 20 line 19 is ok, readReals calls readRecord and the length of the data is 160 and check_size = l etc etc. line 20 is not ok, the 4-byte length value at the start of the record is 160 but the data_str=self.read(l) line only gets 132 bytes and not 160. The data file itself is ok as I've written a small Fortran program to read it. So, am I doing something wrong? I have a little test .py file and data file I can include if it helps. Regards Brennan From alan at ajackson.org Sun Nov 8 19:11:30 2009 From: alan at ajackson.org (alan at ajackson.org) Date: Sun, 8 Nov 2009 18:11:30 -0600 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <4AF6487B.9050808@american.edu> References: <20091107215629.6009523e@ajackson.org> <4AF6487B.9050808@american.edu> Message-ID: <20091108181130.158759b3@ajackson.org> >On 11/7/2009 10:56 PM, alan at ajackson.org wrote: >> I want to build a 2D array of lists, and so I need to initialize the >> array with empty lists : >> >> myarray = array([[[],[],[]] ,[[],[],[]]]) > > >[[[] for i in range(3)] for j in range(2) ] > >fwiw, >Alan Isaac Thanks! I like that - concise and a little strange. 8-) -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From lists_ravi at lavabit.com Sun Nov 8 20:02:35 2009 From: lists_ravi at lavabit.com (Ravi) Date: Sun, 8 Nov 2009 20:02:35 -0500 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <20091107215629.6009523e@ajackson.org> References: <20091107215629.6009523e@ajackson.org> Message-ID: <200911082002.35527.lists_ravi@lavabit.com> On Saturday 07 November 2009 22:56:29 alan at ajackson.org wrote: > I want to build a 2D array of lists, and so I need to initialize the > array with empty lists : > > myarray = array([[[],[],[]] ,[[],[],[]]]) In [1]: [[[]]*3]*2 Out[1]: [[[], [], []], [[], [], []]] Hope this helps. Ravi From kwgoodman at gmail.com Sun Nov 8 20:33:22 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 8 Nov 2009 17:33:22 -0800 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <200911082002.35527.lists_ravi@lavabit.com> References: <20091107215629.6009523e@ajackson.org> <200911082002.35527.lists_ravi@lavabit.com> Message-ID: On Sun, Nov 8, 2009 at 5:02 PM, Ravi wrote: > On Saturday 07 November 2009 22:56:29 alan at ajackson.org wrote: >> I want to build a 2D array of lists, and so I need to initialize the >> array with empty lists : >> >> myarray = array([[[],[],[]] ,[[],[],[]]]) > > > In [1]: [[[]]*3]*2 > Out[1]: [[[], [], []], [[], [], []]] Have to be careful with that one: >> x = [[[]]*3]*2 >> x[0][0] = [9999] >> x [[[9999], [], []], [[9999], [], []]] but not with this one: >> x = [[[] for i in range(3)] for j in range(2)] >> x[0][0] = [9999] >> x [[[9999], [], []], [[], [], []]] From d.l.goldsmith at gmail.com Sun Nov 8 22:03:12 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 19:03:12 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> Message-ID: <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> OK, now I'm trying to wrap my brain around broadcasting in choose when both `a` *and* `choices` need to be (non-trivially) broadcast in order to arrive at a common shape, e.g.: >>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >>> a=np.eye(2, dtype=int) # shape is (2,2) >>> np.choose(a,c) array([[2, 1], [0, 3]]) (Unfortunately, the implementation is in C, so I can't easily insert print statements to see intermediate results.) First, let me confirm that the above is indeed an example of what I think it is, i.e., both `a` and `choices` are broadcast in order for this to work, correct? (And if incorrect, how is one broadcast to the shape of the other?) Second, both are broadcast to shape (2,2,2), correct? But how, precisely, i.e., does c become [[[0, 1], [2, 3]], [[[0, 1], [0, 1]], [[0, 1], [2, 3]]] or [[2, 3], [2, 3]]] and same question for a? Then, once a is broadcast to a (2,2,2) shape, precisely how does it "pick and choose" from c to create a (2,2) result? For example, suppose a is broadcast to: [[[1, 0], [0, 1]], [[1, 0], [0, 1]]] (as indicated above, I'm uncertain at this point if this is indeed what a is broadcast to); how does this create the (2,2) result obtained above? (Obviously this depends in part on precisely how c is broadcast, I do recognize that much.) Finally, a seemingly relevant comment in the C source is: /* Broadcast all arrays to each other, index array at the end.*/ This would appear to confirm that "co-broadcasting" is performed if necessary, but what does the "index array at the end" phrase mean? Thanks for your continued patience and tutelage. DG On Sun, Nov 8, 2009 at 5:36 AM, wrote: > On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith > wrote: > > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> 2009/11/8 David Goldsmith : > >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald > >> > > >> > wrote: > >> >> > >> >> 2009/11/7 David Goldsmith : > >> >> > So in essence, at least as it presently functions, the shape of 'a' > >> >> > *defines* what the individual choices are within 'choices`, and if > >> >> > 'choices' > >> >> > can't be parsed into an integer number of such individual choices, > >> >> > that's > >> >> > when an exception is raised? > >> >> > >> >> Um, I don't think so. > >> >> > >> >> Think of it this way: you provide np.choose with a selector array, a, > >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct > an > >> >> output array, say r, the same shape as a (no matter how many > >> >> dimensions it has). > >> > > >> > Except that I haven't yet seen a working example with 'a' greater than > >> > 1-D, > >> > Josef's last two examples notwithstanding; or is that what you're > saying > >> > is > >> > the bug. > >> > >> There's nothing magic about A being one-dimensional. > >> > >> C = np.random.randn(2,3,5) > >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) > >> > >> R = np.choose(A, (-1, -C, C, 1)) > > > > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and > > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but > > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for choose's > > arguments is that both can be broadcast to a common shape (as you state > > below), but choose won't reshape the arguments for you to make this > > possible, you have to do so yourself first, if necessary. That does > appear > > to be what's happening now; but do we want choose to be smarter than that > > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user > > doesn't need to include the .reshape((3,1)))? > > No, I don't think we want to be that smart. > > If standard broadcasting rules apply, as I think they do, then I wouldn't > want > any special newaxis or reshapes done automatically. It will be confusing, > the function wouldn't know what to do if there are, e.g., as many rows as > columns, and this looks like a big source of errors. > Standard broadcasting is pretty nice (once I got the hang of it), and > adding > a lot of np.newaxis (or some reshapes) to the code is only a small price to > pay. > > Josef > > > > > > > DG > > > >> > >> Requv = np.minimum(np.abs(C),1) > >> > >> or: > >> > >> def wedge(*functions): > >> """Return a function whose value is the minimum of those of > >> functions""" > >> def wedgef(X): > >> fXs = [f(X) for f in functions] > >> A = np.argmin(fXs, axis=0) > >> return np.choose(A,fXs) > >> return wedgef > >> > >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) > >> > >> This works no matter what shape of X the user supplies - so a wedged > >> function can be somewhat ufunclike - by making A the same shape. > >> > >> >> The (i0, i1, ..., iN) element of the output array > >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, which > >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., iN] > >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array > >> >> determines which of the choice arrays to pull the corresponding > >> >> element from. > >> > > >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, see > >> > above. > >> > > >> >> For example, suppose that you are processing an array C, and have > >> >> constructed a selector array A the same shape as C in which a value > is > >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or > too > >> >> big respectively. Then you might do something like: > >> >> > >> >> C = np.choose(A, [-inf, C, inf]) > >> >> > >> >> This is something you might want to do no matter what shape A and C > >> >> have. It's important not to require that the choices be an array of > >> >> choices, because they often have quite different shapes (here, two > are > >> >> scalars) and it would be wasteful to broadcast them up to the same > >> >> shape as C, just to stack them. > >> > > >> > OK, that's a pretty generic use-case, thanks; let me see if I > understand > >> > it > >> > correctly: A is some how created independently with a 0 everywhere C > is > >> > too > >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then > >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere > C > >> > is > >> > too small, inf everywhere C is too large, and C otherwise (and since > >> > -inf > >> > and inf are scalars, this implies broadcasting of these is taking > >> > place). > >> > This is what you're asserting *should* be the behavior. So, unless > >> > there is > >> > disagreement about this (you yourself said the opposite viewpoint > might > >> > rationally be held) np.choose definitely presently has a bug, namely, > >> > the > >> > index array can't be of arbitrary shape. > >> > >> There seems to be some disagreement between versions, but both Josef > >> and I find that the index array *can* be arbitrary shape. In numpy > >> 1.2.1 I find that all the choose items must be the same shape as it, > >> which I think is a bug. > >> > >> What I suggested might be okay was if the index array was not > >> broadcasted, so that the outputs always had exactly the same shape as > >> the index array. But upon reflection it's useful to be able to use a > >> 1-d array to select rows from a set of matrices, so I now think that > >> all of A and the elements of choose should be broadcast to the same > >> shape. This seems to be what Josef observes in his version of numpy, > >> so maybe there's nothing to do. > >> > >> Anne > >> > >> > DG > >> > > >> >> > >> >> Anne > >> >> _______________________________________________ > >> >> NumPy-Discussion mailing list > >> >> NumPy-Discussion at scipy.org > >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Nov 8 22:12:58 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 9 Nov 2009 12:12:58 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF1B6CA.1060608@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> Message-ID: <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> Hi Michael, On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom wrote: > I'm getting the following from r7603 on Solaris Sparc -- somehow related > to not having a long double version of next after available. ?I realise > not everyone has access to (or is dependent on) this platform, so I'm > willing to help in whatever way I can, I'm just not sure I understand > the change yet. Could you check whether r7710 fixes it for you ? David From josef.pktd at gmail.com Sun Nov 8 22:23:44 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Nov 2009 22:23:44 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> Message-ID: <1cd32cbb0911081923v4ad2c291t3db1d040417d1016@mail.gmail.com> On Sun, Nov 8, 2009 at 10:03 PM, David Goldsmith wrote: > OK, now I'm trying to wrap my brain around broadcasting in choose when both > `a` *and* `choices` need to be (non-trivially) broadcast in order to arrive > at a common shape, e.g.: > >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >>>> a=np.eye(2, dtype=int) # shape is (2,2) >>>> np.choose(a,c) > array([[2, 1], > ?????? [0, 3]]) > > (Unfortunately, the implementation is in C, so I can't easily insert print > statements to see intermediate results.) > > First, let me confirm that the above is indeed an example of what I think it > is, i.e., both `a` and `choices` are broadcast in order for this to work, > correct?? (And if incorrect, how is one broadcast to the shape of the > other?)? Second, both are broadcast to shape (2,2,2), correct? Not, You only have one choice array, so first it is interpreted as a sequence with c[0], c[1], then c[0] and c[0] are broadcasted to each other (1,2), then they are broadcasted to a (2,2), then a picks from 0,0 1,1 or 2,2 3,3 which is 2,0 1,3 qed Josef >>> np.choose(a,[c[0],c[1]]) array([[2, 1], [0, 3]]) >>> c[0] array([[0, 1]]) >>> c[1] array([[2, 3]]) >>> c[1].shape (1, 2) >>> c[0].shape (1, 2) >>> a array([[1, 0], [0, 1]]) But how, > precisely, i.e., does c become > > [[[0, 1], [2, 3]],??????? [[[0, 1], [0, 1]], > ?[[0, 1], [2, 3]]]?? or?? [[2, 3], [2, 3]]] > > and same question for a?? Then, once a is broadcast to a (2,2,2) shape, > precisely how does it "pick and choose" from c to create a (2,2) result? > For example, suppose a is broadcast to: > > [[[1, 0], [0, 1]], > ?[[1, 0], [0, 1]]] > > (as indicated above, I'm uncertain at this point if this is indeed what a is > broadcast to); how does this create the (2,2) result obtained above? > (Obviously this depends in part on precisely how c is broadcast, I do > recognize that much.) > > Finally, a seemingly relevant comment in the C source is: > > /* Broadcast all arrays to each other, index array at the end.*/ > > This would appear to confirm that "co-broadcasting" is performed if > necessary, but what does the "index array at the end" phrase mean? > > Thanks for your continued patience and tutelage. > > DG > > On Sun, Nov 8, 2009 at 5:36 AM, wrote: >> >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith >> wrote: >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald >> > >> > wrote: >> >> >> >> 2009/11/8 David Goldsmith : >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >> >> > >> >> > wrote: >> >> >> >> >> >> 2009/11/7 David Goldsmith : >> >> >> > So in essence, at least as it presently functions, the shape of >> >> >> > 'a' >> >> >> > *defines* what the individual choices are within 'choices`, and if >> >> >> > 'choices' >> >> >> > can't be parsed into an integer number of such individual choices, >> >> >> > that's >> >> >> > when an exception is raised? >> >> >> >> >> >> Um, I don't think so. >> >> >> >> >> >> Think of it this way: you provide np.choose with a selector array, >> >> >> a, >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct >> >> >> an >> >> >> output array, say r, the same shape as a (no matter how many >> >> >> dimensions it has). >> >> > >> >> > Except that I haven't yet seen a working example with 'a' greater >> >> > than >> >> > 1-D, >> >> > Josef's last two examples notwithstanding; or is that what you're >> >> > saying >> >> > is >> >> > the bug. >> >> >> >> There's nothing magic about A being one-dimensional. >> >> >> >> C = np.random.randn(2,3,5) >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >> >> >> >> R = np.choose(A, (-1, -C, C, 1)) >> > >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for >> > choose's >> > arguments is that both can be broadcast to a common shape (as you state >> > below), but choose won't reshape the arguments for you to make this >> > possible, you have to do so yourself first, if necessary.? That does >> > appear >> > to be what's happening now; but do we want choose to be smarter than >> > that >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user >> > doesn't need to include the .reshape((3,1)))? >> >> No, I don't think we want to be that smart. >> >> If standard broadcasting rules apply, as I think they do, then I wouldn't >> want >> any special newaxis or reshapes done automatically. It will be confusing, >> the function wouldn't know what to do if there are, e.g., as many rows as >> columns, and this looks like a big source of errors. >> Standard broadcasting is pretty nice (once I got the hang of it), and >> adding >> a lot of np.newaxis (or some reshapes) to the code is only a small price >> to pay. >> >> Josef >> >> >> >> > >> > DG >> > >> >> >> >> Requv = np.minimum(np.abs(C),1) >> >> >> >> or: >> >> >> >> def wedge(*functions): >> >> ? ? """Return a function whose value is the minimum of those of >> >> functions""" >> >> ? ? def wedgef(X): >> >> ? ? ? ? ?fXs = [f(X) for f in functions] >> >> ? ? ? ? ?A = np.argmin(fXs, axis=0) >> >> ? ? ? ? ?return np.choose(A,fXs) >> >> ? ? return wedgef >> >> >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >> >> >> >> This works no matter what shape of X the user supplies - so a wedged >> >> function can be somewhat ufunclike - by making A the same shape. >> >> >> >> >> The (i0, i1, ..., iN) element of the output array >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, which >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., >> >> >> iN] >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array >> >> >> determines which of the choice arrays to pull the corresponding >> >> >> element from. >> >> > >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, see >> >> > above. >> >> > >> >> >> For example, suppose that you are processing an array C, and have >> >> >> constructed a selector array A the same shape as C in which a value >> >> >> is >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or >> >> >> too >> >> >> big respectively. Then you might do something like: >> >> >> >> >> >> C = np.choose(A, [-inf, C, inf]) >> >> >> >> >> >> This is something you might want to do no matter what shape A and C >> >> >> have. It's important not to require that the choices be an array of >> >> >> choices, because they often have quite different shapes (here, two >> >> >> are >> >> >> scalars) and it would be wasteful to broadcast them up to the same >> >> >> shape as C, just to stack them. >> >> > >> >> > OK, that's a pretty generic use-case, thanks; let me see if I >> >> > understand >> >> > it >> >> > correctly: A is some how created independently with a 0 everywhere C >> >> > is >> >> > too >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere >> >> > C >> >> > is >> >> > too small, inf everywhere C is too large, and C otherwise (and since >> >> > -inf >> >> > and inf are scalars, this implies broadcasting of these is taking >> >> > place). >> >> > This is what you're asserting *should* be the behavior.? So, unless >> >> > there is >> >> > disagreement about this (you yourself said the opposite viewpoint >> >> > might >> >> > rationally be held) np.choose definitely presently has a bug, namely, >> >> > the >> >> > index array can't be of arbitrary shape. >> >> >> >> There seems to be some disagreement between versions, but both Josef >> >> and I find that the index array *can* be arbitrary shape. In numpy >> >> 1.2.1 I find that all the choose items must be the same shape as it, >> >> which I think is a bug. >> >> >> >> What I suggested might be okay was if the index array was not >> >> broadcasted, so that the outputs always had exactly the same shape as >> >> the index array. But upon reflection it's useful to be able to use a >> >> 1-d array to select rows from a set of matrices, so I now think that >> >> all of A and the elements of choose should be broadcast to the same >> >> shape. This seems to be what Josef observes in his version of numpy, >> >> so maybe there's nothing to do. >> >> >> >> Anne >> >> >> >> > DG >> >> > >> >> >> >> >> >> Anne >> >> >> _______________________________________________ >> >> >> NumPy-Discussion mailing list >> >> >> NumPy-Discussion at scipy.org >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From nmb at wartburg.edu Sun Nov 8 22:39:04 2009 From: nmb at wartburg.edu (Neil Martinsen-Burrell) Date: Sun, 08 Nov 2009 21:39:04 -0600 Subject: [Numpy-discussion] problem with FortranFile In-Reply-To: <4AF7555B.2010500@visualreservoir.com> References: <4AF0DDDC.3080405@visualreservoir.com> <4AF0E465.3000607@visualreservoir.com> <4AF1BD45.4050608@wartburg.edu> <4AF7555B.2010500@visualreservoir.com> Message-ID: <4AF78ED8.8050100@wartburg.edu> On 2009-11-08 17:33 , Brennan Williams wrote: > I'm using FortranFile to read a binary Fortran file. > It has a bit of header data at the top of the file which I'm reading > with a combination of readString and struct.unpack > This is then followed by a number of lines/records, each of which has 20 > double precision reals/floats. > For some reason it reads the first 19 ok and then things start going wrong. > > To read the double precision data I'm using.... > > for il in range(0,nlines): > try: > darray=f.readReals('d') > except: > print 'problem reading well data line',il > > > I've added print statements to my code and to fortranfile.py and the > print output I get is.... > > line 19 > readRecord:l= 160 > readRecord:len(data_str)= 160 > readRecord:check_size= 160 > readReals:len(data_str)= 160 > calcsizeprec= 8 > num= 20 > numbers > (4.2843303680419922, 0.0, 0.0, 0.0, 4955.73974609375, 0.0, -1000.0, > -1000.0, 0.0, 0.0, 0.0, 21.22749137878418, 0.0, 0.0, 0.0, 0.0, > 0.2054678201675415, 0.0, 6386.78271484375, 6356.27001953125) > il= 19 dsize= 20 > darray= [ 4.28433037e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 > 4.95573975e+03 0.00000000e+00 -1.00000000e+03 -1.00000000e+03 > 0.00000000e+00 0.00000000e+00 0.00000000e+00 2.12274914e+01 > 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 > 2.05467820e-01 0.00000000e+00 6.38678271e+03 6.35627002e+03] > --------------------- > line 20 > readRecord:l= 160 > readRecord:len(data_str)= 132 > problem reading well data line 20 > > > line 19 is ok, readReals calls readRecord and the length of the data is > 160 and check_size = l etc etc. > > line 20 is not ok, the 4-byte length value at the start of the record is > 160 but the data_str=self.read(l) line only gets 132 bytes and not 160. > > The data file itself is ok as I've written a small Fortran program to > read it. > > So, am I doing something wrong? > > I have a little test .py file and data file I can include if it helps. I've run into this problem before, but I don't recall what I did to work around it. The problem is this: python's f.read(SIZE) method returns at most SIZE bytes. There is no guarantee that it will actually return that number of bytes. The right fix is to call self.read in a loop until you get exactly the right number of bytes. The attached version of fortranfile.py should do the trick. Let me know if it does or doesn't help. [As an aside, fortranfile.py is code that I've written that isn't part of Numpy and perhaps the right place for any discussions of it is off-list.] -Neil -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fortranfile.py URL: From peridot.faceted at gmail.com Sun Nov 8 22:40:05 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 8 Nov 2009 22:40:05 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> Message-ID: As Josef said, this is not correct. I think the key point of confusion is this: Do not pass choose two arrays. Pass it one array and a *list* of arrays. The fact that choices can be an array is a quirk we can't change, but you should think of the second argument as a list of arrays, possibly of different shapes. np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) Here the first argument is an array of choices. The second argument is a *list* - if you cast it to an array you'll get an error or nonsense - of arrays to choose from. The broadcasting ensures the first argument and *each element of the list* are the same shape. The only constraint on the number of arrays in the list is that it be larger than the largest value in a. If you try to make the second argument into a single array, for one thing you are throwing away useful generality by forcing each choice to be the same shape (and real shape, not zero-strided fake shape), and for another the broadcasting becomes very hard to understand. Anne 2009/11/8 David Goldsmith : > OK, now I'm trying to wrap my brain around broadcasting in choose when both > `a` *and* `choices` need to be (non-trivially) broadcast in order to arrive > at a common shape, e.g.: > >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >>>> a=np.eye(2, dtype=int) # shape is (2,2) >>>> np.choose(a,c) > array([[2, 1], > ?????? [0, 3]]) > > (Unfortunately, the implementation is in C, so I can't easily insert print > statements to see intermediate results.) > > First, let me confirm that the above is indeed an example of what I think it > is, i.e., both `a` and `choices` are broadcast in order for this to work, > correct?? (And if incorrect, how is one broadcast to the shape of the > other?)? Second, both are broadcast to shape (2,2,2), correct?? But how, > precisely, i.e., does c become > > [[[0, 1], [2, 3]],??????? [[[0, 1], [0, 1]], > ?[[0, 1], [2, 3]]]?? or?? [[2, 3], [2, 3]]] > > and same question for a?? Then, once a is broadcast to a (2,2,2) shape, > precisely how does it "pick and choose" from c to create a (2,2) result? > For example, suppose a is broadcast to: > > [[[1, 0], [0, 1]], > ?[[1, 0], [0, 1]]] > > (as indicated above, I'm uncertain at this point if this is indeed what a is > broadcast to); how does this create the (2,2) result obtained above? > (Obviously this depends in part on precisely how c is broadcast, I do > recognize that much.) > > Finally, a seemingly relevant comment in the C source is: > > /* Broadcast all arrays to each other, index array at the end.*/ > > This would appear to confirm that "co-broadcasting" is performed if > necessary, but what does the "index array at the end" phrase mean? > > Thanks for your continued patience and tutelage. > > DG > > On Sun, Nov 8, 2009 at 5:36 AM, wrote: >> >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith >> wrote: >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald >> > >> > wrote: >> >> >> >> 2009/11/8 David Goldsmith : >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >> >> > >> >> > wrote: >> >> >> >> >> >> 2009/11/7 David Goldsmith : >> >> >> > So in essence, at least as it presently functions, the shape of >> >> >> > 'a' >> >> >> > *defines* what the individual choices are within 'choices`, and if >> >> >> > 'choices' >> >> >> > can't be parsed into an integer number of such individual choices, >> >> >> > that's >> >> >> > when an exception is raised? >> >> >> >> >> >> Um, I don't think so. >> >> >> >> >> >> Think of it this way: you provide np.choose with a selector array, >> >> >> a, >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct >> >> >> an >> >> >> output array, say r, the same shape as a (no matter how many >> >> >> dimensions it has). >> >> > >> >> > Except that I haven't yet seen a working example with 'a' greater >> >> > than >> >> > 1-D, >> >> > Josef's last two examples notwithstanding; or is that what you're >> >> > saying >> >> > is >> >> > the bug. >> >> >> >> There's nothing magic about A being one-dimensional. >> >> >> >> C = np.random.randn(2,3,5) >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >> >> >> >> R = np.choose(A, (-1, -C, C, 1)) >> > >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for >> > choose's >> > arguments is that both can be broadcast to a common shape (as you state >> > below), but choose won't reshape the arguments for you to make this >> > possible, you have to do so yourself first, if necessary.? That does >> > appear >> > to be what's happening now; but do we want choose to be smarter than >> > that >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user >> > doesn't need to include the .reshape((3,1)))? >> >> No, I don't think we want to be that smart. >> >> If standard broadcasting rules apply, as I think they do, then I wouldn't >> want >> any special newaxis or reshapes done automatically. It will be confusing, >> the function wouldn't know what to do if there are, e.g., as many rows as >> columns, and this looks like a big source of errors. >> Standard broadcasting is pretty nice (once I got the hang of it), and >> adding >> a lot of np.newaxis (or some reshapes) to the code is only a small price >> to pay. >> >> Josef >> >> >> >> > >> > DG >> > >> >> >> >> Requv = np.minimum(np.abs(C),1) >> >> >> >> or: >> >> >> >> def wedge(*functions): >> >> ? ? """Return a function whose value is the minimum of those of >> >> functions""" >> >> ? ? def wedgef(X): >> >> ? ? ? ? ?fXs = [f(X) for f in functions] >> >> ? ? ? ? ?A = np.argmin(fXs, axis=0) >> >> ? ? ? ? ?return np.choose(A,fXs) >> >> ? ? return wedgef >> >> >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >> >> >> >> This works no matter what shape of X the user supplies - so a wedged >> >> function can be somewhat ufunclike - by making A the same shape. >> >> >> >> >> The (i0, i1, ..., iN) element of the output array >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, which >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., >> >> >> iN] >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array >> >> >> determines which of the choice arrays to pull the corresponding >> >> >> element from. >> >> > >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, see >> >> > above. >> >> > >> >> >> For example, suppose that you are processing an array C, and have >> >> >> constructed a selector array A the same shape as C in which a value >> >> >> is >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or >> >> >> too >> >> >> big respectively. Then you might do something like: >> >> >> >> >> >> C = np.choose(A, [-inf, C, inf]) >> >> >> >> >> >> This is something you might want to do no matter what shape A and C >> >> >> have. It's important not to require that the choices be an array of >> >> >> choices, because they often have quite different shapes (here, two >> >> >> are >> >> >> scalars) and it would be wasteful to broadcast them up to the same >> >> >> shape as C, just to stack them. >> >> > >> >> > OK, that's a pretty generic use-case, thanks; let me see if I >> >> > understand >> >> > it >> >> > correctly: A is some how created independently with a 0 everywhere C >> >> > is >> >> > too >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere >> >> > C >> >> > is >> >> > too small, inf everywhere C is too large, and C otherwise (and since >> >> > -inf >> >> > and inf are scalars, this implies broadcasting of these is taking >> >> > place). >> >> > This is what you're asserting *should* be the behavior.? So, unless >> >> > there is >> >> > disagreement about this (you yourself said the opposite viewpoint >> >> > might >> >> > rationally be held) np.choose definitely presently has a bug, namely, >> >> > the >> >> > index array can't be of arbitrary shape. >> >> >> >> There seems to be some disagreement between versions, but both Josef >> >> and I find that the index array *can* be arbitrary shape. In numpy >> >> 1.2.1 I find that all the choose items must be the same shape as it, >> >> which I think is a bug. >> >> >> >> What I suggested might be okay was if the index array was not >> >> broadcasted, so that the outputs always had exactly the same shape as >> >> the index array. But upon reflection it's useful to be able to use a >> >> 1-d array to select rows from a set of matrices, so I now think that >> >> all of A and the elements of choose should be broadcast to the same >> >> shape. This seems to be what Josef observes in his version of numpy, >> >> so maybe there's nothing to do. >> >> >> >> Anne >> >> >> >> > DG >> >> > >> >> >> >> >> >> Anne >> >> >> _______________________________________________ >> >> >> NumPy-Discussion mailing list >> >> >> NumPy-Discussion at scipy.org >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Sun Nov 8 22:49:46 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Nov 2009 20:49:46 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> Message-ID: On Sun, Nov 8, 2009 at 8:12 PM, David Cournapeau wrote: > Hi Michael, > > On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom > wrote: > > I'm getting the following from r7603 on Solaris Sparc -- somehow related > > to not having a long double version of next after available. I realise > > not everyone has access to (or is dependent on) this platform, so I'm > > willing to help in whatever way I can, I'm just not sure I understand > > the change yet. > > Could you check whether r7710 fixes it for you ? > > Umm, PPC doesn't have ieee quad, it is an amalgam of two doubles. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Nov 8 22:50:43 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Nov 2009 22:50:43 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> Message-ID: <1cd32cbb0911081950m37a3c7f0pa608da7d1cca7919@mail.gmail.com> On Sun, Nov 8, 2009 at 10:40 PM, Anne Archibald wrote: > As Josef said, this is not correct. I think the key point of confusion is this: > > Do not pass choose two arrays. > > Pass it one array and a *list* of arrays. The fact that choices can be > an array is a quirk we can't change, but you should think of the > second argument as a list of arrays, possibly of different shapes. > > np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) > > Here the first argument is an array of choices. The second argument is > a *list* - if you cast it to an array you'll get an error or nonsense > - of arrays to choose from. The broadcasting ensures the first > argument and *each element of the list* are the same shape. > The only > constraint on the number of arrays in the list is that it be larger > than the largest value in a. and it's possible to get around this constraint with the mode argument. Josef > > If you try to make the second argument into a single array, for one > thing you are throwing away useful generality by forcing each choice > to be the same shape (and real shape, not zero-strided fake shape), > and for another the broadcasting becomes very hard to understand. > > Anne > > > 2009/11/8 David Goldsmith : >> OK, now I'm trying to wrap my brain around broadcasting in choose when both >> `a` *and* `choices` need to be (non-trivially) broadcast in order to arrive >> at a common shape, e.g.: >> >>>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >>>>> a=np.eye(2, dtype=int) # shape is (2,2) >>>>> np.choose(a,c) >> array([[2, 1], >> ?????? [0, 3]]) >> >> (Unfortunately, the implementation is in C, so I can't easily insert print >> statements to see intermediate results.) >> >> First, let me confirm that the above is indeed an example of what I think it >> is, i.e., both `a` and `choices` are broadcast in order for this to work, >> correct?? (And if incorrect, how is one broadcast to the shape of the >> other?)? Second, both are broadcast to shape (2,2,2), correct?? But how, >> precisely, i.e., does c become >> >> [[[0, 1], [2, 3]],??????? [[[0, 1], [0, 1]], >> ?[[0, 1], [2, 3]]]?? or?? [[2, 3], [2, 3]]] >> >> and same question for a?? Then, once a is broadcast to a (2,2,2) shape, >> precisely how does it "pick and choose" from c to create a (2,2) result? >> For example, suppose a is broadcast to: >> >> [[[1, 0], [0, 1]], >> ?[[1, 0], [0, 1]]] >> >> (as indicated above, I'm uncertain at this point if this is indeed what a is >> broadcast to); how does this create the (2,2) result obtained above? >> (Obviously this depends in part on precisely how c is broadcast, I do >> recognize that much.) >> >> Finally, a seemingly relevant comment in the C source is: >> >> /* Broadcast all arrays to each other, index array at the end.*/ >> >> This would appear to confirm that "co-broadcasting" is performed if >> necessary, but what does the "index array at the end" phrase mean? >> >> Thanks for your continued patience and tutelage. >> >> DG >> >> On Sun, Nov 8, 2009 at 5:36 AM, wrote: >>> >>> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith >>> wrote: >>> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald >>> > >>> > wrote: >>> >> >>> >> 2009/11/8 David Goldsmith : >>> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >>> >> > >>> >> > wrote: >>> >> >> >>> >> >> 2009/11/7 David Goldsmith : >>> >> >> > So in essence, at least as it presently functions, the shape of >>> >> >> > 'a' >>> >> >> > *defines* what the individual choices are within 'choices`, and if >>> >> >> > 'choices' >>> >> >> > can't be parsed into an integer number of such individual choices, >>> >> >> > that's >>> >> >> > when an exception is raised? >>> >> >> >>> >> >> Um, I don't think so. >>> >> >> >>> >> >> Think of it this way: you provide np.choose with a selector array, >>> >> >> a, >>> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You construct >>> >> >> an >>> >> >> output array, say r, the same shape as a (no matter how many >>> >> >> dimensions it has). >>> >> > >>> >> > Except that I haven't yet seen a working example with 'a' greater >>> >> > than >>> >> > 1-D, >>> >> > Josef's last two examples notwithstanding; or is that what you're >>> >> > saying >>> >> > is >>> >> > the bug. >>> >> >>> >> There's nothing magic about A being one-dimensional. >>> >> >>> >> C = np.random.randn(2,3,5) >>> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >>> >> >>> >> R = np.choose(A, (-1, -C, C, 1)) >>> > >>> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and >>> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but >>> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for >>> > choose's >>> > arguments is that both can be broadcast to a common shape (as you state >>> > below), but choose won't reshape the arguments for you to make this >>> > possible, you have to do so yourself first, if necessary.? That does >>> > appear >>> > to be what's happening now; but do we want choose to be smarter than >>> > that >>> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user >>> > doesn't need to include the .reshape((3,1)))? >>> >>> No, I don't think we want to be that smart. >>> >>> If standard broadcasting rules apply, as I think they do, then I wouldn't >>> want >>> any special newaxis or reshapes done automatically. It will be confusing, >>> the function wouldn't know what to do if there are, e.g., as many rows as >>> columns, and this looks like a big source of errors. >>> Standard broadcasting is pretty nice (once I got the hang of it), and >>> adding >>> a lot of np.newaxis (or some reshapes) to the code is only a small price >>> to pay. >>> >>> Josef >>> >>> >>> >>> > >>> > DG >>> > >>> >> >>> >> Requv = np.minimum(np.abs(C),1) >>> >> >>> >> or: >>> >> >>> >> def wedge(*functions): >>> >> ? ? """Return a function whose value is the minimum of those of >>> >> functions""" >>> >> ? ? def wedgef(X): >>> >> ? ? ? ? ?fXs = [f(X) for f in functions] >>> >> ? ? ? ? ?A = np.argmin(fXs, axis=0) >>> >> ? ? ? ? ?return np.choose(A,fXs) >>> >> ? ? return wedgef >>> >> >>> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >>> >> >>> >> This works no matter what shape of X the user supplies - so a wedged >>> >> function can be somewhat ufunclike - by making A the same shape. >>> >> >>> >> >> The (i0, i1, ..., iN) element of the output array >>> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, which >>> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., >>> >> >> iN] >>> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array >>> >> >> determines which of the choice arrays to pull the corresponding >>> >> >> element from. >>> >> > >>> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, see >>> >> > above. >>> >> > >>> >> >> For example, suppose that you are processing an array C, and have >>> >> >> constructed a selector array A the same shape as C in which a value >>> >> >> is >>> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or >>> >> >> too >>> >> >> big respectively. Then you might do something like: >>> >> >> >>> >> >> C = np.choose(A, [-inf, C, inf]) >>> >> >> >>> >> >> This is something you might want to do no matter what shape A and C >>> >> >> have. It's important not to require that the choices be an array of >>> >> >> choices, because they often have quite different shapes (here, two >>> >> >> are >>> >> >> scalars) and it would be wasteful to broadcast them up to the same >>> >> >> shape as C, just to stack them. >>> >> > >>> >> > OK, that's a pretty generic use-case, thanks; let me see if I >>> >> > understand >>> >> > it >>> >> > correctly: A is some how created independently with a 0 everywhere C >>> >> > is >>> >> > too >>> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; then >>> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf everywhere >>> >> > C >>> >> > is >>> >> > too small, inf everywhere C is too large, and C otherwise (and since >>> >> > -inf >>> >> > and inf are scalars, this implies broadcasting of these is taking >>> >> > place). >>> >> > This is what you're asserting *should* be the behavior.? So, unless >>> >> > there is >>> >> > disagreement about this (you yourself said the opposite viewpoint >>> >> > might >>> >> > rationally be held) np.choose definitely presently has a bug, namely, >>> >> > the >>> >> > index array can't be of arbitrary shape. >>> >> >>> >> There seems to be some disagreement between versions, but both Josef >>> >> and I find that the index array *can* be arbitrary shape. In numpy >>> >> 1.2.1 I find that all the choose items must be the same shape as it, >>> >> which I think is a bug. >>> >> >>> >> What I suggested might be okay was if the index array was not >>> >> broadcasted, so that the outputs always had exactly the same shape as >>> >> the index array. But upon reflection it's useful to be able to use a >>> >> 1-d array to select rows from a set of matrices, so I now think that >>> >> all of A and the elements of choose should be broadcast to the same >>> >> shape. This seems to be what Josef observes in his version of numpy, >>> >> so maybe there's nothing to do. >>> >> >>> >> Anne >>> >> >>> >> > DG >>> >> > >>> >> >> >>> >> >> Anne >>> >> >> _______________________________________________ >>> >> >> NumPy-Discussion mailing list >>> >> >> NumPy-Discussion at scipy.org >>> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > NumPy-Discussion mailing list >>> >> > NumPy-Discussion at scipy.org >>> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > >>> >> > >>> >> _______________________________________________ >>> >> NumPy-Discussion mailing list >>> >> NumPy-Discussion at scipy.org >>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From d.l.goldsmith at gmail.com Sun Nov 8 22:57:01 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 19:57:01 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <1cd32cbb0911081923v4ad2c291t3db1d040417d1016@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <1cd32cbb0911081923v4ad2c291t3db1d040417d1016@mail.gmail.com> Message-ID: <45d1ab480911081957h1c3f0344hc72542f1f43e8cf8@mail.gmail.com> On Sun, Nov 8, 2009 at 7:23 PM, wrote: > On Sun, Nov 8, 2009 at 10:03 PM, David Goldsmith > wrote: > > OK, now I'm trying to wrap my brain around broadcasting in choose when > both > > `a` *and* `choices` need to be (non-trivially) broadcast in order to > arrive > > at a common shape, e.g.: > > > >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) > >>>> a=np.eye(2, dtype=int) # shape is (2,2) > >>>> np.choose(a,c) > > array([[2, 1], > > [0, 3]]) > > > > (Unfortunately, the implementation is in C, so I can't easily insert > print > > statements to see intermediate results.) > > > > First, let me confirm that the above is indeed an example of what I think > it > > is, i.e., both `a` and `choices` are broadcast in order for this to work, > > correct? (And if incorrect, how is one broadcast to the shape of the > > other?) Second, both are broadcast to shape (2,2,2), correct? > > Not, > > You only have one choice array, so first it is interpreted as a > sequence with c[0], c[1], then c[0] and c[0] are broadcasted to each > other (1,2), then they are broadcasted to a (2,2), then a picks from > > 0,0 > 1,1 > > or > 2,2 > 3,3 > > which is > 2,0 > 1,3 > Except that wasn't the result; is yours a typo? qed > > Josef > > >>> np.choose(a,[c[0],c[1]]) > array([[2, 1], > [0, 3]]) > >>> c[0] > array([[0, 1]]) > >>> c[1] > array([[2, 3]]) > >>> c[1].shape > (1, 2) > >>> c[0].shape > (1, 2) > >>> a > array([[1, 0], > [0, 1]]) > > > But how, > > precisely, i.e., does c become > > > > [[[0, 1], [2, 3]], [[[0, 1], [0, 1]], > > [[0, 1], [2, 3]]] or [[2, 3], [2, 3]]] > > > > and same question for a? Then, once a is broadcast to a (2,2,2) shape, > > precisely how does it "pick and choose" from c to create a (2,2) result? > > For example, suppose a is broadcast to: > > > > [[[1, 0], [0, 1]], > > [[1, 0], [0, 1]]] > > > > (as indicated above, I'm uncertain at this point if this is indeed what a > is > > broadcast to); how does this create the (2,2) result obtained above? > > (Obviously this depends in part on precisely how c is broadcast, I do > > recognize that much.) > > > > Finally, a seemingly relevant comment in the C source is: > > > > /* Broadcast all arrays to each other, index array at the end.*/ > > > > This would appear to confirm that "co-broadcasting" is performed if > > necessary, but what does the "index array at the end" phrase mean? > > > > Thanks for your continued patience and tutelage. > > > > DG > > > > On Sun, Nov 8, 2009 at 5:36 AM, wrote: > >> > >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith < > d.l.goldsmith at gmail.com> > >> wrote: > >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald > >> > > >> > wrote: > >> >> > >> >> 2009/11/8 David Goldsmith : > >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> 2009/11/7 David Goldsmith : > >> >> >> > So in essence, at least as it presently functions, the shape of > >> >> >> > 'a' > >> >> >> > *defines* what the individual choices are within 'choices`, and > if > >> >> >> > 'choices' > >> >> >> > can't be parsed into an integer number of such individual > choices, > >> >> >> > that's > >> >> >> > when an exception is raised? > >> >> >> > >> >> >> Um, I don't think so. > >> >> >> > >> >> >> Think of it this way: you provide np.choose with a selector array, > >> >> >> a, > >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You > construct > >> >> >> an > >> >> >> output array, say r, the same shape as a (no matter how many > >> >> >> dimensions it has). > >> >> > > >> >> > Except that I haven't yet seen a working example with 'a' greater > >> >> > than > >> >> > 1-D, > >> >> > Josef's last two examples notwithstanding; or is that what you're > >> >> > saying > >> >> > is > >> >> > the bug. > >> >> > >> >> There's nothing magic about A being one-dimensional. > >> >> > >> >> C = np.random.randn(2,3,5) > >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) > >> >> > >> >> R = np.choose(A, (-1, -C, C, 1)) > >> > > >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and > >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but > >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for > >> > choose's > >> > arguments is that both can be broadcast to a common shape (as you > state > >> > below), but choose won't reshape the arguments for you to make this > >> > possible, you have to do so yourself first, if necessary. That does > >> > appear > >> > to be what's happening now; but do we want choose to be smarter than > >> > that > >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user > >> > doesn't need to include the .reshape((3,1)))? > >> > >> No, I don't think we want to be that smart. > >> > >> If standard broadcasting rules apply, as I think they do, then I > wouldn't > >> want > >> any special newaxis or reshapes done automatically. It will be > confusing, > >> the function wouldn't know what to do if there are, e.g., as many rows > as > >> columns, and this looks like a big source of errors. > >> Standard broadcasting is pretty nice (once I got the hang of it), and > >> adding > >> a lot of np.newaxis (or some reshapes) to the code is only a small price > >> to pay. > >> > >> Josef > >> > >> > >> > >> > > >> > DG > >> > > >> >> > >> >> Requv = np.minimum(np.abs(C),1) > >> >> > >> >> or: > >> >> > >> >> def wedge(*functions): > >> >> """Return a function whose value is the minimum of those of > >> >> functions""" > >> >> def wedgef(X): > >> >> fXs = [f(X) for f in functions] > >> >> A = np.argmin(fXs, axis=0) > >> >> return np.choose(A,fXs) > >> >> return wedgef > >> >> > >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) > >> >> > >> >> This works no matter what shape of X the user supplies - so a wedged > >> >> function can be somewhat ufunclike - by making A the same shape. > >> >> > >> >> >> The (i0, i1, ..., iN) element of the output array > >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, > which > >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., > >> >> >> iN] > >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array > >> >> >> determines which of the choice arrays to pull the corresponding > >> >> >> element from. > >> >> > > >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, > see > >> >> > above. > >> >> > > >> >> >> For example, suppose that you are processing an array C, and have > >> >> >> constructed a selector array A the same shape as C in which a > value > >> >> >> is > >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or > >> >> >> too > >> >> >> big respectively. Then you might do something like: > >> >> >> > >> >> >> C = np.choose(A, [-inf, C, inf]) > >> >> >> > >> >> >> This is something you might want to do no matter what shape A and > C > >> >> >> have. It's important not to require that the choices be an array > of > >> >> >> choices, because they often have quite different shapes (here, two > >> >> >> are > >> >> >> scalars) and it would be wasteful to broadcast them up to the same > >> >> >> shape as C, just to stack them. > >> >> > > >> >> > OK, that's a pretty generic use-case, thanks; let me see if I > >> >> > understand > >> >> > it > >> >> > correctly: A is some how created independently with a 0 everywhere > C > >> >> > is > >> >> > too > >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; > then > >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf > everywhere > >> >> > C > >> >> > is > >> >> > too small, inf everywhere C is too large, and C otherwise (and > since > >> >> > -inf > >> >> > and inf are scalars, this implies broadcasting of these is taking > >> >> > place). > >> >> > This is what you're asserting *should* be the behavior. So, unless > >> >> > there is > >> >> > disagreement about this (you yourself said the opposite viewpoint > >> >> > might > >> >> > rationally be held) np.choose definitely presently has a bug, > namely, > >> >> > the > >> >> > index array can't be of arbitrary shape. > >> >> > >> >> There seems to be some disagreement between versions, but both Josef > >> >> and I find that the index array *can* be arbitrary shape. In numpy > >> >> 1.2.1 I find that all the choose items must be the same shape as it, > >> >> which I think is a bug. > >> >> > >> >> What I suggested might be okay was if the index array was not > >> >> broadcasted, so that the outputs always had exactly the same shape as > >> >> the index array. But upon reflection it's useful to be able to use a > >> >> 1-d array to select rows from a set of matrices, so I now think that > >> >> all of A and the elements of choose should be broadcast to the same > >> >> shape. This seems to be what Josef observes in his version of numpy, > >> >> so maybe there's nothing to do. > >> >> > >> >> Anne > >> >> > >> >> > DG > >> >> > > >> >> >> > >> >> >> Anne > >> >> >> _______________________________________________ > >> >> >> NumPy-Discussion mailing list > >> >> >> NumPy-Discussion at scipy.org > >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > NumPy-Discussion mailing list > >> >> > NumPy-Discussion at scipy.org > >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> >> > > >> >> > > >> >> _______________________________________________ > >> >> NumPy-Discussion mailing list > >> >> NumPy-Discussion at scipy.org > >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Sun Nov 8 23:08:39 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 20:08:39 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> Message-ID: <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald wrote: > As Josef said, this is not correct. I think the key point of confusion is > this: > > Do not pass choose two arrays. > > Pass it one array and a *list* of arrays. The fact that choices can be > an array is a quirk we can't change, but you should think of the > second argument as a list of arrays, Fine, but as you say, one *can* pass choose an array as the second argument and it doesn't raise an exception, so if someone is stupid/careless enough to pass an array for `choices`, how is choose interpreting it as a list? Is the first dimension "list converted" (so that, e.g., my (2,1,2) example is interpreted as a two element list, each of whose elements is a (1,2) array)? > possibly of different shapes. > > np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) > >>> np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) Traceback (most recent call last): File "", line 1, in File "C:\Python254\lib\site-packages\numpy\core\fromnumeric.py", line 208, in choose return choose(choices, out=out, mode=mode) TypeError: array cannot be safely cast to required type Here the first argument is an array of choices. The second argument is > a *list* - if you cast it to an array you'll get an error or nonsense > So the result of my example was nonsense? > - of arrays to choose from. The broadcasting ensures the first > argument and *each element of the list* are the same shape. The only > constraint on the number of arrays in the list is that it be larger > than the largest value in a. > > If you try to make the second argument into a single array, for one > thing you are throwing away useful generality by forcing each choice > to be the same shape (and real shape, not zero-strided fake shape), > and for another the broadcasting becomes very hard to understand. > Obviously, I find the broadcasting very hard to understand regardless. :-( DG > > Anne > > > 2009/11/8 David Goldsmith : > > OK, now I'm trying to wrap my brain around broadcasting in choose when > both > > `a` *and* `choices` need to be (non-trivially) broadcast in order to > arrive > > at a common shape, e.g.: > > > >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) > >>>> a=np.eye(2, dtype=int) # shape is (2,2) > >>>> np.choose(a,c) > > array([[2, 1], > > [0, 3]]) > > > > (Unfortunately, the implementation is in C, so I can't easily insert > print > > statements to see intermediate results.) > > > > First, let me confirm that the above is indeed an example of what I think > it > > is, i.e., both `a` and `choices` are broadcast in order for this to work, > > correct? (And if incorrect, how is one broadcast to the shape of the > > other?) Second, both are broadcast to shape (2,2,2), correct? But how, > > precisely, i.e., does c become > > > > [[[0, 1], [2, 3]], [[[0, 1], [0, 1]], > > [[0, 1], [2, 3]]] or [[2, 3], [2, 3]]] > > > > and same question for a? Then, once a is broadcast to a (2,2,2) shape, > > precisely how does it "pick and choose" from c to create a (2,2) result? > > For example, suppose a is broadcast to: > > > > [[[1, 0], [0, 1]], > > [[1, 0], [0, 1]]] > > > > (as indicated above, I'm uncertain at this point if this is indeed what a > is > > broadcast to); how does this create the (2,2) result obtained above? > > (Obviously this depends in part on precisely how c is broadcast, I do > > recognize that much.) > > > > Finally, a seemingly relevant comment in the C source is: > > > > /* Broadcast all arrays to each other, index array at the end.*/ > > > > This would appear to confirm that "co-broadcasting" is performed if > > necessary, but what does the "index array at the end" phrase mean? > > > > Thanks for your continued patience and tutelage. > > > > DG > > > > On Sun, Nov 8, 2009 at 5:36 AM, wrote: > >> > >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith < > d.l.goldsmith at gmail.com> > >> wrote: > >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald > >> > > >> > wrote: > >> >> > >> >> 2009/11/8 David Goldsmith : > >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> 2009/11/7 David Goldsmith : > >> >> >> > So in essence, at least as it presently functions, the shape of > >> >> >> > 'a' > >> >> >> > *defines* what the individual choices are within 'choices`, and > if > >> >> >> > 'choices' > >> >> >> > can't be parsed into an integer number of such individual > choices, > >> >> >> > that's > >> >> >> > when an exception is raised? > >> >> >> > >> >> >> Um, I don't think so. > >> >> >> > >> >> >> Think of it this way: you provide np.choose with a selector array, > >> >> >> a, > >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You > construct > >> >> >> an > >> >> >> output array, say r, the same shape as a (no matter how many > >> >> >> dimensions it has). > >> >> > > >> >> > Except that I haven't yet seen a working example with 'a' greater > >> >> > than > >> >> > 1-D, > >> >> > Josef's last two examples notwithstanding; or is that what you're > >> >> > saying > >> >> > is > >> >> > the bug. > >> >> > >> >> There's nothing magic about A being one-dimensional. > >> >> > >> >> C = np.random.randn(2,3,5) > >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) > >> >> > >> >> R = np.choose(A, (-1, -C, C, 1)) > >> > > >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and > >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but > >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for > >> > choose's > >> > arguments is that both can be broadcast to a common shape (as you > state > >> > below), but choose won't reshape the arguments for you to make this > >> > possible, you have to do so yourself first, if necessary. That does > >> > appear > >> > to be what's happening now; but do we want choose to be smarter than > >> > that > >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the user > >> > doesn't need to include the .reshape((3,1)))? > >> > >> No, I don't think we want to be that smart. > >> > >> If standard broadcasting rules apply, as I think they do, then I > wouldn't > >> want > >> any special newaxis or reshapes done automatically. It will be > confusing, > >> the function wouldn't know what to do if there are, e.g., as many rows > as > >> columns, and this looks like a big source of errors. > >> Standard broadcasting is pretty nice (once I got the hang of it), and > >> adding > >> a lot of np.newaxis (or some reshapes) to the code is only a small price > >> to pay. > >> > >> Josef > >> > >> > >> > >> > > >> > DG > >> > > >> >> > >> >> Requv = np.minimum(np.abs(C),1) > >> >> > >> >> or: > >> >> > >> >> def wedge(*functions): > >> >> """Return a function whose value is the minimum of those of > >> >> functions""" > >> >> def wedgef(X): > >> >> fXs = [f(X) for f in functions] > >> >> A = np.argmin(fXs, axis=0) > >> >> return np.choose(A,fXs) > >> >> return wedgef > >> >> > >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) > >> >> > >> >> This works no matter what shape of X the user supplies - so a wedged > >> >> function can be somewhat ufunclike - by making A the same shape. > >> >> > >> >> >> The (i0, i1, ..., iN) element of the output array > >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, > which > >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., > >> >> >> iN] > >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector array > >> >> >> determines which of the choice arrays to pull the corresponding > >> >> >> element from. > >> >> > > >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, > see > >> >> > above. > >> >> > > >> >> >> For example, suppose that you are processing an array C, and have > >> >> >> constructed a selector array A the same shape as C in which a > value > >> >> >> is > >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, or > >> >> >> too > >> >> >> big respectively. Then you might do something like: > >> >> >> > >> >> >> C = np.choose(A, [-inf, C, inf]) > >> >> >> > >> >> >> This is something you might want to do no matter what shape A and > C > >> >> >> have. It's important not to require that the choices be an array > of > >> >> >> choices, because they often have quite different shapes (here, two > >> >> >> are > >> >> >> scalars) and it would be wasteful to broadcast them up to the same > >> >> >> shape as C, just to stack them. > >> >> > > >> >> > OK, that's a pretty generic use-case, thanks; let me see if I > >> >> > understand > >> >> > it > >> >> > correctly: A is some how created independently with a 0 everywhere > C > >> >> > is > >> >> > too > >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; > then > >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf > everywhere > >> >> > C > >> >> > is > >> >> > too small, inf everywhere C is too large, and C otherwise (and > since > >> >> > -inf > >> >> > and inf are scalars, this implies broadcasting of these is taking > >> >> > place). > >> >> > This is what you're asserting *should* be the behavior. So, unless > >> >> > there is > >> >> > disagreement about this (you yourself said the opposite viewpoint > >> >> > might > >> >> > rationally be held) np.choose definitely presently has a bug, > namely, > >> >> > the > >> >> > index array can't be of arbitrary shape. > >> >> > >> >> There seems to be some disagreement between versions, but both Josef > >> >> and I find that the index array *can* be arbitrary shape. In numpy > >> >> 1.2.1 I find that all the choose items must be the same shape as it, > >> >> which I think is a bug. > >> >> > >> >> What I suggested might be okay was if the index array was not > >> >> broadcasted, so that the outputs always had exactly the same shape as > >> >> the index array. But upon reflection it's useful to be able to use a > >> >> 1-d array to select rows from a set of matrices, so I now think that > >> >> all of A and the elements of choose should be broadcast to the same > >> >> shape. This seems to be what Josef observes in his version of numpy, > >> >> so maybe there's nothing to do. > >> >> > >> >> Anne > >> >> > >> >> > DG > >> >> > > >> >> >> > >> >> >> Anne > >> >> >> _______________________________________________ > >> >> >> NumPy-Discussion mailing list > >> >> >> NumPy-Discussion at scipy.org > >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > NumPy-Discussion mailing list > >> >> > NumPy-Discussion at scipy.org > >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> >> > > >> >> > > >> >> _______________________________________________ > >> >> NumPy-Discussion mailing list > >> >> NumPy-Discussion at scipy.org > >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Sun Nov 8 23:12:33 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 20:12:33 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> Message-ID: <45d1ab480911082012u64b63fb0r9559076b7a0eab09@mail.gmail.com> On Sun, Nov 8, 2009 at 8:08 PM, David Goldsmith wrote: > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald wrote: > >> As Josef said, this is not correct. I think the key point of confusion is >> this: >> >> Do not pass choose two arrays. >> >> Pass it one array and a *list* of arrays. The fact that choices can be >> an array is a quirk we can't change, but you should think of the >> second argument as a list of arrays, > > > Fine, but as you say, one *can* pass choose an array as the second argument > and it doesn't raise an exception, so if someone is stupid/careless enough > to pass an array for `choices`, how is choose interpreting it as a list? Is > the first dimension "list converted" (so that, e.g., my (2,1,2) example is > interpreted as a two element list, each of whose elements is a (1,2) array)? > > >> possibly of different shapes. >> >> np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) >> > > >>> np.choose(np.ones((2,1,1)), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) > Traceback (most recent call last): > File "", line 1, in > File "C:\Python254\lib\site-packages\numpy\core\fromnumeric.py", line > 208, in > choose > return choose(choices, out=out, mode=mode) > TypeError: array cannot be safely cast to required type > Never mind, I recognized the problem from my experimenting: a needs to be integer valued, so you need, e.g., np.choose(np.ones((2,1,1),dtype=int), [ np.ones((1,3,1)), np.ones((1,1,5)) ] ) > > Here the first argument is an array of choices. The second argument is >> a *list* - if you cast it to an array you'll get an error or nonsense >> > > So the result of my example was nonsense? > > >> - of arrays to choose from. The broadcasting ensures the first >> argument and *each element of the list* are the same shape. The only >> constraint on the number of arrays in the list is that it be larger >> than the largest value in a. >> >> If you try to make the second argument into a single array, for one >> thing you are throwing away useful generality by forcing each choice >> to be the same shape (and real shape, not zero-strided fake shape), >> and for another the broadcasting becomes very hard to understand. >> > > Obviously, I find the broadcasting very hard to understand regardless. :-( > > DG > > >> >> Anne >> >> >> 2009/11/8 David Goldsmith : >> > OK, now I'm trying to wrap my brain around broadcasting in choose when >> both >> > `a` *and* `choices` need to be (non-trivially) broadcast in order to >> arrive >> > at a common shape, e.g.: >> > >> >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >> >>>> a=np.eye(2, dtype=int) # shape is (2,2) >> >>>> np.choose(a,c) >> > array([[2, 1], >> > [0, 3]]) >> > >> > (Unfortunately, the implementation is in C, so I can't easily insert >> print >> > statements to see intermediate results.) >> > >> > First, let me confirm that the above is indeed an example of what I >> think it >> > is, i.e., both `a` and `choices` are broadcast in order for this to >> work, >> > correct? (And if incorrect, how is one broadcast to the shape of the >> > other?) Second, both are broadcast to shape (2,2,2), correct? But how, >> > precisely, i.e., does c become >> > >> > [[[0, 1], [2, 3]], [[[0, 1], [0, 1]], >> > [[0, 1], [2, 3]]] or [[2, 3], [2, 3]]] >> > >> > and same question for a? Then, once a is broadcast to a (2,2,2) shape, >> > precisely how does it "pick and choose" from c to create a (2,2) result? >> > For example, suppose a is broadcast to: >> > >> > [[[1, 0], [0, 1]], >> > [[1, 0], [0, 1]]] >> > >> > (as indicated above, I'm uncertain at this point if this is indeed what >> a is >> > broadcast to); how does this create the (2,2) result obtained above? >> > (Obviously this depends in part on precisely how c is broadcast, I do >> > recognize that much.) >> > >> > Finally, a seemingly relevant comment in the C source is: >> > >> > /* Broadcast all arrays to each other, index array at the end.*/ >> > >> > This would appear to confirm that "co-broadcasting" is performed if >> > necessary, but what does the "index array at the end" phrase mean? >> > >> > Thanks for your continued patience and tutelage. >> > >> > DG >> > >> > On Sun, Nov 8, 2009 at 5:36 AM, wrote: >> >> >> >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith < >> d.l.goldsmith at gmail.com> >> >> wrote: >> >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald >> >> > >> >> > wrote: >> >> >> >> >> >> 2009/11/8 David Goldsmith : >> >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> >> >> 2009/11/7 David Goldsmith : >> >> >> >> > So in essence, at least as it presently functions, the shape of >> >> >> >> > 'a' >> >> >> >> > *defines* what the individual choices are within 'choices`, and >> if >> >> >> >> > 'choices' >> >> >> >> > can't be parsed into an integer number of such individual >> choices, >> >> >> >> > that's >> >> >> >> > when an exception is raised? >> >> >> >> >> >> >> >> Um, I don't think so. >> >> >> >> >> >> >> >> Think of it this way: you provide np.choose with a selector >> array, >> >> >> >> a, >> >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You >> construct >> >> >> >> an >> >> >> >> output array, say r, the same shape as a (no matter how many >> >> >> >> dimensions it has). >> >> >> > >> >> >> > Except that I haven't yet seen a working example with 'a' greater >> >> >> > than >> >> >> > 1-D, >> >> >> > Josef's last two examples notwithstanding; or is that what you're >> >> >> > saying >> >> >> > is >> >> >> > the bug. >> >> >> >> >> >> There's nothing magic about A being one-dimensional. >> >> >> >> >> >> C = np.random.randn(2,3,5) >> >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >> >> >> >> >> >> R = np.choose(A, (-1, -C, C, 1)) >> >> > >> >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and >> >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but >> >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for >> >> > choose's >> >> > arguments is that both can be broadcast to a common shape (as you >> state >> >> > below), but choose won't reshape the arguments for you to make this >> >> > possible, you have to do so yourself first, if necessary. That does >> >> > appear >> >> > to be what's happening now; but do we want choose to be smarter than >> >> > that >> >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the >> user >> >> > doesn't need to include the .reshape((3,1)))? >> >> >> >> No, I don't think we want to be that smart. >> >> >> >> If standard broadcasting rules apply, as I think they do, then I >> wouldn't >> >> want >> >> any special newaxis or reshapes done automatically. It will be >> confusing, >> >> the function wouldn't know what to do if there are, e.g., as many rows >> as >> >> columns, and this looks like a big source of errors. >> >> Standard broadcasting is pretty nice (once I got the hang of it), and >> >> adding >> >> a lot of np.newaxis (or some reshapes) to the code is only a small >> price >> >> to pay. >> >> >> >> Josef >> >> >> >> >> >> >> >> > >> >> > DG >> >> > >> >> >> >> >> >> Requv = np.minimum(np.abs(C),1) >> >> >> >> >> >> or: >> >> >> >> >> >> def wedge(*functions): >> >> >> """Return a function whose value is the minimum of those of >> >> >> functions""" >> >> >> def wedgef(X): >> >> >> fXs = [f(X) for f in functions] >> >> >> A = np.argmin(fXs, axis=0) >> >> >> return np.choose(A,fXs) >> >> >> return wedgef >> >> >> >> >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >> >> >> >> >> >> This works no matter what shape of X the user supplies - so a wedged >> >> >> function can be somewhat ufunclike - by making A the same shape. >> >> >> >> >> >> >> The (i0, i1, ..., iN) element of the output array >> >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, >> which >> >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., >> >> >> >> iN] >> >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector >> array >> >> >> >> determines which of the choice arrays to pull the corresponding >> >> >> >> element from. >> >> >> > >> >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, >> see >> >> >> > above. >> >> >> > >> >> >> >> For example, suppose that you are processing an array C, and have >> >> >> >> constructed a selector array A the same shape as C in which a >> value >> >> >> >> is >> >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, >> or >> >> >> >> too >> >> >> >> big respectively. Then you might do something like: >> >> >> >> >> >> >> >> C = np.choose(A, [-inf, C, inf]) >> >> >> >> >> >> >> >> This is something you might want to do no matter what shape A and >> C >> >> >> >> have. It's important not to require that the choices be an array >> of >> >> >> >> choices, because they often have quite different shapes (here, >> two >> >> >> >> are >> >> >> >> scalars) and it would be wasteful to broadcast them up to the >> same >> >> >> >> shape as C, just to stack them. >> >> >> > >> >> >> > OK, that's a pretty generic use-case, thanks; let me see if I >> >> >> > understand >> >> >> > it >> >> >> > correctly: A is some how created independently with a 0 everywhere >> C >> >> >> > is >> >> >> > too >> >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; >> then >> >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf >> everywhere >> >> >> > C >> >> >> > is >> >> >> > too small, inf everywhere C is too large, and C otherwise (and >> since >> >> >> > -inf >> >> >> > and inf are scalars, this implies broadcasting of these is taking >> >> >> > place). >> >> >> > This is what you're asserting *should* be the behavior. So, >> unless >> >> >> > there is >> >> >> > disagreement about this (you yourself said the opposite viewpoint >> >> >> > might >> >> >> > rationally be held) np.choose definitely presently has a bug, >> namely, >> >> >> > the >> >> >> > index array can't be of arbitrary shape. >> >> >> >> >> >> There seems to be some disagreement between versions, but both Josef >> >> >> and I find that the index array *can* be arbitrary shape. In numpy >> >> >> 1.2.1 I find that all the choose items must be the same shape as it, >> >> >> which I think is a bug. >> >> >> >> >> >> What I suggested might be okay was if the index array was not >> >> >> broadcasted, so that the outputs always had exactly the same shape >> as >> >> >> the index array. But upon reflection it's useful to be able to use a >> >> >> 1-d array to select rows from a set of matrices, so I now think that >> >> >> all of A and the elements of choose should be broadcast to the same >> >> >> shape. This seems to be what Josef observes in his version of numpy, >> >> >> so maybe there's nothing to do. >> >> >> >> >> >> Anne >> >> >> >> >> >> > DG >> >> >> > >> >> >> >> >> >> >> >> Anne >> >> >> >> _______________________________________________ >> >> >> >> NumPy-Discussion mailing list >> >> >> >> NumPy-Discussion at scipy.org >> >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > >> >> >> > >> >> >> > _______________________________________________ >> >> >> > NumPy-Discussion mailing list >> >> >> > NumPy-Discussion at scipy.org >> >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > >> >> >> > >> >> >> _______________________________________________ >> >> >> NumPy-Discussion mailing list >> >> >> NumPy-Discussion at scipy.org >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Nov 8 23:19:43 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 8 Nov 2009 23:19:43 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911081957h1c3f0344hc72542f1f43e8cf8@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <1cd32cbb0911081923v4ad2c291t3db1d040417d1016@mail.gmail.com> <45d1ab480911081957h1c3f0344hc72542f1f43e8cf8@mail.gmail.com> Message-ID: <1cd32cbb0911082019l7c8817efwe7a91cb5c79056ad@mail.gmail.com> On Sun, Nov 8, 2009 at 10:57 PM, David Goldsmith wrote: > On Sun, Nov 8, 2009 at 7:23 PM, wrote: >> >> On Sun, Nov 8, 2009 at 10:03 PM, David Goldsmith >> wrote: >> > OK, now I'm trying to wrap my brain around broadcasting in choose when >> > both >> > `a` *and* `choices` need to be (non-trivially) broadcast in order to >> > arrive >> > at a common shape, e.g.: >> > >> >>>> c=np.arange(4).reshape((2,1,2)) # shape is (2,1,2) >> >>>> a=np.eye(2, dtype=int) # shape is (2,2) >> >>>> np.choose(a,c) >> > array([[2, 1], >> > ?????? [0, 3]]) >> > >> > (Unfortunately, the implementation is in C, so I can't easily insert >> > print >> > statements to see intermediate results.) >> > >> > First, let me confirm that the above is indeed an example of what I >> > think it >> > is, i.e., both `a` and `choices` are broadcast in order for this to >> > work, >> > correct?? (And if incorrect, how is one broadcast to the shape of the >> > other?)? Second, both are broadcast to shape (2,2,2), correct? >> >> Not, >> >> You only have one choice array, so first it is interpreted as a >> sequence with c[0], c[1], then c[0] and c[0] are broadcasted to each >> other (1,2), then they are broadcasted to a (2,2), then a picks from >> >> 0,0 >> 1,1 >> >> or >> 2,2 >> 3,3 >> >> which is >> 2,0 >> 1,3 > > Except that wasn't the result; is yours a typo? No typo, my mistake, I mixed up rows and columns. I thought c[0] and c[1] are column vectors when I retyped the answer from the interpreter. Sorry, Josef > >> qed >> >> Josef >> >> >>> np.choose(a,[c[0],c[1]]) >> array([[2, 1], >> ? ? ? [0, 3]]) >> >>> c[0] >> array([[0, 1]]) >> >>> c[1] >> array([[2, 3]]) >> >>> c[1].shape >> (1, 2) >> >>> c[0].shape >> (1, 2) >> >>> a >> array([[1, 0], >> ? ? ? [0, 1]]) >> >> >> But how, >> > precisely, i.e., does c become >> > >> > [[[0, 1], [2, 3]],??????? [[[0, 1], [0, 1]], >> > ?[[0, 1], [2, 3]]]?? or?? [[2, 3], [2, 3]]] >> > >> > and same question for a?? Then, once a is broadcast to a (2,2,2) shape, >> > precisely how does it "pick and choose" from c to create a (2,2) result? >> > For example, suppose a is broadcast to: >> > >> > [[[1, 0], [0, 1]], >> > ?[[1, 0], [0, 1]]] >> > >> > (as indicated above, I'm uncertain at this point if this is indeed what >> > a is >> > broadcast to); how does this create the (2,2) result obtained above? >> > (Obviously this depends in part on precisely how c is broadcast, I do >> > recognize that much.) >> > >> > Finally, a seemingly relevant comment in the C source is: >> > >> > /* Broadcast all arrays to each other, index array at the end.*/ >> > >> > This would appear to confirm that "co-broadcasting" is performed if >> > necessary, but what does the "index array at the end" phrase mean? >> > >> > Thanks for your continued patience and tutelage. >> > >> > DG >> > >> > On Sun, Nov 8, 2009 at 5:36 AM, wrote: >> >> >> >> On Sun, Nov 8, 2009 at 5:00 AM, David Goldsmith >> >> >> >> wrote: >> >> > On Sun, Nov 8, 2009 at 12:57 AM, Anne Archibald >> >> > >> >> > wrote: >> >> >> >> >> >> 2009/11/8 David Goldsmith : >> >> >> > On Sat, Nov 7, 2009 at 11:59 PM, Anne Archibald >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> >> >> 2009/11/7 David Goldsmith : >> >> >> >> > So in essence, at least as it presently functions, the shape of >> >> >> >> > 'a' >> >> >> >> > *defines* what the individual choices are within 'choices`, and >> >> >> >> > if >> >> >> >> > 'choices' >> >> >> >> > can't be parsed into an integer number of such individual >> >> >> >> > choices, >> >> >> >> > that's >> >> >> >> > when an exception is raised? >> >> >> >> >> >> >> >> Um, I don't think so. >> >> >> >> >> >> >> >> Think of it this way: you provide np.choose with a selector >> >> >> >> array, >> >> >> >> a, >> >> >> >> and a list (not array!) [c0, c1, ..., cM] of choices. You >> >> >> >> construct >> >> >> >> an >> >> >> >> output array, say r, the same shape as a (no matter how many >> >> >> >> dimensions it has). >> >> >> > >> >> >> > Except that I haven't yet seen a working example with 'a' greater >> >> >> > than >> >> >> > 1-D, >> >> >> > Josef's last two examples notwithstanding; or is that what you're >> >> >> > saying >> >> >> > is >> >> >> > the bug. >> >> >> >> >> >> There's nothing magic about A being one-dimensional. >> >> >> >> >> >> C = np.random.randn(2,3,5) >> >> >> A = (C>-1).astype(int) + (C>0).astype(int) + (C>1).astype(int) >> >> >> >> >> >> R = np.choose(A, (-1, -C, C, 1)) >> >> > >> >> > OK, now I get it: np.choose(A[0,:,:], (-1,-C,C,-1)) and >> >> > np.choose(A[0,:,0].reshape((3,1)), (-1,-C,C,1)), e.g., also work, but >> >> > np.choose(A[0,:,0], (-1,-C,C,-1)) doesn't - what's necessary for >> >> > choose's >> >> > arguments is that both can be broadcast to a common shape (as you >> >> > state >> >> > below), but choose won't reshape the arguments for you to make this >> >> > possible, you have to do so yourself first, if necessary.? That does >> >> > appear >> >> > to be what's happening now; but do we want choose to be smarter than >> >> > that >> >> > (e.g., for np.choose(A[0,:,0], (-1,-C,C,-1)) to work, so that the >> >> > user >> >> > doesn't need to include the .reshape((3,1)))? >> >> >> >> No, I don't think we want to be that smart. >> >> >> >> If standard broadcasting rules apply, as I think they do, then I >> >> wouldn't >> >> want >> >> any special newaxis or reshapes done automatically. It will be >> >> confusing, >> >> the function wouldn't know what to do if there are, e.g., as many rows >> >> as >> >> columns, and this looks like a big source of errors. >> >> Standard broadcasting is pretty nice (once I got the hang of it), and >> >> adding >> >> a lot of np.newaxis (or some reshapes) to the code is only a small >> >> price >> >> to pay. >> >> >> >> Josef >> >> >> >> >> >> >> >> > >> >> > DG >> >> > >> >> >> >> >> >> Requv = np.minimum(np.abs(C),1) >> >> >> >> >> >> or: >> >> >> >> >> >> def wedge(*functions): >> >> >> ? ? """Return a function whose value is the minimum of those of >> >> >> functions""" >> >> >> ? ? def wedgef(X): >> >> >> ? ? ? ? ?fXs = [f(X) for f in functions] >> >> >> ? ? ? ? ?A = np.argmin(fXs, axis=0) >> >> >> ? ? ? ? ?return np.choose(A,fXs) >> >> >> ? ? return wedgef >> >> >> >> >> >> so e.g. np.abs is -wedge(lambda X: X, lambda X: -X) >> >> >> >> >> >> This works no matter what shape of X the user supplies - so a wedged >> >> >> function can be somewhat ufunclike - by making A the same shape. >> >> >> >> >> >> >> The (i0, i1, ..., iN) element of the output array >> >> >> >> is obtained by looking at the (i0, i1, ..., iN) element of a, >> >> >> >> which >> >> >> >> should be an integer no larger than M; say j. Then r[i0, i1, ..., >> >> >> >> iN] >> >> >> >> = cj[i0, i1, ..., iN]. That is, each element of the selector >> >> >> >> array >> >> >> >> determines which of the choice arrays to pull the corresponding >> >> >> >> element from. >> >> >> > >> >> >> > That's pretty clear (thanks for doing my work for me). ;-), Yet, >> >> >> > see >> >> >> > above. >> >> >> > >> >> >> >> For example, suppose that you are processing an array C, and have >> >> >> >> constructed a selector array A the same shape as C in which a >> >> >> >> value >> >> >> >> is >> >> >> >> 0, 1, or 2 depending on whether the C value is too small, okay, >> >> >> >> or >> >> >> >> too >> >> >> >> big respectively. Then you might do something like: >> >> >> >> >> >> >> >> C = np.choose(A, [-inf, C, inf]) >> >> >> >> >> >> >> >> This is something you might want to do no matter what shape A and >> >> >> >> C >> >> >> >> have. It's important not to require that the choices be an array >> >> >> >> of >> >> >> >> choices, because they often have quite different shapes (here, >> >> >> >> two >> >> >> >> are >> >> >> >> scalars) and it would be wasteful to broadcast them up to the >> >> >> >> same >> >> >> >> shape as C, just to stack them. >> >> >> > >> >> >> > OK, that's a pretty generic use-case, thanks; let me see if I >> >> >> > understand >> >> >> > it >> >> >> > correctly: A is some how created independently with a 0 everywhere >> >> >> > C >> >> >> > is >> >> >> > too >> >> >> > small, a 1 everywhere C is OK, and a 2 everywhere C is too big; >> >> >> > then >> >> >> > np.choose(A, [-inf, C, inf]) creates an array that is -inf >> >> >> > everywhere >> >> >> > C >> >> >> > is >> >> >> > too small, inf everywhere C is too large, and C otherwise (and >> >> >> > since >> >> >> > -inf >> >> >> > and inf are scalars, this implies broadcasting of these is taking >> >> >> > place). >> >> >> > This is what you're asserting *should* be the behavior.? So, >> >> >> > unless >> >> >> > there is >> >> >> > disagreement about this (you yourself said the opposite viewpoint >> >> >> > might >> >> >> > rationally be held) np.choose definitely presently has a bug, >> >> >> > namely, >> >> >> > the >> >> >> > index array can't be of arbitrary shape. >> >> >> >> >> >> There seems to be some disagreement between versions, but both Josef >> >> >> and I find that the index array *can* be arbitrary shape. In numpy >> >> >> 1.2.1 I find that all the choose items must be the same shape as it, >> >> >> which I think is a bug. >> >> >> >> >> >> What I suggested might be okay was if the index array was not >> >> >> broadcasted, so that the outputs always had exactly the same shape >> >> >> as >> >> >> the index array. But upon reflection it's useful to be able to use a >> >> >> 1-d array to select rows from a set of matrices, so I now think that >> >> >> all of A and the elements of choose should be broadcast to the same >> >> >> shape. This seems to be what Josef observes in his version of numpy, >> >> >> so maybe there's nothing to do. >> >> >> >> >> >> Anne >> >> >> >> >> >> > DG >> >> >> > >> >> >> >> >> >> >> >> Anne >> >> >> >> _______________________________________________ >> >> >> >> NumPy-Discussion mailing list >> >> >> >> NumPy-Discussion at scipy.org >> >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > >> >> >> > >> >> >> > _______________________________________________ >> >> >> > NumPy-Discussion mailing list >> >> >> > NumPy-Discussion at scipy.org >> >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > >> >> >> > >> >> >> _______________________________________________ >> >> >> NumPy-Discussion mailing list >> >> >> NumPy-Discussion at scipy.org >> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> > _______________________________________________ >> >> > NumPy-Discussion mailing list >> >> > NumPy-Discussion at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > >> >> > >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From peridot.faceted at gmail.com Sun Nov 8 23:19:46 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 8 Nov 2009 23:19:46 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911071653o4cd1c832wb8061d5f94fdd4b4@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> Message-ID: 2009/11/8 David Goldsmith : > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald > wrote: >> >> As Josef said, this is not correct. I think the key point of confusion is >> this: >> >> Do not pass choose two arrays. >> >> Pass it one array and a *list* of arrays. The fact that choices can be >> an array is a quirk we can't change, but you should think of the >> second argument as a list of arrays, > > Fine, but as you say, one *can* pass choose an array as the second argument > and it doesn't raise an exception, so if someone is stupid/careless enough > to pass an array for `choices`, how is choose interpreting it as a list?? Is > the first dimension "list converted" (so that, e.g., my (2,1,2) example is > interpreted as a two element list, each of whose elements is a (1,2) array)? It seems to me that this is the only reasonable interpretation, yes. After all, arrays behave like sequences along the first axis, whose elements are arrays of one less dimension. Thus if you pass an array, any broadcasting happens ignoring the first axis, which is a rather abnormal pattern for numpy broadcasting, but necessary here. As a bonus, I think this is what is implemented in current versions of numpy. (In 1.2.1 it raises an exception if broadcasting is necessary.) Anne From cournape at gmail.com Sun Nov 8 23:43:51 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 9 Nov 2009 13:43:51 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> Message-ID: <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> On Mon, Nov 9, 2009 at 12:49 PM, Charles R Harris wrote: >> > > Umm, PPC doesn't have ieee quad, it is an amalgam of two doubles. Good catch ! I added a new type define for this case, with the corresponding union. I have quickly tested it under rosetta, and it seems that npy_nextafterl gives a reasonable answer for the few values I have tried (i.e. the same as for a double). David From d.l.goldsmith at gmail.com Sun Nov 8 23:50:27 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 8 Nov 2009 20:50:27 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> Message-ID: <45d1ab480911082050r7d18f3c0g6ce12a7c39aba34@mail.gmail.com> OK, let me see if I'm interpreting this example correctly: >>> c1=np.arange(2).reshape(2,1,1); c1 array([[[0]], [[1]]]) >>> c2=2+np.arange(2).reshape(1,1,2); c2 array([[[2, 3]]]) >>> a=np.eye(2,dtype=int) >>> np.choose(a, [c1, c2]) array([[[2, 0], [0, 3]], [[2, 1], [1, 3]]]) First, everything is being broadcast to (2,2,2); a is broadcast to [[[1,0], [0,1]], [[1,0], [0,1]]], c1 is broadcast to [[[0,0], [0,0]], [[1,1], [1,1]]] and c2 is broadcast to [[[2,3], [2,3]], [[2,3], [2,3]]]. Now result is created by "stepping through" broadcast a and using, respectively, the positionally corresponding element from broadcast c1 (resp. c2) if the value in a at the position is 0 (resp. 1). At least, this gives the result above (but I have not examined other possible broadcasts of the arguments to see if they would also give the result - I conjectured what appeared to me to be the most "natural" broadcasts and checked to see if it worked and it does; is there something I should know - e.g., uniqueness of the result, or a rule governing how choose broadcasts - to *know* that the broadcasts above are indeed the broadcasts choose is using?) Thanks again, DG On Sun, Nov 8, 2009 at 8:19 PM, Anne Archibald wrote: > 2009/11/8 David Goldsmith : > > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> As Josef said, this is not correct. I think the key point of confusion > is > >> this: > >> > >> Do not pass choose two arrays. > >> > >> Pass it one array and a *list* of arrays. The fact that choices can be > >> an array is a quirk we can't change, but you should think of the > >> second argument as a list of arrays, > > > > Fine, but as you say, one *can* pass choose an array as the second > argument > > and it doesn't raise an exception, so if someone is stupid/careless > enough > > to pass an array for `choices`, how is choose interpreting it as a list? > Is > > the first dimension "list converted" (so that, e.g., my (2,1,2) example > is > > interpreted as a two element list, each of whose elements is a (1,2) > array)? > > It seems to me that this is the only reasonable interpretation, yes. > After all, arrays behave like sequences along the first axis, whose > elements are arrays of one less dimension. Thus if you pass an array, > any broadcasting happens ignoring the first axis, which is a rather > abnormal pattern for numpy broadcasting, but necessary here. > > As a bonus, I think this is what is implemented in current versions of > numpy. (In 1.2.1 it raises an exception if broadcasting is necessary.) > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Nov 9 00:11:46 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Nov 2009 22:11:46 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> Message-ID: On Sun, Nov 8, 2009 at 9:43 PM, David Cournapeau wrote: > On Mon, Nov 9, 2009 at 12:49 PM, Charles R Harris > wrote: > > >> > > > > Umm, PPC doesn't have ieee quad, it is an amalgam of two doubles. > > Good catch ! I added a new type define for this case, with the > corresponding union. > > I have quickly tested it under rosetta, and it seems that > npy_nextafterl gives a reasonable answer for the few values I have > tried (i.e. the same as for a double). > > I don't remember how PPC does it's arithmetic with the two doubles, but one holds small values that are added to the other double, so I don't think it can be just like a double in some circumstances. What's rosetta? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Nov 9 00:16:25 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Nov 2009 22:16:25 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> Message-ID: On Sun, Nov 8, 2009 at 10:11 PM, Charles R Harris wrote: > > > On Sun, Nov 8, 2009 at 9:43 PM, David Cournapeau wrote: > >> On Mon, Nov 9, 2009 at 12:49 PM, Charles R Harris >> wrote: >> >> >> >> > >> > Umm, PPC doesn't have ieee quad, it is an amalgam of two doubles. >> >> Good catch ! I added a new type define for this case, with the >> corresponding union. >> >> I have quickly tested it under rosetta, and it seems that >> npy_nextafterl gives a reasonable answer for the few values I have >> tried (i.e. the same as for a double). >> >> > I don't remember how PPC does it's arithmetic with the two doubles, but one > holds small values that are added to the other double, so I don't think it > can be just like a double in some circumstances. What's rosetta? > > OTOH, it does look like extended precision on the PPC mac might be doubles: The libm functions declared in math.h provide mathematical library func-tions functions tions in single-, double-, and extended-precision IEEE-754 floating-point formats on Intel macs, and in single- and double-precision IEEE-754 floating-point formats on PowerPC macs. Is that what you see? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Nov 9 00:04:49 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 09 Nov 2009 14:04:49 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> Message-ID: <4AF7A2F1.8000202@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I don't remember how PPC does it's arithmetic with the two doubles, > but one holds small values that are added to the other double, so I > don't think it can be just like a double in some circumstances. That's not how it seems to work on mac os x - when I look at the generated code for long double, the last 8 bytes were always equal to 0. I cannot find a definitive reference, but it looks like it is only 16 bytes to respect the mac os x 16 bytes alignment as required by its ABI. For other OS which run ppc with a different format for long double, I think we could force long double to be the same as double. > What's rosetta? That's the ppc emulator used in mac os x to run old powerpc-only applications on recent Intel mac. It is not 100 % accurate (a few unit tests don't pass, mostly related to FPU exceptions), but it works quite well for testing and is reasonably fast. David From charlesr.harris at gmail.com Mon Nov 9 00:44:03 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Nov 2009 22:44:03 -0700 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF7A2F1.8000202@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911081912t526eb17ch345b9af2905c0cb4@mail.gmail.com> <5b8d13220911082043w35cd9522ye62697b66c51d8ab@mail.gmail.com> <4AF7A2F1.8000202@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Nov 8, 2009 at 10:04 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > I don't remember how PPC does it's arithmetic with the two doubles, > > but one holds small values that are added to the other double, so I > > don't think it can be just like a double in some circumstances. > > That's not how it seems to work on mac os x - when I look at the > generated code for long double, the last 8 bytes were always equal to 0. > I cannot find a definitive reference, but it looks like it is only 16 > bytes to respect the mac os x 16 bytes alignment as required by its ABI. > > For other OS which run ppc with a different format for long double, I > think we could force long double to be the same as double. > > What's rosetta? > > That's the ppc emulator used in mac os x to run old powerpc-only > applications on recent Intel mac. It is not 100 % accurate (a few unit > tests don't pass, mostly related to FPU exceptions), but it works quite > well for testing and is reasonably fast. > > This old thread leads me to believe that OS-X on PPC (G5) is actually different: http://www.mail-archive.com/numpy-discussion at scipy.org/ms16428.html However, there are enough problems with the "real" PPC long double (Nans, etc), that maybe it isn't worth trying to make it work. It isn't ieee and it seems to be fading away in any case. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Mon Nov 9 02:43:01 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 8 Nov 2009 23:43:01 -0800 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: <4AF5DB6C.1070506@molden.no> References: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> <4AF5DB6C.1070506@molden.no> Message-ID: On Sat, Nov 7, 2009 at 12:41 PM, Sturla Molden wrote: > You find a C function pointer wrapped in a CObject in the ._cpointer > attribute. Sorry, in the ._cpointer attribute of what precisely? I tried introspecting in various parts of np.linalg (down to the 'bare' lapack_lite functions) and couldn't find anything with a _cpointer attribute. Thanks, f From david at ar.media.kyoto-u.ac.jp Mon Nov 9 02:43:16 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 09 Nov 2009 16:43:16 +0900 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: References: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> <4AF5DB6C.1070506@molden.no> Message-ID: <4AF7C814.2030809@ar.media.kyoto-u.ac.jp> Fernando Perez wrote: > On Sat, Nov 7, 2009 at 12:41 PM, Sturla Molden wrote: > >> You find a C function pointer wrapped in a CObject in the ._cpointer >> attribute. >> > > Sorry, in the ._cpointer attribute of what precisely? If Sturla refers to CObject as defined in the python C API, then the underlying lapack functions are not wrapped into a C object (this is not done automatically). The lapack_lite functions such as dgeev and co are standard python function (with *self and *args args, i.e. wrappers around the underlying 'true' lapack function). The safe way to access them, since they are not exposed, is to call the function at the python level in your C code, but I don't think that's what you want, cheers, David From pgmdevlist at gmail.com Mon Nov 9 04:02:17 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 9 Nov 2009 04:02:17 -0500 Subject: [Numpy-discussion] Vector interpolation on a 2D grid (with realistic results) In-Reply-To: References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> <1257547522.5931.24.camel@idol> <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> Message-ID: <42BBF0C3-666B-4D64-9F71-89470344445E@gmail.com> On Nov 7, 2009, at 9:14 PM, Ryan May wrote: > On Sat, Nov 7, 2009 at 5:38 PM, Pierre GM > wrote: >> Linear interpolation with the delaunay package doesn't work great for >> my data. I played with the radial basis functions, but I'm afraid >> they're leading me down the dark, dark path of parameter fiddling. In >> particular, I'm not sure how to prevent my interpolated values to be >> bounded by the min and max of my actual observations. >> Ralf' suggestion of smoothing the values afterwards is tempting, but >> sounds a bit too ad-hoc (btw, Ralf, those were relative differences >> of >> monthly average precipitation between El Ni?o and Neutral phases for >> November). > > That was me, not Ralf. :) Oopsie, sorry about that... > And I agree, I the interpolated field does > look a bit noisy for such data. I've been doing the smoothing on top > of natural neighbor for doing some of my own meteorological analysis. > Using the Gaussian kernel isn't really *that* ad hoc considering the > prevalence of Barnes/Cressman weighting for spatial averaging > typically used in meteorology. And if you have no idea what I'm > talking about, Google them, and you'll see. :) I gonna give it a try, thx for the suggestion From pearu.peterson at gmail.com Mon Nov 9 06:23:02 2009 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Mon, 09 Nov 2009 13:23:02 +0200 Subject: [Numpy-discussion] f2py function callback: error while using repeated arguments in function call In-Reply-To: <62e6eafb0911080525x10d31a30le5d67f1500c9ee3e@mail.gmail.com> References: <62e6eafb0911080525x10d31a30le5d67f1500c9ee3e@mail.gmail.com> Message-ID: <4AF7FB96.5070904@cens.ioc.ee> Yves Frederix wrote: > Hi, > > I am doing a simple function callback from fortran to python for which > the actual function call in fortran has repeated arguments. > > ! callback_error.f90: > subroutine testfun(x) > double precision, intent(in) :: x > double precision :: y > !f2py intent(callback) foo > !f2py double precision :: arg1 > !f2py double precision :: arg2 > !f2py double precision :: y > !f2py external y = foo(arg1, arg2) > external foo > y = foo(x, x) ! <-- this causes problems > print *, 'y:', y > end subroutine testfun .. > Is this expected behavior? No. The bug is now fixed in numpy svn (rev 7712). Thanks for pointing out this corner case. Pearu From charlesr.harris at gmail.com Mon Nov 9 10:08:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Nov 2009 08:08:29 -0700 Subject: [Numpy-discussion] Who is ariver and why does it keep making test commits? Message-ID: Just curious, but there have been a lot of test commits to numpy svn by ariver of the form: --- a/trunk/TEST_COMMIT +++ b/trunk/TEST_COMMIT @@ -15,3 +15,3 @@ tim_hochberg: yes jarrod.millman: yes -ariver: 2009.11.04.17.33.56 +ariver: 2009.11.09.08.30.47 Which are 1) pointless, and 2) look almost automated. Anyone know what this is about? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Nov 9 11:12:53 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 9 Nov 2009 10:12:53 -0600 Subject: [Numpy-discussion] Who is ariver and why does it keep making test commits? In-Reply-To: References: Message-ID: <3d375d730911090812j3aa06184q2635cc432273a8f7@mail.gmail.com> On Mon, Nov 9, 2009 at 09:08, Charles R Harris wrote: > Just curious, but there have been a lot of test commits to numpy svn by > ariver of the form: > > --- a/trunk/TEST_COMMIT > +++ b/trunk/TEST_COMMIT > @@ -15,3 +15,3 @@ > tim_hochberg: yes > > jarrod.millman: yes > -ariver: 2009.11.04.17.33.56 > +ariver: 2009.11.09.08.30.47 > > Which are 1) pointless, and 2) look almost automated. Anyone know what this > is about? Aaron River is our sysadmin. He is working on fixing the checkin mailing lists. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sccolbert at gmail.com Mon Nov 9 12:49:41 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 9 Nov 2009 18:49:41 +0100 Subject: [Numpy-discussion] Extremely bizarre behavior I just cant track down... Message-ID: <7f014ea60911090949v1fe8bae8yed986a6789756b5e@mail.gmail.com> I've got an issue where trying to pass a numpy array to one of my cython functions fails, with the exception saying than 'int objects are not iterable'. So somehow, my array is going from being perfectly ok (i can display the image and print its shape and size), to going bad right before the function call (i can still print the size and shape, but not the array itself). I have pastebin'ed a couple of example cases showing the a workaround and some failures that make absolutely no sense. http://pastebin.com/m65b0c718 Could one of the numpy geniuses here take a stab at what this could be? in the call to cvCvtColor() I create a new array which is returned as grayimg, and I properly incref the dtype before the call to PyArray_EMPY(). Cheers! Chris From Chris.Barker at noaa.gov Mon Nov 9 13:00:30 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 09 Nov 2009 10:00:30 -0800 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <20091107215629.6009523e@ajackson.org> References: <20091107215629.6009523e@ajackson.org> Message-ID: <4AF858BE.3080207@noaa.gov> alan at ajackson.org wrote: > myarray = zeros( (xdim,ydim), dtype=object) > and then iterate through the elements initializing then to empty lists, but > surely there is a better way. I tried this: In [3]: a = np.empty((2,3), dtype=np.object) In [5]: a[:,:] = [] but got: ValueError: shape mismatch: objects cannot be broadcast to a single shape Is that a bug? Or is it simply too ambiguous for numpy to figure out what the heck I want? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Nov 9 13:03:06 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 9 Nov 2009 12:03:06 -0600 Subject: [Numpy-discussion] initializing an array of lists In-Reply-To: <4AF858BE.3080207@noaa.gov> References: <20091107215629.6009523e@ajackson.org> <4AF858BE.3080207@noaa.gov> Message-ID: <3d375d730911091003g1465eabfx5d1933a05533978f@mail.gmail.com> On Mon, Nov 9, 2009 at 12:00, Christopher Barker wrote: > alan at ajackson.org wrote: >> myarray = zeros( (xdim,ydim), dtype=object) >> and then iterate through the elements initializing then to empty lists, but >> surely there is a better way. > > I tried this: > > In [3]: a = np.empty((2,3), dtype=np.object) > > In [5]: a[:,:] = [] > > but got: > > ValueError: shape mismatch: objects cannot be broadcast to a single shape > > Is that a bug? Or is it simply too ambiguous for numpy to figure out > what the heck I want? The latter. In any case, you wouldn't want each element to be the same list object. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pav+sp at iki.fi Mon Nov 9 13:09:43 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Mon, 9 Nov 2009 18:09:43 +0000 (UTC) Subject: [Numpy-discussion] initializing an array of lists References: <20091107215629.6009523e@ajackson.org> Message-ID: Sat, 07 Nov 2009 21:56:29 -0600, alan wrote: > I want to build a 2D array of lists, and so I need to initialize the > array with empty lists : > > myarray = array([[[],[],[]] ,[[],[],[]]]) > > Is there a clever way to do this? I could define the array > > myarray = zeros( (xdim,ydim), dtype=object) and then iterate through the > elements initializing then to empty lists, but surely there is a better > way. This question seems to come up from time to time (= maybe should be a FAQ?). You can for example vectorize the list constructor: >>> filler = np.frompyfunc(lambda x: list(), 1, 1) >>> a = np.empty((3, 4), dtype=np.object) >>> filler(a, a); array([[[], [], [], []], [[], [], [], []], [[], [], [], []]], dtype=object) >>> a[0,3].append(9) >>> a array([[[], [], [], [9]], [[], [], [], []], [[], [], [], []]], dtype=object) -- Pauli Virtanen From jakevdp at gmail.com Mon Nov 9 14:08:52 2009 From: jakevdp at gmail.com (Jake VanderPlas) Date: Mon, 9 Nov 2009 11:08:52 -0800 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API Message-ID: <58df6dc20911091108m45cef3a6r5600e8d5d972b790@mail.gmail.com> >The safe way to access them, since they are not exposed, is to call the >function at the python level in your C code, but I don't think that's >what you want, I want to avoid calling functions at the python level, because of the overhead for multiple calls within nested loops. I may have a solution: I've managed to get access to the BLAS fortran library by using in the setup.py file: from distutils.core import Extension from numpy.distutils import system_info myextension = Extension(<...> library_dirs = system_info.blas_opt_info().get_lib_dirs(), <...>) Then in my C++ code I can declare, e.g. extern "C" double ddot_(const int *N, const double *DX, const int *INCX, const double *DY, const int *INCY); ...and directly call the BLAS fortran library. This pattern works on my system (linux, using the system BLAS/LAPACK libraries). Is this a form that will work across different OSs and different installs? -Jake From sccolbert at gmail.com Mon Nov 9 16:52:13 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 9 Nov 2009 22:52:13 +0100 Subject: [Numpy-discussion] Extremely bizarre behavior I just cant track down... In-Reply-To: <7f014ea60911090949v1fe8bae8yed986a6789756b5e@mail.gmail.com> References: <7f014ea60911090949v1fe8bae8yed986a6789756b5e@mail.gmail.com> Message-ID: <7f014ea60911091352v636f8b08hcbfdcef0619830a0@mail.gmail.com> This problem is solved. Lisandro spent a bunch of time with me helping to track it down. Thanks Lisandro! On Mon, Nov 9, 2009 at 6:49 PM, Chris Colbert wrote: > I've got an issue where trying to pass a numpy array to one of my > cython functions fails, with the exception saying than 'int objects > are not iterable'. > > So somehow, my array is going from being perfectly ok (i can display > the image and print its shape and size), to going bad right before the > function call (i can still print the size and shape, but not the array > itself). > > I have pastebin'ed a couple of example cases showing the a workaround > and some failures that make absolutely no sense. > > http://pastebin.com/m65b0c718 > > Could one of the numpy geniuses here take a stab at what this could be? > > in the call to cvCvtColor() I create a new array which is returned as > grayimg, and I properly incref the dtype before the call to > PyArray_EMPY(). > > Cheers! > > Chris > From neilcrighton at gmail.com Mon Nov 9 18:13:42 2009 From: neilcrighton at gmail.com (Neil Crighton) Date: Mon, 9 Nov 2009 23:13:42 +0000 (UTC) Subject: [Numpy-discussion] Release notes for arraysetops changes Message-ID: Hi, I've written some release notes (below) describing the changes to arraysetops.py. If someone with commit access could check that these sound ok and add them to the release notes file, that would be great. Cheers, Neil New features ============ Improved set operations ~~~~~~~~~~~~~~~~~~~~~~~ In previous versions of NumPy some set functions (intersect1d, setxor1d, setdiff1d and setmember1d) could return incorrect results if the input arrays contained duplicate items. These now work correctly for input arrays with duplicates. setmember1d has been renamed to in1d, as with the change to accept arrays with duplicates it is no longer a set operation, and is conceptually similar to an elementwise version of the Python operator 'in'. All of these functions now accept the boolean keyword assume_unique. This is False by default, but can be set True if the input arrays are known not to contain duplicates, which can increase the functions' execution speed. Deprecations ============ #. unique1d: use unique instead. unique1d raises a deprecation warning in 1.4, and will be removed in 1.5. #. intersect1d_nu: use intersect1d instead. intersect1d_nu raises a deprecation warning in 1.4, and will be removed in 1.5. #. setmember1d: use in1d instead. setmember1d raises a deprecation warning in 1.4, and will be removed in 1.5. From cournape at gmail.com Mon Nov 9 18:24:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 10 Nov 2009 08:24:39 +0900 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: <58df6dc20911091108m45cef3a6r5600e8d5d972b790@mail.gmail.com> References: <58df6dc20911091108m45cef3a6r5600e8d5d972b790@mail.gmail.com> Message-ID: <5b8d13220911091524n2cfaef2bx314f47e873f30c06@mail.gmail.com> On Tue, Nov 10, 2009 at 4:08 AM, Jake VanderPlas wrote: >>The safe way to access them, since they are not exposed, is to call the >>function at the python level in your C code, but I don't think that's >>what you want, > > I want to avoid calling functions at the python level, because of the > overhead for multiple calls within nested loops. Yes, there is no question this would be useful, but we don't have the mechanism (yet) to make that possible. We have added some infrastructure at the build level to make C libraries available to 3rd parties, but more is needed for blas/lapack. > ...and directly call the BLAS fortran library. ?This pattern works on > my system (linux, using the system BLAS/LAPACK libraries). ?Is this a > form that will work across different OSs and different installs? Not at all. What you have done here is simply link against whatever blas/lapack is found on your system. It will not work on platforms without development package of blas/lapack, with different fortran name mangling, etc... That's certainly a valid option until we support this better in numpy/scipy, though. David From pav at iki.fi Mon Nov 9 18:32:01 2009 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 10 Nov 2009 01:32:01 +0200 Subject: [Numpy-discussion] Release notes for arraysetops changes In-Reply-To: References: Message-ID: <1257809520.4973.0.camel@idol> ma, 2009-11-09 kello 23:13 +0000, Neil Crighton kirjoitti: > I've written some release notes (below) describing the changes to > arraysetops.py. If someone with commit access could check that these sound ok > and add them to the release notes file, that would be great. Thanks, added! Pauli From d.l.goldsmith at gmail.com Mon Nov 9 19:54:23 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Mon, 9 Nov 2009 16:54:23 -0800 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911082050r7d18f3c0g6ce12a7c39aba34@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911080033x41aa4eb2r7fd8bb97ca0a282c@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> <45d1ab480911082050r7d18f3c0g6ce12a7c39aba34@mail.gmail.com> Message-ID: <45d1ab480911091654g7a543b5evbce93bf6224c33b0@mail.gmail.com> May I infer from the sudden silence that I finally have it? DG On Sun, Nov 8, 2009 at 8:50 PM, David Goldsmith wrote: > OK, let me see if I'm interpreting this example correctly: > > >>> c1=np.arange(2).reshape(2,1,1); c1 > array([[[0]], > > [[1]]]) > >>> c2=2+np.arange(2).reshape(1,1,2); c2 > array([[[2, 3]]]) > > >>> a=np.eye(2,dtype=int) > >>> np.choose(a, [c1, c2]) > array([[[2, 0], > [0, 3]], > > [[2, 1], > [1, 3]]]) > > First, everything is being broadcast to (2,2,2); a is broadcast to [[[1,0], > [0,1]], [[1,0], [0,1]]], c1 is broadcast to [[[0,0], [0,0]], [[1,1], [1,1]]] > and c2 is broadcast to [[[2,3], [2,3]], [[2,3], [2,3]]]. Now result is > created by "stepping through" broadcast a and using, respectively, the > positionally corresponding element from broadcast c1 (resp. c2) if the value > in a at the position is 0 (resp. 1). At least, this gives the result above > (but I have not examined other possible broadcasts of the arguments to see > if they would also give the result - I conjectured what appeared to me to be > the most "natural" broadcasts and checked to see if it worked and it does; > is there something I should know - e.g., uniqueness of the result, or a rule > governing how choose broadcasts - to *know* that the broadcasts above are > indeed the broadcasts choose is using?) > > Thanks again, > > DG > > On Sun, Nov 8, 2009 at 8:19 PM, Anne Archibald wrote: > >> 2009/11/8 David Goldsmith : >> > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald < >> peridot.faceted at gmail.com> >> > wrote: >> >> >> >> As Josef said, this is not correct. I think the key point of confusion >> is >> >> this: >> >> >> >> Do not pass choose two arrays. >> >> >> >> Pass it one array and a *list* of arrays. The fact that choices can be >> >> an array is a quirk we can't change, but you should think of the >> >> second argument as a list of arrays, >> > >> > Fine, but as you say, one *can* pass choose an array as the second >> argument >> > and it doesn't raise an exception, so if someone is stupid/careless >> enough >> > to pass an array for `choices`, how is choose interpreting it as a >> list? Is >> > the first dimension "list converted" (so that, e.g., my (2,1,2) example >> is >> > interpreted as a two element list, each of whose elements is a (1,2) >> array)? >> >> It seems to me that this is the only reasonable interpretation, yes. >> After all, arrays behave like sequences along the first axis, whose >> elements are arrays of one less dimension. Thus if you pass an array, >> any broadcasting happens ignoring the first axis, which is a rather >> abnormal pattern for numpy broadcasting, but necessary here. >> >> As a bonus, I think this is what is implemented in current versions of >> numpy. (In 1.2.1 it raises an exception if broadcasting is necessary.) >> >> Anne >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Nov 9 19:59:29 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Nov 2009 19:59:29 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <45d1ab480911091654g7a543b5evbce93bf6224c33b0@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> <45d1ab480911082050r7d18f3c0g6ce12a7c39aba34@mail.gmail.com> <45d1ab480911091654g7a543b5evbce93bf6224c33b0@mail.gmail.com> Message-ID: <1cd32cbb0911091659j5c8aaed4i80e4d1437255bcd0@mail.gmail.com> On Mon, Nov 9, 2009 at 7:54 PM, David Goldsmith wrote: > May I infer from the sudden silence that I finally have it? I think so, I assume that the result of broadcasting is unique, I haven't seen an example yet where broadcasting would depend on the sequence in which it is done. Josef > > DG > > On Sun, Nov 8, 2009 at 8:50 PM, David Goldsmith > wrote: >> >> OK, let me see if I'm interpreting this example correctly: >> >> >>> c1=np.arange(2).reshape(2,1,1); c1 >> array([[[0]], >> >> ?????? [[1]]]) >> >>> c2=2+np.arange(2).reshape(1,1,2); c2 >> array([[[2, 3]]]) >> >>> a=np.eye(2,dtype=int) >> >>> np.choose(a, [c1, c2]) >> array([[[2, 0], >> ??????? [0, 3]], >> >> ?????? [[2, 1], >> ??????? [1, 3]]]) >> >> First, everything is being broadcast to (2,2,2); a is broadcast to >> [[[1,0], [0,1]], [[1,0], [0,1]]], c1 is broadcast to [[[0,0], [0,0]], >> [[1,1], [1,1]]] and c2 is broadcast to [[[2,3], [2,3]], [[2,3], [2,3]]]. >> Now result is created by "stepping through" broadcast a and using, >> respectively, the positionally corresponding element from broadcast c1 >> (resp. c2) if the value in a at the position is 0 (resp. 1).? At least, this >> gives the result above (but I have not examined other possible broadcasts of >> the arguments to see if they would also give the result - I conjectured what >> appeared to me to be the most "natural" broadcasts and checked to see if it >> worked and it does; is there something I should know - e.g., uniqueness of >> the result, or a rule governing how choose broadcasts - to *know* that the >> broadcasts above are indeed the broadcasts choose is using?) >> >> Thanks again, >> >> DG >> On Sun, Nov 8, 2009 at 8:19 PM, Anne Archibald >> wrote: >>> >>> 2009/11/8 David Goldsmith : >>> > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald >>> > >>> > wrote: >>> >> >>> >> As Josef said, this is not correct. I think the key point of confusion >>> >> is >>> >> this: >>> >> >>> >> Do not pass choose two arrays. >>> >> >>> >> Pass it one array and a *list* of arrays. The fact that choices can be >>> >> an array is a quirk we can't change, but you should think of the >>> >> second argument as a list of arrays, >>> > >>> > Fine, but as you say, one *can* pass choose an array as the second >>> > argument >>> > and it doesn't raise an exception, so if someone is stupid/careless >>> > enough >>> > to pass an array for `choices`, how is choose interpreting it as a >>> > list?? Is >>> > the first dimension "list converted" (so that, e.g., my (2,1,2) example >>> > is >>> > interpreted as a two element list, each of whose elements is a (1,2) >>> > array)? >>> >>> It seems to me that this is the only reasonable interpretation, yes. >>> After all, arrays behave like sequences along the first axis, whose >>> elements are arrays of one less dimension. Thus if you pass an array, >>> any broadcasting happens ignoring the first axis, which is a rather >>> abnormal pattern for numpy broadcasting, but necessary here. >>> >>> As a bonus, I think this is what is implemented in current versions of >>> numpy. (In 1.2.1 it raises an exception if broadcasting is necessary.) >>> >>> Anne >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Mon Nov 9 20:16:58 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Nov 2009 20:16:58 -0500 Subject: [Numpy-discussion] Use-case for np.choose In-Reply-To: <1cd32cbb0911091659j5c8aaed4i80e4d1437255bcd0@mail.gmail.com> References: <45d1ab480911071304ubd8c718g8ab66ade2b4025e7@mail.gmail.com> <45d1ab480911080200t58461b10n730418fec56d2ad7@mail.gmail.com> <1cd32cbb0911080536y1d0dfc8ej5e5707590bc749cd@mail.gmail.com> <45d1ab480911081903h17975cb9r7acd9a347a9de9a7@mail.gmail.com> <45d1ab480911082008n15164023v9cbc24d1fe357506@mail.gmail.com> <45d1ab480911082050r7d18f3c0g6ce12a7c39aba34@mail.gmail.com> <45d1ab480911091654g7a543b5evbce93bf6224c33b0@mail.gmail.com> <1cd32cbb0911091659j5c8aaed4i80e4d1437255bcd0@mail.gmail.com> Message-ID: <1cd32cbb0911091716p62b76cecoa832e3810333eb2e@mail.gmail.com> On Mon, Nov 9, 2009 at 7:59 PM, wrote: > On Mon, Nov 9, 2009 at 7:54 PM, David Goldsmith wrote: >> May I infer from the sudden silence that I finally have it? > > I think so, > I assume that the result of broadcasting is unique, I haven't seen an > example yet where broadcasting would depend on the sequence in which > it is done. on a related note: I looked at the doc editor discussion for numpy.where and it seems broadcasting has been fixed also for where (in 1.3.0). I don't know whether the c implementation is related to choose. I only tried Paulis example. Josef > > Josef > > >> >> DG >> >> On Sun, Nov 8, 2009 at 8:50 PM, David Goldsmith >> wrote: >>> >>> OK, let me see if I'm interpreting this example correctly: >>> >>> >>> c1=np.arange(2).reshape(2,1,1); c1 >>> array([[[0]], >>> >>> ?????? [[1]]]) >>> >>> c2=2+np.arange(2).reshape(1,1,2); c2 >>> array([[[2, 3]]]) >>> >>> a=np.eye(2,dtype=int) >>> >>> np.choose(a, [c1, c2]) >>> array([[[2, 0], >>> ??????? [0, 3]], >>> >>> ?????? [[2, 1], >>> ??????? [1, 3]]]) >>> >>> First, everything is being broadcast to (2,2,2); a is broadcast to >>> [[[1,0], [0,1]], [[1,0], [0,1]]], c1 is broadcast to [[[0,0], [0,0]], >>> [[1,1], [1,1]]] and c2 is broadcast to [[[2,3], [2,3]], [[2,3], [2,3]]]. >>> Now result is created by "stepping through" broadcast a and using, >>> respectively, the positionally corresponding element from broadcast c1 >>> (resp. c2) if the value in a at the position is 0 (resp. 1).? At least, this >>> gives the result above (but I have not examined other possible broadcasts of >>> the arguments to see if they would also give the result - I conjectured what >>> appeared to me to be the most "natural" broadcasts and checked to see if it >>> worked and it does; is there something I should know - e.g., uniqueness of >>> the result, or a rule governing how choose broadcasts - to *know* that the >>> broadcasts above are indeed the broadcasts choose is using?) >>> >>> Thanks again, >>> >>> DG >>> On Sun, Nov 8, 2009 at 8:19 PM, Anne Archibald >>> wrote: >>>> >>>> 2009/11/8 David Goldsmith : >>>> > On Sun, Nov 8, 2009 at 7:40 PM, Anne Archibald >>>> > >>>> > wrote: >>>> >> >>>> >> As Josef said, this is not correct. I think the key point of confusion >>>> >> is >>>> >> this: >>>> >> >>>> >> Do not pass choose two arrays. >>>> >> >>>> >> Pass it one array and a *list* of arrays. The fact that choices can be >>>> >> an array is a quirk we can't change, but you should think of the >>>> >> second argument as a list of arrays, >>>> > >>>> > Fine, but as you say, one *can* pass choose an array as the second >>>> > argument >>>> > and it doesn't raise an exception, so if someone is stupid/careless >>>> > enough >>>> > to pass an array for `choices`, how is choose interpreting it as a >>>> > list?? Is >>>> > the first dimension "list converted" (so that, e.g., my (2,1,2) example >>>> > is >>>> > interpreted as a two element list, each of whose elements is a (1,2) >>>> > array)? >>>> >>>> It seems to me that this is the only reasonable interpretation, yes. >>>> After all, arrays behave like sequences along the first axis, whose >>>> elements are arrays of one less dimension. Thus if you pass an array, >>>> any broadcasting happens ignoring the first axis, which is a rather >>>> abnormal pattern for numpy broadcasting, but necessary here. >>>> >>>> As a bonus, I think this is what is implemented in current versions of >>>> numpy. (In 1.2.1 it raises an exception if broadcasting is necessary.) >>>> >>>> Anne >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From scott.sinclair.za at gmail.com Tue Nov 10 01:48:53 2009 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Tue, 10 Nov 2009 08:48:53 +0200 Subject: [Numpy-discussion] Vector interpolation on a 2D grid (with realistic results) In-Reply-To: <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> References: <9C0274B1-9147-4A8D-817C-C79367AFF7C5@gmail.com> <1257547522.5931.24.camel@idol> <25011892-2FDF-42C6-8D64-376936297CDB@gmail.com> Message-ID: <6a17e9ee0911092248g3e1e4370p8010098419e42c56@mail.gmail.com> >2009/11/8 Pierre GM : > Chris, I gonna poke around and try to find some kriging algorithms. > I'll report in a few. In the meantime, if anybody has anythng already > implemented, please just let us know. A little late with the reply. I've used gstat (http://www.gstat.org/) in two ways 1) by running the executable from Python using os.system() and 2) using the rpy interface to the gstat R package. It's a little clunky but works and is quick and easy to set up. If you need to see some code for option 2 I can dig it up. You might also look at SGEMS http://sgems.sourceforge.net/ there is a Python interface, but I haven't used it. Cheers, Scott From fperez.net at gmail.com Tue Nov 10 02:43:31 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 9 Nov 2009 23:43:31 -0800 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: <4AF7C814.2030809@ar.media.kyoto-u.ac.jp> References: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> <4AF5DB6C.1070506@molden.no> <4AF7C814.2030809@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Nov 8, 2009 at 11:43 PM, David Cournapeau wrote: > Fernando Perez wrote: >> On Sat, Nov 7, 2009 at 12:41 PM, Sturla Molden wrote: >> >>> You find a C function pointer wrapped in a CObject in the ._cpointer >>> attribute. >>> >> >> Sorry, in the ._cpointer attribute of what precisely? > > If Sturla refers to CObject as defined in the python C API, then the > underlying lapack functions are not wrapped into a C object (this is not > done automatically). The lapack_lite functions such as dgeev and co are > standard python function (with *self and *args args, i.e. wrappers > around the underlying 'true' lapack function). Thanks, David. My question was because Sturla's language seemed to suggest (at least it did to me) that there was some python object foo where foo._cpointer would be a poniter to the raw C function one could then manipulate with ctypes. Perhaps I just misunderstood, but thanks for clarifying that at least this isn't possible today from pure python. Cheers, f From robert.kern at gmail.com Tue Nov 10 02:50:25 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 10 Nov 2009 02:50:25 -0500 Subject: [Numpy-discussion] Accessing LAPACK and BLAS from the numpy C API In-Reply-To: References: <58df6dc20911071212k693b4c73wcbaa827b9799d258@mail.gmail.com> <4AF5DB6C.1070506@molden.no> <4AF7C814.2030809@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911092350u7f4bcffcxa67fb0c1df1e183d@mail.gmail.com> On Tue, Nov 10, 2009 at 02:43, Fernando Perez wrote: > On Sun, Nov 8, 2009 at 11:43 PM, David Cournapeau > wrote: >> Fernando Perez wrote: >>> On Sat, Nov 7, 2009 at 12:41 PM, Sturla Molden wrote: >>> >>>> You find a C function pointer wrapped in a CObject in the ._cpointer >>>> attribute. >>>> >>> >>> Sorry, in the ._cpointer attribute of what precisely? >> >> If Sturla refers to CObject as defined in the python C API, then the >> underlying lapack functions are not wrapped into a C object (this is not >> done automatically). The lapack_lite functions such as dgeev and co are >> standard python function (with *self and *args args, i.e. wrappers >> around the underlying 'true' lapack function). > > Thanks, David. ?My question was because Sturla's language seemed to > suggest (at least it did to me) that there was some python object foo > where foo._cpointer would be a poniter to the raw C function one could > then manipulate with ctypes. ?Perhaps I just misunderstood, but thanks > for clarifying that at least this isn't possible today from pure > python. Only f2py functions have the ._cpointer attribute. In [3]: from scipy import linalg In [4]: linalg.flapack.dgeev._cpointer Out[4]: -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From darryl.wallace at prosensus.ca Tue Nov 10 13:09:30 2009 From: darryl.wallace at prosensus.ca (Darryl Wallace) Date: Tue, 10 Nov 2009 13:09:30 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> Message-ID: <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> Hello again, The best way so far that's come to my attention is to use: numpy.ma.masked_object The problem with this is that it's looking for a specific instance of an object. So if the user had some elements of their array that were, for example, "randomString" , then it would not be picked up e.g. --- from numpy import * mixedArray=array([1,2, '', 3, 4, 'randomString'], dtype=object) mixedArrayMask = ma.masked_object(mixedArray, 'randomString').mask --- then mixedArrayMask will yield: array([ False, False, False, False, False, True]) Can anyone help me so that all strings are found in the array without having to explicitly loop through them in Python? Thanks, Darryl On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace wrote: > What I'm doing is importing some data from excel and sometimes there are > strings in the worksheet. Often times a user will use an empty cell or a > string to represent data that is missing. > > e.g. > from numpy import * > mixedArray=array([1, 2, '', 3, 4, 'String'], dtype=object) > > Two questions: > 1) Is there a quick way to find the elements in the array that are the > strings without iterating over each element in the array? > > or > > 2) Could I quickly turn it into a masked array of type float where all > string elements are set as missing points? > > I've been struggling with this for a while and can't come across a method > that will all me to do it without iterating over each element. > > Any help or pointers in the right direction would be greatly appreciated. > > Thanks, > Darryl > -- ______________________________________ Darryl Wallace: Project Leader ProSensus Inc. McMaster Innovation Park 175 Longwood Road South, Suite 301 Hamilton, Ontario, L8P 0A1 Canada (GMT -05:00) Tel: 1-905-528-9136 Fax: 1-905-546-1372 Web site: http://www.prosensus.ca/ ______________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Tue Nov 10 13:32:15 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 10 Nov 2009 12:32:15 -0600 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> Message-ID: <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> On Tue, Nov 10, 2009 at 12:09 PM, Darryl Wallace wrote: > Hello again, > The best way so far that's come to my attention is to use: > numpy.ma.masked_object > The problem with this is that it's looking for a specific instance of an > object. ?So if the user had some elements of their array that were, for > example, "randomString" , then it would not be picked up > e.g. > --- > from numpy import * > mixedArray=array([1,2, '', 3, 4, 'randomString'], dtype=object) > mixedArrayMask = ma.masked_object(mixedArray, 'randomString').mask > --- > then mixedArrayMask will yield: > > array([ False, False, False, False, False, True]) > Can anyone help me so that all strings are found in the array without having > to explicitly loop through them in Python? > Thanks, > Darryl Why not stick to a same Missing-Value-Code or for all the non-valid data? I don't know how MA module would handle mixed MVCs in a same array without modifying the existing code. Otherwise looping over the array an masking the str instances as NaN would be my alternative solution. > > On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace > wrote: >> >> What I'm doing is importing some data from excel and sometimes there are >> strings in the worksheet. ?Often times a user will use an empty cell or a >> string to represent data that is missing. >> e.g. >> from numpy import * >> mixedArray=array([1, 2, '', 3, 4, 'String'], dtype=object) >> Two questions: >> 1) Is there a quick way to find the elements in the array that are the >> strings without iterating over each element in the array? >> or >> 2) Could I quickly turn it into a masked array of type float where all >> string elements are set as missing points? >> I've been struggling with this for a while and can't come across a method >> that will all me to do it without iterating over each element. >> Any help or pointers in the right direction would be greatly appreciated. >> Thanks, >> Darryl > > > > -- > ______________________________________ > Darryl Wallace: Project Leader > ProSensus Inc. > McMaster Innovation Park > 175 Longwood Road South, Suite 301 > Hamilton, Ontario, L8P 0A1 > Canada ? ? ? ?(GMT -05:00) > > Tel: ? ? ? 1-905-528-9136 > Fax: ? ? ? 1-905-546-1372 > > Web site: ?http://www.prosensus.ca/ > ______________________________________ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan From darryl.wallace at prosensus.ca Tue Nov 10 13:53:21 2009 From: darryl.wallace at prosensus.ca (Darryl Wallace) Date: Tue, 10 Nov 2009 13:53:21 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> Message-ID: <7abc62af0911101053t7c3bac8m23c9d681cfd9708b@mail.gmail.com> Hello, On Tue, Nov 10, 2009 at 1:32 PM, G?khan Sever wrote: > On Tue, Nov 10, 2009 at 12:09 PM, Darryl Wallace > wrote: > > Hello again, > > The best way so far that's come to my attention is to use: > > numpy.ma.masked_object > > The problem with this is that it's looking for a specific instance of an > > object. So if the user had some elements of their array that were, for > > example, "randomString" , then it would not be picked up > > e.g. > > --- > > from numpy import * > > mixedArray=array([1,2, '', 3, 4, 'randomString'], dtype=object) > > mixedArrayMask = ma.masked_object(mixedArray, 'randomString').mask > > --- > > then mixedArrayMask will yield: > > > > array([ False, False, False, False, False, True]) > > Can anyone help me so that all strings are found in the array without > having > > to explicitly loop through them in Python? > > Thanks, > > Darryl > > Why not stick to a same Missing-Value-Code or for all the non-valid > data? I don't know how MA module would handle mixed MVCs in a same > array without modifying the existing code. Otherwise looping over the > array an masking the str instances as NaN would be my alternative > solution. > The reason I don't stick to a standard missing value code is because a user may import other things in the datasheet that we need, like row or column labels, or maybe getting data from a specific source which reports missing data as a specific string. I currently do as you suggested. But when the dataset size becomes large, it gets to be quite slow due to the overhead of python looping. Thanks > > > > > > On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace < > darryl.wallace at prosensus.ca> > > wrote: > >> > >> What I'm doing is importing some data from excel and sometimes there are > >> strings in the worksheet. Often times a user will use an empty cell or > a > >> string to represent data that is missing. > >> e.g. > >> from numpy import * > >> mixedArray=array([1, 2, '', 3, 4, 'String'], dtype=object) > >> Two questions: > >> 1) Is there a quick way to find the elements in the array that are the > >> strings without iterating over each element in the array? > >> or > >> 2) Could I quickly turn it into a masked array of type float where all > >> string elements are set as missing points? > >> I've been struggling with this for a while and can't come across a > method > >> that will all me to do it without iterating over each element. > >> Any help or pointers in the right direction would be greatly > appreciated. > >> Thanks, > >> Darryl > > > > > > > > -- > > ______________________________________ > > Darryl Wallace: Project Leader > > ProSensus Inc. > > McMaster Innovation Park > > 175 Longwood Road South, Suite 301 > > Hamilton, Ontario, L8P 0A1 > > Canada (GMT -05:00) > > > > Tel: 1-905-528-9136 > > Fax: 1-905-546-1372 > > > > Web site: http://www.prosensus.ca/ > > ______________________________________ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > -- > G?khan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- ______________________________________ Darryl Wallace: Project Leader ProSensus Inc. McMaster Innovation Park 175 Longwood Road South, Suite 301 Hamilton, Ontario, L8P 0A1 Canada (GMT -05:00) Tel: 1-905-528-9136 Fax: 1-905-546-1372 Web site: http://www.prosensus.ca/ ______________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Nov 10 14:28:41 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 10 Nov 2009 11:28:41 -0800 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> <7abc62af0911101053t7c3bac8m23c9d681cfd9708b@mail.gmail.com> Message-ID: On Tue, Nov 10, 2009 at 11:14 AM, Keith Goodman wrote: > On Tue, Nov 10, 2009 at 10:53 AM, Darryl Wallace > wrote: >> I currently do as you suggested. ?But when the dataset size becomes large, >> it gets to be quite slow due to the overhead of python looping. > > Are you using a for loop? Is so, something like this might be faster: > >>> x = [1, 2, '', 3, 4, 'String'] >>> from numpy import nan >>> [(z, nan)[type(z) is str] for z in x] > ? [1, 2, nan, 3, 4, nan] > > I use something similar in my code, so I'm interested to see if anyone > can speed things up using python or numpy, or both. I run it on each > row of the file replacing '' with None. Here's the benchmark code: > >>> x = [1, 2, '', 4, 5, '', 7, 8, 9, 10] >>> timeit [(z, None)[z is ''] for z in x] > 100000 loops, best of 3: 2.32 ?s per loop If there are few missing values (my use case), this seems to be faster: def myfunc(x): while '' in x: x[x.index('')] = None return x >> timeit myfunc(x) 1000000 loops, best of 3: 697 ns per loop Note that it works inplace. From kwgoodman at gmail.com Tue Nov 10 14:34:52 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 10 Nov 2009 11:34:52 -0800 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> <7abc62af0911101053t7c3bac8m23c9d681cfd9708b@mail.gmail.com> Message-ID: On Tue, Nov 10, 2009 at 11:28 AM, Keith Goodman wrote: > On Tue, Nov 10, 2009 at 11:14 AM, Keith Goodman wrote: >> On Tue, Nov 10, 2009 at 10:53 AM, Darryl Wallace >> wrote: >>> I currently do as you suggested. ?But when the dataset size becomes large, >>> it gets to be quite slow due to the overhead of python looping. >> >> Are you using a for loop? Is so, something like this might be faster: >> >>>> x = [1, 2, '', 3, 4, 'String'] >>>> from numpy import nan >>>> [(z, nan)[type(z) is str] for z in x] >> ? [1, 2, nan, 3, 4, nan] >> >> I use something similar in my code, so I'm interested to see if anyone >> can speed things up using python or numpy, or both. I run it on each >> row of the file replacing '' with None. Here's the benchmark code: >> >>>> x = [1, 2, '', 4, 5, '', 7, 8, 9, 10] >>>> timeit [(z, None)[z is ''] for z in x] >> 100000 loops, best of 3: 2.32 ?s per loop > > If there are few missing values (my use case), this seems to be faster: > > def myfunc(x): > ? ?while '' in x: > ? ? ? ?x[x.index('')] = None > ? ?return x > >>> timeit myfunc(x) > 1000000 loops, best of 3: 697 ns per loop > > Note that it works inplace. Oops. It's hard to time functions that change the input. Making a copy of x at the top of the functions takes away the speed advantage. Is there a way to get timeit to restore the input on each cycle? OK, I'll stop spamming the list. From darryl.wallace at prosensus.ca Tue Nov 10 15:08:27 2009 From: darryl.wallace at prosensus.ca (Darryl Wallace) Date: Tue, 10 Nov 2009 15:08:27 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> <49d6b3500911101032w7b9b817x733583ad50f3ef85@mail.gmail.com> <7abc62af0911101053t7c3bac8m23c9d681cfd9708b@mail.gmail.com> Message-ID: <7abc62af0911101208u398692abr62df9f35fc8fe721@mail.gmail.com> Thanks for the help, I'll test out this simple example. On Tue, Nov 10, 2009 at 2:28 PM, Keith Goodman wrote: > On Tue, Nov 10, 2009 at 11:14 AM, Keith Goodman > wrote: > > On Tue, Nov 10, 2009 at 10:53 AM, Darryl Wallace > > wrote: > >> I currently do as you suggested. But when the dataset size becomes > large, > >> it gets to be quite slow due to the overhead of python looping. > > > > Are you using a for loop? Is so, something like this might be faster: > > > >>> x = [1, 2, '', 3, 4, 'String'] > >>> from numpy import nan > >>> [(z, nan)[type(z) is str] for z in x] > > [1, 2, nan, 3, 4, nan] > > > > I use something similar in my code, so I'm interested to see if anyone > > can speed things up using python or numpy, or both. I run it on each > > row of the file replacing '' with None. Here's the benchmark code: > > > >>> x = [1, 2, '', 4, 5, '', 7, 8, 9, 10] > >>> timeit [(z, None)[z is ''] for z in x] > > 100000 loops, best of 3: 2.32 ?s per loop > > If there are few missing values (my use case), this seems to be faster: > > def myfunc(x): > while '' in x: > x[x.index('')] = None > return x > > >> timeit myfunc(x) > 1000000 loops, best of 3: 697 ns per loop > > Note that it works inplace. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- ______________________________________ Darryl Wallace: Project Leader ProSensus Inc. McMaster Innovation Park 175 Longwood Road South, Suite 301 Hamilton, Ontario, L8P 0A1 Canada (GMT -05:00) Tel: 1-905-528-9136 Fax: 1-905-546-1372 Web site: http://www.prosensus.ca/ ______________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Nov 10 14:23:32 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 10 Nov 2009 14:23:32 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> Message-ID: <31ECC99C-AA62-48CC-BBA7-4B63DF9D9DA5@gmail.com> On Nov 10, 2009, at 1:09 PM, Darryl Wallace wrote: > Hello again, > > The best way so far that's come to my attention is to use: > > numpy.ma.masked_object Will only work for masking one specific string, as you've noticed. > > Can anyone help me so that all strings are found in the array without having to explicitly loop through them in Python? This looks like it's working: >>> mask = (mixed >= '') >>> mixed = ma.array(np.where(~mask,mixed,np.nan),mask=mask,dtype=float) > Thanks, > Darryl > > > On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace wrote: > What I'm doing is importing some data from excel and sometimes there are strings in the worksheet. Now, what are you using to read the Excel file ? Do you get a list that you transform into an array, or an ndarray straightaway ? From darryl.wallace at prosensus.ca Tue Nov 10 15:57:22 2009 From: darryl.wallace at prosensus.ca (Darryl Wallace) Date: Tue, 10 Nov 2009 15:57:22 -0500 Subject: [Numpy-discussion] Numpy Array of dtype=object with strings and floats question In-Reply-To: <31ECC99C-AA62-48CC-BBA7-4B63DF9D9DA5@gmail.com> References: <7abc62af0911061256i681535and29dd0e86a0647de@mail.gmail.com> <7abc62af0911101009x194d517fn581c40d57bc52bb3@mail.gmail.com> <31ECC99C-AA62-48CC-BBA7-4B63DF9D9DA5@gmail.com> Message-ID: <7abc62af0911101257l4521e93fg96dd4e764c4de971@mail.gmail.com> Hello! On Tue, Nov 10, 2009 at 2:23 PM, Pierre GM wrote: > > On Nov 10, 2009, at 1:09 PM, Darryl Wallace wrote: > > > Hello again, > > > > The best way so far that's come to my attention is to use: > > > > numpy.ma.masked_object > > Will only work for masking one specific string, as you've noticed. > > > > Can anyone help me so that all strings are found in the array without > having to explicitly loop through them in Python? > > This looks like it's working: > >>> mask = (mixed >= '') > >>> mixed = ma.array(np.where(~mask,mixed,np.nan),mask=mask,dtype=float) > This method worked the best and fastest. Thanks for your help on this one :) > > > On Fri, Nov 6, 2009 at 3:56 PM, Darryl Wallace < > darryl.wallace at prosensus.ca> wrote: > > What I'm doing is importing some data from excel and sometimes there are > strings in the worksheet. > > Now, what are you using to read the Excel file ? Do you get a list that you > transform into an array, or an ndarray straightaway ? It depends, I will use Excel if it's available otherwise I will use XLRD. Using win32com, excel returns a tuple of tuples from the range that is selected. XLRD returns a list of lists. To keep things consistent I return a numpy array straight away. darryl -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Nov 10 16:18:07 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 10 Nov 2009 16:18:07 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> Message-ID: <4AF9D88F.2000906@stsci.edu> I don't know if your 'long double' detection code is complete yet, but I thought I'd share the current build output on one of our Solaris machines. It looks like it may just be a typo difference between 'IEEE_QUAD_BE' in long_double_representation() and 'IEEE_QUAD_16B_BE' in setup.py, but I wanted to confirm. > uname -a SunOS grail.stsci.edu 5.8 Generic_117350-62 sun4u sparc SUNW,Ultra-5_10 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/stsci/pyssgdev/Python-2.5.4/include/python2.5 -c' cc: _configtest.c removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/install.py", line 52, in run r = old_install.run(self) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/stsci/pyssgdev/Python-2.5.4/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/grail/data1/iraf/work/build.grail.numpy.iraf/numpy/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 425, in generate_config_h raise ValueError("Unrecognized long double format: %s" % rep) ValueError: Unrecognized long double format: IEEE_QUAD_BE David Cournapeau wrote: > On Thu, Nov 5, 2009 at 4:55 AM, Michael Droettboom wrote: > >> David Cournapeau wrote: >> >>> On Thu, Nov 5, 2009 at 2:15 AM, Michael Droettboom wrote: >>> >>> >>>> I'm getting the following from r7603 on Solaris Sparc -- somehow related >>>> to not having a long double version of next after available. I realise >>>> not everyone has access to (or is dependent on) this platform, so I'm >>>> willing to help in whatever way I can, I'm just not sure I understand >>>> the change yet. >>>> >>>> >>> The only way to implement nextafter that I know of requires to know >>> the exact representation of the floating point number, and long double >>> is unfortunately platform dependent. >>> >>> What is the long double format on solaris sparc ? (big endian I >>> suppose, but how many bits for the mantissa and exponent ? Does it >>> follow IEER754 ?) >>> >>> >> I honestly don't know -- I've never had to use them. It would be great >> to solve this properly but it's difficult to find definitive information >> about these things. >> >> Assuming we can't solve this the "right" way before the next release, >> would it be possible for this to raise a runtime "NotImplemented" error >> (by not defining the LONGDOUBLE_nextafter ufunc) rather than raising a >> compiler error which prevents the build from completing? >> > > To be honest, I thought this condition would never arise (I am quite > surprised that solaris does not have nextafter - both BSD and GNU libc > use the Sun implementation for this function). > > If this is not fixed within the code freeze (16th november), the > feature will be removed altogether for 1.4.0. I don't want to go down > the road of different feature set for different platforms. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From cournape at gmail.com Tue Nov 10 18:37:35 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 11 Nov 2009 08:37:35 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AF9D88F.2000906@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> Message-ID: <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> On Wed, Nov 11, 2009 at 6:18 AM, Michael Droettboom wrote: > I don't know if your 'long double' detection code is complete yet, but I > thought I'd share the current build output on one of our Solaris > machines. ?It looks like it may just be a typo difference between > 'IEEE_QUAD_BE' in long_double_representation() and 'IEEE_QUAD_16B_BE' in > setup.py, but I wanted to confirm. Yes, you're right - I don't have any machine with a 128 bits long double type, so I let the typo slipped. Could you try last revision (r7726) ? David From Chris.Barker at noaa.gov Tue Nov 10 19:07:32 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 10 Nov 2009 16:07:32 -0800 Subject: [Numpy-discussion] finding close together points. Message-ID: <4AFA0044.6020106@noaa.gov> Hi all, I have a bunch of points in 2-d space, and I need to find out which pairs of points are within a certain distance of one-another (regular old Euclidean norm). scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. However, I'm a bit confused. The first argument is a kdtree, but I'm calling it as a method of a kdtree -- I want to know which points in the tree I already have are closer that some r from each-other. If I call it as: tree.query_ball_tree(tree, r) I get a big list, that has all the points in it (some of them paired up with close neighbors.) It appears I'm getting the distances between all the points in the tree and itself, as though they were different trees. This is slow, takes a bunch of memory, and I then have to parse out the list to find the ones that are paired up. Is there a way to get just the close ones from the single tree? thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From peridot.faceted at gmail.com Tue Nov 10 19:48:14 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 10 Nov 2009 19:48:14 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA0044.6020106@noaa.gov> References: <4AFA0044.6020106@noaa.gov> Message-ID: 2009/11/10 Christopher Barker : > Hi all, > > I have a bunch of points in 2-d space, and I need to find out which > pairs of points are within a certain distance of one-another (regular > old Euclidean norm). This is an eminently reasonable thing to want, and KDTree should support it. Unfortunately it doesn't. > scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. > > However, I'm a bit confused. The first argument is a kdtree, but I'm > calling it as a method of a kdtree -- I want to know which points in the > tree I already have are closer that some r from each-other. > > If I call it as: > > tree.query_ball_tree(tree, r) > > I get a big list, that has all the points in it (some of them paired up > with close neighbors.) It appears I'm getting the distances between all > the points in the tree and itself, as though they were different trees. > > This is slow, takes a bunch of memory, and I then have to parse out the > list to find the ones that are paired up. > > Is there a way to get just the close ones from the single tree? Unfortunately not at the moment. It's slow both because you're using the python implementation rather than the C, and because you're getting all "pairs" where "pair" includes pairing a point with itself (and also each distinct pair in both orders). The tree really should allow self-queries that don't return the point and itself. The one good thing about using the python implementation rather than the Cython one is that you can subclass it, providing a new method. There's still a certain amount of boilerplate code to write, but it shouldn't be too bad. If this is still too slow, I have no problem incorporating additional code into cKDTree; the only reason none of the ball queries are in there is because ball queries must return variable-size results, so you lose a certain amount of speed because you're forced to manipulate python objects. But if there are relatively few result points, this need not be much of a slowdown. Anne > thanks, > > -Chris > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bergstrj at iro.umontreal.ca Tue Nov 10 20:08:40 2009 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Tue, 10 Nov 2009 20:08:40 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA0044.6020106@noaa.gov> References: <4AFA0044.6020106@noaa.gov> Message-ID: <7f1eaee30911101708va5d7498m8bf950076e11bdcc@mail.gmail.com> On Tue, Nov 10, 2009 at 7:07 PM, Christopher Barker wrote: > Hi all, > > I have a bunch of points in 2-d space, and I need to find out which > pairs of points are within a certain distance of one-another (regular > old Euclidean norm). > > scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. > In some cases a brute-force approach is also good. If r is a matrix of shape Nx2: (r*r).sum(axis=1) -2 * numpy.dot(r, r.T) + (r*r).sum(axis=1).reshape((r.shape[0], 1)) < thresh**2 It's brute force, but it takes advantage of fast matrix multiplication. -- http://www-etud.iro.umontreal.ca/~bergstrj From Chris.Barker at noaa.gov Tue Nov 10 20:17:50 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 10 Nov 2009 17:17:50 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <7f1eaee30911101708va5d7498m8bf950076e11bdcc@mail.gmail.com> References: <4AFA0044.6020106@noaa.gov> <7f1eaee30911101708va5d7498m8bf950076e11bdcc@mail.gmail.com> Message-ID: <4AFA10BE.4000507@noaa.gov> James Bergstra wrote: > In some cases a brute-force approach is also good. true. > If r is a matrix of shape Nx2: > > (r*r).sum(axis=1) -2 * numpy.dot(r, r.T) + > (r*r).sum(axis=1).reshape((r.shape[0], 1)) < thresh**2 > > It's brute force, but it takes advantage of fast matrix multiplication. I'm more concerned about memory -- doesn't this use N^2 memory? Which could be an issue here. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bergstrj at iro.umontreal.ca Tue Nov 10 20:22:11 2009 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Tue, 10 Nov 2009 20:22:11 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA10BE.4000507@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <7f1eaee30911101708va5d7498m8bf950076e11bdcc@mail.gmail.com> <4AFA10BE.4000507@noaa.gov> Message-ID: <7f1eaee30911101722i7c627cf1y265bbf58e7a7cec@mail.gmail.com> On Tue, Nov 10, 2009 at 8:17 PM, Christopher Barker wrote: > James Bergstra wrote: >> In some cases a brute-force approach is also good. > > true. > >> If r is a matrix of shape Nx2: >> >> (r*r).sum(axis=1) -2 * numpy.dot(r, r.T) + >> (r*r).sum(axis=1).reshape((r.shape[0], 1)) < thresh**2 >> >> It's brute force, but it takes advantage of fast matrix multiplication. > > I'm more concerned about memory -- doesn't this use N^2 memory? Which > could be an issue here. Yes, this uses N^2 time and space. It's not a good algorithm when N is large. -- http://www-etud.iro.umontreal.ca/~bergstrj From d.l.goldsmith at gmail.com Tue Nov 10 20:22:52 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Tue, 10 Nov 2009 17:22:52 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA10BE.4000507@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <7f1eaee30911101708va5d7498m8bf950076e11bdcc@mail.gmail.com> <4AFA10BE.4000507@noaa.gov> Message-ID: <45d1ab480911101722o5f61d0eag4cae312a0c5ba5aa@mail.gmail.com> Also, is it not returning distances between points and themselves? Or am I misinterpreting it? DG On Tue, Nov 10, 2009 at 5:17 PM, Christopher Barker wrote: > James Bergstra wrote: > > In some cases a brute-force approach is also good. > > true. > > > If r is a matrix of shape Nx2: > > > > (r*r).sum(axis=1) -2 * numpy.dot(r, r.T) + > > (r*r).sum(axis=1).reshape((r.shape[0], 1)) < thresh**2 > > > > It's brute force, but it takes advantage of fast matrix multiplication. > > I'm more concerned about memory -- doesn't this use N^2 memory? Which > could be an issue here. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 10 21:30:47 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Nov 2009 21:30:47 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> Message-ID: <1cd32cbb0911101830s4a600a8x8ce5d6f47ac9a98e@mail.gmail.com> On Tue, Nov 10, 2009 at 7:48 PM, Anne Archibald wrote: > 2009/11/10 Christopher Barker : >> Hi all, >> >> I have a bunch of points in 2-d space, and I need to find out which >> pairs of points are within a certain distance of one-another (regular >> old Euclidean norm). > > This is an eminently reasonable thing to want, and KDTree should > support it. Unfortunately it doesn't. > >> scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. >> >> However, I'm a bit confused. The first argument is a kdtree, but I'm >> calling it as a method of a kdtree -- I want to know which points in the >> tree I already have are closer that some r from each-other. >> >> If I call it as: >> >> tree.query_ball_tree(tree, r) >> >> I get a big list, that has all the points in it (some of them paired up >> with close neighbors.) It appears I'm getting the distances between all >> the points in the tree and itself, as though they were different trees. >> >> This is slow, takes a bunch of memory, and I then have to parse out the >> list to find the ones that are paired up. >> >> Is there a way to get just the close ones from the single tree? I used sparse_distance_matrix for the distance of a kdtree to itself in the past. Since it's a matrix it should be possible to just get the lower or upper triangle, and it's sparse so memory is not so much of a problem. But I remember it was also slow and only worth using if the matrix is large. Josef > > Unfortunately not at the moment. > > It's slow both because you're using the python implementation rather > than the C, and because you're getting all "pairs" where "pair" > includes pairing a point with itself (and also each distinct pair in > both orders). The tree really should allow self-queries that don't > return the point and itself. > > The one good thing about using the python implementation rather than > the Cython one is that you can subclass it, providing a new method. > There's still a certain amount of boilerplate code to write, but it > shouldn't be too bad. > > If this is still too slow, I have no problem incorporating additional > code into cKDTree; the only reason none of the ball queries are in > there is because ball queries must return variable-size results, so > you lose a certain amount of speed because you're forced to manipulate > python objects. But if there are relatively few result points, this > need not be much of a slowdown. > > Anne > >> thanks, >> >> -Chris >> >> >> >> >> >> >> -- >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice >> 7600 Sand Point Way NE ? (206) 526-6329 ? fax >> Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception >> >> Chris.Barker at noaa.gov >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From fonnesbeck at gmail.com Tue Nov 10 21:55:48 2009 From: fonnesbeck at gmail.com (Chris) Date: Wed, 11 Nov 2009 02:55:48 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 Message-ID: I am building Numpy on OSX 10.6 using a recent update from SVN (r7726). Though I was able to build the package successfully, the resulting package generates an ImportError: import umath ImportError: dlopen(/Library/Python/2.6/site-packages/ numpy-1.4.0.dev7726-py2.6-macosx-10.6- universal.egg/numpy/core/umath.so, 2): Symbol not found : _npy_cexp Referenced from: /Library/Python/2.6/site-packages/ numpy-1.4.0.dev7726-py2.6-macosx-10.6-universal.egg/ numpy/core/umath.so Expected in: flat namespace in /Library/Python/2.6/site-packages/ numpy-1.4.0.dev7726-py2.6-macosx-10.6-universal.egg /numpy/core/umath.so Note that this did not occur previously (r7542). For some reason umath is not being built properly. From david at ar.media.kyoto-u.ac.jp Tue Nov 10 21:58:34 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 11 Nov 2009 11:58:34 +0900 Subject: [Numpy-discussion] Import error in builds of 7726 In-Reply-To: References: Message-ID: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> Chris wrote: > I am building Numpy on OSX 10.6 using a recent update > from SVN (r7726). Though I was able to build the package > successfully, the resulting package generates an ImportError: > > import umath > ImportError: dlopen(/Library/Python/2.6/site-packages/ > numpy-1.4.0.dev7726-py2.6-macosx-10.6- > universal.egg/numpy/core/umath.so, 2): Symbol not found > : _npy_cexp > Referenced from: /Library/Python/2.6/site-packages/ > numpy-1.4.0.dev7726-py2.6-macosx-10.6-universal.egg/ > numpy/core/umath.so > Expected in: flat namespace > in /Library/Python/2.6/site-packages/ > numpy-1.4.0.dev7726-py2.6-macosx-10.6-universal.egg > /numpy/core/umath.so > > Note that this did not occur previously (r7542). For some reason > umath is not being built properly. > Did you make sure to build from scratch and clean the working directory first ? I don't see the error on my macbook: umath.so has npy_cexp* functions defined. David From Chris.Barker at noaa.gov Wed Nov 11 00:51:16 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 10 Nov 2009 21:51:16 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> Message-ID: <4AFA50D4.70403@noaa.gov> Anne Archibald wrote: > 2009/11/10 Christopher Barker : >> I have a bunch of points in 2-d space, and I need to find out which >> pairs of points are within a certain distance of one-another (regular >> old Euclidean norm). > > This is an eminently reasonable thing to want, and KDTree should > support it. Unfortunately it doesn't. darn. > It's slow both because you're using the python implementation rather > than the C, I noticed that. > The one good thing about using the python implementation rather than > the Cython one is that you can subclass it, providing a new method. > There's still a certain amount of boilerplate code to write, but it > shouldn't be too bad. Can you give me a hint as to where to start? I have no idea how kd trees work. > If this is still too slow, I have no problem incorporating additional > code into cKDTree; the only reason none of the ball queries are in > there is because ball queries must return variable-size results, so > you lose a certain amount of speed because you're forced to manipulate > python objects. But if there are relatively few result points, this > need not be much of a slowdown. And certainly in this case, there would not be very many results, and lists are pretty fast. However, this does bring up the need for a growable numpy array. I've been working on one in Python, but it would be really nice to have one that could be accessed by C (or cython) code. It wouldn't be that hard to do (for someone familiar with the numpy code, anyway), I don't think. It is mostly boiler plate in Python. I'm imagining it would only support contiguous arrays, and could only grow is the first dimension. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Nov 11 01:00:13 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 Nov 2009 23:00:13 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA50D4.70403@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> Message-ID: On Tue, Nov 10, 2009 at 10:51 PM, Christopher Barker wrote: > Anne Archibald wrote: > > 2009/11/10 Christopher Barker : > > >> I have a bunch of points in 2-d space, and I need to find out which > >> pairs of points are within a certain distance of one-another (regular > >> old Euclidean norm). > > > > This is an eminently reasonable thing to want, and KDTree should > > support it. Unfortunately it doesn't. > > darn. > > > It's slow both because you're using the python implementation rather > > than the C, > > I noticed that. > > > The one good thing about using the python implementation rather than > > the Cython one is that you can subclass it, providing a new method. > > There's still a certain amount of boilerplate code to write, but it > > shouldn't be too bad. > > Can you give me a hint as to where to start? I have no idea how kd > trees work. > > > If this is still too slow, I have no problem incorporating additional > > code into cKDTree; the only reason none of the ball queries are in > > there is because ball queries must return variable-size results, so > > you lose a certain amount of speed because you're forced to manipulate > > python objects. But if there are relatively few result points, this > > need not be much of a slowdown. > > And certainly in this case, there would not be very many results, and > lists are pretty fast. > > However, this does bring up the need for a growable numpy array. I've > been working on one in Python, but it would be really nice to have one > that could be accessed by C (or cython) code. > > It wouldn't be that hard to do (for someone familiar with the numpy > code, anyway), I don't think. It is mostly boiler plate in Python. I'm > imagining it would only support contiguous arrays, and could only grow > is the first dimension. > > > I think Python lists are basically just expanding arrays and pointers are cheap. Where you might lose is in creating python objects to put in the list and not having ufuncs and the rest of the numpy machinery. If you don't need the machinery, lists are probably not a bad way to go. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Nov 11 01:15:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 11 Nov 2009 15:15:39 +0900 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> Message-ID: <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I think Python lists are basically just expanding arrays and pointers > are cheap. Where you might lose is in creating python objects to put > in the list and not having ufuncs and the rest of the numpy machinery. > If you don't need the machinery, lists are probably not a bad way to go. I am not familiar enough with the kdtree code: if you need to manipulate core datatypes (float, etc...), I think python lists are a significant cost, because of pointer chasing in particular. It may or may not apply, but Stefan used some tricks to avoid using python list and use basic C structures in some custom code (written in cython), maybe he has something to say. I would love having a core C library of containers - there are a few candidates we could take code from: - http://github.com/stevedekorte/basekit: used in IO programming language. Quite a few goodies, from list to trees to sets to portable code to load dynamically loaded libraries. BSD - The Apple Core Foundation (I have to check the Apple license is ok). It is more complex, but is designed with objective-C in mind, meaning integration with the C python API may be easier (both Obj-C and Python C API being based on ref counting). - Another one mentioned by Robert Kern, I forgot the reference. I hope to make it my main focus for the 1.5 release, cheers, David From robert.kern at gmail.com Wed Nov 11 01:42:42 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 11 Nov 2009 01:42:42 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911102242k46f3d87dge0a150e26a27fc81@mail.gmail.com> On Wed, Nov 11, 2009 at 01:15, David Cournapeau wrote: > ? ?- The Apple Core Foundation (I have to check the Apple license is > ok). No. The APSL is not DFSG-free. > It is more complex, but is designed with objective-C in mind, > meaning integration with the C python API may be easier (both Obj-C and > Python C API being based on ref counting). > ? ?- Another one mentioned by Robert Kern, I forgot the reference. http://c-algorithms.sourceforge.net/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Wed Nov 11 01:40:53 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 11 Nov 2009 15:40:53 +0900 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <3d375d730911102242k46f3d87dge0a150e26a27fc81@mail.gmail.com> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <3d375d730911102242k46f3d87dge0a150e26a27fc81@mail.gmail.com> Message-ID: <4AFA5C75.7080203@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > No. The APSL is not DFSG-free. > It was too good to be true, I guess. > > http://c-algorithms.sourceforge.net/ > Thanks for the link, David From pschmidtke at mmb.pcb.ub.es Wed Nov 11 02:30:57 2009 From: pschmidtke at mmb.pcb.ub.es (Peter Schmidtke) Date: Wed, 11 Nov 2009 08:30:57 +0100 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA0044.6020106@noaa.gov> References: <4AFA0044.6020106@noaa.gov> Message-ID: <94d043d59126cf48e77b4b1a0014d243@mmb.pcb.ub.es> On Tue, 10 Nov 2009 16:07:32 -0800, Christopher Barker wrote: > Hi all, > > I have a bunch of points in 2-d space, and I need to find out which > pairs of points are within a certain distance of one-another (regular > old Euclidean norm). How big is your set of points? > > scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. > > However, I'm a bit confused. The first argument is a kdtree, but I'm > calling it as a method of a kdtree -- I want to know which points in the > tree I already have are closer that some r from each-other. > > If I call it as: > > tree.query_ball_tree(tree, r) > > I get a big list, that has all the points in it (some of them paired up > with close neighbors.) It appears I'm getting the distances between all > the points in the tree and itself, as though they were different trees. > > This is slow, takes a bunch of memory, and I then have to parse out the > list to find the ones that are paired up. > > Is there a way to get just the close ones from the single tree? > > thanks, > > -Chris -- Peter Schmidtke ---------------------- PhD Student at the Molecular Modeling and Bioinformatics Group Dep. Physical Chemistry Faculty of Pharmacy University of Barcelona From seb.haase at gmail.com Wed Nov 11 03:30:55 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 11 Nov 2009 09:30:55 +0100 Subject: [Numpy-discussion] unittest for np.random.poisson with lambda=0 Message-ID: Hi, maybe this is an obsolete concern, but at some point in the past N.random.poisson would not always return 0 for lambda being zero. (My post at the time should be in the (numarray) list archive) So the question is, is this still a valid concern for the current numpy? Could there a unittest added that does something like this: >>> a=N.random.poisson(0, 10000) >>> N.alltrue(a==0) True Regards, Sebastian Haase From lou_boog2000 at yahoo.com Wed Nov 11 08:35:52 2009 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Wed, 11 Nov 2009 05:35:52 -0800 (PST) Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA0044.6020106@noaa.gov> References: <4AFA0044.6020106@noaa.gov> Message-ID: <348618.28546.qm@web34403.mail.mud.yahoo.com> ----- Original Message ---- From: Christopher Barker To: Discussion of Numerical Python Sent: Tue, November 10, 2009 7:07:32 PM Subject: [Numpy-discussion] finding close together points. Hi all, I have a bunch of points in 2-d space, and I need to find out which pairs of points are within a certain distance of one-another (regular old Euclidean norm). scipy.spatial.KDTree.query_ball_tree() seems like it's built for this. ---- Chris, Maybe I'm missing something simple, but if your array of 2D points is static, a KD tree for 2D nearest neighbor seems like over kill. You might want to try the simple approach of using boxes of points to narrow things down by sorting on the first component. If your distances are, say, 10% of the variance, then you'll *roughly* decrease the remaining points to search by a factor of 10. This can get more sophisticated and is useful in higher dimensions (see: Sameer A. Nene and Shree K. Nayar, A Simple Algorithm for Nearest Neighbor Search in High Dimensions, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 19 (9), 989 (1997).) where for a static data set it can match KD trees in speed, but is SO much easier to program. -- Lou Pecora, my views are my own. From charlesr.harris at gmail.com Wed Nov 11 10:14:57 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 11 Nov 2009 08:14:57 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Nov 10, 2009 at 11:15 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Charles R Harris wrote: > > > > I think Python lists are basically just expanding arrays and pointers > > are cheap. Where you might lose is in creating python objects to put > > in the list and not having ufuncs and the rest of the numpy machinery. > > If you don't need the machinery, lists are probably not a bad way to go. > > I am not familiar enough with the kdtree code: if you need to manipulate > core datatypes (float, etc...), I think python lists are a significant > cost, because of pointer chasing in particular. It may or may not apply, > but Stefan used some tricks to avoid using python list and use basic C > structures in some custom code (written in cython), maybe he has > something to say. > > I would love having a core C library of containers - there are a few > candidates we could take code from: > I've got a cythonized structure for union-find (relations) and we could put together a heap pretty easily from heap sort. I've also got one for the problem at hand, but it is for several thousand points (stars) scattered over an image and uses c++ lists. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Wed Nov 11 10:33:59 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 11 Nov 2009 10:33:59 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> Message-ID: <4AFAD967.7030207@stsci.edu> Thanks. Behind that, however, it runs into this compiler shortcoming: cc: build/src.solaris-2.8-sun4u-2.5/numpy/core/src/npymath/npy_math.c "numpy/core/src/npymath/npy_math_private.h", line 229: invalid type for bit-field: manh "numpy/core/src/npymath/npy_math_private.h", line 230: invalid type for bit-field: manl It seems that Sun compilers (before Sun Studio 12, C compiler 5.9), don't support bit fields larger than 32-bits. Welcome to my world :) You can see a discussion of this here: http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02634.html and the release notes for Sun C compiler 5.9 here: http://docs.sun.com/source/820-4155/c.html The best solution I've been able to dream up is to use a macro to get at the manh bitfield on these older compilers. This fix does seem to pass the "test_umath.py" unit tests. It's not as clean as the bit-field method, but it should be more portable. You can see what I'm getting at in the attached (rough) patch. Can you think of a better solution? If this is *the* solution, we may want to add getters/setters for all the members (sign, exp, manl) and fix nextafterl in terms of those, just for consistency. Cheers, Mike David Cournapeau wrote: > On Wed, Nov 11, 2009 at 6:18 AM, Michael Droettboom wrote: > >> I don't know if your 'long double' detection code is complete yet, but I >> thought I'd share the current build output on one of our Solaris >> machines. It looks like it may just be a typo difference between >> 'IEEE_QUAD_BE' in long_double_representation() and 'IEEE_QUAD_16B_BE' in >> setup.py, but I wanted to confirm. >> > > Yes, you're right - I don't have any machine with a 128 bits long > double type, so I let the typo slipped. > > Could you try last revision (r7726) ? > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From mdroe at stsci.edu Wed Nov 11 10:36:08 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 11 Nov 2009 10:36:08 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AFAD967.7030207@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> Message-ID: <4AFAD9E8.70401@stsci.edu> Forgot to attach the patch. Mike Michael Droettboom wrote: > Thanks. Behind that, however, it runs into this compiler shortcoming: > > cc: build/src.solaris-2.8-sun4u-2.5/numpy/core/src/npymath/npy_math.c > "numpy/core/src/npymath/npy_math_private.h", line 229: invalid type > for bit-field: manh > "numpy/core/src/npymath/npy_math_private.h", line 230: invalid type > for bit-field: manl > > It seems that Sun compilers (before Sun Studio 12, C compiler 5.9), > don't support bit fields larger than 32-bits. Welcome to my world :) > > You can see a discussion of this here: > > http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02634.html > > and the release notes for Sun C compiler 5.9 here: > > http://docs.sun.com/source/820-4155/c.html > > The best solution I've been able to dream up is to use a macro to get > at the manh bitfield on these older compilers. This fix does seem to > pass the "test_umath.py" unit tests. It's not as clean as the > bit-field method, but it should be more portable. You can see what > I'm getting at in the attached (rough) patch. > > Can you think of a better solution? If this is *the* solution, we may > want to add getters/setters for all the members (sign, exp, manl) and > fix nextafterl in terms of those, just for consistency. > > Cheers, > Mike > > David Cournapeau wrote: >> On Wed, Nov 11, 2009 at 6:18 AM, Michael Droettboom >> wrote: >> >>> I don't know if your 'long double' detection code is complete yet, >>> but I >>> thought I'd share the current build output on one of our Solaris >>> machines. It looks like it may just be a typo difference between >>> 'IEEE_QUAD_BE' in long_double_representation() and >>> 'IEEE_QUAD_16B_BE' in >>> setup.py, but I wanted to confirm. >>> >> >> Yes, you're right - I don't have any machine with a 128 bits long >> double type, so I let the typo slipped. >> >> Could you try last revision (r7726) ? >> >> David >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: sun_long_double.patch URL: From cournape at gmail.com Wed Nov 11 10:45:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 12 Nov 2009 00:45:39 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AFAD967.7030207@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> Message-ID: <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> On Thu, Nov 12, 2009 at 12:33 AM, Michael Droettboom wrote: > Thanks. ?Behind that, however, it runs into this compiler shortcoming: > > cc: build/src.solaris-2.8-sun4u-2.5/numpy/core/src/npymath/npy_math.c > "numpy/core/src/npymath/npy_math_private.h", line 229: invalid type for bit-field: manh > "numpy/core/src/npymath/npy_math_private.h", line 230: invalid type for bit-field: manl > > It seems that Sun compilers (before Sun Studio 12, C compiler 5.9), > don't support bit fields larger than 32-bits. ?Welcome to my world :) Damn... > > The best solution I've been able to dream up is to use a macro to get at > the manh bitfield on these older compilers. ?This fix does seem to pass > the "test_umath.py" unit tests. ?It's not as clean as the bit-field > method, but it should be more portable. I think it is the only solution, actually. Actually, I wanted to use bit mask at first, because I thought bitfield was a C99 feature, not supported by MS compilers (although it actually is). > Can you think of a better solution? ?If this is *the* solution, we may > want to add getters/setters for all the members (sign, exp, manl) and > fix nextafterl in terms of those, just for consistency. I will implement this, but I would prefer using this method everywhere for every compiler, it would be more robust. David From perfreem at gmail.com Wed Nov 11 12:53:43 2009 From: perfreem at gmail.com (per freem) Date: Wed, 11 Nov 2009 12:53:43 -0500 Subject: [Numpy-discussion] parsing tab separated files into dictionaries - alternative to genfromtxt? Message-ID: hi all, i've been using genfromtxt to parse tab separated files for plotting purposes in matplotlib. the problem is that genfromtxt seems to give only two ways to access the contents of the file: one is by column, where you can use: d = genfromtxt(...) and then do d['header_name1'] to access the column named by 'header_name1', d['header_name2'] to access the column named by 'header_name2', etc. Or it will allow you to traverse the file line by line, and then access each header by number, i.e. for line in d: field1 = d[0] field2 = d[1] # etc. the problem is that the second method relies on knowing the order of the fields rather than just their name, and the first method does not allow line by line iteration. ideally what i would like is to be able to traverse each line of the parsed file, and then refer to each of its fields by header name, so that if the column order in the file changes my program will be unaffected: for line in d: field1 = ['header_name1'] field2 = ['header_name2'] is there a way to do this using standard matplotlib/numpy/scipy utilities? i could write my own code to do this but it seems like something somebody probably already thought of a good representation for and has implemented a more optimized version than i could write on my own. does such a thing exist? thanks very much From gokhansever at gmail.com Wed Nov 11 13:16:22 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 11 Nov 2009 12:16:22 -0600 Subject: [Numpy-discussion] parsing tab separated files into dictionaries - alternative to genfromtxt? In-Reply-To: References: Message-ID: <49d6b3500911111016x3e341ce9m66d21fed8d91f441@mail.gmail.com> On Wed, Nov 11, 2009 at 11:53 AM, per freem wrote: > hi all, > > i've been using genfromtxt to parse tab separated files for plotting > purposes in matplotlib. the problem is that genfromtxt seems to give > only two ways to access the contents of the file: one is by column, > where you can use: > > d = genfromtxt(...) > > and then do d['header_name1'] to access the column named by > 'header_name1', d['header_name2'] to access the column named by > 'header_name2', etc. ?Or it will allow you to traverse the file line > by line, and then access each header by number, i.e. > > for line in d: > ?field1 = d[0] > ?field2 = d[1] > ?# etc. > > the problem is that the second method relies on knowing the order of > the fields rather than just their name, and the first method does not > allow line by line iteration. > ideally what i would like is to be able to traverse each line of the > parsed file, and then refer to each of its fields by header name, so > that if the column order in the file changes my program will be > unaffected: > > for line in d: > ?field1 = ['header_name1'] > ?field2 = ['header_name2'] > > is there a way to do this using standard matplotlib/numpy/scipy > utilities? i could write my own code to do this but it seems like > something somebody probably already thought of a good representation > for and has implemented a more optimized version than i could write on > my own. does such a thing exist? > > thanks very much I have a constructor class to read space-delimited ASCII files. class NasaFile(object): def __init__(self, filename): ... # Reading data _data = np.loadtxt(filename, dtype='float', skiprows=self.NLHEAD).T # Read using data['Time'] syntax self.data = dict(zip(self.VDESC, _data)) ... There is a meta-header in this type of data and NLHEAD is the variable telling me how many lines to skip to reach the actual data. VDESC tells me what each columns are (starting with Time variable and many other different measurement results.) There is not any column dependence in this case, and generically read any length specifically formatted data. For instance: from nasafile import NasaFile c = NasaFile("mydata") c.data['Time'] gets me the whole Time column as an ndarray . Why do you think dictionaries are not sufficient for your case? I was using locals() to create automatic names but that was not a very wise approach. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- G?khan From robert.kern at gmail.com Wed Nov 11 14:53:32 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 11 Nov 2009 13:53:32 -0600 Subject: [Numpy-discussion] unittest for np.random.poisson with lambda=0 In-Reply-To: References: Message-ID: <3d375d730911111153t4477a575ob9a3c6d77fb32e11@mail.gmail.com> On Wed, Nov 11, 2009 at 02:30, Sebastian Haase wrote: > Hi, > maybe this is an obsolete concern, > but at some point in the past > N.random.poisson would not always return 0 for lambda being zero. > (My post at the time should be in the (numarray) ?list archive) > > So the question is, is this still a valid concern for the current numpy? No. > Could there a unittest added that does something like this: > >>>> a=N.random.poisson(0, 10000) >>>> N.alltrue(a==0) > True Write up a patch. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mdroe at stsci.edu Wed Nov 11 16:03:30 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 11 Nov 2009 16:03:30 -0500 Subject: [Numpy-discussion] Solaris atan2 differences Message-ID: <4AFB26A2.7030804@stsci.edu> There are a set of three unit test failures on Solaris all related to differences in the atan2 implementation, reported as bugs 1201, 1202 and 1203. They boil down to the following three differences between Solaris (C compiler 5.3) and "other" platforms -- when I say "other" I mean at least Linux x86 and x86_64, and Mac OS X x86. >>> import numpy as np >>> import numpy.core.umath as ncu >>> ncu.arctan2(np.PZERO, np.NZERO) 0.0 (Solaris) 3.1415926535897931 (other) >>> ncu.arctan2(np.NZERO, np.NZERO) 0.0 (Solaris) -3.1415926535897931 (other) >>> np.signbit(ncu.arctan2(np.NZERO, np.PZERO)) False (Solaris) True (other) The easy fix seems to be to force it to use the npy_math version of atan2 on Solaris (as we already do for MS Windows). (Patch attached). Indeed this fixes the unit tests. Does that seem right? Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: solaris_atan.patch URL: From seb.haase at gmail.com Wed Nov 11 16:47:48 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 11 Nov 2009 22:47:48 +0100 Subject: [Numpy-discussion] unittest for np.random.poisson with lambda=0 In-Reply-To: <3d375d730911111153t4477a575ob9a3c6d77fb32e11@mail.gmail.com> References: <3d375d730911111153t4477a575ob9a3c6d77fb32e11@mail.gmail.com> Message-ID: Thanks Robert, Just for the archive: I hope the problem I was thinking of is what is referred to here: http://mail.scipy.org/pipermail/numpy-discussion/2005-July/004992.html i.e. it should have been fixed as bug #1123145 in Numeric. (see also here: https://nanohub.org/infrastructure/rappture-runtime/browser/trunk/Numeric-24.2/changes.txt?rev=42 ) I still haven't learned how to write unittests .... (sorry) - Sebastian Haase On Wed, Nov 11, 2009 at 8:53 PM, Robert Kern wrote: > On Wed, Nov 11, 2009 at 02:30, Sebastian Haase wrote: >> Hi, >> maybe this is an obsolete concern, >> but at some point in the past >> N.random.poisson would not always return 0 for lambda being zero. >> (My post at the time should be in the (numarray) ?list archive) >> >> So the question is, is this still a valid concern for the current numpy? > > No. > >> Could there a unittest added that does something like this: >> >>>>> a=N.random.poisson(0, 10000) >>>>> N.alltrue(a==0) >> True > > Write up a patch. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Wed Nov 11 16:51:14 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 11 Nov 2009 15:51:14 -0600 Subject: [Numpy-discussion] unittest for np.random.poisson with lambda=0 In-Reply-To: References: <3d375d730911111153t4477a575ob9a3c6d77fb32e11@mail.gmail.com> Message-ID: <3d375d730911111351k595876f9wde2aca41d2c23d9e@mail.gmail.com> On Wed, Nov 11, 2009 at 15:47, Sebastian Haase wrote: > Thanks Robert, > Just for the archive: I hope the problem I was thinking of is what is > referred to here: > http://mail.scipy.org/pipermail/numpy-discussion/2005-July/004992.html > i.e. it should have been fixed as bug #1123145 in Numeric. > > (see also here: > https://nanohub.org/infrastructure/rappture-runtime/browser/trunk/Numeric-24.2/changes.txt?rev=42 > ) The PRNG is completely different in numpy. > I still haven't learned how to write unittests .... (sorry) If this issue is important to you, now might be a good time to start. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fonnesbeck at gmail.com Wed Nov 11 16:57:09 2009 From: fonnesbeck at gmail.com (Chris) Date: Wed, 11 Nov 2009 21:57:09 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > Did you make sure to build from scratch and clean the working directory > first ? > > I don't see the error on my macbook: umath.so has npy_cexp* functions > defined. > > David > Yeah, here is my build script -- it removes the build directory entirely: #!/bin/sh export MACOSX_DEPLOYMENT_TARGET=10.6 export CFLAGS="-arch i386 -arch x86_64" export FFLAGS="-arch i386 -arch x86_64" export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch i386 -arch x86_64" rm -rf build python setupegg.py config_fc --fcompiler gfortran config - L/Developer/src/freetype-2.3.5 -L/Developer/src/libpng-1.2.24 build bdist_egg From fonnesbeck at gmail.com Wed Nov 11 17:13:00 2009 From: fonnesbeck at gmail.com (Chris) Date: Wed, 11 Nov 2009 22:13:00 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> Message-ID: David Cournapeau ar.media.kyoto-u.ac.jp> writes: > > Did you make sure to build from scratch and clean the working directory > first ? > > I don't see the error on my macbook: umath.so has npy_cexp* functions > defined. > > David > Just now tried deleting everything and pulling down numpy again from SVN. Same result. From david at ar.media.kyoto-u.ac.jp Thu Nov 12 00:15:20 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 12 Nov 2009 14:15:20 +0900 Subject: [Numpy-discussion] unittest for np.random.poisson with lambda=0 In-Reply-To: References: <3d375d730911111153t4477a575ob9a3c6d77fb32e11@mail.gmail.com> Message-ID: <4AFB99E8.1090300@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > Thanks Robert, > Just for the archive: I hope the problem I was thinking of is what is > referred to here: > http://mail.scipy.org/pipermail/numpy-discussion/2005-July/004992.html > i.e. it should have been fixed as bug #1123145 in Numeric. > > (see also here: > https://nanohub.org/infrastructure/rappture-runtime/browser/trunk/Numeric-24.2/changes.txt?rev=42 > ) > > I still haven't learned how to write unittests .... (sorry) That's very easy. Assuming you have a new random generator foo which should always return one, it may be as easy as writing: from numpy.random import foo def test_foo(): assert foo() == 1 In numpy/random/test_random. You can follow the nose doc (or other unit tests) for more complex examples. If you are unsure about how to write something, just create an issue to numpy trac with a patch, and regularly bother us until we review it. cheers, David From seb.haase at gmail.com Thu Nov 12 04:38:37 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 12 Nov 2009 10:38:37 +0100 Subject: [Numpy-discussion] does an ndarray have a "hook" for a dictionary attribute ? Message-ID: Hi, I hope my subject line is not entirely incomprehensible: I remember there was a discussion (some time ago) that every ndarray instance should get an extra dictionary (or just a "hook" for it, to minimize the memory foot print) for "application specific" / "auxiliary" data. What is the current state on this ? The issue at hand is, that my image analysis development platform supports various file types by now, and I usually load images straight into an ndarray (e.g. using PIL for standard types). This is fine for most needs depending only on the pixel values. However, now I would like to save images back after performing some operation on it. So it would be nice if I could "attach" the original filename to the respective ndarray. Is this possible with standard ndarray (using the mentioned dictionary attribute) or do I need to subclass ndarray to make such a ndarray_with_dict ? (For the latter it would be very helpful to get a hint on how to make such a "minimal" ndarray-subclass . I remember some issues involving some special __new__ methods, that I don't entirely remember by heard..) Thanks, Sebastian Haase From seb.haase at gmail.com Thu Nov 12 04:47:18 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 12 Nov 2009 10:47:18 +0100 Subject: [Numpy-discussion] does an ndarray have a "hook" for a dictionary attribute ? In-Reply-To: References: Message-ID: On Thu, Nov 12, 2009 at 10:38 AM, Sebastian Haase wrote: > Hi, > > I hope my subject line is not entirely incomprehensible: > I remember there was a discussion (some time ago) that every ndarray > instance should get an extra dictionary (or just a "hook" for it, to > minimize the memory foot print) for "application specific" / > "auxiliary" data. > What is the current state on this ? > > The issue at hand is, that my image analysis development platform > supports various file types by now, and I usually load images straight > into an ndarray (e.g. using PIL for standard types). > This is fine for most needs depending only on the pixel values. > However, now I would like to save images back after performing some > operation on it. > So it would be nice if I could "attach" the original filename to the > respective ndarray. > Is this possible with standard ndarray (using the mentioned dictionary > attribute) > or > do I need to subclass ndarray to make such a ndarray_with_dict > ? > > (For the latter it would be very helpful to get a hint on how to make > such a "minimal" ndarray-subclass . I remember some issues involving > some special __new__ methods, that I don't entirely remember by > heard..) > I found the thread I was thinking of in the archive: http://www.mail-archive.com/numpy-discussion at scipy.org/msg11898.html However, what was the outcome ... ? Sebastian Haase From josemaria.alkala at gmail.com Thu Nov 12 06:21:01 2009 From: josemaria.alkala at gmail.com (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Garc=EDa_P=E9rez?=) Date: Thu, 12 Nov 2009 12:21:01 +0100 Subject: [Numpy-discussion] Failed installation on Windows XP Message-ID: Good morning, I used to have Python 2.6, Numpy-1.3.0, ... on Windows 2000 in the computer at work (which has the typical clampdown policies). I have been updated to Windows XP SP3 and now the same installation files fail to install. My configuration is: - Windows XP Serivice Pack 3 - Processor: T7300 (Intel Core 2 Duo) - Python 2.6.4 r264:75708 The installation of: "numpy-1.3.0-win32-superpack-python2.6.exe" fails saying: Executing numpy installer failed When I press Show Details: Output folder: C:\DOCUME~1\user1\LOCALS~1\Temp Install dir for actual installers is C:\DOCUME~1\user1\LOCALS~1\Temp "Target CPU handles SSE2" "Target CPU handles SSE3" "native install (arch value: native)" "Install SSE 3" Extract: numpy-1.3.0-sse3.exe... 100% Execute: "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" Completed I suspect it might be a lack of a library. Openining "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" with "Dependency Walker" shows: "Error: The Side-by-Side configuration information for "c:\documents and settings\user1\local settings\temp\NUMPY-1.3.0-SSE3.EXE" contains errors. No se pudo iniciar la aplicacin porque su configuracin es incorrecta. Reinstalar la aplicacin puede solucionar el problema (14001). Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module." The spanish bit says: "The application couldn't be started because the configuration is not right. Reinstalling the application cas resolve the issue (14001)." Do you have any clue about how can i resolve the issue? Cheers, Jos? M. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Nov 12 06:37:15 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 12 Nov 2009 20:37:15 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> Message-ID: <5b8d13220911120337y27450b09x91e0fe346d81582e@mail.gmail.com> On Thu, Nov 12, 2009 at 12:45 AM, David Cournapeau wrote: > > I will implement this, but I would prefer using this method everywhere > for every compiler, it would be more robust. Done for r7727. I have not tested it much (I have only checked that the bit twiddling macros to get sign/exp/mantissa works as expected on sparc linux). David From sole at esrf.fr Thu Nov 12 07:42:47 2009 From: sole at esrf.fr (=?ISO-8859-1?Q?=22V=2E_Armando_Sol=E9=22?=) Date: Thu, 12 Nov 2009 13:42:47 +0100 Subject: [Numpy-discussion] Failed installation on Windows XP In-Reply-To: References: Message-ID: <4AFC02C7.2050005@esrf.fr> Hola, I am not an expert, but I had a similar issue with a program of main that I could trace to not having installed VS9 runtime libraries in the target computer: http://www.microsoft.com/downloads/details.aspx?FamilyID=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en Perhaps you can give a shot at it. Armando Jos? Mar?a Garc?a P?rez wrote: > Good morning, > I used to have Python 2.6, Numpy-1.3.0, ... on Windows 2000 in the > computer at work (which has the typical clampdown policies). I have > been updated to Windows XP SP3 and now the same installation files > fail to install. > > My configuration is: > > * Windows XP Serivice Pack 3 > * Processor: T7300 (Intel Core 2 Duo) > * Python 2.6.4 r264:75708 > > The installation of: "numpy-1.3.0-win32-superpack-python2.6.exe" fails > saying: > Executing numpy installer failed > > When I press Show Details: > Output folder: C:\DOCUME~1\user1\LOCALS~1\Temp > Install dir for actual installers is C:\DOCUME~1\user1\LOCALS~1\Temp > "Target CPU handles SSE2" > "Target CPU handles SSE3" > "native install (arch value: native)" > "Install SSE 3" > Extract: numpy-1.3.0-sse3.exe... 100% > Execute: "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" > Completed > > I suspect it might be a lack of a library. Openining > "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" with > "Dependency Walker" shows: > "Error: The Side-by-Side configuration information for "c:\documents > and settings\user1\local settings\temp\NUMPY-1.3.0-SSE3.EXE" contains > errors. No se pudo iniciar la aplicacin porque su configuracin es > incorrecta. Reinstalar la aplicacin puede solucionar el problema (14001). > Warning: At least one module has an unresolved import due to a missing > export function in a delay-load dependent module." > > The spanish bit says: "The application couldn't be started because the > configuration is not right. Reinstalling the application cas resolve > the issue (14001)." > > Do you have any clue about how can i resolve the issue? > > Cheers, > Jos? M. > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Thu Nov 12 07:45:34 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 12 Nov 2009 07:45:34 -0500 Subject: [Numpy-discussion] does an ndarray have a "hook" for a dictionary attribute ? In-Reply-To: References: Message-ID: On Nov 12, 2009, at 4:47 AM, Sebastian Haase wrote: > On Thu, Nov 12, 2009 at 10:38 AM, Sebastian Haase wrote: >> Hi, >> >> I hope my subject line is not entirely incomprehensible: >> I remember there was a discussion (some time ago) that every ndarray >> instance should get an extra dictionary (or just a "hook" for it, to >> minimize the memory foot print) for "application specific" / >> "auxiliary" data. >> What is the current state on this ? Nothing so far, I think... >> do I need to subclass ndarray to make such a ndarray_with_dict Why go for a full dictionary ? But anyway, yes, subclassing might be the fastest solution for you >> >> (For the latter it would be very helpful to get a hint on how to make >> such a "minimal" ndarray-subclass . I remember some issues involving >> some special __new__ methods, that I don't entirely remember by >> heard..) What kind of issues ? Anyhow, by now I'm sure you already googled "numpy subclassing" and checked the first two entries (one in the documentation, one on the cookbook). The examples given should be clear enough , but let me know if you have any problem. Cheers P. From cournape at gmail.com Thu Nov 12 08:16:34 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 12 Nov 2009 22:16:34 +0900 Subject: [Numpy-discussion] Import error in builds of 7726 In-Reply-To: References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> On Thu, Nov 12, 2009 at 6:57 AM, Chris wrote: > > Yeah, here is my build script -- it removes the build directory entirely Ah, that's not enough. You need to clean the working tree as well. git has the clean option for that, you can also use a quick script to do this with parsing svn st output. > > > #!/bin/sh > export MACOSX_DEPLOYMENT_TARGET=10.6 > export CFLAGS="-arch i386 -arch x86_64" > export FFLAGS="-arch i386 -arch x86_64" > export LDFLAGS="-Wall -undefined dynamic_lookup -bundle > -arch i386 -arch x86_64" > rm -rf build > python setupegg.py config_fc --fcompiler gfortran config - > L/Developer/src/freetype-2.3.5 -L/Developer/src/libpng-1.2.24 > build bdist_egg I have used the exact same options, and has no issue. David From mdroe at stsci.edu Thu Nov 12 09:34:45 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Thu, 12 Nov 2009 09:34:45 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911120337y27450b09x91e0fe346d81582e@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> <5b8d13220911120337y27450b09x91e0fe346d81582e@mail.gmail.com> Message-ID: <4AFC1D05.70003@stsci.edu> The new macros contain non-constant initializers for a type declared constant. const union IEEEl2bitsrep u = {x}; AFAIK, ANSI dictates that the initializer must be a literal or another constant variable. We could declare 'x' as constant in the function signature, but it actually does get changed in the body of the function. cc: acomp failed for build/src.solaris-2.8-sun4u-2.5/numpy/core/src/npymath/ieee754.c "numpy/core/src/npymath/ieee754.c.src", line 166: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 166: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 167: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 167: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 168: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 168: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 169: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 169: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 171: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 171: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 172: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 172: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 173: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 173: left operand must be modifiable lvalue: op "=" "numpy/core/src/npymath/ieee754.c.src", line 174: warning: non-constant initializer: op "NAME" "numpy/core/src/npymath/ieee754.c.src", line 174: left operand must be modifiable lvalue: op "=" I replaced: const union IEEEl2bitsrep u = {x}; with union IEEEl2bitsrep u; u.e = x; Hopefully a good compiler optimizer will alias this away. Alternatively, we could do what C++ would call a "reinterpret cast", which would avoid the need for the "do { ... } while (0);" trick: ((IEEEl2bitsrep *)(&(x)))->a Also, it seems LDBL_NBIT wasn't defined for the HAVE_LDOUBLE_IEEE_QUAD_BE case: "numpy/core/src/npymath/ieee754.c.src", line 176: undefined symbol: LDBL_NBIT I defined it as 0. Is that correct? So that got it to compile. Unfortunately, then I run into assertion errors in the unit test: > /home/mdroe/numpy/numpy/core/tests/test_umath.py(834)test_nextafter() -> assert np.nextafter(one, two) - one == eps (Pdb) p t (Pdb) p eps 1.9259299443872358531e-34 (Pdb) p np.nextafter(one, two) 1.0 (Pdb) p np.nextafter(one, two) - one 0.0 (Pdb) p (np.nextafter(one, two) - one) == 0.0 True It looks as if the nextafterl implementation isn't ever packing fields back into the long double at all. I can confirm this error even when using the numpy nextafterl on x86 Linux with gcc (by undef'ing HAVE_NEXTAFTERL on that platform). Take for example the following block: if (x == 0.0) { xmanh = 0; /* return +-minsubnormal */ xmanl = 1; xsign = ysign; t = x * x; if (t == x) { return t; } else { return x; /* raise underflow flag */ } } 'xmanh', 'xmanl' and 'xsign' are completely disconnected variables from the original long double 'x'. It seems the function needs to be written in terms of both getter and setter macros for each of the fields (as in my patch from yesterday), or it needs a macro re-pack the fields, e.g. LDOUBLE_PACK(x, sign, exp, manh, manl) which could be inserted before the "t = x * x" line above. Personally, I think the getter/setter approach is cleaner, even if more verbose, because it's not otherwise terribly obvious when the value needs to be re-packed. I'm happy to make these changes, since I've got access to the "finicky" platform, but want to confirm how you would prefer it done first. Mike David Cournapeau wrote: > On Thu, Nov 12, 2009 at 12:45 AM, David Cournapeau wrote: > > >> I will implement this, but I would prefer using this method everywhere >> for every compiler, it would be more robust. >> > > Done for r7727. I have not tested it much (I have only checked that > the bit twiddling macros to get sign/exp/mantissa works as expected on > sparc linux). > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From Chris.Barker at noaa.gov Thu Nov 12 12:01:33 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 09:01:33 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <94d043d59126cf48e77b4b1a0014d243@mmb.pcb.ub.es> References: <4AFA0044.6020106@noaa.gov> <94d043d59126cf48e77b4b1a0014d243@mmb.pcb.ub.es> Message-ID: <4AFC3F6D.7010400@noaa.gov> Peter Schmidtke wrote: > On Tue, 10 Nov 2009 16:07:32 -0800, Christopher Barker > wrote: >> I have a bunch of points in 2-d space, and I need to find out which >> pairs of points are within a certain distance of one-another (regular >> old Euclidean norm). > > How big is your set of points? Could be 100s of thousands, maybe ever millions. That's why O(N^2) is not good. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Nov 12 12:32:50 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 09:32:50 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> Message-ID: <4AFC46C2.5070303@noaa.gov> David Cournapeau wrote: > I would love having a core C library of containers - I'm all for that. However, I think that a very, very common need is simply for a growable numpy array. It seems this would actually be pretty darn easy (again, for someone familiar with the code!). IIUC, it would be exactly like a regular old ndarray, except: 1) The data block could be larger than the current size of the array: - this would require an additional attribute to carry that size 2) There would be an "append" method, or some other name: - this would increase the size of the array, manipulating .dimensions, and, if needed, doing a reallocation of the data pointer. - This is not very complex code, really, you simply allocate more than required, choosing some (user-resettable?) scale: 25% more than before, or whatever. re-allocating memory is actually pretty cheap these days, so it's not all that critical how you do it. 3) we'd probably want a .fit() method that would resize the data block to fit the current size of the array. In fact, as I write this, I wonder if these features could simply be added to the regular old ndarray -- if not used, they wouldn't be much overhead. Issues: - views on the same data block: as the data pointer would need to be re-allocated, you couldn't have views on the data block. Maybe this could be handles in the same way that ndarray.resize is handles now: you'd get an error if you try to resize an array that has a view to its data block. This is a bit tricky, as it's really easy to create a view without thinking about it (like by using ipython, for instance) - You could only grow the array in the first dimension (C-order, anyway). I think it could still be very useful, handling many of the common cases. - if you append to an array of more than one dimension, do you need to give the entire appended slice? Again, restrictive, but still useful. - Others? -- I'm not thinking of any right now. David Cournapeau wrote: > If you need to manipulate > core datatypes (float, etc...), I think python lists are a significant > cost, because of pointer chasing in particular. Exactly, and add custom numpy dtypes as well. Putting python types and/or tuples of python types into a list can be far more memory intensive than putting them in a numpy array. That's why I built a python implementation of a growable array -- it does save memory, but it's a fair bit slower than lists -- when adding to it in python, you've got python types already, so the additional overhead of an extra function call, etc. is substantial. That's why I'd like a C version. Even more important is having one that is not only written in C (or Cython) but can be accessed directly from C (or Cython), so you can fill an array with core data types efficiently. Going from C-type to python object to put it in a list is substantial overhead. > Stefan used some tricks to avoid using python list and use basic C > structures in some custom code. Exactly -- and he's not he only one -- we're all re-inventing the wheel here. By the way -- what does the fromstring/fromfile code do when you don't give it a size to start with? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Thu Nov 12 12:31:55 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 12 Nov 2009 10:31:55 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFC3F6D.7010400@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <94d043d59126cf48e77b4b1a0014d243@mmb.pcb.ub.es> <4AFC3F6D.7010400@noaa.gov> Message-ID: On Thu, Nov 12, 2009 at 10:01 AM, Christopher Barker wrote: > Peter Schmidtke wrote: > > On Tue, 10 Nov 2009 16:07:32 -0800, Christopher Barker > > wrote: > > >> I have a bunch of points in 2-d space, and I need to find out which > >> pairs of points are within a certain distance of one-another (regular > >> old Euclidean norm). > > > > How big is your set of points? > > Could be 100s of thousands, maybe ever millions. That's why O(N^2) is > not good. > > Another question is density. Are the points randomly scattered or do they cluster? The order N^2 part can be taken care of if you can easily eliminate a large fraction of the points, say by gridding the 2D part. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Nov 12 12:37:37 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 09:37:37 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <348618.28546.qm@web34403.mail.mud.yahoo.com> References: <4AFA0044.6020106@noaa.gov> <348618.28546.qm@web34403.mail.mud.yahoo.com> Message-ID: <4AFC47E1.5020607@noaa.gov> Lou Pecora wrote: > Maybe I'm missing something simple, but if your array of 2D points is > static, well, not quite. > a KD tree for 2D nearest neighbor seems like over kill. You > might want to try the simple approach of using boxes of points to > narrow things down by sorting on the first component. yeah, we'll probably do something like that if we have to write the code ourselves. At the moment, we're using geohash: http://pypi.python.org/pypi/Geohash/ (this is for points on the earth) and it's working OK. I was just hoping kdtree would work out of the box! > where for a > static data set it can match KD trees in speed Why does it have to be static -- it doesn't look hard to insert/remove points. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Thu Nov 12 12:39:04 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 12 Nov 2009 10:39:04 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFC46C2.5070303@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> Message-ID: On Thu, Nov 12, 2009 at 10:32 AM, Christopher Barker wrote: > David Cournapeau wrote: > > > I would love having a core C library of containers - > > I'm all for that. However, I think that a very, very common need is > simply for a growable numpy array. > > It seems this would actually be pretty darn easy (again, for someone > familiar with the code!). > > IIUC, it would be exactly like a regular old ndarray, except: > > 1) The data block could be larger than the current size of the array: > - this would require an additional attribute to carry that size > > 2) There would be an "append" method, or some other name: > - this would increase the size of the array, manipulating > .dimensions, and, if needed, doing a reallocation of the data pointer. > - This is not very complex code, really, you simply allocate more > than required, choosing some (user-resettable?) scale: 25% more than > before, or whatever. re-allocating memory is actually pretty cheap these > days, so it's not all that critical how you do it. > > 3) we'd probably want a .fit() method that would resize the data block > to fit the current size of the array. > > In fact, as I write this, I wonder if these features could simply be > added to the regular old ndarray -- if not used, they wouldn't be much > overhead. > > I'm rapidly losing interest here. Perhaps you could supply some code implementing this new array? Why not just a class using an array that doubles the array size when an index is out of bounds and copies over the old data. That is pretty much what realloc does. As to python lists, do you have any benchmarks showing how bad python lists are compared to arrays? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Nov 12 13:09:36 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 12 Nov 2009 12:09:36 -0600 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> Message-ID: <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> On Thu, Nov 12, 2009 at 11:39, Charles R Harris wrote: > > > On Thu, Nov 12, 2009 at 10:32 AM, Christopher Barker > wrote: >> >> David Cournapeau wrote: >> >> > I would love having a core C library of containers - >> >> I'm all for that. However, I think that a very, very common need is >> simply for a growable numpy array. >> >> It seems this would actually be pretty darn easy (again, for someone >> familiar with the code!). >> >> IIUC, it would be exactly like a regular old ndarray, except: >> >> ?1) The data block could be larger than the current size of the array: >> ? ? - this would require an additional attribute to carry that size >> >> ?2) There would be an "append" method, or some other name: >> ? ? - this would increase the size of the array, manipulating >> .dimensions, and, if needed, doing a reallocation of the data pointer. >> ? ? - This is not very complex code, really, you simply allocate more >> than required, choosing some (user-resettable?) scale: 25% more than >> before, or whatever. re-allocating memory is actually pretty cheap these >> days, so it's not all that critical how you do it. >> >> ?3) we'd probably want a .fit() method that would resize the data block >> to fit the current size of the array. >> >> In fact, as I write this, I wonder if these features could simply be >> added to the regular old ndarray -- if not used, they wouldn't be much >> overhead. >> > > I'm rapidly losing interest here. Perhaps you could supply some code > implementing this new array? Why not just a class using an array that > doubles the array size when an index is out of bounds and copies over the > old data. That is pretty much what realloc does. As to python lists, do you > have any benchmarks showing how bad python lists are compared to arrays? Didn't we already do this? http://www.mail-archive.com/numpy-discussion at scipy.org/msg21010.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lou_boog2000 at yahoo.com Thu Nov 12 13:40:53 2009 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Thu, 12 Nov 2009 10:40:53 -0800 (PST) Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFC47E1.5020607@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <348618.28546.qm@web34403.mail.mud.yahoo.com> <4AFC47E1.5020607@noaa.gov> Message-ID: <824047.6032.qm@web34408.mail.mud.yahoo.com> ----- Original Message ---- From: Christopher Barker To: Discussion of Numerical Python Sent: Thu, November 12, 2009 12:37:37 PM Subject: Re: [Numpy-discussion] finding close together points. Lou Pecora wrote: > a KD tree for 2D nearest neighbor seems like over kill. You > might want to try the simple approach of using boxes of points to > narrow things down by sorting on the first component. yeah, we'll probably do something like that if we have to write the code ourselves. At the moment, we're using geohash: http://pypi.python.org/pypi/Geohash/ (this is for points on the earth) and it's working OK. I was just hoping kdtree would work out of the box! > where for a > static data set it can match KD trees in speed Why does it have to be static -- it doesn't look hard to insert/remove points. --- Lou answers ---------------------- It doesn't have to be static, but if I remember correctly, when you add points you have to resort or do a "smart" insert (which may require a lot of data "shifting"). Not hard, but the original paper claimed that if there are a lot of changes to the data set this becomes more expensive than the kd-tree. If you can get the original paper I mentioned, it will have more info. It's pretty clearly written and contains some nice comparisons to kd-trees for finding near neighbors. Bottom line is that you can still try it with changing data. See what the run times are like. I've used the approach for up to eight dimensions with about 10^5 data points. It worked nicely. I remember looking for a kd-tree library that would work out of the box (C or C++), but everything I found was not as simple as I would have liked. The slice searching was a nice solution. But maybe I just didn't know where to look or I'm not understanding some of the libraries I did find. Let us know how it goes. -- Lou Pecora From peridot.faceted at gmail.com Thu Nov 12 14:19:58 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 12 Nov 2009 14:19:58 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <824047.6032.qm@web34408.mail.mud.yahoo.com> References: <4AFA0044.6020106@noaa.gov> <348618.28546.qm@web34403.mail.mud.yahoo.com> <4AFC47E1.5020607@noaa.gov> <824047.6032.qm@web34408.mail.mud.yahoo.com> Message-ID: 2009/11/12 Lou Pecora : > > > ----- Original Message ---- > From: Christopher Barker > To: Discussion of Numerical Python > Sent: Thu, November 12, 2009 12:37:37 PM > Subject: Re: [Numpy-discussion] finding close together points. > > Lou Pecora wrote: >> a KD tree for 2D nearest neighbor seems like over kill. ?You >> might want to try the simple approach of using boxes of points to >> narrow things down by sorting on the first component. > > yeah, we'll probably do something like that if we have to write the code > ourselves. At the moment, we're using geohash: > > http://pypi.python.org/pypi/Geohash/ > > (this is for points on the earth) > > and it's working OK. I was just hoping kdtree would work out of the box! > >> where for a >> static data set it can match KD trees in speed > > Why does it have to be static -- it doesn't look hard to insert/remove > points. > > --- Lou answers ---------------------- > > > It doesn't have to be static, but if I remember correctly, when you add points you have to resort or do a "smart" insert (which may require a lot of data "shifting"). ?Not hard, but the original paper claimed that if there are a lot of changes to the data set this becomes more expensive than the kd-tree. ?If you can get the original paper I mentioned, it will have more info. ?It's pretty clearly written and contains some nice comparisons to kd-trees for finding near neighbors. > > Bottom line is that you can still try it with changing data. ?See what the run times are like. ?I've used the approach for up to eight dimensions with about 10^5 data points. ?It worked nicely. ?I remember looking for a kd-tree library that would work out of the box (C or C++), but everything I found was not as simple as I would have liked. ?The slice searching was a nice solution. ?But maybe I just didn't know where to look or I'm not understanding some of the libraries I did find. > > Let us know how it goes. Just a quick comment: none of the KDTrees in scipy.spatial support any kind of modification right now, so if your data set changes you will need to rebuild them completely. Construction is fairly cheap, but it's very unlikely to compete with a data structure that supports modification algorithms. Anne From jakevdp at gmail.com Thu Nov 12 14:40:46 2009 From: jakevdp at gmail.com (Jake VanderPlas) Date: Thu, 12 Nov 2009 11:40:46 -0800 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 52 In-Reply-To: References: Message-ID: <58df6dc20911121140t35aa7a34v66598dc09b4af3d3@mail.gmail.com> > I'm rapidly losing interest here. Perhaps you could supply some code > implementing this new array? Why not just a class using an array that > doubles the array size when an index is out of bounds and copies over the > old data. That is pretty much what realloc does. As to python lists, do you > have any benchmarks showing how bad python lists are compared to arrays? > > Chuck It sounds like all of this could be done very simply without going to C, using a class based on numpy.ndarray. The following works for 1D arrays, behaves like a regular 1D numpy array, and could be easily improved with a little care. Is this what you had in mind? import numpy #easy scalable array class class scarray: def __init__(self,*args,**kwargs): self.__data = numpy.ndarray(*args,**kwargs) def append(self,val): tmp = self.__data self.__data = numpy.ndarray(tmp.size+1) self.__data[:-1] = tmp self.__data[-1] = val del tmp def __getattr__(self,attr): return getattr(self.__data,attr) x = scarray(5) x[:] = numpy.arange(5) print x x.append(5) print x From Chris.Barker at noaa.gov Thu Nov 12 14:52:39 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 11:52:39 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> Message-ID: <4AFC6787.4080805@noaa.gov> Robert Kern wrote: > Didn't we already do this? > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg21010.html Indeed we did. What I posted then ( and have improved a bit now). Is a Python version. Written in Python, it has an advantage of using less memory for a big array, but is slower in other respects than a Python list. This is probably why we all use lists for this when we need it! What I'd like (and I have seen interest from others) is a C version, one that could be used from C code directly as well. I can't benchmark that, as it hasn't been written! I'm afraid I'm out of my depth as to how to write such a thing in C (at least so that it is well integrated into numpy). I suppose I could write a simple C accumulating array, and only convert it into a numpy array at the end (indeed, I have done that in the past), and benchmark that. Charles R Harris wrote: > I'm rapidly losing interest here. Maybe that's why it hasn't been done yet -- it's not an interesting enough problem! > Why not just a class using an array that > doubles the array size when an index is out of bounds and copies over > the old data. Yes, it pretty much is that simple -- that's my point. Simple and useful, and why should we all re-invent this wheel each time (even it if is a simple wheel)? At the C(ython) level, it's not that hard to simply do your accumulating in regular old C, and then convert to a numpy array, but it would be nice for it to be easier, particularly for various data-neutral code, particularly numpy custom-defined ones. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fonnesbeck at gmail.com Thu Nov 12 14:56:28 2009 From: fonnesbeck at gmail.com (Chris) Date: Thu, 12 Nov 2009 19:56:28 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> Message-ID: David Cournapeau gmail.com> writes: > > On Thu, Nov 12, 2009 at 6:57 AM, Chris gmail.com> wrote: > > > > > Yeah, here is my build script -- it removes the build directory entirely > > Ah, that's not enough. You need to clean the working tree as well. git > has the clean option for that, you can also use a quick script to do > this with parsing svn st output. I tried scrapping the whole directory and starting again. Same thing though. I do notice a bunch of configtest warnings though: gcc-4.2: _configtest.c _configtest.c:4: warning: function declaration isn?t a prototype _configtest.c: In function ?main?: _configtest.c:5: error: size of array ?test_array? is negative _configtest.c:4: warning: function declaration isn?t a prototype _configtest.c: In function ?main?: _configtest.c:5: error: size of array ?test_array? is negative lipo: can't open input file: /var/folders/Cp/Cp1bvHSvFwuPM0 9XVxsVF++++TI/-Tmp-//ccO4OC35.out (No such file or directory) _configtest.c:4: warning: function declaration isn?t a prototype _configtest.c: In function ?main?: _configtest.c:5: error: size of array ?test_array? is negative _configtest.c:4: warning: function declaration isn?t a prototype _configtest.c: In function ?main?: _configtest.c:5: error: size of array ?test_array? is negative lipo: can't open input file: /var/folders/Cp/Cp1bvHSvFwuPM09XVx sVF++++TI/-Tmp-//ccO4OC35.out (No such file or directory) C compiler: gcc-4.2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -pipe Not sure this has anything to do with it. Again, none of this happened until this week, and I am using the same build script as always. From josemaria.alkala at gmail.com Thu Nov 12 16:01:52 2009 From: josemaria.alkala at gmail.com (=?ISO-8859-1?Q?Jos=E9_Mar=EDa_Garc=EDa_P=E9rez?=) Date: Thu, 12 Nov 2009 22:01:52 +0100 Subject: [Numpy-discussion] Failed installation on Windows XP In-Reply-To: <4AFC02C7.2050005@esrf.fr> References: <4AFC02C7.2050005@esrf.fr> Message-ID: Thanks for your answer, but it didn't resolve the issue. The problem is that I would need an administrator account to install it. Nevertheless, your suggestion might be helpful to others. I am not an expert either, but it looks like a dependency problem. Reading from: http://www.scipy.org/Installing_SciPy/Windows It says: 2009/11/12 "V. Armando Sol?" > Hola, > > I am not an expert, but I had a similar issue with a program of main > that I could trace to not having installed VS9 runtime libraries in the > target computer: > > > http://www.microsoft.com/downloads/details.aspx?FamilyID=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en > > Perhaps you can give a shot at it. > > Armando > > Jos? Mar?a Garc?a P?rez wrote: > > Good morning, > > I used to have Python 2.6, Numpy-1.3.0, ... on Windows 2000 in the > > computer at work (which has the typical clampdown policies). I have > > been updated to Windows XP SP3 and now the same installation files > > fail to install. > > > > My configuration is: > > > > * Windows XP Serivice Pack 3 > > * Processor: T7300 (Intel Core 2 Duo) > > * Python 2.6.4 r264:75708 > > > > The installation of: "numpy-1.3.0-win32-superpack-python2.6.exe" fails > > saying: > > Executing numpy installer failed > > > > When I press Show Details: > > Output folder: C:\DOCUME~1\user1\LOCALS~1\Temp > > Install dir for actual installers is C:\DOCUME~1\user1\LOCALS~1\Temp > > "Target CPU handles SSE2" > > "Target CPU handles SSE3" > > "native install (arch value: native)" > > "Install SSE 3" > > Extract: numpy-1.3.0-sse3.exe... 100% > > Execute: "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" > > Completed > > > > I suspect it might be a lack of a library. Openining > > "C:\DOCUME~1\user1\LOCALS~1\Temp\numpy-1.3.0-sse3.exe" with > > "Dependency Walker" shows: > > "Error: The Side-by-Side configuration information for "c:\documents > > and settings\user1\local settings\temp\NUMPY-1.3.0-SSE3.EXE" contains > > errors. No se pudo iniciar la aplicaci n porque su configuraci n es > > incorrecta. Reinstalar la aplicaci n puede solucionar el problema > (14001). > > Warning: At least one module has an unresolved import due to a missing > > export function in a delay-load dependent module." > > > > The spanish bit says: "The application couldn't be started because the > > configuration is not right. Reinstalling the application cas resolve > > the issue (14001)." > > > > Do you have any clue about how can i resolve the issue? > > > > Cheers, > > Jos? M. > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 12 17:30:48 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 12 Nov 2009 15:30:48 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFC6787.4080805@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> <4AFC6787.4080805@noaa.gov> Message-ID: On Thu, Nov 12, 2009 at 12:52 PM, Christopher Barker wrote: > Robert Kern wrote: > > > Didn't we already do this? > > > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg21010.html > > Indeed we did. What I posted then ( and have improved a bit now). Is a > Python version. Written in Python, it has an advantage of using less > memory for a big array, but is slower in other respects than a Python > list. This is probably why we all use lists for this when we need it! > > IIRC, python lists *are* expanding arrays of contiguous memory, not linked lists. They are efficient for appending and indexing, less so for deletion and insertion. It's one of those tradeoffs. So what you buy with an array implementation is the space/time efficiency of not having to allocate python types to put on the list. But you probably need to go through a python type at some point anyway, so... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Nov 12 17:37:36 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 12 Nov 2009 15:37:36 -0700 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> <4AFC6787.4080805@noaa.gov> Message-ID: On Thu, Nov 12, 2009 at 3:30 PM, Charles R Harris wrote: > > > On Thu, Nov 12, 2009 at 12:52 PM, Christopher Barker < > Chris.Barker at noaa.gov> wrote: > >> Robert Kern wrote: >> >> > Didn't we already do this? >> > >> > http://www.mail-archive.com/numpy-discussion at scipy.org/msg21010.html >> >> Indeed we did. What I posted then ( and have improved a bit now). Is a >> Python version. Written in Python, it has an advantage of using less >> memory for a big array, but is slower in other respects than a Python >> list. This is probably why we all use lists for this when we need it! >> >> > IIRC, python lists *are* expanding arrays of contiguous memory, not linked > lists. They are efficient for appending and indexing, less so for deletion > and insertion. It's one of those tradeoffs. So what you buy with an array > implementation is the space/time efficiency of not having to allocate python > types to put on the list. But you probably need to go through a python type > at some point anyway, so... > > And here is a pep for a Blist with a table of some of the performance tradeoffs. I think you could use small ndarrays in the leaf nodes and get around having to handle all the type stuff. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Nov 12 17:56:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 13 Nov 2009 07:56:09 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4AFC1D05.70003@stsci.edu> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> <5b8d13220911120337y27450b09x91e0fe346d81582e@mail.gmail.com> <4AFC1D05.70003@stsci.edu> Message-ID: <5b8d13220911121456g13239d0bub347ea22fdb63126@mail.gmail.com> On Thu, Nov 12, 2009 at 11:34 PM, Michael Droettboom wrote: > The new macros contain non-constant initializers for a type declared > constant. > > ? ?const union IEEEl2bitsrep u = {x}; > > AFAIK, ANSI dictates that the initializer must be a literal or another > constant variable. ?We could declare 'x' as constant in the function > signature, but it actually does get changed in the body of the function. yep, the const should not have been set in the first place here. > Hopefully a good compiler optimizer will alias this away. > Alternatively, we could do what C++ would call a "reinterpret cast", > which would avoid the need for the "do { ... } while (0);" trick: > > ? ((IEEEl2bitsrep *)(&(x)))->a Nope, this is not guaranteed to work, and is exactly what I am trying to avoid with union :) > > Also, it seems LDBL_NBIT wasn't defined for the > HAVE_LDOUBLE_IEEE_QUAD_BE case: > > ? "numpy/core/src/npymath/ieee754.c.src", line 176: undefined symbol: > LDBL_NBIT > > I defined it as 0. ?Is that correct? Yes. > > So that got it to compile. ?Unfortunately, then I run into assertion > errors in the unit test: > > ?> /home/mdroe/numpy/numpy/core/tests/test_umath.py(834)test_nextafter() > -> assert np.nextafter(one, two) - one == eps > (Pdb) p t > > (Pdb) p eps > 1.9259299443872358531e-34 > (Pdb) p np.nextafter(one, two) > 1.0 > (Pdb) p np.nextafter(one, two) - one > 0.0 > (Pdb) p (np.nextafter(one, two) - one) == 0.0 > True > > It looks as if the nextafterl implementation isn't ever packing fields > back into the long double at all. ?I can confirm this error even when > using the numpy nextafterl on x86 Linux with gcc (by undef'ing > HAVE_NEXTAFTERL on that platform). > > Take for example the following block: > > ? ?if (x == 0.0) { > ? ? ? ?xmanh = 0; ? ? ? ? ? ? ?/* return +-minsubnormal */ > ? ? ? ?xmanl = 1; > ? ? ? ?xsign = ysign; > ? ? ? ?t = x * x; > ? ? ? ?if (t == x) { > ? ? ? ? ? ?return t; > ? ? ? ?} else { > ? ? ? ? ? ?return x; ? ? ? ? ? /* raise underflow flag */ > ? ? ? ?} > ? ?} > > 'xmanh', 'xmanl' and 'xsign' are completely disconnected variables from > the original long double 'x'. Yes, you're right, the current code does not make any sense. I don't know what I was thinking (probably not much). > ?Personally, > I think the getter/setter approach is cleaner, even if more verbose, > because it's not otherwise terribly obvious when the value needs to be > re-packed. Yes, GETTER/SETTER is also more similar to the other cases (single/double precision). > I'm happy to make these changes, since I've got access to the "finicky" > platform, but want to confirm how you would prefer it done first. Cool. The changes should be done for all platforms, and pointer cast should be done through union, not (*(new type*)(&x)). thanks, David From plaes at plaes.org Thu Nov 12 18:00:57 2009 From: plaes at plaes.org (Priit Laes) Date: Fri, 13 Nov 2009 01:00:57 +0200 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() Message-ID: <1258066857.10034.13.camel@chi> Hey, I cooked up an initial implementation for one-dimensional histogram_discrete(). Example: >> import numpy >> numpy.histogram_discrete([-1, 9, 9, 0, 3, 5, 3]) array([1, 1, 0, 0, 2, 0, 1, 0, 0, 0, 2]) >> numpy.histogram_discrete([-99999, 99999]) array([1, 0, 0, ..., 0, 0, 1]) Suggestions, criticism? :) Priit :) -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-histogram_discrete.patch Type: text/x-patch Size: 3780 bytes Desc: URL: From josef.pktd at gmail.com Thu Nov 12 18:09:06 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 12 Nov 2009 18:09:06 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1258066857.10034.13.camel@chi> References: <1258066857.10034.13.camel@chi> Message-ID: <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> On Thu, Nov 12, 2009 at 6:00 PM, Priit Laes wrote: > Hey, > > I cooked up an initial implementation for one-dimensional > histogram_discrete(). > > Example: > >>> import numpy >>> numpy.histogram_discrete([-1, 9, 9, 0, 3, 5, 3]) > array([1, 1, 0, 0, 2, 0, 1, 0, 0, 0, 2]) >>> numpy.histogram_discrete([-99999, 99999]) > array([1, 0, 0, ..., 0, 0, 1]) > > Suggestions, criticism? :) Is there a difference to numpy.bincount ? Josef > > Priit :) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From dwf at cs.toronto.edu Thu Nov 12 18:16:09 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 12 Nov 2009 18:16:09 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> Message-ID: On 12-Nov-09, at 6:09 PM, josef.pktd at gmail.com wrote: > On Thu, Nov 12, 2009 at 6:00 PM, Priit Laes wrote: >> Hey, >> >> I cooked up an initial implementation for one-dimensional >> histogram_discrete(). >> >> Example: >> >>>> import numpy >>>> numpy.histogram_discrete([-1, 9, 9, 0, 3, 5, 3]) >> array([1, 1, 0, 0, 2, 0, 1, 0, 0, 0, 2]) >>>> numpy.histogram_discrete([-99999, 99999]) >> array([1, 0, 0, ..., 0, 0, 1]) >> >> Suggestions, criticism? :) > Is there a difference to numpy.bincount ? Time to add to this to the FAQ (FWIW I've reinvented the wheel a number of times too). Though, np.bincount does not handle negatives -- I kind of like it that way though. David From josef.pktd at gmail.com Thu Nov 12 18:25:34 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 12 Nov 2009 18:25:34 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> Message-ID: <1cd32cbb0911121525wa60d6ecv5bcfcc1216de80a@mail.gmail.com> On Thu, Nov 12, 2009 at 6:16 PM, David Warde-Farley wrote: > On 12-Nov-09, at 6:09 PM, josef.pktd at gmail.com wrote: > >> On Thu, Nov 12, 2009 at 6:00 PM, Priit Laes wrote: >>> Hey, >>> >>> I cooked up an initial implementation for one-dimensional >>> histogram_discrete(). >>> >>> Example: >>> >>>>> import numpy >>>>> numpy.histogram_discrete([-1, 9, 9, 0, 3, 5, 3]) >>> array([1, 1, 0, 0, 2, 0, 1, 0, 0, 0, 2]) >>>>> numpy.histogram_discrete([-99999, 99999]) >>> array([1, 0, 0, ..., 0, 0, 1]) >>> >>> Suggestions, criticism? :) > >> Is there a difference to numpy.bincount ? > > > Time to add to this to the FAQ (FWIW I've reinvented the wheel a > number of times too). > > Though, np.bincount does not handle negatives -- I kind of like it > that way though. If I could make a related wish, I wish np.bincount to take also a 2d array as weights. Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Nov 12 18:31:18 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 12 Nov 2009 17:31:18 -0600 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1cd32cbb0911121525wa60d6ecv5bcfcc1216de80a@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1cd32cbb0911121525wa60d6ecv5bcfcc1216de80a@mail.gmail.com> Message-ID: <3d375d730911121531w48d08c74i579c7594490b2e8d@mail.gmail.com> On Thu, Nov 12, 2009 at 17:25, wrote: > If I could make a related wish, I wish np.bincount to take also a 2d > array as weights. It does. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From alexey.tigarev at gmail.com Thu Nov 12 18:38:25 2009 From: alexey.tigarev at gmail.com (Alexey Tigarev) Date: Fri, 13 Nov 2009 01:38:25 +0200 Subject: [Numpy-discussion] Multiple Regression Message-ID: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> Hi All! I have implemented multiple regression in a following way: def multipleRegression(x, y): """ Perform linear regression using least squares method. X - matrix containing inputs for observations, y - vector containing one of outputs for every observation """ mulregLogger.debug("multipleRegression(x=%s, y=%s)" % (x, y)) xt = transpose(x) a = dot(xt, x) # A = xt * x b = dot(xt, y) # B = xt * y try: return linalg.solve(a, b) except linalg.LinAlgError, lae: mulregLogger.warn("Singular matrix:\n%s" % (a)) mulregLogger.warn(lae) mulregLogger.warn("Determinant: %f" % (linalg.det(a))) raise lae Can you suggest me something to optimize it? I am using it on large number of observations so it is common to have "x" matrix of about 5000x20 and "y" vector of length 5000, and more. I also have to run that multiple times for different "y" vectors and same "x" matrix. Thanks, Alexey From robert.kern at gmail.com Thu Nov 12 18:44:09 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 12 Nov 2009 17:44:09 -0600 Subject: [Numpy-discussion] Multiple Regression In-Reply-To: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> References: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> Message-ID: <3d375d730911121544n7a0734ai847436ed5844a241@mail.gmail.com> On Thu, Nov 12, 2009 at 17:38, Alexey Tigarev wrote: > Hi All! > > I have implemented multiple regression in a following way: > > def multipleRegression(x, y): > ? ?""" Perform linear regression using least squares method. > > ? ?X - matrix containing inputs for observations, > ? ?y - vector containing one of outputs for every observation """ > ? ?mulregLogger.debug("multipleRegression(x=%s, y=%s)" % (x, y)) > ? ?xt = transpose(x) > ? ?a = dot(xt, x) ? ? # A = xt * x > ? ?b = dot(xt, y) ? ? # B = xt * y > ? ?try: > ? ? ? ?return linalg.solve(a, b) Never, ever use the normal equations. :-) Use linalg.lstsq(x, y) instead. > ? ?except linalg.LinAlgError, lae: > ? ? ? ?mulregLogger.warn("Singular matrix:\n%s" % (a)) > ? ? ? ?mulregLogger.warn(lae) > ? ? ? ?mulregLogger.warn("Determinant: %f" % (linalg.det(a))) > ? ? ? ?raise lae > > Can you suggest me something to optimize it? > > I am using it on large number of observations so it is common to have > "x" matrix of about 5000x20 and "y" vector of length 5000, and more. > I also have to run that multiple times for different "y" vectors and > same "x" matrix. Just make a matrix "y" such that each column vector is a different output vector (e.g. y.shape == (5000, number_of_different_y_vectors)) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dwf at cs.toronto.edu Thu Nov 12 19:01:50 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 12 Nov 2009 19:01:50 -0500 Subject: [Numpy-discussion] Multiple Regression In-Reply-To: <3d375d730911121544n7a0734ai847436ed5844a241@mail.gmail.com> References: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> <3d375d730911121544n7a0734ai847436ed5844a241@mail.gmail.com> Message-ID: On 12-Nov-09, at 6:44 PM, Robert Kern wrote: >> I am using it on large number of observations so it is common to have >> "x" matrix of about 5000x20 and "y" vector of length 5000, and more. >> I also have to run that multiple times for different "y" vectors and >> same "x" matrix. > > Just make a matrix "y" such that each column vector is a different > output vector (e.g. y.shape == (5000, number_of_different_y_vectors)) Hmm, I just noticed that numpy.linalg.solve's docstring is wrong. Parameters ---------- a : array_like, shape (M, M) Coefficient matrix. b : array_like, shape (M,) Ordinate or "dependent variable" values. Whereas it *does* take multiple right-hand-side vectors, just like it's scipy.linalg counterpart (I found it somewhat odd that it wouldn't considering the LAPACK functions do). So I guess that should read Parameters ---------- a : array_like, shape (M, M) Coefficient matrix. b : array_like, shape (M,) or (M, N) Ordinate or "dependent variable" values. I'll update it in the doc editor when I get home, if no one beats me to it. David From Chris.Barker at noaa.gov Thu Nov 12 20:13:41 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 17:13:41 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> <4AFC6787.4080805@noaa.gov> Message-ID: <4AFCB2C5.2040802@noaa.gov> Charles R Harris wrote: > So what you buy > with an array implementation is the space/time efficiency of not having > to allocate python types to put on the list. But you probably need to go > through a python type at some point anyway, When writing Python, yes (though maybe not if you are appending a lot of data at once from a numpy array). That's whey my python implementation is slower, and I suspect a C implementation would be similar in performance to a list. But if you are writing a C extension in C or Cython, then you could get a real gain by not having to go through Python types. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josef.pktd at gmail.com Thu Nov 12 20:12:42 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 12 Nov 2009 20:12:42 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <3d375d730911121531w48d08c74i579c7594490b2e8d@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1cd32cbb0911121525wa60d6ecv5bcfcc1216de80a@mail.gmail.com> <3d375d730911121531w48d08c74i579c7594490b2e8d@mail.gmail.com> Message-ID: <1cd32cbb0911121712h2838831ch291a0e3f261b875e@mail.gmail.com> On Thu, Nov 12, 2009 at 6:31 PM, Robert Kern wrote: > On Thu, Nov 12, 2009 at 17:25, ? wrote: > >> If I could make a related wish, I wish np.bincount to take also a 2d >> array as weights. > > It does. If it does, I don't manage to make it work, and the new docs say 1dimension. Josef >>> np.bincount(np.array([[1,1],[0,0]]), weights=np.arange(4).reshape((2,2))) Traceback (most recent call last): ValueError: object too deep for desired array >>> np.bincount(np.array([[1,1],[0,0]])) Traceback (most recent call last): ValueError: object too deep for desired array >>> np.bincount(np.array([[1,1],[0,0]]).ravel()) array([2, 2]) >>> np.bincount(np.array([1,1,0,0]),np.arange(8).reshape((2,4))) Traceback (most recent call last): ValueError: object too deep for desired array >>> np.bincount(np.array([1,1,0,0]),np.arange(8).reshape((4,2))) Traceback (most recent call last): ValueError: object too deep for desired array >>> np.bincount(np.array([1,1,0,0]),np.arange(4)) array([ 5., 1.]) > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Nov 12 20:16:27 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 12 Nov 2009 17:16:27 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFA568B.3010005@ar.media.kyoto-u.ac.jp> <4AFC46C2.5070303@noaa.gov> <3d375d730911121009k4e46105bwf18cc2282b5a742d@mail.gmail.com> <4AFC6787.4080805@noaa.gov> Message-ID: <4AFCB36B.3040503@noaa.gov> Charles R Harris wrote: > And here is a pep That link was broken, try this one: http://www.python.org/dev/peps/pep-3128/ -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josef.pktd at gmail.com Thu Nov 12 20:20:35 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 12 Nov 2009 20:20:35 -0500 Subject: [Numpy-discussion] Multiple Regression In-Reply-To: <3d375d730911121544n7a0734ai847436ed5844a241@mail.gmail.com> References: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> <3d375d730911121544n7a0734ai847436ed5844a241@mail.gmail.com> Message-ID: <1cd32cbb0911121720o30c8edfdobcd4bd9c6c28aa80@mail.gmail.com> On Thu, Nov 12, 2009 at 6:44 PM, Robert Kern wrote: > On Thu, Nov 12, 2009 at 17:38, Alexey Tigarev wrote: >> Hi All! >> >> I have implemented multiple regression in a following way: >> >> def multipleRegression(x, y): >> ? ?""" Perform linear regression using least squares method. >> >> ? ?X - matrix containing inputs for observations, >> ? ?y - vector containing one of outputs for every observation """ >> ? ?mulregLogger.debug("multipleRegression(x=%s, y=%s)" % (x, y)) >> ? ?xt = transpose(x) >> ? ?a = dot(xt, x) ? ? # A = xt * x >> ? ?b = dot(xt, y) ? ? # B = xt * y >> ? ?try: >> ? ? ? ?return linalg.solve(a, b) > > Never, ever use the normal equations. :-) > > Use linalg.lstsq(x, y) instead. > >> ? ?except linalg.LinAlgError, lae: >> ? ? ? ?mulregLogger.warn("Singular matrix:\n%s" % (a)) >> ? ? ? ?mulregLogger.warn(lae) >> ? ? ? ?mulregLogger.warn("Determinant: %f" % (linalg.det(a))) >> ? ? ? ?raise lae >> >> Can you suggest me something to optimize it? >> >> I am using it on large number of observations so it is common to have >> "x" matrix of about 5000x20 and "y" vector of length 5000, and more. >> I also have to run that multiple times for different "y" vectors and >> same "x" matrix. > > Just make a matrix "y" such that each column vector is a different > output vector (e.g. y.shape == (5000, number_of_different_y_vectors)) or if you want to do it sequentially, this should work xpinv = linalg.pinv(x) for y in all_ys: beta = np.dot(xpinv, y) but this works for singular problems without warning Josef > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Nov 12 21:26:05 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 12 Nov 2009 21:26:05 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1cd32cbb0911121712h2838831ch291a0e3f261b875e@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1cd32cbb0911121525wa60d6ecv5bcfcc1216de80a@mail.gmail.com> <3d375d730911121531w48d08c74i579c7594490b2e8d@mail.gmail.com> <1cd32cbb0911121712h2838831ch291a0e3f261b875e@mail.gmail.com> Message-ID: <3d375d730911121826v27ca38bbh97071c67e8393250@mail.gmail.com> On Thu, Nov 12, 2009 at 20:12, wrote: > On Thu, Nov 12, 2009 at 6:31 PM, Robert Kern wrote: >> On Thu, Nov 12, 2009 at 17:25, ? wrote: >> >>> If I could make a related wish, I wish np.bincount to take also a 2d >>> array as weights. >> >> It does. > > If it does, I don't manage to make it work, and the new docs say 1dimension. Oh, sorry. I read "2d" as "second". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sturla at molden.no Thu Nov 12 22:14:04 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Nov 2009 04:14:04 +0100 Subject: [Numpy-discussion] Multiple Regression In-Reply-To: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> References: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> Message-ID: <4AFCCEFC.4000806@molden.no> Alexey Tigarev skrev: > I have implemented multiple regression in a following way: > > You should be using QR or SVD for this. Sturla From gokhansever at gmail.com Thu Nov 12 22:33:09 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 12 Nov 2009 21:33:09 -0600 Subject: [Numpy-discussion] Multiple Regression In-Reply-To: <4AFCCEFC.4000806@molden.no> References: <16bfdd890911121538t2ea5c186x64a86b3972ca2eb4@mail.gmail.com> <4AFCCEFC.4000806@molden.no> Message-ID: <49d6b3500911121933i2f756edav6b36eb4c75e4309e@mail.gmail.com> On Thu, Nov 12, 2009 at 9:14 PM, Sturla Molden wrote: > Alexey Tigarev skrev: >> I have implemented multiple regression in a following way: >> >> > You should be using QR or SVD for this. > > Sturla > Seeing this QR and SVD terms I recalled the answer to the "I am the very model for a student mathematical" poem. I am just quoting the answer part for the full poem see Google :) My apologies for the noise in the subject... "When you have learnt just what is meant by 'Jacobian' and 'Abelian'; When you at sight can estimate, for the modal, mean and median; When describing normal subgroups is much more than recitation; When you understand precisely what is 'quantum excitation'; When you know enough statistics that you can recognise RV; When you have learnt all advances that have been made in SVD; And when you can spot the transform that solves some tricky PDE, You will feel no better student has ever sat for a degree. Your accumulated knowledge, whilst extensive and exemplary, Will have only been brought down to the beginning of last century, But still in matters rational, and logical and practical, You'll be the very model of a student mathematical." * *K. F. Riley, with apologies to W.S. Gilbert * -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Nov 12 22:50:58 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 13 Nov 2009 12:50:58 +0900 Subject: [Numpy-discussion] Import error in builds of 7726 In-Reply-To: References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> Message-ID: <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> On Fri, Nov 13, 2009 at 4:56 AM, Chris wrote: > David Cournapeau gmail.com> writes: > >> >> On Thu, Nov 12, 2009 at 6:57 AM, Chris gmail.com> wrote: >> >> > >> > Yeah, here is my build script -- it removes the build directory entirely >> >> Ah, that's not enough. You need to clean the working tree as well. git >> has the clean option for that, you can also use a quick script to do >> this with parsing svn st output. > > I tried scrapping the whole directory and starting again. Same thing though. > I do notice a bunch of configtest warnings though: > > gcc-4.2: _configtest.c > _configtest.c:4: warning: function declaration isn?t a prototype > _configtest.c: In function ?main?: > _configtest.c:5: error: size of array ?test_array? is negative > _configtest.c:4: warning: function declaration isn?t a prototype > _configtest.c: In function ?main?: > _configtest.c:5: error: size of array ?test_array? is negative > lipo: can't open input file: /var/folders/Cp/Cp1bvHSvFwuPM0 > 9XVxsVF++++TI/-Tmp-//ccO4OC35.out (No such file or directory) > _configtest.c:4: warning: function declaration isn?t a prototype > _configtest.c: In function ?main?: > _configtest.c:5: error: size of array ?test_array? is negative > _configtest.c:4: warning: function declaration isn?t a prototype > _configtest.c: In function ?main?: > _configtest.c:5: error: size of array ?test_array? is negative > lipo: can't open input file: /var/folders/Cp/Cp1bvHSvFwuPM09XVx > sVF++++TI/-Tmp-//ccO4OC35.out (No such file or directory) > C compiler: gcc-4.2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes > -arch i386 -arch x86_64 -pipe > > Not sure this has anything to do with it. No, this is expected - those are related to size check, and compilation failure is part of the process. Unfortunately, the output of those checks is not easy to control. I am afraid I don't see where the problem may come from. I used the exact same build script as you, on the same version of mac os x, and I don't see the problem. Is there anything non standard in your environment that you can think of ? David From peridot.faceted at gmail.com Fri Nov 13 02:31:47 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 13 Nov 2009 02:31:47 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFA50D4.70403@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> Message-ID: 2009/11/11 Christopher Barker : > Anne Archibald wrote: >> 2009/11/10 Christopher Barker : > >>> I have a bunch of points in 2-d space, and I need to find out which >>> pairs of points are within a certain distance of one-another (regular >>> old Euclidean norm). >> >> This is an eminently reasonable thing to want, and KDTree should >> support it. Unfortunately it doesn't. > > darn. > >> It's slow both because you're using the python implementation rather >> than the C, > > I noticed that. > >> The one good thing about using the python implementation rather than >> the Cython one is that you can subclass it, providing a new method. >> There's still a certain amount of boilerplate code to write, but it >> shouldn't be too bad. > > Can you give me a hint as to where to start? ?I have no idea how kd > trees work. > >> If this is still too slow, I have no problem incorporating additional >> code into cKDTree; the only reason none of the ball queries are in >> there is because ball queries must return variable-size results, so >> you lose a certain amount of speed because you're forced to manipulate >> python objects. But if there are relatively few result points, this >> need not be much of a slowdown. > > And certainly in this case, there would not be very many results, and > lists are pretty fast. This is now implemented in SVN. I (tentatively?) used a set to store the collection of pairs, because my tree traversal is not smart enough to reliably uniquify the pairs without using sets. With more time and energy, I'm sure the algorithm could be improved to avoid using sets (both internally and on return), but I think that's something to save for the Cython version. Anne From plaes at plaes.org Fri Nov 13 02:41:19 2009 From: plaes at plaes.org (Priit Laes) Date: Fri, 13 Nov 2009 09:41:19 +0200 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> Message-ID: <1258098079.10759.30.camel@chi> ?hel kenal p?eval, N, 2009-11-12 kell 18:16, kirjutas David Warde-Farley: > On 12-Nov-09, at 6:09 PM, josef.pktd at gmail.com wrote: > > > On Thu, Nov 12, 2009 at 6:00 PM, Priit Laes wrote: > >> Hey, > >> > >> I cooked up an initial implementation for one-dimensional > >> histogram_discrete(). > >> > >> Example: > >> > >>>> import numpy > >>>> numpy.histogram_discrete([-1, 9, 9, 0, 3, 5, 3]) > >> array([1, 1, 0, 0, 2, 0, 1, 0, 0, 0, 2]) > >>>> numpy.histogram_discrete([-99999, 99999]) > >> array([1, 0, 0, ..., 0, 0, 1]) > >> > >> Suggestions, criticism? :) > > > Is there a difference to numpy.bincount ? Yes, it allows negative numbers to be in your list, otherwise it is almost identical to bincount. (except I stripped the weights option) > > Time to add to this to the FAQ (FWIW I've reinvented the wheel a > number of times too). Does anyone have a scenario where one would actually have both negative and positive numbers (integers) in the list? Of course, I really didn't know about bincount before I really started coding histogram_discrete() so maybe it would be nice to have an alias for it that calls bincount directly, so it would be easier to find? FAQ entry would be fine too, although I cannot find the FAQ itself from Numpy homepage. Also I spent some time on IRC (#scipy) complaining about my woes with histogram(), so please do not be that hard on me for reinventing a wheel... :P So, how about numpy.histogram_discrete() that returns data the way histogram() does: a list containing histogram values (ie counts) and list of sorted items from min(input)...max(input). ? Priit :) From cimrman3 at ntc.zcu.cz Fri Nov 13 05:54:51 2009 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 13 Nov 2009 11:54:51 +0100 Subject: [Numpy-discussion] allclose() does not check shape of inputs Message-ID: <4AFD3AFB.5000605@ntc.zcu.cz> Hi, I think this is a bug: In [16]: np.allclose([1.0, 1.0], [1.1], rtol=0.1, atol=0.0) Out[16]: True Shall I create a ticket? r. From pav+sp at iki.fi Fri Nov 13 06:08:31 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Fri, 13 Nov 2009 11:08:31 +0000 (UTC) Subject: [Numpy-discussion] allclose() does not check shape of inputs References: <4AFD3AFB.5000605@ntc.zcu.cz> Message-ID: Fri, 13 Nov 2009 11:54:51 +0100, Robert Cimrman wrote: > I think this is a bug: > > In [16]: np.allclose([1.0, 1.0], [1.1], rtol=0.1, atol=0.0) > Out[16]: True It's broadcasting. I'm not sure it is a bug: >>> np.allclose([1.0, 1.0], [1.1, 1.1, 1.1], rtol=0.1, atol=0.0) False -- Pauli Virtanen From cimrman3 at ntc.zcu.cz Fri Nov 13 06:12:21 2009 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Fri, 13 Nov 2009 12:12:21 +0100 Subject: [Numpy-discussion] allclose() does not check shape of inputs In-Reply-To: References: <4AFD3AFB.5000605@ntc.zcu.cz> Message-ID: <4AFD3F15.8090201@ntc.zcu.cz> Pauli Virtanen wrote: > Fri, 13 Nov 2009 11:54:51 +0100, Robert Cimrman wrote: >> I think this is a bug: >> >> In [16]: np.allclose([1.0, 1.0], [1.1], rtol=0.1, atol=0.0) >> Out[16]: True > > It's broadcasting. I'm not sure it is a bug: > >>>> np.allclose([1.0, 1.0], [1.1, 1.1, 1.1], rtol=0.1, atol=0.0) > False > I see, I hit the corner case... Sorry for the noise! r. From ralf.gommers at googlemail.com Fri Nov 13 06:25:42 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 13 Nov 2009 12:25:42 +0100 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Wed, Nov 4, 2009 at 11:05 AM, Jarrod Millman wrote: > On Wed, Nov 4, 2009 at 1:37 AM, Ralf Gommers > wrote: > > It would be good if we could also have one more merge of the work in the > doc > > editor (close to 300 new/changed docstrings now). I can have it all > reviewed > > by the 13th. > > That would be great. Thanks for taking care of that. > > All wiki changes are now reviewed and can be merged. Under numpy/doc there is a file HOWTO_MERGE_WIKI_DOCS.txt with details on how this is done. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Fri Nov 13 07:36:48 2009 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Fri, 13 Nov 2009 13:36:48 +0100 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1258098079.10759.30.camel@chi> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> Message-ID: <20091113123648.GA18442@doriath.local> 13/11/09 @ 09:41 (+0200), thus spake Priit Laes: > Does anyone have a scenario where one would actually have both negative > and positive numbers (integers) in the list? Yes: when you have a random variable that is the difference of two (discrete) random variables. For example, if you measure the difference in number of days off per week because of sickness between two groups of people, you would end up with a discrete variable with both positive and negative integers. > So, how about numpy.histogram_discrete() that returns data the way > histogram() does: a list containing histogram values (ie counts) and > list of sorted items from min(input)...max(input). ? In my humble opinion, it would be nice. Regards. From robince at gmail.com Fri Nov 13 12:23:19 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 17:23:19 +0000 Subject: [Numpy-discussion] bus error in embedded numpy Message-ID: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> Hi, I'm trying to embed Python in a MATLAB mex file. I've been coming under some pressure to make my Python code available to my MATLAB colleagues so I am trying to come up with a relatively general way of calling numerical python code from Matlab. I am making some progress - but get a reliable crash from numpy. This only occurs the second time I am loading it. So I Py_Initialize, import numpy, Py_Finalize (all works fine), but then if I clear the mex file (clear funcname in matlab - which calls Py_Finalize through a mexAtExit handler) and try to run the function again (which will reinitialize interpreter and import numpy again) I get the followin stack trace from multisarray.so. Wondered if anyone could through any light. I have already run into this bug http://bugs.python.org/issue6869 which prevents me using ctypes... I wondered if this was related. For now its not such a big problem - I will just avoid unloading the mex function (with clear function). But I thought it might be indicative of a memory leak or some other problem since I think in theory it should work (It does if numpy isn't imported). Cheers Robin ------------------------------------------------------------------------ Bus error detected at Fri Nov 13 17:11:57 2009 ------------------------------------------------------------------------ Configuration: MATLAB Version: 7.8.0.347 (R2009a) MATLAB License: 161051 Operating System: Darwin 10.0.0 Darwin Kernel Version 10.0.0: Fri Jul 31 22:47:34 PDT 2009; root:xnu-1456.1.25~1/RELEASE_I386 i386 Window System: The X.Org Foundation (10402000), display /tmp/launch-2p1ZWg/:0 Current Visual: 0x24 (class 4, depth 24) Processor ID: x86 Family 6 Model 15 Stepping 10, GenuineIntel Virtual Machine: Java 1.6.0_15-b03-219 with Apple Inc. Java HotSpot(TM) Client VM mixed mode Default Encoding: ISO-8859-1 Fault Count: 1 Register State: eax = 00000001 ebx = 307b12ab ecx = 00000000 edx = 305ef148 esi = 305ef140 edi = 32a79d60 ebp = b097c6c8 esp = b097c670 eip = 307b14be flg = 00010202 Stack Trace: [0] multiarray.so:PyArray_FromScalar~(0x305ef140, 0, 2, 0x32e23287) + 542 bytes [1] multiarray.so:gentype_nonzero_number~(0x305ef140, 0x2eaa6db0, 0, 0x32e73cfe) + 36 bytes [2] Python:PyObject_IsTrue~(0x305ef140, 0x2ea95de0, 2, 0) + 63 bytes [3] Python:PyEval_EvalFrameEx~(0x33c95160, 0, 0x332941e0, 0) + 12598 bytes [4] Python:PyEval_EvalFrameEx~(0x32a70f80, 0, 0x332941e0, 0) + 24217 bytes [5] Python:PyEval_EvalCodeEx~(0x33f05068, 0x332941e0, 0, 0x32a96794) + 1819 bytes [6] Python:PyEval_EvalFrameEx~(0x32a96640, 0, 0x332941e0, 0) + 16561 bytes [7] Python:PyEval_EvalCodeEx~(0x33f051d0, 0x332941e0, 0, 0x32a6fcc0) + 1819 bytes [8] Python:PyEval_EvalFrameEx~(0x32a6fb60, 0, 0x33bc99c0, 0) + 16561 bytes [9] Python:PyEval_EvalCodeEx~(0x334abda0, 0x33bc99c0, 0, 0x32ad3b24) + 1819 bytes [10] Python:PyEval_EvalFrameEx~(0x32ad39d0, 0, 0x33bc94b0, 0) + 16561 bytes [11] Python:PyEval_EvalCodeEx~(0x332919b0, 0x33bc94b0, 0, 0x33ce5294) + 1819 bytes [12] Python:PyEval_EvalFrameEx~(0x33ce5150, 0, 0x33bc94b0, 0) + 16561 bytes [13] Python:PyEval_EvalCodeEx~(0x332918d8, 0x33bc94b0, 0, 0x34d5156c) + 1819 bytes [14] Python:PyEval_EvalFrameEx~(0x34d51430, 0, 0x33bc9300, 0x33bc9300) + 16561 bytes [15] Python:PyEval_EvalCodeEx~(0x33291968, 0x33bc9300, 0x33bc9300, 0) + 1819 bytes [16] Python:PyEval_EvalCode~(0x33291968, 0x33bc9300, 0x33bc9300, 0x9325378f) + 87 bytes [17] Python:PyImport_ExecCodeModuleEx~(0xb097e4cb "numpy.core._internal", 0x33291968, 0xb097dbbf "/Library/Frameworks/Python.frame..", 0x332a2000) + 193 bytes [18] Python:load_source_module~(1, 0, 0xb097e418 "X?C?", 0xb097e41c) + 726 bytes [19] Python:import_submodule~(0xb097e4d6 "_internal", 0x33d1ce1c "_internal", 9, 0x32e2cfc9) + 293 bytes [20] Python:load_next~(0xb097e4cb "numpy.core._internal", 0xb097e8cc, 0xb097e578, 0x32e8b5e6) + 195 bytes [21] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, 0xee9e70b3, 0x32e6df4d "?E?") + 142 bytes [22] Python:PyImport_ImportModuleLevel~(0x33d1ce1c "_internal", 0x33b25150, 0x33b25150, 0x32ee4aa0 "TD") + 45 bytes [23] Python:builtin___import__~(0, 0x348854e0, 0, 0x32e73cfe) + 156 bytes [24] Python:PyObject_Call~(0x33da9a08, 0x348854e0, 0, 0x32e233cd) + 45 bytes [25] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348854e0, 0, 0x33b25150) + 112 bytes [26] Python:PyEval_EvalFrameEx~(0x33cd2da0, 0, 0x33b25150, 0x33b25150) + 8138 bytes [27] Python:PyEval_EvalCodeEx~(0x33332068, 0x33b25150, 0x33b25150, 0) + 1819 bytes [28] Python:PyEval_EvalCode~(0x33332068, 0x33b25150, 0x33b25150, 0x9325378f) + 87 bytes [29] Python:PyImport_ExecCodeModuleEx~(0xb097fadb "numpy.core", 0x33332068, 0xb097ed7f "/Library/Frameworks/Python.frame..", 0x4adc73cc) + 193 bytes [30] Python:load_source_module~(1, 0, 0xb097f5cc, 0) + 726 bytes [31] Python:load_package~(5, 0, 0xb097fa28, 0xb097fa2c) + 427 bytes [32] Python:import_submodule~(0xb097fae1 "core", 0x3488536a "core.numeric", 4, 0x32e2cfc9) + 293 bytes [33] Python:load_next~(0xb097fadb "numpy.core", 0xb097fedc, 9, 0x32e8b5e6) + 195 bytes [34] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, 0x746e6920, 0x32e6df4d "?E?") + 213 bytes [35] Python:PyImport_ImportModuleLevel~(0x34885364 "numpy.core.numeric", 0x33b25a50, 0x33b25a50, 0x32ee4aa0 "TD") + 45 bytes [36] Python:builtin___import__~(0, 0x34885780, 0, 0x32e73cfe) + 156 bytes [37] Python:PyObject_Call~(0x33da9a08, 0x34885780, 0, 0x32e233cd) + 45 bytes [38] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x34885780, 0, 0x33b25a50) + 112 bytes [39] Python:PyEval_EvalFrameEx~(0x33cd07a0, 0, 0x33b25a50, 0x33b25a50) + 8138 bytes [40] Python:PyEval_EvalCodeEx~(0x33332140, 0x33b25a50, 0x33b25a50, 0) + 1819 bytes [41] Python:PyEval_EvalCode~(0x33332140, 0x33b25a50, 0x33b25a50, 0x9325378f) + 87 bytes [42] Python:PyImport_ExecCodeModuleEx~(0xb0980c9b "numpy.lib.type_check", 0x33332140, 0xb098038f "/Library/Frameworks/Python.frame..", 6771) + 193 bytes [43] Python:load_source_module~(1, 0, 0xb0980be8 "??C?", 0xb0980bec) + 726 bytes [44] Python:import_submodule~(0xb0980ca5 "type_check", 0x33d182b4 "type_check", 10, 0x32e2cfc9) + 293 bytes [45] Python:load_next~(0xb0980c9b "numpy.lib.type_check", 0xb098109c, 0xb0980d48, 0x32e8b5e6) + 195 bytes [46] Python:import_module_level~(0x3082c1b0, 0xffffffff, 0xee9e70b3, 0x32e6df4d "?E?") + 142 bytes [47] Python:PyImport_ImportModuleLevel~(0x33d182b4 "type_check", 0x33b25390, 0x33b25390, 0x3082c1b0) + 45 bytes [48] Python:builtin___import__~(0, 0x348d0d20, 0, 0x32e73cfe) + 156 bytes [49] Python:PyObject_Call~(0x33da9a08, 0x348d0d20, 0, 0x32e233cd) + 45 bytes [50] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348d0d20, 0, 0x33b25390) + 112 bytes [51] Python:PyEval_EvalFrameEx~(0x33ce5580, 0, 0x33b25390, 0x33b25390) + 8138 bytes [52] Python:PyEval_EvalCodeEx~(0x337b14e8, 0x33b25390, 0x33b25390, 0) + 1819 bytes [53] Python:PyEval_EvalCode~(0x337b14e8, 0x33b25390, 0x33b25390, 0x9325378f) + 87 bytes [54] Python:PyImport_ExecCodeModuleEx~(0xb09822ab "numpy.lib", 0x337b14e8, 0xb098154f "/Library/Frameworks/Python.frame..", 0x4adc73cd) + 193 bytes [55] Python:load_source_module~(1, 0, 0xb0981d9c, 0) + 726 bytes [56] Python:load_package~(5, 0, 0xb09821f8, 0xb09821fc) + 427 bytes [57] Python:import_submodule~(0xb09822b1 "lib", 0x2ea9f074 "lib", 3, 0x32e2cfc9) + 293 bytes [58] Python:load_next~(0xb09822ab "numpy.lib", 0xb09826ac, 5, 0x32e8b5e6) + 195 bytes [59] Python:import_module_level~(0x33768ab0, 0xffffffff, 0xb0982738, 0x32e6df4d "?E?") + 142 bytes [60] Python:PyImport_ImportModuleLevel~(0x2ea9f074 "lib", 0x33b25420, 0x33b25420, 0x33768ab0) + 45 bytes [61] Python:builtin___import__~(0, 0x348d0f30, 0, 0x32e73cfe) + 156 bytes [62] Python:PyObject_Call~(0x33da9a08, 0x348d0f30, 0, 0x32e233cd) + 45 bytes [63] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348d0f30, 0, 0x33b25420) + 112 bytes [64] Python:PyEval_EvalFrameEx~(0x32a90300, 0, 0x33b25420, 0x33b25420) + 8138 bytes [65] Python:PyEval_EvalCodeEx~(0x337b13c8, 0x33b25420, 0x33b25420, 0) + 1819 bytes [66] Python:PyEval_EvalCode~(0x337b13c8, 0x33b25420, 0x33b25420, 0x9325378f) + 87 bytes [67] Python:PyImport_ExecCodeModuleEx~(0xb098346b "numpy.add_newdocs", 0x337b13c8, 0xb0982b5f "/Library/Frameworks/Python.frame..", 0x0358d500) + 193 bytes [68] Python:load_source_module~(1, 0, 0xb09833b8 "??C?", 0xb09833bc) + 726 bytes [69] Python:import_submodule~(0xb0983471 "add_newdocs", 0x33d0a7b4 "add_newdocs", 11, 0x32e2cfc9) + 293 bytes [70] Python:load_next~(0xb098346b "numpy.add_newdocs", 0xb098386c, 0xb0983518, 0x32e8b5e6) + 195 bytes [71] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, 0xe17a9024, 0x32e6df4d "?E?") + 142 bytes [72] Python:PyImport_ImportModuleLevel~(0x33d0a7b4 "add_newdocs", 0x33b25b70, 0x33b25b70, 0x32ee4aa0 "TD") + 45 bytes [73] Python:builtin___import__~(0, 0x348c7ed0, 0, 0x32e73cfe) + 156 bytes [74] Python:PyObject_Call~(0x33da9a08, 0x348c7ed0, 0, 0x32e233cd) + 45 bytes [75] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348c7ed0, 0, 0x33b25b70) + 112 bytes [76] Python:PyEval_EvalFrameEx~(0x34c489a0, 0, 0x33b25b70, 0x33b25b70) + 8138 bytes [77] Python:PyEval_EvalCodeEx~(0x337562f0, 0x33b25b70, 0x33b25b70, 0) + 1819 bytes [78] Python:PyEval_EvalCode~(0x337562f0, 0x33b25b70, 0x33b25b70, 0x9325378f) + 87 bytes [79] Python:PyImport_ExecCodeModuleEx~(0xb0984a7b "numpy", 0x337562f0, 0xb0983d1f "/Library/Frameworks/Python.frame..", 0x4adc73cd) + 193 bytes [80] Python:load_source_module~(1, 0, 0xb098456c, 0) + 726 bytes [81] Python:load_package~(5, 0, 0xb09849c8, 0xb09849cc) + 427 bytes [82] Python:import_submodule~(0xb0984a7b "numpy", 0x304f6134 "numpy", 5, 0x32e21553) + 293 bytes [83] Python:load_next~(0xb0984a7b "numpy", 0xb0984e7c, 0xb0984b28, 0x32e8b5e6) + 195 bytes [84] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, 0x30173290, 0x32e6df4d "?E?") + 142 bytes [85] Python:PyImport_ImportModuleLevel~(0x304f6134 "numpy", 0x33f19420, 0x33f19420, 0x32ee4aa0 "TD") + 45 bytes [86] Python:builtin___import__~(0, 0x33341bd0, 0, 0x32e73cfe) + 156 bytes [87] Python:PyObject_Call~(0x33da9a08, 0x33341bd0, 0, 0x32e233cd) + 45 bytes [88] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x33341bd0, 0, 0x33f19420) + 112 bytes [89] Python:PyEval_EvalFrameEx~(0x336e5be0, 0, 0x33f19420, 0x33f19420) + 8138 bytes [90] Python:PyEval_EvalCodeEx~(0x2eabb9b0, 0x33f19420, 0x33f19420, 0) + 1819 bytes [91] Python:PyEval_EvalCode~(0x2eabb9b0, 0x33f19420, 0x33f19420, 0x33684d60) + 87 bytes [92] Python:PyRun_StringFlags~(0x2dd56f19 "import numpy; numpy.test()", 257, 0x33f19420, 0x33f19420) + 243 bytes [93] Python:PyRun_SimpleStringFlags~(0x2dd56f19 "import numpy; numpy.test()", 0, 0, 0xb0985b88) + 72 bytes [94] libmex.dylib:mexRunMexFile(0, 0xb0985b28, 0, 0xb0985b88) + 107 bytes [95] libmex.dylib:Mfh_mex::runMexFileWithSignalProtection(int, mxArray_tag**, int, mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 111 bytes [96] libmex.dylib:Mfh_mex::dispatch_file(int, mxArray_tag**, int, mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 250 bytes [97] libmwm_dispatcher.dylib:Mfh_file::dispatch_fh(int, mxArray_tag**, int, mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 256 bytes [98] libmwm_interpreter.dylib:ResolverFunctionDesc::CallFunction(int, mxArray_tag**, int, mxArray_tag**)(0xb0986118 "?Gs", 0, 0xb0985b28, 0) + 793 bytes [99] libmwm_interpreter.dylib:Resolver::CallMFunction(int, int, _m_operand*, m_operand_storage*, int, _m_operand*, m_operand_storage*, int*)(0xb0985cf4, 0, 1, 0x301e3f50) + 1446 bytes [100] libmwm_interpreter.dylib:inResolveMFunctionCall(_m_function_desc*, int, int, _m_operand*, m_operand_storage*, int, _m_operand*, m_operand_storage*, int*, inMarshalType*, int, mpsTypeSequenceNlhs const*, mxArray_tag* (*)(int))(0x301beef0, 0, 1, 0x301e3f50) + 474 bytes [101] libmwm_interpreter.dylib:accelImpl::MFunctionCall(_accelOp**)(0xb0986264, 0xb0986278 "XT$.", 0x336380b0, 0x34c89470 "?::operator()() const(0x2b44dc54 "??A", 0, 0xb0986c68, 171953) + 41 bytes [117] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::NoResultInvocationRequest::run()(0x2b44dc40, 0, 0xb0986be8, 172001) + 21 bytes [118] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::invocation_request_handler(long)(0x2b44dc40, 0, 0x027a1a00, 0x3005f880) + 24 bytes [119] libmwuix.dylib:uix_DispatchOrProcess(_XEvent*, _XtAppStruct*, int, bool)(0, 15, 0x000f4240 "put_target_string_with_length", 0x4afd9271) + 476 bytes [120] libmwuix.dylib:ws_ProcessPendingEventsHelper(int, int, bool)(1, 0xffffffff, 0, 852585) + 469 bytes [121] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::process_events(boost::shared_ptr const&)(0x04c3b8e0, 0xb0986e68, 0, 0x04c3b890 "/Applications/MATLAB_R2009a.app/..") + 376 bytes [122] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::run(boost::shared_ptr const&, mcr::runtime::InterpreterThread::Impl::init_context*)(0x04c3b8e0, 0xb0986e68, 0xb0a06cac, 4078061) + 410 bytes [123] libmwmcr.dylib:run_init_and_handle_events(void*)(0xb0a06cac, 0, 0, 0) + 52 bytes [124] MATLAB:create_mcrInstance_and_run_mnParser(0xb0986f00 "@?A", 8816, 0, 0) + 553 bytes [125] MATLAB:start(2, 0xbffff4d0 "????????", 0xb0987000 "DRHT", 0xffffffff) + -2223 bytes [126] libmwmcr.dylib:runMcrMain(void*)(0xbffff440 "?%", 0x04000000, 1, 0) + 39 bytes [127] libSystem.B.dylib:_pthread_start~(0xb0987000 "DRHT", 14083, 0x004075f0, 0xbffff440 "?%") + 345 bytes [128] libSystem.B.dylib:thread_start~(0, 0, 0, 0x54485244) + 34 bytes From robince at gmail.com Fri Nov 13 12:31:31 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 17:31:31 +0000 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> Message-ID: <2d5132a50911130931o15bcc22dtfdcae177802f6642@mail.gmail.com> I forgot to add it doesn't happen if Py_Finalize isn't called - if I take that out and just let the OS/matlab unload the mex function then I can run it many times without the error, but it does leak a bit of memory each time. Cheers Robin On Fri, Nov 13, 2009 at 5:23 PM, Robin wrote: > Hi, > > I'm trying to embed Python in a MATLAB mex file. I've been coming > under some pressure to make my Python code available to my MATLAB > colleagues so I am trying to come up with a relatively general way of > calling numerical python code from Matlab. > > I am making some progress - but get a reliable crash from numpy. This > only occurs the second time I am loading it. So I Py_Initialize, > import numpy, Py_Finalize (all works fine), but then if I clear the > mex file (clear funcname in matlab - which calls Py_Finalize through a > mexAtExit handler) ?and try to run the function again (which will > reinitialize interpreter and import numpy again) I get the followin > stack trace from multisarray.so. Wondered if anyone could through any > light. I have already run into this bug > http://bugs.python.org/issue6869 which prevents me using ctypes... I > wondered if this was related. > > For now its not such a big problem - I will just avoid unloading the > mex function (with clear function). But I thought it might be > indicative of a memory leak or some other problem since I think in > theory it should work (It does if numpy isn't imported). > > Cheers > > Robin > > ------------------------------------------------------------------------ > ? ? ? ? ? ? Bus error detected at Fri Nov 13 17:11:57 2009 > ------------------------------------------------------------------------ > > Configuration: > ?MATLAB Version: ? 7.8.0.347 (R2009a) > ?MATLAB License: ? 161051 > ?Operating System: Darwin 10.0.0 Darwin Kernel Version 10.0.0: Fri > Jul 31 22:47:34 PDT 2009; root:xnu-1456.1.25~1/RELEASE_I386 i386 > ?Window System: ? ?The X.Org Foundation (10402000), display > /tmp/launch-2p1ZWg/:0 > ?Current Visual: ? 0x24 (class 4, depth 24) > ?Processor ID: ? ? x86 Family 6 Model 15 Stepping 10, GenuineIntel > ?Virtual Machine: ?Java 1.6.0_15-b03-219 with Apple Inc. Java > HotSpot(TM) Client VM mixed mode > ?Default Encoding: ?ISO-8859-1 > > Fault Count: 1 > > Register State: > ?eax = 00000001 ?ebx = 307b12ab > ?ecx = 00000000 ?edx = 305ef148 > ?esi = 305ef140 ?edi = 32a79d60 > ?ebp = b097c6c8 ?esp = b097c670 > ?eip = 307b14be ?flg = 00010202 > > Stack Trace: > ?[0] multiarray.so:PyArray_FromScalar~(0x305ef140, 0, 2, 0x32e23287) > + 542 bytes > ?[1] multiarray.so:gentype_nonzero_number~(0x305ef140, 0x2eaa6db0, 0, > 0x32e73cfe) + 36 bytes > ?[2] Python:PyObject_IsTrue~(0x305ef140, 0x2ea95de0, 2, 0) + 63 bytes > ?[3] Python:PyEval_EvalFrameEx~(0x33c95160, 0, 0x332941e0, 0) + 12598 bytes > ?[4] Python:PyEval_EvalFrameEx~(0x32a70f80, 0, 0x332941e0, 0) + 24217 bytes > ?[5] Python:PyEval_EvalCodeEx~(0x33f05068, 0x332941e0, 0, 0x32a96794) > + 1819 bytes > ?[6] Python:PyEval_EvalFrameEx~(0x32a96640, 0, 0x332941e0, 0) + 16561 bytes > ?[7] Python:PyEval_EvalCodeEx~(0x33f051d0, 0x332941e0, 0, 0x32a6fcc0) > + 1819 bytes > ?[8] Python:PyEval_EvalFrameEx~(0x32a6fb60, 0, 0x33bc99c0, 0) + 16561 bytes > ?[9] Python:PyEval_EvalCodeEx~(0x334abda0, 0x33bc99c0, 0, 0x32ad3b24) > + 1819 bytes > ?[10] Python:PyEval_EvalFrameEx~(0x32ad39d0, 0, 0x33bc94b0, 0) + 16561 bytes > ?[11] Python:PyEval_EvalCodeEx~(0x332919b0, 0x33bc94b0, 0, > 0x33ce5294) + 1819 bytes > ?[12] Python:PyEval_EvalFrameEx~(0x33ce5150, 0, 0x33bc94b0, 0) + 16561 bytes > ?[13] Python:PyEval_EvalCodeEx~(0x332918d8, 0x33bc94b0, 0, > 0x34d5156c) + 1819 bytes > ?[14] Python:PyEval_EvalFrameEx~(0x34d51430, 0, 0x33bc9300, > 0x33bc9300) + 16561 bytes > ?[15] Python:PyEval_EvalCodeEx~(0x33291968, 0x33bc9300, 0x33bc9300, > 0) + 1819 bytes > ?[16] Python:PyEval_EvalCode~(0x33291968, 0x33bc9300, 0x33bc9300, > 0x9325378f) + 87 bytes > ?[17] Python:PyImport_ExecCodeModuleEx~(0xb097e4cb > "numpy.core._internal", 0x33291968, 0xb097dbbf > "/Library/Frameworks/Python.frame..", 0x332a2000) + 193 bytes > ?[18] Python:load_source_module~(1, 0, 0xb097e418 "X?C?", 0xb097e41c) > + 726 bytes > ?[19] Python:import_submodule~(0xb097e4d6 "_internal", 0x33d1ce1c > "_internal", 9, 0x32e2cfc9) + 293 bytes > ?[20] Python:load_next~(0xb097e4cb "numpy.core._internal", > 0xb097e8cc, 0xb097e578, 0x32e8b5e6) + 195 bytes > ?[21] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, > 0xee9e70b3, 0x32e6df4d "?E?") + 142 bytes > ?[22] Python:PyImport_ImportModuleLevel~(0x33d1ce1c "_internal", > 0x33b25150, 0x33b25150, 0x32ee4aa0 "TD") + 45 bytes > ?[23] Python:builtin___import__~(0, 0x348854e0, 0, 0x32e73cfe) + 156 bytes > ?[24] Python:PyObject_Call~(0x33da9a08, 0x348854e0, 0, 0x32e233cd) + 45 bytes > ?[25] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348854e0, > 0, 0x33b25150) + 112 bytes > ?[26] Python:PyEval_EvalFrameEx~(0x33cd2da0, 0, 0x33b25150, > 0x33b25150) + 8138 bytes > ?[27] Python:PyEval_EvalCodeEx~(0x33332068, 0x33b25150, 0x33b25150, > 0) + 1819 bytes > ?[28] Python:PyEval_EvalCode~(0x33332068, 0x33b25150, 0x33b25150, > 0x9325378f) + 87 bytes > ?[29] Python:PyImport_ExecCodeModuleEx~(0xb097fadb "numpy.core", > 0x33332068, 0xb097ed7f "/Library/Frameworks/Python.frame..", > 0x4adc73cc) + 193 bytes > ?[30] Python:load_source_module~(1, 0, 0xb097f5cc, 0) + 726 bytes > ?[31] Python:load_package~(5, 0, 0xb097fa28, 0xb097fa2c) + 427 bytes > ?[32] Python:import_submodule~(0xb097fae1 "core", 0x3488536a > "core.numeric", 4, 0x32e2cfc9) + 293 bytes > ?[33] Python:load_next~(0xb097fadb "numpy.core", 0xb097fedc, 9, > 0x32e8b5e6) + 195 bytes > ?[34] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, > 0x746e6920, 0x32e6df4d "?E?") + 213 bytes > ?[35] Python:PyImport_ImportModuleLevel~(0x34885364 > "numpy.core.numeric", 0x33b25a50, 0x33b25a50, 0x32ee4aa0 "TD") + 45 > bytes > ?[36] Python:builtin___import__~(0, 0x34885780, 0, 0x32e73cfe) + 156 bytes > ?[37] Python:PyObject_Call~(0x33da9a08, 0x34885780, 0, 0x32e233cd) + 45 bytes > ?[38] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x34885780, > 0, 0x33b25a50) + 112 bytes > ?[39] Python:PyEval_EvalFrameEx~(0x33cd07a0, 0, 0x33b25a50, > 0x33b25a50) + 8138 bytes > ?[40] Python:PyEval_EvalCodeEx~(0x33332140, 0x33b25a50, 0x33b25a50, > 0) + 1819 bytes > ?[41] Python:PyEval_EvalCode~(0x33332140, 0x33b25a50, 0x33b25a50, > 0x9325378f) + 87 bytes > ?[42] Python:PyImport_ExecCodeModuleEx~(0xb0980c9b > "numpy.lib.type_check", 0x33332140, 0xb098038f > "/Library/Frameworks/Python.frame..", 6771) + 193 bytes > ?[43] Python:load_source_module~(1, 0, 0xb0980be8 "??C?", 0xb0980bec) > + 726 bytes > ?[44] Python:import_submodule~(0xb0980ca5 "type_check", 0x33d182b4 > "type_check", 10, 0x32e2cfc9) + 293 bytes > ?[45] Python:load_next~(0xb0980c9b "numpy.lib.type_check", > 0xb098109c, 0xb0980d48, 0x32e8b5e6) + 195 bytes > ?[46] Python:import_module_level~(0x3082c1b0, 0xffffffff, 0xee9e70b3, > 0x32e6df4d "?E?") + 142 bytes > ?[47] Python:PyImport_ImportModuleLevel~(0x33d182b4 "type_check", > 0x33b25390, 0x33b25390, 0x3082c1b0) + 45 bytes > ?[48] Python:builtin___import__~(0, 0x348d0d20, 0, 0x32e73cfe) + 156 bytes > ?[49] Python:PyObject_Call~(0x33da9a08, 0x348d0d20, 0, 0x32e233cd) + 45 bytes > ?[50] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348d0d20, > 0, 0x33b25390) + 112 bytes > ?[51] Python:PyEval_EvalFrameEx~(0x33ce5580, 0, 0x33b25390, > 0x33b25390) + 8138 bytes > ?[52] Python:PyEval_EvalCodeEx~(0x337b14e8, 0x33b25390, 0x33b25390, > 0) + 1819 bytes > ?[53] Python:PyEval_EvalCode~(0x337b14e8, 0x33b25390, 0x33b25390, > 0x9325378f) + 87 bytes > ?[54] Python:PyImport_ExecCodeModuleEx~(0xb09822ab "numpy.lib", > 0x337b14e8, 0xb098154f "/Library/Frameworks/Python.frame..", > 0x4adc73cd) + 193 bytes > ?[55] Python:load_source_module~(1, 0, 0xb0981d9c, 0) + 726 bytes > ?[56] Python:load_package~(5, 0, 0xb09821f8, 0xb09821fc) + 427 bytes > ?[57] Python:import_submodule~(0xb09822b1 "lib", 0x2ea9f074 "lib", 3, > 0x32e2cfc9) + 293 bytes > ?[58] Python:load_next~(0xb09822ab "numpy.lib", 0xb09826ac, 5, > 0x32e8b5e6) + 195 bytes > ?[59] Python:import_module_level~(0x33768ab0, 0xffffffff, 0xb0982738, > 0x32e6df4d "?E?") + 142 bytes > ?[60] Python:PyImport_ImportModuleLevel~(0x2ea9f074 "lib", > 0x33b25420, 0x33b25420, 0x33768ab0) + 45 bytes > ?[61] Python:builtin___import__~(0, 0x348d0f30, 0, 0x32e73cfe) + 156 bytes > ?[62] Python:PyObject_Call~(0x33da9a08, 0x348d0f30, 0, 0x32e233cd) + 45 bytes > ?[63] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348d0f30, > 0, 0x33b25420) + 112 bytes > ?[64] Python:PyEval_EvalFrameEx~(0x32a90300, 0, 0x33b25420, > 0x33b25420) + 8138 bytes > ?[65] Python:PyEval_EvalCodeEx~(0x337b13c8, 0x33b25420, 0x33b25420, > 0) + 1819 bytes > ?[66] Python:PyEval_EvalCode~(0x337b13c8, 0x33b25420, 0x33b25420, > 0x9325378f) + 87 bytes > ?[67] Python:PyImport_ExecCodeModuleEx~(0xb098346b > "numpy.add_newdocs", 0x337b13c8, 0xb0982b5f > "/Library/Frameworks/Python.frame..", 0x0358d500) + 193 bytes > ?[68] Python:load_source_module~(1, 0, 0xb09833b8 "??C?", 0xb09833bc) > + 726 bytes > ?[69] Python:import_submodule~(0xb0983471 "add_newdocs", 0x33d0a7b4 > "add_newdocs", 11, 0x32e2cfc9) + 293 bytes > ?[70] Python:load_next~(0xb098346b "numpy.add_newdocs", 0xb098386c, > 0xb0983518, 0x32e8b5e6) + 195 bytes > ?[71] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, > 0xe17a9024, 0x32e6df4d "?E?") + 142 bytes > ?[72] Python:PyImport_ImportModuleLevel~(0x33d0a7b4 "add_newdocs", > 0x33b25b70, 0x33b25b70, 0x32ee4aa0 "TD") + 45 bytes > ?[73] Python:builtin___import__~(0, 0x348c7ed0, 0, 0x32e73cfe) + 156 bytes > ?[74] Python:PyObject_Call~(0x33da9a08, 0x348c7ed0, 0, 0x32e233cd) + 45 bytes > ?[75] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x348c7ed0, > 0, 0x33b25b70) + 112 bytes > ?[76] Python:PyEval_EvalFrameEx~(0x34c489a0, 0, 0x33b25b70, > 0x33b25b70) + 8138 bytes > ?[77] Python:PyEval_EvalCodeEx~(0x337562f0, 0x33b25b70, 0x33b25b70, > 0) + 1819 bytes > ?[78] Python:PyEval_EvalCode~(0x337562f0, 0x33b25b70, 0x33b25b70, > 0x9325378f) + 87 bytes > ?[79] Python:PyImport_ExecCodeModuleEx~(0xb0984a7b "numpy", > 0x337562f0, 0xb0983d1f "/Library/Frameworks/Python.frame..", > 0x4adc73cd) + 193 bytes > ?[80] Python:load_source_module~(1, 0, 0xb098456c, 0) + 726 bytes > ?[81] Python:load_package~(5, 0, 0xb09849c8, 0xb09849cc) + 427 bytes > ?[82] Python:import_submodule~(0xb0984a7b "numpy", 0x304f6134 > "numpy", 5, 0x32e21553) + 293 bytes > ?[83] Python:load_next~(0xb0984a7b "numpy", 0xb0984e7c, 0xb0984b28, > 0x32e8b5e6) + 195 bytes > ?[84] Python:import_module_level~(0x32ee4aa0 "TD", 0xffffffff, > 0x30173290, 0x32e6df4d "?E?") + 142 bytes > ?[85] Python:PyImport_ImportModuleLevel~(0x304f6134 "numpy", > 0x33f19420, 0x33f19420, 0x32ee4aa0 "TD") + 45 bytes > ?[86] Python:builtin___import__~(0, 0x33341bd0, 0, 0x32e73cfe) + 156 bytes > ?[87] Python:PyObject_Call~(0x33da9a08, 0x33341bd0, 0, 0x32e233cd) + 45 bytes > ?[88] Python:PyEval_CallObjectWithKeywords~(0x33da9a08, 0x33341bd0, > 0, 0x33f19420) + 112 bytes > ?[89] Python:PyEval_EvalFrameEx~(0x336e5be0, 0, 0x33f19420, > 0x33f19420) + 8138 bytes > ?[90] Python:PyEval_EvalCodeEx~(0x2eabb9b0, 0x33f19420, 0x33f19420, > 0) + 1819 bytes > ?[91] Python:PyEval_EvalCode~(0x2eabb9b0, 0x33f19420, 0x33f19420, > 0x33684d60) + 87 bytes > ?[92] Python:PyRun_StringFlags~(0x2dd56f19 "import numpy; > numpy.test()", 257, 0x33f19420, 0x33f19420) + 243 bytes > ?[93] Python:PyRun_SimpleStringFlags~(0x2dd56f19 "import numpy; > numpy.test()", 0, 0, 0xb0985b88) + 72 bytes > ?[94] libmex.dylib:mexRunMexFile(0, 0xb0985b28, 0, 0xb0985b88) + 107 bytes > ?[95] libmex.dylib:Mfh_mex::runMexFileWithSignalProtection(int, > mxArray_tag**, int, mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 111 > bytes > ?[96] libmex.dylib:Mfh_mex::dispatch_file(int, mxArray_tag**, int, > mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 250 bytes > ?[97] libmwm_dispatcher.dylib:Mfh_file::dispatch_fh(int, > mxArray_tag**, int, mxArray_tag**)(0x301beb70, 0, 0xb0985b28, 0) + 256 > bytes > ?[98] libmwm_interpreter.dylib:ResolverFunctionDesc::CallFunction(int, > mxArray_tag**, int, mxArray_tag**)(0xb0986118 "?Gs", 0, 0xb0985b28, 0) > + 793 bytes > ?[99] libmwm_interpreter.dylib:Resolver::CallMFunction(int, int, > _m_operand*, m_operand_storage*, int, _m_operand*, m_operand_storage*, > int*)(0xb0985cf4, 0, 1, 0x301e3f50) + 1446 bytes > ?[100] libmwm_interpreter.dylib:inResolveMFunctionCall(_m_function_desc*, > int, int, _m_operand*, m_operand_storage*, int, _m_operand*, > m_operand_storage*, int*, inMarshalType*, int, mpsTypeSequenceNlhs > const*, mxArray_tag* (*)(int))(0x301beef0, 0, 1, 0x301e3f50) + 474 > bytes > ?[101] libmwm_interpreter.dylib:accelImpl::MFunctionCall(_accelOp**)(0xb0986264, > 0xb0986278 "XT$.", 0x336380b0, 0x34c89470 "? ?[102] libmwm_interpreter.dylib:accelImpl::Exec()(0xb0986264, > 0x0358a000, 0xb09862a8, 0x336bb9b4) + 199 bytes > ?[103] libmwm_interpreter.dylib:accelCode::Call(inMarshalType*, int*) > const(0x336cd430, 0xb09863d8, 0xb09863d4, 0x009d6e60) + 100 bytes > ?[104] libmwm_interpreter.dylib:inJit::ExecuteHotSegment(_inJitAccelInfo*, > opcodes*, int*, int*)(0xb0986458, 0xb0986468, 0xb0986464 "????", > 0xb0986460) + 1338 bytes > ?[105] libmwm_interpreter.dylib:inExecuteMFunctionOrScript(Mfh_mp*, > bool)(0x336e3cb0, 1, 0xb098699c, 0) + 704 bytes > ?[106] libmwm_interpreter.dylib:inRunMfile(int, mxArray_tag**, int, > mxArray_tag**, Mfh_mp*, inWorkSpace_tag*)(0, 0xb098699c, 0, 0) + 696 > bytes > ?[107] libmwm_interpreter.dylib:Mfh_mp::dispatch_file(int, > mxArray_tag**, int, mxArray_tag**)(0x336e3cb0, 0, 0xb098699c, 0) + 56 > bytes > ?[108] libmwm_dispatcher.dylib:Mfh_file::dispatch_fh(int, > mxArray_tag**, int, mxArray_tag**)(0x336e3cb0, 0, 0xb098699c, 0) + 256 > bytes > ?[109] libmwm_interpreter.dylib:inEvalPcodeHeaderToWord(_memory_context*, > int, mxArray_tag**, _pcodeheader*, Mfh_mp*, unsigned int)(0x000a2ac8 > "?*\n", 0, 0xb098699c, 0xb098683c) + 252 bytes > ?[110] libmwm_interpreter.dylib:inEvalStringWithIsVarFcn(_memory_context*, > char const*, EvalType, int, mxArray_tag**, inDebugCheck, > _pcodeheader*, int*, bool (*)(void*, char const*), void*, bool, > bool)(0, 0xb098699c, 0, 0) + 1835 bytes > ?[111] libmwm_interpreter.dylib:inEvalCmdWithLocalReturn(char const*, > int*, bool, bool, bool (*)(void*, char const*))(1, 0x004aeb20, 0, 0) + > 148 bytes > ?[112] libmwm_interpreter.dylib:inEvalCmdWithLocalReturn(0x2e272200 > "pytest()\n", 0, 0, 1) + 66 bytes > ?[113] libmwbridge.dylib:evalCommandWithLongjmpSafety(char > const*)(0x2e272200 "pytest()\n", 2, 0x3012ff98, 4073223) + 108 bytes > ?[114] libmwbridge.dylib:mnParser(0xb0986b34, 0x0502cc00, 1, 0) + 666 bytes > ?[115] libmwmcr.dylib:mcrInstance::mnParser_on_interpreter_thread()(0x0502cc00, > 8, 0x0384ce00, 4) + 43 bytes > ?[116] libmwmcr.dylib:boost::function0::operator()() > const(0x2b44dc54 "??A", 0, 0xb0986c68, 171953) + 41 bytes > ?[117] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::NoResultInvocationRequest::run()(0x2b44dc40, > 0, 0xb0986be8, 172001) + 21 bytes > ?[118] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::invocation_request_handler(long)(0x2b44dc40, > 0, 0x027a1a00, 0x3005f880) + 24 bytes > ?[119] libmwuix.dylib:uix_DispatchOrProcess(_XEvent*, _XtAppStruct*, > int, bool)(0, 15, 0x000f4240 "put_target_string_with_length", > 0x4afd9271) + 476 bytes > ?[120] libmwuix.dylib:ws_ProcessPendingEventsHelper(int, int, > bool)(1, 0xffffffff, 0, 852585) + 469 bytes > ?[121] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::process_events(boost::shared_ptr > const&)(0x04c3b8e0, 0xb0986e68, 0, 0x04c3b890 > "/Applications/MATLAB_R2009a.app/..") + 376 bytes > ?[122] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::run(boost::shared_ptr > const&, mcr::runtime::InterpreterThread::Impl::init_context*)(0x04c3b8e0, > 0xb0986e68, 0xb0a06cac, 4078061) + 410 bytes > ?[123] libmwmcr.dylib:run_init_and_handle_events(void*)(0xb0a06cac, > 0, 0, 0) + 52 bytes > ?[124] MATLAB:create_mcrInstance_and_run_mnParser(0xb0986f00 "@?A", > 8816, 0, 0) + 553 bytes > ?[125] MATLAB:start(2, 0xbffff4d0 "????????", 0xb0987000 "DRHT", > 0xffffffff) + -2223 bytes > ?[126] libmwmcr.dylib:runMcrMain(void*)(0xbffff440 "?%", 0x04000000, > 1, 0) + 39 bytes > ?[127] libSystem.B.dylib:_pthread_start~(0xb0987000 "DRHT", 14083, > 0x004075f0, 0xbffff440 "?%") + 345 bytes > ?[128] libSystem.B.dylib:thread_start~(0, 0, 0, 0x54485244) + 34 bytes > From millman at berkeley.edu Fri Nov 13 12:51:06 2009 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 13 Nov 2009 09:51:06 -0800 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Fri, Nov 13, 2009 at 3:25 AM, Ralf Gommers wrote: > All wiki changes are now reviewed and can be merged. Under numpy/doc there > is a file HOWTO_MERGE_WIKI_DOCS.txt with details on how this is done. I checked in the majority of the doc changes, but there are some minor problems with the remaining diffs. I have to run now, but I will look at it later today. Thanks, -- Jarrod Millman Helen Wills Neuroscience Institute 10 Giannini Hall, UC Berkeley http://cirl.berkeley.edu/ From robince at gmail.com Fri Nov 13 12:56:01 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 17:56:01 +0000 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911130931o15bcc22dtfdcae177802f6642@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <2d5132a50911130931o15bcc22dtfdcae177802f6642@mail.gmail.com> Message-ID: <2d5132a50911130956u603244abk14386ec52c857924@mail.gmail.com> >> I'm trying to embed Python in a MATLAB mex file. I've been coming >> under some pressure to make my Python code available to my MATLAB >> colleagues so I am trying to come up with a relatively general way of >> calling numerical python code from Matlab. I get a similar but different error trying to do the same with scipy - (importing scipy after a Py_Finalize and matlab function clear) - this time in umath.so: ------------------------------------------------------------------------ Bus error detected at Fri Nov 13 17:53:43 2009 ------------------------------------------------------------------------ Configuration: MATLAB Version: 7.8.0.347 (R2009a) MATLAB License: 161051 Operating System: Darwin 10.0.0 Darwin Kernel Version 10.0.0: Fri Jul 31 22:47:34 PDT 2009; root:xnu-1456.1.25~1/RELEASE_I386 i386 Window System: The X.Org Foundation (10402000), display /tmp/launch-2p1ZWg/:0 Current Visual: 0x24 (class 4, depth 24) Processor ID: x86 Family 6 Model 15 Stepping 10, GenuineIntel Virtual Machine: Java 1.6.0_15-b03-219 with Apple Inc. Java HotSpot(TM) Client VM mixed mode Default Encoding: ISO-8859-1 Fault Count: 1 Register State: eax = 33028046 ebx = 327464cb ecx = 3382b110 edx = 330180b0 esi = 304d6ac0 edi = 336e6660 ebp = b08fa8f8 esp = b08fa8cc eip = 23810006 flg = 00010a92 Stack Trace: [0] 0x23810006(0x336e6660, 0x3352e415 "remainder", 0x32824d74, 0) [1] umath.so:initumath~(0xb08faebb "numpy.core.umath", 0xb08faec6 "umath", 0xb08faa07 "/Library/Frameworks/Python.frame..", 0xa043aab0) + 2152 bytes [2] Python:_PyImport_LoadDynamicModule~(0xb08faebb "numpy.core.umath", 0xb08faa07 "/Library/Frameworks/Python.frame..", 0xa043aab0, 0x331ae7d3) + 153 bytes [3] Python:load_module~(3, 0, 0xb08fae08 "??C?", 0xb08fae0c) + 203 bytes [4] Python:import_submodule~(0xb08faec6 "umath", 0x326eecd4 "umath", 5, 0x33148fc9) + 293 bytes [5] Python:load_next~(0xb08faebb "numpy.core.umath", 0xb08fb2bc, 0xb08faf68, 0x331a75e6) + 195 bytes [6] Python:import_module_level~(0x33200aa0, 0xffffffff, 0xee9e70b3, 0x33189f4d "?E?") + 142 bytes [7] Python:PyImport_ImportModuleLevel~(0x326eecd4 "umath", 0x336340c0, 0x336340c0, 0x33200aa0) + 45 bytes [8] Python:builtin___import__~(0, 0x33845ea0, 0, 0x3318fcfe) + 156 bytes [9] Python:PyObject_Call~(0x326ba7b0, 0x33845ea0, 0, 0x3313f3cd) + 45 bytes [10] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x33845ea0, 0, 0x336340c0) + 112 bytes [11] Python:PyEval_EvalFrameEx~(0x304d5fd0, 0, 0x336340c0, 0x336340c0) + 8138 bytes [12] Python:PyEval_EvalCodeEx~(0x30386e30, 0x336340c0, 0x336340c0, 0) + 1819 bytes [13] Python:PyEval_EvalCode~(0x30386e30, 0x336340c0, 0x336340c0, 0x9325378f) + 87 bytes [14] Python:PyImport_ExecCodeModuleEx~(0xb08fc4cb "numpy.core", 0x30386e30, 0xb08fb76f "/Library/Frameworks/Python.frame..", 0x4adc73cc) + 193 bytes [15] Python:load_source_module~(1, 0, 0xb08fbfbc "X?C?0D80?370", 0) + 726 bytes [16] Python:load_package~(5, 0, 0xb08fc418, 0xb08fc41c) + 427 bytes [17] Python:import_submodule~(0xb08fc4d1 "core", 0x3384512a "core.numeric", 4, 0x33148fc9) + 293 bytes [18] Python:load_next~(0xb08fc4cb "numpy.core", 0xb08fc8cc, 9, 0x331a75e6) + 195 bytes [19] Python:import_module_level~(0x33200aa0, 0xffffffff, 0x746e6920, 0x33189f4d "?E?") + 213 bytes [20] Python:PyImport_ImportModuleLevel~(0x33845124 "numpy.core.numeric", 0x33634ae0, 0x33634ae0, 0x33200aa0) + 45 bytes [21] Python:builtin___import__~(0, 0x33845f30, 0, 0x3318fcfe) + 156 bytes [22] Python:PyObject_Call~(0x326ba7b0, 0x33845f30, 0, 0x3313f3cd) + 45 bytes [23] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x33845f30, 0, 0x33634ae0) + 112 bytes [24] Python:PyEval_EvalFrameEx~(0x304caf70, 0, 0x33634ae0, 0x33634ae0) + 8138 bytes [25] Python:PyEval_EvalCodeEx~(0x30386e78, 0x33634ae0, 0x33634ae0, 0) + 1819 bytes [26] Python:PyEval_EvalCode~(0x30386e78, 0x33634ae0, 0x33634ae0, 0x9325378f) + 87 bytes [27] Python:PyImport_ExecCodeModuleEx~(0xb08fd68b "numpy.lib.type_check", 0x30386e78, 0xb08fcd7f "/Library/Frameworks/Python.frame..", 6771) + 193 bytes [28] Python:load_source_module~(1, 0, 0xb08fd5d8, 0xb08fd5dc) + 726 bytes [29] Python:import_submodule~(0xb08fd695 "type_check", 0x3038a494 "type_check", 10, 0x33148fc9) + 293 bytes [30] Python:load_next~(0xb08fd68b "numpy.lib.type_check", 0xb08fda8c, 0xb08fd738, 0x331a75e6) + 195 bytes [31] Python:import_module_level~(0x32703e30, 0xffffffff, 0xee9e70b3, 0x33189f4d "?E?") + 142 bytes [32] Python:PyImport_ImportModuleLevel~(0x3038a494 "type_check", 0x33634150, 0x33634150, 0x32703e30) + 45 bytes [33] Python:builtin___import__~(0, 0x33845540, 0, 0x3318fcfe) + 156 bytes [34] Python:PyObject_Call~(0x326ba7b0, 0x33845540, 0, 0x3313f3cd) + 45 bytes [35] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x33845540, 0, 0x33634150) + 112 bytes [36] Python:PyEval_EvalFrameEx~(0x304b4fd0, 0, 0x33634150, 0x33634150) + 8138 bytes [37] Python:PyEval_EvalCodeEx~(0x32727bf0, 0x33634150, 0x33634150, 0) + 1819 bytes [38] Python:PyEval_EvalCode~(0x32727bf0, 0x33634150, 0x33634150, 0x9325378f) + 87 bytes [39] Python:PyImport_ExecCodeModuleEx~(0xb08fec9b "numpy.lib", 0x32727bf0, 0xb08fdf3f "/Library/Frameworks/Python.frame..", 0x4adc73cd) + 193 bytes [40] Python:load_source_module~(1, 0, 0xb08fe78c "??C?0D80?370", 0) + 726 bytes [41] Python:load_package~(5, 0, 0xb08febe8, 0xb08febec) + 427 bytes [42] Python:import_submodule~(0xb08feca1 "lib", 0x3271b4b4 "lib", 3, 0x33148fc9) + 293 bytes [43] Python:load_next~(0xb08fec9b "numpy.lib", 0xb08ff09c, 5, 0x331a75e6) + 195 bytes [44] Python:import_module_level~(0x303975f0, 0xffffffff, 0xb08ff128, 0x33189f4d "?E?") + 142 bytes [45] Python:PyImport_ImportModuleLevel~(0x3271b4b4 "lib", 0x336346f0, 0x336346f0, 0x303975f0) + 45 bytes [46] Python:builtin___import__~(0, 0x33845ae0, 0, 0x3318fcfe) + 156 bytes [47] Python:PyObject_Call~(0x326ba7b0, 0x33845ae0, 0, 0x3313f3cd) + 45 bytes [48] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x33845ae0, 0, 0x336346f0) + 112 bytes [49] Python:PyEval_EvalFrameEx~(0x304b6570, 0, 0x336346f0, 0x336346f0) + 8138 bytes [50] Python:PyEval_EvalCodeEx~(0x327274a0, 0x336346f0, 0x336346f0, 0) + 1819 bytes [51] Python:PyEval_EvalCode~(0x327274a0, 0x336346f0, 0x336346f0, 0x9325378f) + 87 bytes [52] Python:PyImport_ExecCodeModuleEx~(0xb08ffe5b "numpy.add_newdocs", 0x327274a0, 0xb08ff54f "/Library/Frameworks/Python.frame..", 0x331491e4) + 193 bytes [53] Python:load_source_module~(1, 0, 0xb08ffda8 "P?C?", 0xb08ffdac) + 726 bytes [54] Python:import_submodule~(0xb08ffe61 "add_newdocs", 0x33629df4 "add_newdocs", 11, 0x33148fc9) + 293 bytes [55] Python:load_next~(0xb08ffe5b "numpy.add_newdocs", 0xb090025c, 0xb08fff08, 0x331a75e6) + 195 bytes [56] Python:import_module_level~(0x33200aa0, 0xffffffff, 0xe17a9024, 0x33189f4d "?E?") + 142 bytes [57] Python:PyImport_ImportModuleLevel~(0x33629df4 "add_newdocs", 0x3384e780, 0x3384e780, 0x33200aa0) + 45 bytes [58] Python:builtin___import__~(0, 0x33845750, 0, 0x3318fcfe) + 156 bytes [59] Python:PyObject_Call~(0x326ba7b0, 0x33845750, 0, 0x3313f3cd) + 45 bytes [60] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x33845750, 0, 0x3384e780) + 112 bytes [61] Python:PyEval_EvalFrameEx~(0x304bcad0, 0, 0x3384e780, 0x3384e780) + 8138 bytes [62] Python:PyEval_EvalCodeEx~(0x33821848, 0x3384e780, 0x3384e780, 0) + 1819 bytes [63] Python:PyEval_EvalCode~(0x33821848, 0x3384e780, 0x3384e780, 0x9325378f) + 87 bytes [64] Python:PyImport_ExecCodeModuleEx~(0xb0901471 "numpy", 0x33821848, 0xb090070f "/Library/Frameworks/Python.frame..", 0x4adc73cd) + 193 bytes [65] Python:load_source_module~(1, 0, 0xb0900f5c "??C? D80 D80", 0) + 726 bytes [66] Python:load_package~(5, 0, 0xb09013b8, 0xb09013bc) + 427 bytes [67] Python:import_submodule~(0xb0901471 "numpy", 0x326b0474 "numpy", 5, 0x33148fc9) + 293 bytes [68] Python:load_next~(0xb090146b "scipy.numpy", 0xb090186c, 0xb0901518, 0x331a75e6) + 313 bytes [69] Python:import_module_level~(0x30392cd0, 0xffffffff, 0x304bf7e0, 0x33189f4d "?E?") + 142 bytes [70] Python:PyImport_ImportModuleLevel~(0x326b0474 "numpy", 0x33584a50, 0x33584a50, 0x30392cd0) + 45 bytes [71] Python:builtin___import__~(0, 0x3360c510, 0, 0x3318fcfe) + 156 bytes [72] Python:PyObject_Call~(0x326ba7b0, 0x3360c510, 0, 0x3313f3cd) + 45 bytes [73] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x3360c510, 0, 0x33584a50) + 112 bytes [74] Python:PyEval_EvalFrameEx~(0x304bc7a0, 0, 0x33584a50, 0x33584a50) + 8138 bytes [75] Python:PyEval_EvalCodeEx~(0x338213c8, 0x33584a50, 0x33584a50, 0) + 1819 bytes [76] Python:PyEval_EvalCode~(0x338213c8, 0x33584a50, 0x33584a50, 0x9325378f) + 87 bytes [77] Python:PyImport_ExecCodeModuleEx~(0xb0902a7b "scipy", 0x338213c8, 0xb0901d1f "/Library/Frameworks/Python.frame..", 0x4adc73ef) + 193 bytes [78] Python:load_source_module~(1, 0, 0xb090256c, 0) + 726 bytes [79] Python:load_package~(5, 0, 0xb09029c8, 0xb09029cc) + 427 bytes [80] Python:import_submodule~(0xb0902a7b "scipy", 0x326b00f4 "scipy", 5, 0x3313d553) + 293 bytes [81] Python:load_next~(0xb0902a7b "scipy", 0xb0902e7c, 0xb0902b28, 0x331a75e6) + 195 bytes [82] Python:import_module_level~(0x33200aa0, 0xffffffff, 0x3049d1e0, 0x33189f4d "?E?") + 142 bytes [83] Python:PyImport_ImportModuleLevel~(0x326b00f4 "scipy", 0x30376540, 0x30376540, 0x33200aa0) + 45 bytes [84] Python:builtin___import__~(0, 0x3360c960, 0, 0x3318fcfe) + 156 bytes [85] Python:PyObject_Call~(0x326ba7b0, 0x3360c960, 0, 0x3313f3cd) + 45 bytes [86] Python:PyEval_CallObjectWithKeywords~(0x326ba7b0, 0x3360c960, 0, 0x30376540) + 112 bytes [87] Python:PyEval_EvalFrameEx~(0x304bdee0, 0, 0x30376540, 0x30376540) + 8138 bytes [88] Python:PyEval_EvalCodeEx~(0x33551530, 0x30376540, 0x30376540, 0) + 1819 bytes [89] Python:PyEval_EvalCode~(0x33551530, 0x30376540, 0x30376540, 0x304d8ae0) + 87 bytes [90] Python:PyRun_StringFlags~(0x304c9910 "import scipy", 257, 0x30376540, 0x30376540) + 243 bytes [91] Python:PyRun_SimpleStringFlags~(0x304c9910 "import scipy", 0, 1, 0xb0903b88 "?!Z(") + 72 bytes [92] libmex.dylib:mexRunMexFile(0, 0xb0903b28, 1, 0xb0903b88 "?!Z(") + 107 bytes [93] libmex.dylib:Mfh_mex::runMexFileWithSignalProtection(int, mxArray_tag**, int, mxArray_tag**)(0x304b1290, 0, 0xb0903b28, 1) + 111 bytes [94] libmex.dylib:Mfh_mex::dispatch_file(int, mxArray_tag**, int, mxArray_tag**)(0x304b1290, 0, 0xb0903b28, 1) + 250 bytes [95] libmwm_dispatcher.dylib:Mfh_file::dispatch_fh(int, mxArray_tag**, int, mxArray_tag**)(0x304b1290, 0, 0xb0903b28, 1) + 256 bytes [96] libmwm_interpreter.dylib:ResolverFunctionDesc::CallFunction(int, mxArray_tag**, int, mxArray_tag**)(0xb0904118 "?Gs", 0, 0xb0903b28, 1) + 793 bytes [97] libmwm_interpreter.dylib:Resolver::CallMFunction(int, int, _m_operand*, m_operand_storage*, int, _m_operand*, m_operand_storage*, int*)(0xb0903cf4, 0, 1, 0x304a8f50) + 1446 bytes [98] libmwm_interpreter.dylib:inResolveMFunctionCall(_m_function_desc*, int, int, _m_operand*, m_operand_storage*, int, _m_operand*, m_operand_storage*, int*, inMarshalType*, int, mpsTypeSequenceNlhs const*, mxArray_tag* (*)(int))(0x3049d5c0, 0, 1, 0x304a8f50) + 474 bytes [99] libmwm_interpreter.dylib:accelImpl::MFunctionCall(_accelOp**)(0xb0904264, 0xb0904278, 0x304d32f0, 0x2fff5b20 "`?I0") + 269 bytes [100] libmwm_interpreter.dylib:accelImpl::Exec()(0xb0904264, 0x0358a000, 0xb09042a8, 0x3049ca14) + 199 bytes [101] libmwm_interpreter.dylib:accelCode::Call(inMarshalType*, int*) const(0x3049e8c0, 0xb09043d8, 0xb09043d4, 0x009d6e60) + 100 bytes [102] libmwm_interpreter.dylib:inJit::ExecuteHotSegment(_inJitAccelInfo*, opcodes*, int*, int*)(0xb0904458, 0xb0904468, 0xb0904464 "????", 0xb0904460) + 1338 bytes [103] libmwm_interpreter.dylib:inExecuteMFunctionOrScript(Mfh_mp*, bool)(0x3049e660, 1, 0xb090499c, 0) + 704 bytes [104] libmwm_interpreter.dylib:inRunMfile(int, mxArray_tag**, int, mxArray_tag**, Mfh_mp*, inWorkSpace_tag*)(0, 0xb090499c, 0, 0) + 696 bytes [105] libmwm_interpreter.dylib:Mfh_mp::dispatch_file(int, mxArray_tag**, int, mxArray_tag**)(0x3049e660, 0, 0xb090499c, 0) + 56 bytes [106] libmwm_dispatcher.dylib:Mfh_file::dispatch_fh(int, mxArray_tag**, int, mxArray_tag**)(0x3049e660, 0, 0xb090499c, 0) + 256 bytes [107] libmwm_interpreter.dylib:inEvalPcodeHeaderToWord(_memory_context*, int, mxArray_tag**, _pcodeheader*, Mfh_mp*, unsigned int)(0x000a2ac8 "?*\n", 0, 0xb090499c, 0xb090483c) + 252 bytes [108] libmwm_interpreter.dylib:inEvalStringWithIsVarFcn(_memory_context*, char const*, EvalType, int, mxArray_tag**, inDebugCheck, _pcodeheader*, int*, bool (*)(void*, char const*), void*, bool, bool)(0, 0xb090499c, 0, 0) + 1835 bytes [109] libmwm_interpreter.dylib:inEvalCmdWithLocalReturn(char const*, int*, bool, bool, bool (*)(void*, char const*))(1, 0x004aeb20, 0, 0) + 148 bytes [110] libmwm_interpreter.dylib:inEvalCmdWithLocalReturn(0x05531000 "pytest('import scipy')\n", 0, 0, 1) + 66 bytes [111] libmwbridge.dylib:evalCommandWithLongjmpSafety(char const*)(0x05531000 "pytest('import scipy')\n", 1, 0x3041e7c8, 4073223) + 108 bytes [112] libmwbridge.dylib:mnParser(0xb0904b34, 0x0384fc00, 1, 0) + 666 bytes [113] libmwmcr.dylib:mcrInstance::mnParser_on_interpreter_thread()(0x0384fc00, 8, 0x05035000, 4) + 43 bytes [114] libmwmcr.dylib:boost::function0::operator()() const(0x3041beb4 "??A", 0, 0xb0904c68, 171953) + 41 bytes [115] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::NoResultInvocationRequest::run()(0x3041bea0, 0, 0xb0904be8, 172001) + 21 bytes [116] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::invocation_request_handler(long)(0x3041bea0, 0, 0xb0904c38, 3636709) + 24 bytes [117] libmwuix.dylib:uix_DispatchOrProcess(_XEvent*, _XtAppStruct*, int, bool)(0, 15, 0x000f4240 "put_target_string_with_length", 0x4afd9d18) + 476 bytes [118] libmwuix.dylib:ws_ProcessPendingEventsHelper(int, int, bool)(1, 0xffffffff, 0, 852585) + 469 bytes [119] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::process_events(boost::shared_ptr const&)(0x0363b7a0, 0xb0904e68, 0, 0x036148e0 "/Applications/MATLAB_R2009a.app/..") + 376 bytes [120] libmwmcr.dylib:mcr::runtime::InterpreterThread::Impl::run(boost::shared_ptr const&, mcr::runtime::InterpreterThread::Impl::init_context*)(0x0363b7a0, 0xb0904e68, 0xb0984cac, 4078061) + 410 bytes [121] libmwmcr.dylib:run_init_and_handle_events(void*)(0xb0984cac, 0, 0, 0) + 52 bytes [122] MATLAB:create_mcrInstance_and_run_mnParser(0xb0904f00 "@?A", 8816, 0, 0) + 553 bytes [123] MATLAB:start(2, 0xbffff4d0 "????????", 0xb0905000 "DRHT", 0xffffffff) + -2223 bytes [124] libmwmcr.dylib:runMcrMain(void*)(0xbffff440 "?%", 0x04000000, 1, 0) + 39 bytes [125] libSystem.B.dylib:_pthread_start~(0xb0905000 "DRHT", 13827, 0x004075f0, 0xbffff440 "?%") + 345 bytes [126] libSystem.B.dylib:thread_start~(0, 0, 0, 0x54485244) + 34 bytes From oliphant at enthought.com Fri Nov 13 13:05:42 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 13 Nov 2009 12:05:42 -0600 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> Message-ID: <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> On Nov 13, 2009, at 11:23 AM, Robin wrote: > Hi, > > I'm trying to embed Python in a MATLAB mex file. I've been coming > under some pressure to make my Python code available to my MATLAB > colleagues so I am trying to come up with a relatively general way of > calling numerical python code from Matlab. > > I am making some progress - but get a reliable crash from numpy. This > only occurs the second time I am loading it. So I Py_Initialize, > import numpy, Py_Finalize (all works fine), but then if I clear the > mex file (clear funcname in matlab - which calls Py_Finalize through a > mexAtExit handler) and try to run the function again (which will > reinitialize interpreter and import numpy again) I get the followin > stack trace from multisarray.so. Wondered if anyone could through any > light. I have already run into this bug > http://bugs.python.org/issue6869 which prevents me using ctypes... I > wondered if this was related. > > For now its not such a big problem - I will just avoid unloading the > mex function (with clear function). But I thought it might be > indicative of a memory leak or some other problem since I think in > theory it should work (It does if numpy isn't imported). I wonder if this is related to the fact that you can't "unload" a dynamically linked module (like NumPy has). So, when you call Py_Finalize you are not really "finalizing" your usage of Python extension modules. I'm not sure though. -Travis From mdroe at stsci.edu Fri Nov 13 13:11:50 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Fri, 13 Nov 2009 13:11:50 -0500 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911121456g13239d0bub347ea22fdb63126@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911041111g2dc60877p69f9e8afcbeaeba9@mail.gmail.com> <4AF1DC28.6030403@stsci.edu> <5b8d13220911041757o5273bf30t530f253c52793c3d@mail.gmail.com> <4AF9D88F.2000906@stsci.edu> <5b8d13220911101537y4bfc12dcy421973adc0e3b3a8@mail.gmail.com> <4AFAD967.7030207@stsci.edu> <5b8d13220911110745s71625afeo54f899738ffce2b1@mail.gmail.com> <5b8d13220911120337y27450b09x91e0fe346d81582e@mail.gmail.com> <4AFC1D05.70003@stsci.edu> <5b8d13220911121456g13239d0bub347ea22fdb63126@mail.gmail.com> Message-ID: <4AFDA166.9060709@stsci.edu> On 11/12/2009 05:56 PM, David Cournapeau wrote: > On Thu, Nov 12, 2009 at 11:34 PM, Michael Droettboom wrote: > >> I'm happy to make these changes, since I've got access to the "finicky" >> platform, but want to confirm how you would prefer it done first. >> > Cool. The changes should be done for all platforms, and pointer cast > should be done through union, not (*(new type*)(&x)). > I've committed this change in r7731. It's been tested in x86/gcc, x86_64/gcc (both by "forcing" it to use the npymath version of nextafterl) and sparc/Sun Studio. It probably could use some testing on platforms with other "long double" types. Mike From ralf.gommers at googlemail.com Fri Nov 13 13:12:49 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 13 Nov 2009 19:12:49 +0100 Subject: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december. In-Reply-To: References: <5b8d13220911020029q1d9f1bd7ia1770e3b93e6e653@mail.gmail.com> Message-ID: On Fri, Nov 13, 2009 at 6:51 PM, Jarrod Millman wrote: > On Fri, Nov 13, 2009 at 3:25 AM, Ralf Gommers > wrote: > > All wiki changes are now reviewed and can be merged. Under numpy/doc > there > > is a file HOWTO_MERGE_WIKI_DOCS.txt with details on how this is done. > > I checked in the majority of the doc changes, but there are some minor > problems with the remaining diffs. I have to run now, but I will look > at it later today. > > Thanks Jarrod. With minor problems I assume you are talking about the error messages at the top of the diff. Most of those are not terribly important items, some are for things like constants for which only the html docs make sense anyway. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Nov 13 13:13:39 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 13 Nov 2009 12:13:39 -0600 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> Message-ID: <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> On Fri, Nov 13, 2009 at 12:05, Travis Oliphant wrote: > > On Nov 13, 2009, at 11:23 AM, Robin wrote: > >> Hi, >> >> I'm trying to embed Python in a MATLAB mex file. I've been coming >> under some pressure to make my Python code available to my MATLAB >> colleagues so I am trying to come up with a relatively general way of >> calling numerical python code from Matlab. >> >> I am making some progress - but get a reliable crash from numpy. This >> only occurs the second time I am loading it. So I Py_Initialize, >> import numpy, Py_Finalize (all works fine), but then if I clear the >> mex file (clear funcname in matlab - which calls Py_Finalize through a >> mexAtExit handler) ?and try to run the function again (which will >> reinitialize interpreter and import numpy again) I get the followin >> stack trace from multisarray.so. Wondered if anyone could through any >> light. I have already run into this bug >> http://bugs.python.org/issue6869 which prevents me using ctypes... I >> wondered if this was related. >> >> For now its not such a big problem - I will just avoid unloading the >> mex function (with clear function). But I thought it might be >> indicative of a memory leak or some other problem since I think in >> theory it should work (It does if numpy isn't imported). > > I wonder if this is related to the fact that you can't "unload" a > dynamically linked module (like NumPy has). ? So, when you call > Py_Finalize you are not really "finalizing" your usage of Python > extension modules. > > I'm not sure though. Right. We do some global things when numpy is imported. Since there is no unload step for extension modules, we can't undo them. The second time the interpreter starts up, it doesn't know that numpy has already been loaded and that numpy shouldn't try to do those global things again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robince at gmail.com Fri Nov 13 13:24:34 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 18:24:34 +0000 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> Message-ID: <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> On Fri, Nov 13, 2009 at 6:13 PM, Robert Kern wrote: > On Fri, Nov 13, 2009 at 12:05, Travis Oliphant wrote: >> I wonder if this is related to the fact that you can't "unload" a >> dynamically linked module (like NumPy has). ? So, when you call >> Py_Finalize you are not really "finalizing" your usage of Python >> extension modules. >> >> I'm not sure though. > > Right. We do some global things when numpy is imported. Since there is > no unload step for extension modules, we can't undo them. The second > time the interpreter starts up, it doesn't know that numpy has already > been loaded and that numpy shouldn't try to do those global things > again. Thanks for the quick responses... I'm sure you're right - in fact it looks like there was a similar issue here: http://www.mathworks.co.uk/matlabcentral/newsreader/view_thread/262089 I had assumed when matlab unloads the mex function it would also unload python - but it looks like other dynamic libs pulled in from the mex function (in this case python and in turn numpy) aren't unloaded... I wonder if it would be possible to link python statically to my mex function, so it really is unloaded when the mex function is... but I'm getting a bit out of my depth with linker options, and I guess numpy is always loaded dynamically anyway and will stick around. Easy enough to work around it anyway - but just wanted to check what was going on. Cheers Robin From pav+sp at iki.fi Fri Nov 13 13:50:15 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Fri, 13 Nov 2009 18:50:15 +0000 (UTC) Subject: [Numpy-discussion] bus error in embedded numpy References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> Message-ID: Fri, 13 Nov 2009 17:23:19 +0000, Robin wrote: > I'm trying to embed Python in a MATLAB mex file. I've been coming under > some pressure to make my Python code available to my MATLAB colleagues > so I am trying to come up with a relatively general way of calling > numerical python code from Matlab. Out of curiosity, how are you handling the Matlab RTLD_GLOBAL issue. Last time [1] I did something similar, I had to hack around it in an ugly way. .. [1] http://www.iki.fi/pav/software/pythoncall/index.html -- Pauli Virtanen From robince at gmail.com Fri Nov 13 14:12:26 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 19:12:26 +0000 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> Message-ID: <2d5132a50911131112t28436052w5b93a30be2027136@mail.gmail.com> On Fri, Nov 13, 2009 at 6:50 PM, Pauli Virtanen wrote: > Fri, 13 Nov 2009 17:23:19 +0000, Robin wrote: >> I'm trying to embed Python in a MATLAB mex file. I've been coming under >> some pressure to make my Python code available to my MATLAB colleagues >> so I am trying to come up with a relatively general way of calling >> numerical python code from Matlab. > > Out of curiosity, how are you handling the Matlab RTLD_GLOBAL issue. Last > time [1] I did something similar, I had to hack around it in an ugly way. > > .. [1] http://www.iki.fi/pav/software/pythoncall/index.html Actually I was completely unaware of it (the RTLD_GLOBAL thing). I had googled for a while trying to find a similar project (I had assumed someone must have done it) but somehow didn't find your pythoncall project. It's great - the interface is very close to what I had in mind, but it's much more comprehensive then anything I thought of (I was hoping to handle just contiguous numpy arrays). How does the RTLD_GLOBAL thing manifest itself? So far I have only a very basic start which basically consists of: cmd = mxArrayToString(prhs[0]); PyRun_SimpleString(cmd); but I haven't noticed anything not working - I can run numpy testsuite, and do simple commands as expected (initiliase arrays in the interpreter, run numpy functions on them). Perhaps recent versions of Matlab behave differently (I am using R2009a on a mac). So far the only remotely tricky thing I did was redirect sys.stdout and sys.stderr to a wrapper that uses mexPrintf so output goes to the matlab console. Cheers Robin From pav+sp at iki.fi Fri Nov 13 14:26:23 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Fri, 13 Nov 2009 19:26:23 +0000 (UTC) Subject: [Numpy-discussion] bus error in embedded numpy References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <2d5132a50911131112t28436052w5b93a30be2027136@mail.gmail.com> Message-ID: Fri, 13 Nov 2009 19:12:26 +0000, Robin wrote: [clip] > How does the RTLD_GLOBAL thing manifest itself? So far I have only a > very basic start which basically consists of: > cmd = mxArrayToString(prhs[0]); > PyRun_SimpleString(cmd); > but I haven't noticed anything not working - I can run numpy testsuite, > and do simple commands as expected (initiliase arrays in the > interpreter, run numpy functions on them). Perhaps recent versions of > Matlab behave differently (I am using R2009a on a mac). The RTLD_GLOBAL issue prevented Numpy from being imported. I think everything was well, until you tried to run "import numpy" in the embedded process -- loading multiarray.so would fail because of missing symbols. But if it worked for you without the hack, then it must have been changed in the Matlab versions since then (and Pythoncall needs updating...). -- Pauli Virtanen From sturla at molden.no Fri Nov 13 14:48:50 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Nov 2009 20:48:50 +0100 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> Message-ID: <4AFDB822.6060202@molden.no> Robin skrev: > I had assumed when matlab unloads the mex function it would also > unload python - but it looks like other dynamic libs pulled in from > the mex function (in this case python and in turn numpy) aren't > unloaded... > Matlab MEX functions are DLLs, Python interpreter is a DLL, NumPy .pyd files are DLLs... If you cannot unload Python or NumPy, you cannot unload MEX functions either. The same OS constraints apply. If you are using Windows, I would recommend that you expose your NumPy code as a COM object using pywin32, and make a Matlab wrapper (ordinary .m file) that calls the COM object. If you are not using Windows, you could create an XMLRPC server in Python and call that using Matlab's built-in Java VM. Or you can spawn a separate Python process from Matlab, and communicate using pipes or a tcp/ip socket (Matlab's Java or MEX). There are many solutions that are easier than embedding Python in Matlab. Sturla From sturla at molden.no Fri Nov 13 14:58:53 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Nov 2009 20:58:53 +0100 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911131112t28436052w5b93a30be2027136@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <2d5132a50911131112t28436052w5b93a30be2027136@mail.gmail.com> Message-ID: <4AFDBA7D.8000304@molden.no> Robin skrev: > So far the only remotely tricky thing I did was redirect sys.stdout > and sys.stderr to a wrapper that uses mexPrintf so output goes to the > matlab console. > Be careful when you are using file handles. You have to be sure that Matlab, Python and NumPy are all linked against the same CRT. Sturla From robince at gmail.com Fri Nov 13 15:10:54 2009 From: robince at gmail.com (Robin) Date: Fri, 13 Nov 2009 20:10:54 +0000 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <4AFDB822.6060202@molden.no> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> <4AFDB822.6060202@molden.no> Message-ID: <2d5132a50911131210h190be556gef7d8b21681af787@mail.gmail.com> On Fri, Nov 13, 2009 at 7:48 PM, Sturla Molden wrote: > Robin skrev: >> I had assumed when matlab unloads the mex function it would also >> unload python - but it looks like other dynamic libs pulled in from >> the mex function (in this case python and in turn numpy) aren't >> unloaded... >> > Matlab MEX functions are DLLs, Python interpreter is a DLL, NumPy .pyd > files are DLLs... If you cannot unload Python or NumPy, you cannot > unload MEX functions either. The same OS constraints apply. Ah, I hadn't realised it was an OS constraint - I thought it was possible to unload dlls - and that was why matlab provides the clear function. mex automatically clears a function when you rebuild it - I thought that was how you can rebuild and reload mex functions without restarting matlab. I thought it was just a quirk of matlab that there was no way to unload other libraries pulled in through the mex function. I must admit to being a bit out of my depth with this stuff though so I stand corrected. > If you are using Windows, I would recommend that you expose your NumPy > code as a COM object using pywin32, and make a Matlab wrapper (ordinary > .m file) that calls the COM object. If you are not using Windows, you > could create an XMLRPC server in Python and call that using Matlab's > built-in Java VM. > > Or you can spawn a separate Python process from Matlab, and communicate > using pipes or a tcp/ip socket (Matlab's Java or MEX). > > There are many solutions that are easier than embedding Python in Matlab. I really want a cross platform solution so that rules out COM. I thought about using web services or something which I guess would be the easiest way to do communication through a tcp socket (least work since I could use existing web services libraries on both sides). But actually I have found it pretty easy to embed Python so far... with about 5 lines of code I'm able run simple strings and with a small cython module I get stdout to the console. I haven't tried passing data back and forth yet but from Pauli's pythoncall it doesn't look like that is too difficult. I was sort of hoping that eventually I could create a numpy array from a matlab data pointer - at least for read only input - so that I could use python code to work on large amounts of data without so much memory overhead (returning smaller results by copying). Do you think I'm likely to run into more problems? I got the feeling from asking questions on IRC that embedding Python is kind of discouraged but I thought in this case it seemed the neatest way. > Be careful when you are using file handles. You have to be sure that > Matlab, Python and NumPy are all linked against the same CRT. I was aware of this - I thought I would be OK on the mac - at least Python and Numpy and my mex function are built with apple gcc although I'm not sure about Matlab. I guess Windows will be more difficult... But in any case I don't plan to pass any file handles around. Cheers Robin From Chris.Barker at noaa.gov Fri Nov 13 15:27:22 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 13 Nov 2009 12:27:22 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> Message-ID: <4AFDC12A.4070304@noaa.gov> Anne Archibald wrote: >>> 2009/11/10 Christopher Barker : >>>> I have a bunch of points in 2-d space, and I need to find out which >>>> pairs of points are within a certain distance of one-another (regular >>>> old Euclidean norm). > This is now implemented in SVN. Wow! great -- you sounded interested, but I had no idea you'd run out and do it! thanks! we'll check it out. > I (tentatively?) used a set to store > the collection of pairs, because my tree traversal is not smart enough > to reliably uniquify the pairs without using sets. With more time and > energy, I'm sure the algorithm could be improved to avoid using sets > (both internally and on return), but I think that's something to save > for the Cython version. I agree -- what's wrong with using a set? Thanks, we'll let you know how it works for us. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From sturla at molden.no Fri Nov 13 16:36:00 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Nov 2009 22:36:00 +0100 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <2d5132a50911131210h190be556gef7d8b21681af787@mail.gmail.com> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> <4AFDB822.6060202@molden.no> <2d5132a50911131210h190be556gef7d8b21681af787@mail.gmail.com> Message-ID: <4AFDD140.1010602@molden.no> Robin skrev: > Ah, I hadn't realised it was an OS constraint - I thought it was > possible to unload dlls - and that was why matlab provides the clear > function. mex automatically clears a function when you rebuild it - I > thought that was how you can rebuild and reload mex functions without > restarting matlab. It is OS dependent if DLLs can be unloaded or not. In Windows you can call CloseHandle on the DLL handle and it's gone. If you can unload a MEX function you can also unload Python or NumPy. But when you unload the MEX function, the DLLs loaded by the MEX are not automatically cleared. There is no garbage collector. In Windows, a load and an unload will result in calls to a function called "DllMain" in the DLL. And if the DLL has other DLLs to load or clear, that is where you need to place it. You can put a custom DllMain function in a MEX file. But be careful: a DLL is only loaded once, i.e. DLLs are imported as singletons in the process. If two MEX functions load Python26.dll, they get handles to the same instance. If you unload Python26.dll in one MEX, it is unloaded globally from the process, so the other MEX will crash. There is no reference counting by the OS kernel. The solution to this type of DLL Hell in Windows is COM. A COM object is a DLL but not a singleton. You can have multiple instances of the save COM object in one process. On Mac I guess your options are to either statically link everything to the MEX file, or find a way for multiple MEX files to share Python interpreter, e.g. implement some sort of reference counting scheme, which by the way is what COM does on Windows. > I really want a cross platform solution so that rules out COM. CORBA or XMLRPC seem to be the standards. I'm not sure I would use either. > Do you think I'm likely to run into more problems? I got the feeling > from asking questions on IRC that embedding Python is kind of > discouraged but I thought in this case it seemed the neatest way. > > It is discouraged because you get a singleton. You can create subinterpreters (cf. Apache's mod_python), but that will bring problems of it's own. For example you cannot use extensions that require the simplified GIL API (e.g. ctypes). I think this is a major design flaw of CPython. For example with Java's JNI, you get a context pointer to the VM, so you don't have any of these problems. But with CPython, both the interpreter and extensions are basically implemented to be loaded as singletons. > I was aware of this - I thought I would be OK on the mac - at least > Python and Numpy and my mex function are built with apple gcc although > I'm not sure about Matlab. I guess Windows will be more difficult... > But in any case I don't plan to pass any file handles around. > > It applies to any CRT resource, not just files. Compiler is not important, but which CRT is loaded. And if you statically link the same CRT twice, that becomes two CRTs that cannot share resources. In Windows, Microsoft has made sure there are multiple versions of their CRT (msvcrt.dll, msvcr71.dll, msvcr80.dll, msvcr90.dll, ...) that cannot share resources. And anyone not careful with this can experice crashes at random locations. From peridot.faceted at gmail.com Fri Nov 13 16:39:02 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 13 Nov 2009 16:39:02 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4AFDC12A.4070304@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFDC12A.4070304@noaa.gov> Message-ID: 2009/11/13 Christopher Barker : > Anne Archibald wrote: >>>> 2009/11/10 Christopher Barker : >>>>> I have a bunch of points in 2-d space, and I need to find out which >>>>> pairs of points are within a certain distance of one-another (regular >>>>> old Euclidean norm). > >> This is now implemented in SVN. > > Wow! great -- you sounded interested, but I had no idea you'd run out > and do it! thanks! we'll check it out. No problem, it was a fairly minor modification of the two-tree code. >> I (tentatively?) used a set to store >> the collection of pairs, because my tree traversal is not smart enough >> to reliably uniquify the pairs without using sets. With more time and >> energy, I'm sure the algorithm could be improved to avoid using sets >> (both internally and on return), but I think that's something to save >> for the Cython version. > > I agree -- what's wrong with using a set? It should be possible to arrange the tree traversal algorithm so that each pair of subtrees is automatically traversed only once - akin to for i in range(n): for j in range(i): This would avoid having to build and modify a set every time you traverse the tree (sets are fairly cheap hash tables, so the cost is probably minor compared to other python overhead), and it would also naturally allow the result to be a list/inflatable array (which would save memory, if nothing else). But it would also require carefully thinking through the tree traversal algorithm. > Thanks, we'll let you know how it works for us. Good luck! Anne > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Fri Nov 13 18:10:46 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 14 Nov 2009 01:10:46 +0200 Subject: [Numpy-discussion] bus error in embedded numpy In-Reply-To: <4AFDD140.1010602@molden.no> References: <2d5132a50911130923qe563029y973e6df33a23a43c@mail.gmail.com> <40FA107F-C9E0-4687-AF3C-96FFB8A44302@enthought.com> <3d375d730911131013u52472048s64a89818cc79a541@mail.gmail.com> <2d5132a50911131024u6335b82cwb49ac8efa13bc068@mail.gmail.com> <4AFDB822.6060202@molden.no> <2d5132a50911131210h190be556gef7d8b21681af787@mail.gmail.com> <4AFDD140.1010602@molden.no> Message-ID: <1258153843.4869.5.camel@idol> pe, 2009-11-13 kello 22:36 +0100, Sturla Molden kirjoitti: [clip] > > I was aware of this - I thought I would be OK on the mac - at least > > Python and Numpy and my mex function are built with apple gcc although > > I'm not sure about Matlab. I guess Windows will be more difficult... > > But in any case I don't plan to pass any file handles around. > > It applies to any CRT resource, not just files. Compiler is not > important, but which CRT is loaded. And if you statically link the same > CRT twice, that becomes two CRTs that cannot share resources. In > Windows, Microsoft has made sure there are multiple versions of their > CRT (msvcrt.dll, msvcr71.dll, msvcr80.dll, msvcr90.dll, ...) that cannot > share resources. And anyone not careful with this can experice crashes > at random locations. Well, for Matlab/Python integration meant for numerical computations I would be surprised if CRT is really a problem. You essentially can pass data to Matlab only via Matlab's own interface -- CRT stuff like ownership of pointers to allocated memory, file handles etc. typically do not cross this boundary. -- Pauli Virtanen From ellisonbg.net at gmail.com Fri Nov 13 20:33:57 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 13 Nov 2009 17:33:57 -0800 Subject: [Numpy-discussion] Is a 4 -bit int dtype possible Message-ID: <6ce0ac130911131733s162fbbe2td6906661c24a16a@mail.gmail.com> Hi, I have a large binary data set that has 4-bit integers in it. It is possible to create a numpy dtype for a 4-bit integer? I can read the data fine using np.fromfile with a dtype of byte, but to get the 4-bit ints out I have to bit twiddle which is a pain. Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Nov 13 20:38:54 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 13 Nov 2009 19:38:54 -0600 Subject: [Numpy-discussion] Is a 4 -bit int dtype possible In-Reply-To: <6ce0ac130911131733s162fbbe2td6906661c24a16a@mail.gmail.com> References: <6ce0ac130911131733s162fbbe2td6906661c24a16a@mail.gmail.com> Message-ID: <3d375d730911131738p5f94d670k7d8211a3011f3788@mail.gmail.com> On Fri, Nov 13, 2009 at 19:33, Brian Granger wrote: > Hi, > > I have a large binary data set that has 4-bit integers in it.? It is > possible to create a numpy dtype for a 4-bit integer? Not really. We have 1-byte atomicity. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Fri Nov 13 20:54:22 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 13 Nov 2009 17:54:22 -0800 Subject: [Numpy-discussion] Is a 4 -bit int dtype possible In-Reply-To: <3d375d730911131738p5f94d670k7d8211a3011f3788@mail.gmail.com> References: <6ce0ac130911131733s162fbbe2td6906661c24a16a@mail.gmail.com> <3d375d730911131738p5f94d670k7d8211a3011f3788@mail.gmail.com> Message-ID: <6ce0ac130911131754r488c7dcbp1d03cb3b88231c82@mail.gmail.com> Robert, Thanks for the quick reply. I will just keep twiddling my bits then. Cheers, Brian On Fri, Nov 13, 2009 at 5:38 PM, Robert Kern wrote: > On Fri, Nov 13, 2009 at 19:33, Brian Granger > wrote: > > Hi, > > > > I have a large binary data set that has 4-bit integers in it. It is > > possible to create a numpy dtype for a 4-bit integer? > > Not really. We have 1-byte atomicity. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Nov 13 21:54:55 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 13 Nov 2009 19:54:55 -0700 Subject: [Numpy-discussion] Howto document constants? Message-ID: Hi All, The documentation documentation says to document constants like functions. So if a module defines a constant is it documented like so: myconstant = 1 """Blah and blah""" That doesn't seem quite right, but what is the proper method? Also, do we have a way to document static class methods? Class attributes? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Fri Nov 13 22:04:43 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 13 Nov 2009 19:04:43 -0800 Subject: [Numpy-discussion] Howto document constants? In-Reply-To: References: Message-ID: <45d1ab480911131904v20a7c16dke62265465c9c29e9@mail.gmail.com> On Fri, Nov 13, 2009 at 6:54 PM, Charles R Harris wrote: > Hi All, > > The documentation documentation says to document constants like functions. > So if a module defines a constant is it documented like so: > > myconstant = 1 > """Blah and blah""" > > That doesn't seem quite right, but what is the proper method? > > Also, do we have a way to document static class methods? Class attributes? > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Hi, Chuck. If you do not get a definitive answer (I'm afraid I don't have one) to these very reasonable questions, would you mind please filing a ticket for enhancement of the docstring standard. Thanks! DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Nov 13 22:30:23 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 13 Nov 2009 22:30:23 -0500 Subject: [Numpy-discussion] Howto document constants? In-Reply-To: References: Message-ID: <491D98E5-0664-4A8F-A301-C8434545DFF1@gmail.com> On Nov 13, 2009, at 9:54 PM, Charles R Harris wrote: > Hi All, > > The documentation documentation says to document constants like functions. So if a module defines a constant is it documented like so: > > myconstant = 1 > """Blah and blah""" > > That doesn't seem quite right, but what is the proper method? Why wouldn't you document the constant in the docstring of the module where it's defined ? Personally, I used ..data to describe the constants in numpy.ma http://docs.scipy.org/doc/numpy/_sources/reference/maskedarray.baseclass.txt From ralf.gommers at googlemail.com Sat Nov 14 04:37:18 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Nov 2009 10:37:18 +0100 Subject: [Numpy-discussion] Howto document constants? In-Reply-To: <491D98E5-0664-4A8F-A301-C8434545DFF1@gmail.com> References: <491D98E5-0664-4A8F-A301-C8434545DFF1@gmail.com> Message-ID: On Sat, Nov 14, 2009 at 4:30 AM, Pierre GM wrote: > > On Nov 13, 2009, at 9:54 PM, Charles R Harris wrote: > > > Hi All, > > > > The documentation documentation says to document constants like > functions. So if a module defines a constant is it documented like so: > > > > myconstant = 1 > > """Blah and blah""" > > > > That doesn't seem quite right, but what is the proper method? > > Document with the same structure (where relevant) as a function is for the html docs. See for example http://docs.scipy.org/numpy/docs/numpy.e/ They then end up in np.doc.constants, so that would be the place to add the docstring if you are not using the doc editor. Assigning a docstring to a constant at runtime is not possible so there it indeed does not make much sense. Why wouldn't you document the constant in the docstring of the module where > it's defined ? > Personally, I used ..data to describe the constants in numpy.ma > > http://docs.scipy.org/doc/numpy/_sources/reference/maskedarray.baseclass.txt > > For some constants this is fine, but there can also be constants that go in __all__ and end up in a different module or namespace. Or are defined in C code. Of course you can have constants in your module docstring if it makes sense there. You can also just refer to the html page (or to doc.constants) in the module docstring. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Sat Nov 14 04:56:45 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 14 Nov 2009 11:56:45 +0200 Subject: [Numpy-discussion] numpy distutils and fcompiler error again Message-ID: <710F2847B0018641891D9A21602763605AD213@ex3.envision.co.il> I encountered again the error "AttributeError: fcompiler" this time when tried to install scikits.image-0.2.2. The full log from "python setup.py instal" is: Warning: Assuming default configuration (scikits/{setup_scikits,setup}.py was not found) Appending scikits configuration to Ignoring attempt to set 'name' (from '' to 'scikits') cython -o /dev/shm/scikits.image-0.2.2/scikits/image/opencv/opencv_backend.c.new /dev/shm/scikits.image-0.2.2/scikits/image/opencv/opencv_backend.pyx warning: /usr/lib64/python2.6/site-packages/Cython-0.11.3-py2.6-linux-x86_64.egg/Cython/Includes/numpy.pxd:170:33: String literals will no longer be Py3 bytes in Cython 0.12. cython -o /dev/shm/scikits.image-0.2.2/scikits/image/opencv/opencv_cv.c.new /dev/shm/scikits.image-0.2.2/scikits/image/opencv/opencv_cv.pyx warning: /usr/lib64/python2.6/site-packages/Cython-0.11.3-py2.6-linux-x86_64.egg/Cython/Includes/numpy.pxd:170:33: String literals will no longer be Py3 bytes in Cython 0.12. Appending scikits.image.opencv configuration to scikits.image Ignoring attempt to set 'name' (from 'scikits.image' to 'scikits.image.opencv') cython -o /dev/shm/scikits.image-0.2.2/scikits/image/graph/spath.c.new /dev/shm/scikits.image-0.2.2/scikits/image/graph/spath.pyx warning: /usr/lib64/python2.6/site-packages/Cython-0.11.3-py2.6-linux-x86_64.egg /Cython/Includes/numpy.pxd:170:33: String literals will no longer be Py3 bytes i n Cython 0.12. Appending scikits.image.graph configuration to scikits.image Ignoring attempt to set 'name' (from 'scikits.image' to 'scikits.image.graph') cython -o /dev/shm/scikits.image-0.2.2/scikits/image/io/_plugins/_colormixer.c.n ew /dev/shm/scikits.image-0.2.2/scikits/image/io/_plugins/_colormixer.pyx warning: /usr/lib64/python2.6/site-packages/Cython-0.11.3-py2.6-linux-x86_64.egg /Cython/Includes/numpy.pxd:170:33: String literals will no longer be Py3 bytes i n Cython 0.12. cython -o /dev/shm/scikits.image-0.2.2/scikits/image/io/_plugins/_histograms.c.n ew /dev/shm/scikits.image-0.2.2/scikits/image/io/_plugins/_histograms.pyx warning: /usr/lib64/python2.6/site-packages/Cython-0.11.3-py2.6-linux-x86_64.egg /Cython/Includes/numpy.pxd:170:33: String literals will no longer be Py3 bytes in Cython 0.12. Appending scikits.image.io configuration to scikits.image Ignoring attempt to set 'name' (from 'scikits.image' to 'scikits.image.io') Appending scikits.image configuration to Ignoring attempt to set 'name' (from '' to 'scikits.image') running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options Traceback (most recent call last): File "setup.py", line 80, in zip_safe=False, # the package can run out of an .egg file File "/usr/lib64/python2.6/site-packages/numpy/distutils/core.py", line 186, i n setup return old_setup(**new_attr) File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/usr/lib64/python2.6/site-packages/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib64/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.6/distutils/dist.py", line 994, in run_command cmd_obj.ensure_finalized() File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized self.finalize_options() File "/usr/lib64/python2.6/site-packages/numpy/distutils/command/config_compiler.py", line 66, in finalize_options v = getattr(c,a) File "/usr/lib64/python2.6/distutils/cmd.py", line 112, in __getattr__ raise AttributeError, attr AttributeError: fcompiler I am using numpy 1.4.0.dev7728, python 2.6.4, gcc-4.3.4 on gentoo linux. * Is it a result of a bad configuration file(s)? * Would a fall back to an older version of numpy or python should solve the problem? * Any other idea? Nadav From ralf.gommers at googlemail.com Sat Nov 14 05:10:44 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 14 Nov 2009 11:10:44 +0100 Subject: [Numpy-discussion] Howto document constants? In-Reply-To: References: Message-ID: On Sat, Nov 14, 2009 at 3:54 AM, Charles R Harris wrote: > > Also, do we have a way to document static class methods? Class attributes? > > Class attributes can be documented in the class docstring as described here: http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#documenting-classes. If you need a long docstring you could maybe make it a property, which can have its own docstring. For static class methods do you mean methods defined with @classmethod or @staticmethod? I thought they could have their own docstring. Or is the problem you have static methods defined from C and you can not use add_newdoc? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From plaes at plaes.org Sat Nov 14 06:53:35 2009 From: plaes at plaes.org (Priit Laes) Date: Sat, 14 Nov 2009 13:53:35 +0200 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <20091113123648.GA18442@doriath.local> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> Message-ID: <1258199615.19279.17.camel@chi> ?hel kenal p?eval, R, 2009-11-13 kell 13:36, kirjutas Ernest Adrogu?: > 13/11/09 @ 09:41 (+0200), thus spake Priit Laes: > > Does anyone have a scenario where one would actually have both negative > > and positive numbers (integers) in the list? > > Yes: when you have a random variable that is the difference > of two (discrete) random variables. For example, if you measure > the difference in number of days off per week because of sickness > between two groups of people, you would end up with a discrete > variable with both positive and negative integers. > > > So, how about numpy.histogram_discrete() that returns data the way > > histogram() does: a list containing histogram values (ie counts) and > > list of sorted items from min(input)...max(input). ? > > In my humble opinion, it would be nice. \o/ I have pushed the preliminary version to: http://github.com/plaes/numpy/commits/histogram_discrete It can currently handle datasets with negative items and weights. I'm also planning to add optional range argument to the function, but I first need to figure out how to parse the range=(min, max) using C API... ;) numpy.histogram_discrete() returns list containing histogram value and bins (hopefully this is the right definition) hist, bins = numpy.histogram_discrete(data) Example: In [1]: import numpy In [2]: data = numpy.random.poisson(3, 300) In [3]: numpy.histogram_discrete(data) Out[3]: [array([15, 50, 72, 59, 52, 34, 8, 7, 3]), array([0, 1, 2, 3, 4, 5, 6, 7, 8])] In [4]: In [5]: data = [-1, 5] In [6]: numpy.histogram_discrete(data, weights=[2, 0]) Out[6]: [array([ 2., 0., 0., 0., 0., 0., 0.]), array([-1, 0, 1, 2, 3, 4, 5])] Priit :) From josef.pktd at gmail.com Sat Nov 14 07:10:38 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 14 Nov 2009 07:10:38 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1258199615.19279.17.camel@chi> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> <1258199615.19279.17.camel@chi> Message-ID: <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> On Sat, Nov 14, 2009 at 6:53 AM, Priit Laes wrote: > ?hel kenal p?eval, R, 2009-11-13 kell 13:36, kirjutas Ernest Adrogu?: >> 13/11/09 @ 09:41 (+0200), thus spake Priit Laes: >> > Does anyone have a scenario where one would actually have both negative >> > and positive numbers (integers) in the list? >> >> Yes: when you have a random variable that is the difference >> of two (discrete) random variables. For example, if you measure >> the difference in number of days off per week because of sickness >> between two groups of people, you would end up with a discrete >> variable with both positive and negative integers. >> >> > So, how about numpy.histogram_discrete() that returns data the way >> > histogram() does: a list containing histogram values (ie counts) and >> > list of sorted items from min(input)...max(input). ? >> >> In my humble opinion, it would be nice. > \o/ > > I have pushed the preliminary version to: > http://github.com/plaes/numpy/commits/histogram_discrete > > It can currently handle datasets with negative items and weights. I'm > also planning to add optional range argument to the function, but I > first need to figure out how to parse the range=(min, max) using C > API... ;) > > numpy.histogram_discrete() returns list containing histogram value and > bins (hopefully this is the right definition) > > hist, bins = numpy.histogram_discrete(data) > > Example: > In [1]: import numpy > In [2]: data = numpy.random.poisson(3, 300) > In [3]: numpy.histogram_discrete(data) > Out[3]: > [array([15, 50, 72, 59, 52, 34, ?8, ?7, ?3]), > ?array([0, 1, 2, 3, 4, 5, 6, 7, 8])] > In [4]: > In [5]: data = [-1, 5] > In [6]: numpy.histogram_discrete(data, weights=[2, 0]) > Out[6]: > [array([ 2., ?0., ?0., ?0., ?0., ?0., ?0.]), > ?array([-1, ?0, ?1, ?2, ?3, ?4, ?5])] Sorry, I still don't see much reason to do this in c >>> data = [-1, 5] >>> c=np.bincount(data-np.min(data),weights=[2,0]) >>> x=np.arange(np.min(data),np.min(data)+len(c)) >>> c,x (array([ 2., 0., 0., 0., 0., 0., 0.]), array([-1, 0, 1, 2, 3, 4, 5])) >>> data = [11,5] >>> np.bincount(data,weights=[2,0]) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 2.]) >>> np.arange(max(data)+1) array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) >>> c=np.bincount(data-np.min(data),weights=[2,0]) >>> x=np.arange(np.min(data),np.min(data)+len(c)) >>> c,x (array([ 0., 0., 0., 0., 0., 0., 2.]), array([ 5, 6, 7, 8, 9, 10, 11])) Josef > > Priit :) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Sat Nov 14 07:40:49 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 14 Nov 2009 07:40:49 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> <1258199615.19279.17.camel@chi> <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> Message-ID: <1cd32cbb0911140440h6e0f1531qfb6455c381af68de@mail.gmail.com> On Sat, Nov 14, 2009 at 7:10 AM, wrote: > On Sat, Nov 14, 2009 at 6:53 AM, Priit Laes wrote: >> ?hel kenal p?eval, R, 2009-11-13 kell 13:36, kirjutas Ernest Adrogu?: >>> 13/11/09 @ 09:41 (+0200), thus spake Priit Laes: >>> > Does anyone have a scenario where one would actually have both negative >>> > and positive numbers (integers) in the list? >>> >>> Yes: when you have a random variable that is the difference >>> of two (discrete) random variables. For example, if you measure >>> the difference in number of days off per week because of sickness >>> between two groups of people, you would end up with a discrete >>> variable with both positive and negative integers. >>> >>> > So, how about numpy.histogram_discrete() that returns data the way >>> > histogram() does: a list containing histogram values (ie counts) and >>> > list of sorted items from min(input)...max(input). ? >>> >>> In my humble opinion, it would be nice. >> \o/ >> >> I have pushed the preliminary version to: >> http://github.com/plaes/numpy/commits/histogram_discrete >> >> It can currently handle datasets with negative items and weights. I'm >> also planning to add optional range argument to the function, but I >> first need to figure out how to parse the range=(min, max) using C >> API... ;) >> >> numpy.histogram_discrete() returns list containing histogram value and >> bins (hopefully this is the right definition) >> >> hist, bins = numpy.histogram_discrete(data) >> >> Example: >> In [1]: import numpy >> In [2]: data = numpy.random.poisson(3, 300) >> In [3]: numpy.histogram_discrete(data) >> Out[3]: >> [array([15, 50, 72, 59, 52, 34, ?8, ?7, ?3]), >> ?array([0, 1, 2, 3, 4, 5, 6, 7, 8])] >> In [4]: >> In [5]: data = [-1, 5] >> In [6]: numpy.histogram_discrete(data, weights=[2, 0]) >> Out[6]: >> [array([ 2., ?0., ?0., ?0., ?0., ?0., ?0.]), >> ?array([-1, ?0, ?1, ?2, ?3, ?4, ?5])] > > > Sorry, I still don't see much reason to do this in c > >>>> data = [-1, 5] >>>> c=np.bincount(data-np.min(data),weights=[2,0]) >>>> x=np.arange(np.min(data),np.min(data)+len(c)) >>>> c,x > (array([ 2., ?0., ?0., ?0., ?0., ?0., ?0.]), array([-1, ?0, ?1, ?2, > 3, ?4, ?5])) >>>> data = [11,5] >>>> np.bincount(data,weights=[2,0]) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?2.]) >>>> np.arange(max(data)+1) > array([ 0, ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, 10, 11]) >>>> c=np.bincount(data-np.min(data),weights=[2,0]) >>>> x=np.arange(np.min(data),np.min(data)+len(c)) >>>> c,x > (array([ 0., ?0., ?0., ?0., ?0., ?0., ?2.]), array([ 5, ?6, ?7, ?8, > 9, 10, 11])) > > Josef I think histogram in the name is misleading to me, because it suggests binning, not counting of all occurrences individually. In some cases, I also remove the zero counts: >>> data = [11,5,20] >>> c=np.bincount(data-np.min(data),weights=[2,1,0]) >>> x=np.arange(np.min(data),np.min(data)+len(c)) >>> c array([ 1., 0., 0., 0., 0., 0., 2., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) >>> x array([ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]) >>> cind = np.nonzero(c) >>> cs = c[cind] >>> xs = x[cind] >>> cs array([ 1., 2.]) >>> xs array([ 5, 11]) Josef > > >> >> Priit :) >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From dsdale24 at gmail.com Sat Nov 14 10:29:16 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Sat, 14 Nov 2009 10:29:16 -0500 Subject: [Numpy-discussion] numpy distutils and distribute Message-ID: Please excuse the cross-post. I have installed distribute-0.6.8 and numpy-svn into my ~/.local/lib/python2.6/site-packages (using "python setup.py install --user"). I am now trying to install Enthought's Enable from a fresh svn checkout on ubuntu karmic: $ python setup.py develop --user [...] building library "agg24_src" sources building library "kiva_src" sources building extension "enthought.kiva.agg._agg" sources building extension "enthought.kiva.agg._plat_support" sources building data_files sources build_src: building npy-pkg config files running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib running build_ext build_clib already run, it is too late to ensure in-place build of build_clib Traceback (most recent call last): File "setup.py", line 327, in **config File "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/command/build_ext.py", line 74, in run self.library_dirs.append(build_clib.build_clib) UnboundLocalError: local variable 'build_clib' referenced before assignment I am able to run "python setup.py install --user". Incidentally, "python setup.py develop --user" worked for TraitsGui, EnthoughtBase, TraitsBackendQt4. I have been (sort of) following the discussion on distutils-sig. Thank you Robert, David, Pauli, for all your effort. From gokhansever at gmail.com Sat Nov 14 10:42:15 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sat, 14 Nov 2009 09:42:15 -0600 Subject: [Numpy-discussion] numpy distutils and distribute In-Reply-To: References: Message-ID: <49d6b3500911140742wfa33b62jb4a27b05690b271e@mail.gmail.com> On Sat, Nov 14, 2009 at 9:29 AM, Darren Dale wrote: > Please excuse the cross-post. I have installed distribute-0.6.8 and > numpy-svn into my ~/.local/lib/python2.6/site-packages (using "python > setup.py install --user"). I am now trying to install Enthought's > Enable from a fresh svn checkout on ubuntu karmic: > > $ python setup.py develop --user > [...] > building library "agg24_src" sources > building library "kiva_src" sources > building extension "enthought.kiva.agg._agg" sources > building extension "enthought.kiva.agg._plat_support" sources > building data_files sources > build_src: building npy-pkg config files > running build_clib > customize UnixCCompiler > customize UnixCCompiler using build_clib > running build_ext > build_clib already run, it is too late to ensure in-place build of > build_clib > Traceback (most recent call last): > File "setup.py", line 327, in > **config > File > "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/core.py", > line 186, in setup > return old_setup(**new_attr) > File "/usr/lib/python2.6/distutils/core.py", line 152, in setup > dist.run_commands() > File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands > self.run_command(cmd) > File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command > cmd_obj.run() > File > "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/command/build_ext.py", > line 74, in run > self.library_dirs.append(build_clib.build_clib) > UnboundLocalError: local variable 'build_clib' referenced before assignment > > Darren, I had a similar installation error. Could you try the solution that was given in this thread? http://www.mail-archive.com/numpy-discussion at scipy.org/msg19798.html > > I am able to run "python setup.py install --user". Incidentally, > "python setup.py develop --user" worked for TraitsGui, EnthoughtBase, > TraitsBackendQt4. > > I have been (sort of) following the discussion on distutils-sig. Thank > you Robert, David, Pauli, for all your effort. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Sat Nov 14 10:57:51 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 14 Nov 2009 09:57:51 -0600 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1cd32cbb0911140440h6e0f1531qfb6455c381af68de@mail.gmail.com> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> <1258199615.19279.17.camel@chi> <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> <1cd32cbb0911140440h6e0f1531qfb6455c381af68de@mail.gmail.com> Message-ID: On Sat, Nov 14, 2009 at 6:40 AM, wrote: > On Sat, Nov 14, 2009 at 7:10 AM, ? wrote: >> On Sat, Nov 14, 2009 at 6:53 AM, Priit Laes wrote: >>> ?hel kenal p?eval, R, 2009-11-13 kell 13:36, kirjutas Ernest Adrogu?: >>>> 13/11/09 @ 09:41 (+0200), thus spake Priit Laes: >>>> > Does anyone have a scenario where one would actually have both negative >>>> > and positive numbers (integers) in the list? >>>> >>>> Yes: when you have a random variable that is the difference >>>> of two (discrete) random variables. For example, if you measure >>>> the difference in number of days off per week because of sickness >>>> between two groups of people, you would end up with a discrete >>>> variable with both positive and negative integers. >>>> >>>> > So, how about numpy.histogram_discrete() that returns data the way >>>> > histogram() does: a list containing histogram values (ie counts) and >>>> > list of sorted items from min(input)...max(input). ? >>>> >>>> In my humble opinion, it would be nice. >>> \o/ >>> >>> I have pushed the preliminary version to: >>> http://github.com/plaes/numpy/commits/histogram_discrete >>> >>> It can currently handle datasets with negative items and weights. I'm >>> also planning to add optional range argument to the function, but I >>> first need to figure out how to parse the range=(min, max) using C >>> API... ;) >>> >>> numpy.histogram_discrete() returns list containing histogram value and >>> bins (hopefully this is the right definition) >>> >>> hist, bins = numpy.histogram_discrete(data) >>> >>> Example: >>> In [1]: import numpy >>> In [2]: data = numpy.random.poisson(3, 300) >>> In [3]: numpy.histogram_discrete(data) >>> Out[3]: >>> [array([15, 50, 72, 59, 52, 34, ?8, ?7, ?3]), >>> ?array([0, 1, 2, 3, 4, 5, 6, 7, 8])] >>> In [4]: >>> In [5]: data = [-1, 5] >>> In [6]: numpy.histogram_discrete(data, weights=[2, 0]) >>> Out[6]: >>> [array([ 2., ?0., ?0., ?0., ?0., ?0., ?0.]), >>> ?array([-1, ?0, ?1, ?2, ?3, ?4, ?5])] >> >> >> Sorry, I still don't see much reason to do this in c >> >>>>> data = [-1, 5] >>>>> c=np.bincount(data-np.min(data),weights=[2,0]) >>>>> x=np.arange(np.min(data),np.min(data)+len(c)) >>>>> c,x >> (array([ 2., ?0., ?0., ?0., ?0., ?0., ?0.]), array([-1, ?0, ?1, ?2, >> 3, ?4, ?5])) >>>>> data = [11,5] >>>>> np.bincount(data,weights=[2,0]) >> array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?2.]) >>>>> np.arange(max(data)+1) >> array([ 0, ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, 10, 11]) >>>>> c=np.bincount(data-np.min(data),weights=[2,0]) >>>>> x=np.arange(np.min(data),np.min(data)+len(c)) >>>>> c,x >> (array([ 0., ?0., ?0., ?0., ?0., ?0., ?2.]), array([ 5, ?6, ?7, ?8, >> 9, 10, 11])) >> >> Josef > > I think histogram in the name is misleading to me, because it suggests > binning, not counting of all occurrences individually. > In some cases, I also remove the zero counts: > >>>> data = [11,5,20] >>>> c=np.bincount(data-np.min(data),weights=[2,1,0]) >>>> x=np.arange(np.min(data),np.min(data)+len(c)) >>>> c > array([ 1., ?0., ?0., ?0., ?0., ?0., ?2., ?0., ?0., ?0., ?0., ?0., ?0., > ? ? ? ?0., ?0., ?0.]) >>>> x > array([ 5, ?6, ?7, ?8, ?9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]) >>>> cind = np.nonzero(c) >>>> cs = c[cind] >>>> xs = x[cind] >>>> cs > array([ 1., ?2.]) >>>> xs > array([ 5, 11]) > > Josef >> >> >>> >>> Priit :) >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Hi, My concern is to avoid some of the issues that arose with changing histogram. So lets get it right from the start. Basically my question is why do we need yet another histogram function? What is the difference between your histogram_discrete and histogram or bincount? Is it just that bincount does not count negative numbers? If so, then I would strongly argue that is insufficient for creating new function. Rather you need to provide a suitable patch to fix bincount or replace bincount with a better version. Thanks Bruce From dsdale24 at gmail.com Sat Nov 14 11:13:31 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Sat, 14 Nov 2009 11:13:31 -0500 Subject: [Numpy-discussion] numpy distutils and distribute In-Reply-To: <49d6b3500911140742wfa33b62jb4a27b05690b271e@mail.gmail.com> References: <49d6b3500911140742wfa33b62jb4a27b05690b271e@mail.gmail.com> Message-ID: On Sat, Nov 14, 2009 at 10:42 AM, G?khan Sever wrote: > On Sat, Nov 14, 2009 at 9:29 AM, Darren Dale wrote: >> >> Please excuse the cross-post. I have installed distribute-0.6.8 and >> numpy-svn into my ~/.local/lib/python2.6/site-packages (using "python >> setup.py install --user"). I am now trying to install Enthought's >> Enable from a fresh svn checkout on ubuntu karmic: >> >> $ python setup.py develop --user >> [...] >> building library "agg24_src" sources >> building library "kiva_src" sources >> building extension "enthought.kiva.agg._agg" sources >> building extension "enthought.kiva.agg._plat_support" sources >> building data_files sources >> build_src: building npy-pkg config files >> running build_clib >> customize UnixCCompiler >> customize UnixCCompiler using build_clib >> running build_ext >> build_clib already run, it is too late to ensure in-place build of >> build_clib >> Traceback (most recent call last): >> ?File "setup.py", line 327, in >> ? ?**config >> ?File >> "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/core.py", >> line 186, in setup >> ? ?return old_setup(**new_attr) >> ?File "/usr/lib/python2.6/distutils/core.py", line 152, in setup >> ? ?dist.run_commands() >> ?File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands >> ? ?self.run_command(cmd) >> ?File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command >> ? ?cmd_obj.run() >> ?File >> "/home/darren/.local/lib/python2.6/site-packages/numpy/distutils/command/build_ext.py", >> line 74, in run >> ? ?self.library_dirs.append(build_clib.build_clib) >> UnboundLocalError: local variable 'build_clib' referenced before >> assignment >> > > Darren, > > I had a similar installation error. Could you try the solution that was > given in this thread? > > http://www.mail-archive.com/numpy-discussion at scipy.org/msg19798.html Thanks! Here is the diff, could someone with knowledge of numpy's distutils have a look and consider committing it? Index: numpy/distutils/command/build_ext.py =================================================================== --- numpy/distutils/command/build_ext.py (revision 7734) +++ numpy/distutils/command/build_ext.py (working copy) @@ -61,6 +61,7 @@ if self.distribution.have_run.get('build_clib'): log.warn('build_clib already run, it is too late to ' \ 'ensure in-place build of build_clib') + build_clib = self.distribution.get_command_obj('build_clib') else: build_clib = self.distribution.get_command_obj('build_clib') build_clib.inplace = 1 From charlesr.harris at gmail.com Sat Nov 14 14:17:56 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 Nov 2009 12:17:56 -0700 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. Message-ID: Hi All, I would like some advise on the best way to add the new functions. I've added a new package polynomial, and that package contains four new modules: chebyshev, polynomial, polytemplate, polyutils. The question is whether or not to include all of the functions in these packages in the __init__.py, or to just import the modules. It is a namespace question, i.e., to get to the chebyshev functions should one do import numpy.polynomial.chebyshev or just import numpy.polynomial Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From amenity at enthought.com Sat Nov 14 14:18:04 2009 From: amenity at enthought.com (Amenity Applewhite) Date: Sat, 14 Nov 2009 13:18:04 -0600 Subject: [Numpy-discussion] November 20 Webinar: Interpolation with NumPy/SciPy References: <1102826606498.1102424111856.4101.9.1814156C@scheduler> Message-ID: Having trouble viewing this email? Click here Friday, November 20: Interpolation with NumPy/SciPy Dear Amenity, It's time for our mid-month Scientific Computing with Python webinar! This month's topic is sure to prove very useful for data analysts: Interpolation with NumPy and SciPy. In many data-processing scenarios, it is necessary to use a discrete set of available data-points to infer the value of a function at a new data-point. One approach to this problem is interpolation, which constructs a new model-function that goes through the original data- points. There are many forms of interpolation - polynomial, spline, kriging, radial basis function, etc. - and SciPy includes some of these interpolation forms. This webinar will review the interpolation modules available in SciPy and in the larger Python community and provide instruction on their use via example. Scientific Computing with Python Webinar: Interpolation with NumPy/SciPy Friday, November 20 1pm CDT/7pm UTC Register at GoToMeeting We look forward to seeing you Friday! As always, feel free to contact us with questions, concerns, or suggestions for future webinar topics. Thanks, The Enthought Team QUICK LINKS ::: www.enthought.com code.enthought.com Facebook Enthought Blog Forward email This email was sent to amenity at enthought.com by amenity at enthought.com. Update Profile/Email Address | Instant removal with SafeUnsubscribe? | Privacy Policy. Enthought, Inc. | 515 Congress Ave. | Suite 2100 | Austin | TX | 78701 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fonnesbeck at gmail.com Sat Nov 14 19:20:11 2009 From: fonnesbeck at gmail.com (Chris) Date: Sun, 15 Nov 2009 00:20:11 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> Message-ID: David Cournapeau gmail.com> writes: > I am afraid I don't see where the problem may come from. I used the > exact same build script as you, on the same version of mac os x, and I > don't see the problem. Is there anything non standard in your > environment that you can think of ? > Not that I can think of, no. Here is my env: TERM_PROGRAM=Apple_Terminal TERM=xterm-color SHELL=/bin/bash TMPDIR=/var/folders/Cp/Cp1bvHSvFwuPM09XVxsVF++++TI/-Tmp-/ Apple_PubSub_Socket_Render=/tmp/launch-o1hxIu/Render TERM_PROGRAM_VERSION=272 USER=fonnesbeck COMMAND_MODE=unix2003 SSH_AUTH_SOCK=/tmp/launch-4DXe3p/Listeners __CF_USER_TEXT_ENCODING=0x1F5:0:0 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin: /usr/X11/bin:/opt/local/bin:/usr/local/git/bin:/usr/local/git/bin PWD=/Users/fonnesbeck EDITOR=mate_wait LANG=en_NZ.UTF-8 SHLVL=1 HOME=/Users/fonnesbeck LESS=-R LOGNAME=fonnesbeck DISPLAY=/tmp/launch-3LiwGx/:0 _GOOGLE_GILD_RUNTIME_FRAMEWORK_ABSOLUTE_PATH_= /Library/Google/Frameworks/Gild.framework _=/usr/bin/env Nothing looks out of order, but I still get the Symbol not found: _npy_cexp errors. From dwf at cs.toronto.edu Sun Nov 15 02:16:44 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sun, 15 Nov 2009 02:16:44 -0500 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> <1258199615.19279.17.camel@chi> <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> <1cd32cbb0911140440h6e0f1531qfb6455c381af68de@mail.gmail.com> Message-ID: <1FAC1346-F7FE-4E45-AF0F-E99FD81A27F2@cs.toronto.edu> On 14-Nov-09, at 10:57 AM, Bruce Southey wrote: > Is it just that bincount does not count negative numbers? > If so, then I would strongly argue that is insufficient for creating > new function. Rather you need to provide a suitable patch to fix > bincount or replace bincount with a better version. The whole point of bincount is that in the result array, the indices correspond to _values_ in the original array, i.e. bc = bincount(arr) then bc[3] gives me the number of 3's that appear in arr. This is lost if bincount starts reporting negatives. I suppose you could return a tuple of two arrays, optionally, but I don't know how much use it would get. If you really need to do bincount on an array with negative ints, you can just add -arr.min() to the array first and subtract the same value when done. Assuming you don't have a huge array this will be very fast, and can even be done in place. David From josef.pktd at gmail.com Sun Nov 15 15:26:46 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 15 Nov 2009 15:26:46 -0500 Subject: [Numpy-discussion] failure building trunk with mingw Message-ID: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> Are there new changes to the configuration needed? mingw 3.4.5, WindowsXP, Python 2.5.2 Python.h is not picked up anymore: _configtest.c:1:20: Python.h: No such file or directory Josef C:\Josef\_progs\Subversion\numpy-trunk>setup.py bdist Running from numpy source directory. F2PY Version 2_7741 numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 3, with checks um 59750b518272c8987f02d66445afd3f1, but recorded checksum for C API version 3 i n codegen_dir/cversions.txt is bf22c0d05b31625d2a7015988d61ce5a. If functions we re added in the C API, you have to update C_API_VERSION in numpy\core\setup_com mon.pyc. MismatchCAPIWarning) blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in C:/Programs/Intel/MKL/10.0.1.015/ia32/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Programs\Python25\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Programs\Python25\libs NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in C:\Programs\Python25\lib libraries f77blas,cblas,atlas not found in C:\ FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\Programs\\Python25\\libs'] language = c C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\config.py:394: De precationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of get_output is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) No module named msvccompiler in numpy.distutils; trying from distutils FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\Programs\\Python25\\libs'] language = c define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in C:/Programs/Intel/MKL/10.0.1.015/ia32/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in C:\Programs\Python25\lib libraries lapack_atlas not found in C:\Programs\Python25\lib libraries ptf77blas,ptcblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries ptf77blas,ptcblas,atlas not found in C:\Programs\Python25\libs libraries lapack_atlas not found in C:\Programs\Python25\libs numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in C:\Programs\Python25\lib libraries lapack_atlas not found in C:\Programs\Python25\lib libraries f77blas,cblas,atlas not found in C:\ libraries lapack_atlas not found in C:\ libraries lapack_atlas not found in C:\Programs\Python25\libs numpy.distutils.system_info.atlas_info FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\Programs\\Python25\\libs'] language = f77 No module named msvccompiler in numpy.distutils; trying from distutils FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\Programs\\Python25\\libs'] language = f77 define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] running bdist running bdist_dumb running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src build_src building py_modules sources building library "npymath" sources customize GnuFCompiler Found executable C:\Programs\MinGW\bin\g77.exe gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found Found executable C:\Programs\MinGW\bin\g77.exe customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy \core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\cor e\include -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Inumpy\core\src\private -Inumpy\c ore\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inump y\core\src\umath -Inumpy\core\include -c _configtest.c -o _configtest.o Found executable C:\Programs\MinGW\bin\gcc.exe g++ -mno-cygwin _configtest.o -lmsvcr71 -o _configtest.exe Found executable C:\Programs\MinGW\bin\g++.exe success! removing: _configtest.c _configtest.o _configtest.exe building extension "numpy.core._sort" sources Generating build\src.win32-2.5\numpy\core\include/numpy\config.h C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy \core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\cor e\include -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Inumpy\core\src\private -Inumpy\c ore\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inump y\core\src\umath -Inumpy\core\include -c _configtest.c -o _configtest.o g++ -mno-cygwin _configtest.o -lmsvcr71 -o _configtest.exe success! removing: _configtest.c _configtest.o _configtest.exe C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes compile options: '-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy \core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\cor e\include -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Inumpy\core\src\private -Inumpy\c ore\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inump y\core\src\umath -Inumpy\core\include -c _configtest.c -o _configtest.o _configtest.c:1:20: Python.h: No such file or directory failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\setup.py", line 187, in setup_package() File "C:\Josef\_progs\Subversion\numpy-trunk\setup.py", line 180, in setup_pac kage configuration=configuration ) File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\core.py", line 18 6, in setup return old_setup(**new_attr) File "C:\Programs\Python25\lib\distutils\core.py", line 151, in setup dist.run_commands() File "C:\Programs\Python25\lib\distutils\dist.py", line 974, in run_commands self.run_command(cmd) File "C:\Programs\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Programs\Python25\lib\distutils\command\bdist.py", line 146, in run self.run_command(cmd_name) File "C:\Programs\Python25\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Programs\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Programs\Python25\lib\distutils\command\bdist_dumb.py", line 85, in r un self.run_command('build') File "C:\Programs\Python25\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Programs\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\build.py" , line 37, in run old_build.run(self) File "C:\Programs\Python25\lib\distutils\command\build.py", line 113, in run self.run_command(cmd_name) File "C:\Programs\Python25\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Programs\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\build_src .py", line 152, in run self.build_sources() File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\build_src .py", line 169, in build_sources self.build_extension_sources(ext) File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\build_src .py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "C:\Josef\_progs\Subversion\numpy-trunk\numpy\distutils\command\build_src .py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 386, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy\core\setup.py", line 41, in check_types out = check_types(*a, **kw) File "numpy\core\setup.py", line 246, in check_types "Cannot compiler 'Python.h'. Perhaps you need to "\ SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev| python-devel. From pgmdevlist at gmail.com Mon Nov 16 00:11:43 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 16 Nov 2009 00:11:43 -0500 Subject: [Numpy-discussion] Summing an array with dtype=object Message-ID: All, An issue was recently raised about summing a MaskedArray with a np.object dtype. Turns out that the problem is numpy based: Let's sum using integers >>> type(np.sum([1,2,3], dtype=np.int32)) Now, with a np.object dtype: >>> type(np.sum([1,2,3],dtype=object)) And we no longer have a np.object, but a regular Python int. Which, unfortunately for MaskedArray, does not have a view object (hence the bug). Is it the expected behavior ? From robert.kern at gmail.com Mon Nov 16 00:16:44 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 15 Nov 2009 23:16:44 -0600 Subject: [Numpy-discussion] Summing an array with dtype=object In-Reply-To: References: Message-ID: <3d375d730911152116r6b236bdcx723905a01e519b3f@mail.gmail.com> On Sun, Nov 15, 2009 at 23:11, Pierre GM wrote: > All, > An issue was recently raised about summing a MaskedArray with a np.object dtype. Turns out that the problem is numpy based: > Let's sum using integers >>>> type(np.sum([1,2,3], dtype=np.int32)) > > > Now, with a np.object dtype: >>>> ?type(np.sum([1,2,3],dtype=object)) > ? > > And we no longer have a np.object, but a regular Python int. Which, unfortunately for MaskedArray, does not have a view object (hence the bug). > Is it the expected behavior ? Yes. Wherever we can, we return the object itself rather than np.object_ scalars. In [2]: a Out[2]: array([0, 1, 2], dtype=object) In [5]: type(a[0]) Out[5]: int -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Mon Nov 16 00:30:29 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 16 Nov 2009 00:30:29 -0500 Subject: [Numpy-discussion] Summing an array with dtype=object In-Reply-To: <3d375d730911152116r6b236bdcx723905a01e519b3f@mail.gmail.com> References: <3d375d730911152116r6b236bdcx723905a01e519b3f@mail.gmail.com> Message-ID: On Nov 16, 2009, at 12:16 AM, Robert Kern wrote: > On Sun, Nov 15, 2009 at 23:11, Pierre GM wrote: >> All, >> An issue was recently raised about summing a MaskedArray with a np.object dtype. Turns out that the problem is numpy based: >> Let's sum using integers >>>>> type(np.sum([1,2,3], dtype=np.int32)) >> >> >> Now, with a np.object dtype: >>>>> type(np.sum([1,2,3],dtype=object)) >> >> >> And we no longer have a np.object, but a regular Python int. Which, unfortunately for MaskedArray, does not have a view object (hence the bug). >> Is it the expected behavior ? > > Yes. Wherever we can, we return the object itself rather than > np.object_ scalars. OK, thank you Robert! From david at ar.media.kyoto-u.ac.jp Mon Nov 16 00:48:39 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 16 Nov 2009 14:48:39 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> Message-ID: <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> josef.pktd at gmail.com wrote: > Are there new changes to the configuration needed? mingw 3.4.5, > WindowsXP, Python 2.5.2 > > Python.h is not picked up anymore: > > _configtest.c:1:20: Python.h: No such file or directory > > Josef > > C:\Josef\_progs\Subversion\numpy-trunk>setup.py bdist Could you try with python setup.py build -c mingw32 bdist I have just tried on the same configuration as you, and I cannot reproduce the problem with the above command. David From david at ar.media.kyoto-u.ac.jp Mon Nov 16 04:29:33 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 16 Nov 2009 18:29:33 +0900 Subject: [Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0 Message-ID: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> Hi, A quick remainder: the trunk will be closed for 1.4.0 changes within a few hours. After that time, the trunk should only contain things which will be in 1.5.0, and the 1.4.0 changes will be in the 1.4.0 branch, which should contain only bug fixes. cheers, David From bsouthey at gmail.com Mon Nov 16 10:10:26 2009 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 16 Nov 2009 09:10:26 -0600 Subject: [Numpy-discussion] Initial implementation of histogram_discrete() In-Reply-To: <1FAC1346-F7FE-4E45-AF0F-E99FD81A27F2@cs.toronto.edu> References: <1258066857.10034.13.camel@chi> <1cd32cbb0911121509u19b0e02fm88d8ed3707fed9c6@mail.gmail.com> <1258098079.10759.30.camel@chi> <20091113123648.GA18442@doriath.local> <1258199615.19279.17.camel@chi> <1cd32cbb0911140410v1d462926x8747c6ac8c93af6c@mail.gmail.com> <1cd32cbb0911140440h6e0f1531qfb6455c381af68de@mail.gmail.com> <1FAC1346-F7FE-4E45-AF0F-E99FD81A27F2@cs.toronto.edu> Message-ID: <4B016B62.1080209@gmail.com> On 11/15/2009 01:16 AM, David Warde-Farley wrote: > On 14-Nov-09, at 10:57 AM, Bruce Southey wrote: > > >> Is it just that bincount does not count negative numbers? >> If so, then I would strongly argue that is insufficient for creating >> new function. Rather you need to provide a suitable patch to fix >> bincount or replace bincount with a better version. >> > The whole point of bincount is that in the result array, the indices > correspond to _values_ in the original array, i.e. bc = bincount(arr) > then bc[3] gives me the number of 3's that appear in arr. This is lost > if bincount starts reporting negatives. I suppose you could return a > tuple of two arrays, optionally, but I don't know how much use it > would get. > Isn't this indexing invalid if you have a negative value in the original array in the first place? While I have not tried Priit's code (as it is in C), the provided output with negative values does not appear to maintain this property. Really without changing the output type, I do not see how you can keep this indexing property with negative values. So really the comparison has to be with histogram(). > If you really need to do bincount on an array with negative ints, you > can just add -arr.min() to the array first and subtract the same value > when done. Assuming you don't have a huge array this will be very > fast, and can even be done in place. > > David > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Josef provided the histogram solution using np.arange for the bins that appears to be handle negative values except that the indexing is refers to position not the value. That is the index is value minus the minimum of (zero, min(array)) - which is the indexing as required by Priit's code. Bruce From Chris.Barker at noaa.gov Mon Nov 16 12:19:07 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 09:19:07 -0800 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 52 In-Reply-To: <58df6dc20911121140t35aa7a34v66598dc09b4af3d3@mail.gmail.com> References: <58df6dc20911121140t35aa7a34v66598dc09b4af3d3@mail.gmail.com> Message-ID: <4B01898B.7060001@noaa.gov> Jake VanderPlas wrote: > It sounds like all of this could be done very simply without going to > C, using a class based on numpy.ndarray. The following works for 1D > arrays, behaves like a regular 1D numpy array, and could be easily > improved with a little care. Is this what you had in mind? > > import numpy > > #easy scalable array class > class scarray: > def __init__(self,*args,**kwargs): > self.__data = numpy.ndarray(*args,**kwargs) > > def append(self,val): > tmp = self.__data > self.__data = numpy.ndarray(tmp.size+1) > self.__data[:-1] = tmp > self.__data[-1] = val > del tmp > > def __getattr__(self,attr): > return getattr(self.__data,attr) The problem here is that it's re-allocating memory with every single addition of an element. It's pretty common to pre-allocate some extra space for this kind of thing (python lists, std vectors, etc, etc, etc), so I assumed that it would be performance killer. However, you know what they say about premature optimization, so a test or two is in order. This is incrementally adding 10000 integers, one point at a time: Using the suggested code: In [21]: timeit p.scarray1(10000) 10 loops, best of 3: 1.71 s per loop Using my accumulator code: In [23]: timeit p.accum1(10000) 10 loops, best of 3: 25.6 ms per loop So all that memory re-allocation really does kill performance. In [24]: timeit p.list1(10000) 100 loops, best of 3: 9.96 ms per loop But, of course, lists are still faster. I think this is because I'm adding python integers, which are already python objects, so that's exactly what lists are for -- you can't beat them. This wouldn't apply to using them from C, however. Also, I see a change when we add chunks of data already in a numpy array, with .extend(): # adding a sequence of ten integers at a time: In [40]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 6.36 ms per loop In [41]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 6.22 ms per loop # about the same speed # but when I add 100 elements at a time: In [46]: timeit profile_accumulator.list_extend1(10000) 10 loops, best of 3: 56.6 ms per loop In [47]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 13.3 ms per loop # I start to see a real advantage to the numpy accumulator approach. Is # that a real use-case? I'm not sure. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Nov 16 12:35:53 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 09:35:53 -0800 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 38, Issue 52 In-Reply-To: <4B01898B.7060001@noaa.gov> References: <58df6dc20911121140t35aa7a34v66598dc09b4af3d3@mail.gmail.com> <4B01898B.7060001@noaa.gov> Message-ID: <4B018D79.3060702@noaa.gov> oops, I meant to include my code with that last note. Here it is. accumulator.py: is my implementation. easy_scale.py: is Jake's suggested implementation. profile_accumulator.py: contains some test functions that can be run with ipython's "timeit" test_accumulator.py: is test code that can be run with nosetests. Any comments, suggestions, etc. welcome. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: accumulator.py Type: application/x-python Size: 4112 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: easy_scale.py Type: application/x-python Size: 542 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: profile_accumulator.py Type: application/x-python Size: 1899 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_accumulator.py Type: application/x-python Size: 5154 bytes Desc: not available URL: From Chris.Barker at noaa.gov Mon Nov 16 12:43:43 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 09:43:43 -0800 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: References: Message-ID: <4B018F4F.7070701@noaa.gov> Charles R Harris wrote: > I would like some advise on the best way to add the new functions. I've > added a new package polynomial, and that package contains four new > modules: chebyshev, polynomial, polytemplate, polyutils. This seems to belong more in scipy than numpy, but I'll leave that to others to decide. > whether or not to include all of the functions in these packages in the > __init__.py, or to just import the modules. Are any of them compiled code? I've been very frustrated when I can't use some pure python stuff in scipy because broken compiled fortran extensions are getting imported that I don't even need. If that isn't an issue, and the polynomial package would end up with only a handful of names, then I say import them all. Another way to ask this: would there by ANY names in the polynomial package if you don't import the modules? If there is compiled code, the import could fail gracefully, and then you could still pull it all in. OTOH, what this does is bring stuff into memory unnecessarily, and also brings it into stand-alone bundles (py2exe, py2app, etc). So if these modules are not small, then it's probably better to have to import them explicitly. Also -- do you foresee many more polynomial types in the future? I know I'd like to see Hermite. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Mon Nov 16 12:50:12 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Nov 2009 10:50:12 -0700 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <4B018F4F.7070701@noaa.gov> References: <4B018F4F.7070701@noaa.gov> Message-ID: On Mon, Nov 16, 2009 at 10:43 AM, Christopher Barker wrote: > Charles R Harris wrote: > > I would like some advise on the best way to add the new functions. I've > > added a new package polynomial, and that package contains four new > > modules: chebyshev, polynomial, polytemplate, polyutils. > > This seems to belong more in scipy than numpy, but I'll leave that to > others to decide. > > > whether or not to include all of the functions in these packages in the > > __init__.py, or to just import the modules. > > Are any of them compiled code? I've been very frustrated when I can't > use some pure python stuff in scipy because broken compiled fortran > extensions are getting imported that I don't even need. > > It's all python. > If that isn't an issue, and the polynomial package would end up with > only a handful of names, then I say import them all. Another way to ask > this: would there by ANY names in the polynomial package if you don't > import the modules? > > That's what I ended up doing. You still need to do "import numpy.polynomial" to get to them, they aren't automatically imported into the numpy namespace. > If there is compiled code, the import could fail gracefully, and then > you could still pull it all in. > > OTOH, what this does is bring stuff into memory unnecessarily, and also > brings it into stand-alone bundles (py2exe, py2app, etc). So if these > modules are not small, then it's probably better to have to import them > explicitly. > > Also -- do you foresee many more polynomial types in the future? I know > I'd like to see Hermite. > > Been thinking about it. Division/multiplication can get hard, but a more restricted set of operations -- division/multiplication by x -- would cover most of the common uses. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Nov 16 13:33:19 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 Nov 2009 13:33:19 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> Message-ID: <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> On Mon, Nov 16, 2009 at 12:48 AM, David Cournapeau wrote: > josef.pktd at gmail.com wrote: >> Are there new changes to the configuration needed? ? mingw 3.4.5, >> WindowsXP, Python 2.5.2 >> >> Python.h is not picked up anymore: >> >> _configtest.c:1:20: Python.h: No such file or directory >> >> Josef >> >> C:\Josef\_progs\Subversion\numpy-trunk>setup.py bdist > > Could you try with > > python setup.py build -c mingw32 bdist > > I have just tried on the same configuration as you, and I cannot > reproduce the problem with the above command. > > David It was a change I did. Following a recommendation on the cython list I had added include_dirs in distutils.cfg [build_ext] include_dirs=c:\programs\python25\lib\site-packages\numpy\core\include As a consequence numpy build had dropped the python include directory from the build options. Now, the numpy build runs for a while then breaks while building umath. Any ideas? setup.py bdist and python setup.py build -c mingw32 bdist give the same results. Thanks, Josef <...> building 'numpy.core.umath' extension compiling C sources C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes creating build\temp.win32-2.5\Release\numpy\core\src\umath compile options: '-Ibuild\src.win32-2.5\numpy\core\src\umath -Inumpy\core\includ e -Ibuild\src.win32-2.5\numpy\core\include/numpy -Inumpy\core\src\private -Inump y\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -In umpy\core\src\umath -Inumpy\core\include -IC:\Programs\Python25\include -IC:\Pro grams\Python25\PC -Ibuild\src.win32-2.5\numpy\core\src\multiarray -Ibuild\src.wi n32-2.5\numpy\core\src\umath -c' gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\s rc\umath -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core\include/numpy -In umpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -In umpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Progra ms\Python25\include -IC:\Programs\Python25\PC -Ibuild\src.win32-2.5\numpy\core\s rc\multiarray -Ibuild\src.win32-2.5\numpy\core\src\umath -c numpy\core\src\umath \umathmodule_onefile.c -o build\temp.win32-2.5\Release\numpy\core\src\umath\umat hmodule_onefile.o g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\src\umath\umathm odule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python25\PCBuild -Lbui ld\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.win32-2.5\numpy\co re\umath.pyd build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xce07): undefined reference to `npy_spacingf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xcefc): undefined reference to `npy_nextafterf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdeb7): undefined reference to `npy_spacing' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xdfae): undefined reference to `npy_nextafter' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xef75): undefined reference to `npy_spacingl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0xf0b0): undefined reference to `npy_nextafterl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c3c6): undefined reference to `npy_csqrtf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c436): undefined reference to `npy_clogf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c4d6): undefined reference to `npy_cexpf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1c78c): undefined reference to `npy_cpowf' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d28b): undefined reference to `npy_csqrt' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d30b): undefined reference to `npy_clog' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d3ab): undefined reference to `npy_cexp' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1d711): undefined reference to `npy_cpow' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e379): undefined reference to `npy_csqrtl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e429): undefined reference to `npy_clogl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1e4e9): undefined reference to `npy_cexpl' build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod ule_onefile.c:(.text+0x1ea15): undefined reference to `npy_cpowl' collect2: ld returned 1 exit status error: Command "g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\ src\umath\umathmodule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python 25\PCBuild -Lbuild\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.wi n32-2.5\numpy\core\umath.pyd" failed with exit status 1 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Mon Nov 16 17:06:00 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 16 Nov 2009 17:06:00 -0500 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <4B018F4F.7070701@noaa.gov> References: <4B018F4F.7070701@noaa.gov> Message-ID: 2009/11/16 Christopher Barker : > Charles R Harris wrote: >> I would like some advise on the best way to add the new functions. I've >> added a new package polynomial, and that package contains four new >> modules: chebyshev, polynomial, polytemplate, polyutils. > > This seems to belong more in scipy than numpy, but I'll leave that to > others to decide. > >> whether or not to include all of the functions in these packages in the >> __init__.py, or to just import the modules. > > Are any of them compiled code? I've been very frustrated when I can't > use some pure python stuff in scipy because broken compiled fortran > extensions are getting imported that I don't even need. > > If that isn't an issue, and the polynomial package would end up with > only a handful of names, then I say import them all. Another way to ask > this: would there by ANY names in the polynomial package if you don't > import the modules? > > If there is compiled code, the import could fail gracefully, and then > you could still pull it all in. > > OTOH, what this does is bring stuff into memory unnecessarily, and also > brings it into stand-alone bundles (py2exe, py2app, etc). So if these > modules are not small, then it's probably better to have to import them > explicitly. > > Also -- do you foresee many more polynomial types in the future? I know > I'd like to see Hermite. I have kind of gone silent on this issue, in part because I have been busy with other things. I think that Charles Harris' framework for working with polynomials is perfectly sufficient for adding Chebyshev polynomials, and since they're finished and working and fill a concrete need, they should probably go into numpy or scipy as is. But if you want to start introducing other types of polynomials - Hermite, Lagrange interpolation based, Bernstein based, or other - I think we would need to revive the discussion about how to unify all these different types. Anne > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Nov 16 17:31:58 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Nov 2009 15:31:58 -0700 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: References: <4B018F4F.7070701@noaa.gov> Message-ID: On Mon, Nov 16, 2009 at 3:06 PM, Anne Archibald wrote: > 2009/11/16 Christopher Barker : > > Charles R Harris wrote: > >> I would like some advise on the best way to add the new functions. I've > >> added a new package polynomial, and that package contains four new > >> modules: chebyshev, polynomial, polytemplate, polyutils. > > > > This seems to belong more in scipy than numpy, but I'll leave that to > > others to decide. > > > >> whether or not to include all of the functions in these packages in the > >> __init__.py, or to just import the modules. > > > > Are any of them compiled code? I've been very frustrated when I can't > > use some pure python stuff in scipy because broken compiled fortran > > extensions are getting imported that I don't even need. > > > > If that isn't an issue, and the polynomial package would end up with > > only a handful of names, then I say import them all. Another way to ask > > this: would there by ANY names in the polynomial package if you don't > > import the modules? > > > > If there is compiled code, the import could fail gracefully, and then > > you could still pull it all in. > > > > OTOH, what this does is bring stuff into memory unnecessarily, and also > > brings it into stand-alone bundles (py2exe, py2app, etc). So if these > > modules are not small, then it's probably better to have to import them > > explicitly. > > > > Also -- do you foresee many more polynomial types in the future? I know > > I'd like to see Hermite. > > I have kind of gone silent on this issue, in part because I have been > busy with other things. I think that Charles Harris' framework for > working with polynomials is perfectly sufficient for adding Chebyshev > polynomials, and since they're finished and working and fill a > concrete need, they should probably go into numpy or scipy as is. But > if you want to start introducing other types of polynomials - Hermite, > Lagrange interpolation based, Bernstein based, or other - I think we > would need to revive the discussion about how to unify all these > different types. > > Yes, that still needs thinking about. The recursion for the Chebyshev polynomials (and powers) has constant coefficients apart from the x, and that makes a lot of simplifications possible. In particular, the resulting polynomials can be represented as constant combinations of powers of the two roots of the recursion relation. That simplifies a lot of things that can be difficult in the more general case. And that isn't even getting started on non-graded things like the Bernstein and Lagrange interpolation based schemes. There is a lot of interesting combinatorial work going on in the polynomial business that I am not familiar with. Some of that may be of use to us in looking for a framework. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Nov 16 19:05:35 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 16:05:35 -0800 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: References: <4B018F4F.7070701@noaa.gov> Message-ID: <4B01E8CF.7090507@noaa.gov> Charles R Harris wrote: > That's what I ended up doing. You still need to do "import > numpy.polynomial" to get to them, they aren't automatically imported > into the numpy namespace. good start. This brings up a semi-off-topic question: Is there a way to avoid importing everything when importing a module deep in a big package? My example: we just started using scipy.spatial.ckdtree in a project that doesn't (yet) use anything else in scipy. The import line: from scipy.spatial.ckdtree import cKDTree takes a LONG time and adds 262 modules to sys.modules It also added 15MB to our py2exe package and who knows how much memory it is using up. As far as I can tell, it is perfectly functional with only ckdtree.so and kdtree.py Is there a solution to this short of a complete re-structuring of scipy? -Chris Tests: Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> mem use: 37404 >>> import sys >>> len(sys.modules) 29 >>> import numpy >>> len(sys.modules) 174 mem use: 45700 >>> len(sys.modules) 291 mem use: 52384 So I guess memory use isn't that bad (by modern standards!), but I'd still rather not have 115 or so modules in there! -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Nov 16 19:14:37 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 16 Nov 2009 18:14:37 -0600 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <4B01E8CF.7090507@noaa.gov> References: <4B018F4F.7070701@noaa.gov> <4B01E8CF.7090507@noaa.gov> Message-ID: <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> On Mon, Nov 16, 2009 at 18:05, Christopher Barker wrote: > Charles R Harris wrote: >> That's what I ended up doing. You still need to do "import >> numpy.polynomial" to get to them, they aren't automatically imported >> into the numpy namespace. > > good start. This brings up a semi-off-topic question: > > Is there a way to avoid importing everything when importing a module > deep in a big package? The package authors need to keep the __init__.py files clear. There is nothing you can do as a user. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Mon Nov 16 19:17:56 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 16:17:56 -0800 Subject: [Numpy-discussion] finding close together points. In-Reply-To: References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFDC12A.4070304@noaa.gov> Message-ID: <4B01EBB4.6030506@noaa.gov> Anne Archibald wrote: > 2009/11/13 Christopher Barker : >> Wow! great -- you sounded interested, but I had no idea you'd run out >> and do it! thanks! we'll check it out. well, it turns out the Python version is unacceptably slow. However, we found we could use ckdtree, and use: tree.query(points, 2, # Number of points to return per point given. distance_upper_bound = distance_tolerance, ) This gives us a set of pairs of points closer than our distance tolerance -- it includes duplicates, of course, but even when we filter those in pure python, it's is nice and speedy. It looks like it would be pretty easy to add this to the Cython code, but it's working now for us, so... Thanks again, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From peridot.faceted at gmail.com Mon Nov 16 19:25:26 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 16 Nov 2009 19:25:26 -0500 Subject: [Numpy-discussion] finding close together points. In-Reply-To: <4B01EBB4.6030506@noaa.gov> References: <4AFA0044.6020106@noaa.gov> <4AFA50D4.70403@noaa.gov> <4AFDC12A.4070304@noaa.gov> <4B01EBB4.6030506@noaa.gov> Message-ID: 2009/11/16 Christopher Barker : > Anne Archibald wrote: >> 2009/11/13 Christopher Barker : >>> Wow! great -- you sounded interested, but I had no idea you'd run out >>> and do it! thanks! we'll check it out. > > well, it turns out the Python version is unacceptably slow. However, we > found we could use ckdtree, and use: > > tree.query(points, > ? ? ? ? ? ? 2, # Number of points to return per point given. > ? ? ? ? ? ? distance_upper_bound = distance_tolerance, > ? ? ? ? ? ?) > > This gives us a set of pairs of points closer than our distance > tolerance -- it includes duplicates, of course, but even when we filter > those in pure python, it's is nice and speedy. > > It looks like it would be pretty easy to add this to the Cython code, > but it's working now for us, so... You probably noticed this, but I should remind you that if a point has more than one nearby neighbor, this may not give you what you wanted. In particular, if there are three points all within the fixed distance from each other, you may not find all three pairs this way. Anne > Thanks again, > > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Mon Nov 16 19:29:09 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 16 Nov 2009 19:29:09 -0500 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> References: <4B018F4F.7070701@noaa.gov> <4B01E8CF.7090507@noaa.gov> <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> Message-ID: 2009/11/16 Robert Kern : > On Mon, Nov 16, 2009 at 18:05, Christopher Barker wrote: >> Charles R Harris wrote: >>> That's what I ended up doing. You still need to do "import >>> numpy.polynomial" to get to them, they aren't automatically imported >>> into the numpy namespace. >> >> good start. This brings up a semi-off-topic question: >> >> Is there a way to avoid importing everything when importing a module >> deep in a big package? > > The package authors need to keep the __init__.py files clear. There is > nothing you can do as a user. The reason numpy and scipy don't do this is largely historical - Numeric had a nearly flat namespace, and imported all the submodules in any case, and numpy is trying to remain somewhat compatible; scipy originally tried to reexport all the numpy symbols, for some reason, and there too it is maintaining compatibility. Since spatial is new, though, it should be pretty good about not inflating your imports unnecessarily. It does, of course, import various things itself, which may balloon out the imports. Anne > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Mon Nov 16 19:39:28 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 16:39:28 -0800 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> References: <4B018F4F.7070701@noaa.gov> <4B01E8CF.7090507@noaa.gov> <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> Message-ID: <4B01F0C0.2050006@noaa.gov> Robert Kern wrote: >> Is there a way to avoid importing everything when importing a module >> deep in a big package? > > The package authors need to keep the __init__.py files clear. There is > nothing you can do as a user. I figured. so, to bring this back on-topic: I recommend that no package imports anything it doesn't need to. It's just not that hard to import what you need. numpy appears to be pretty bad in this regard: >>> len(sys.modules) 29 >>> import numpy >>> len(sys.modules) 174 with a simple "import numpy", I get all kinds of stuff I'm unlikely to need in every app, and if I do, I import it anyway: numpy.testing numpy.ma numpy.random numpy.linalg numpy.lib.financial numpy.lib.polynomial numpy.fft It's probably way too late from a backward compatibility perspective to change any of this, but I'd love to see that cleaned up. Maybe we can clean up scipy, though? We could have a two-API apporach, ike what MPL has been doing. One that import every darn thing for ease of us on the command line, and one that only imports what you ask for. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Nov 16 19:44:12 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 16 Nov 2009 16:44:12 -0800 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: References: <4B018F4F.7070701@noaa.gov> <4B01E8CF.7090507@noaa.gov> <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> Message-ID: <4B01F1DC.6050301@noaa.gov> Anne Archibald wrote: > The reason numpy and scipy don't do this is largely historical - > Numeric had a nearly flat namespace, I know. Despite namespaces being "one honking great idea", it seems to have taken a while to catch on. > Since spatial is new, though, it should be pretty good about not > inflating your imports unnecessarily. well, maybe -- it seems to import everything when you do an import scipy.spatial Since kdtree is about all that's there, it's not a big deal, but much like the polynomial namespace, if we envision a day when there may be all sorts of lovely spatial indexing options in there, maybe it shouldn't bring them all into the "spatial" namespace. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Nov 16 19:49:11 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 16 Nov 2009 18:49:11 -0600 Subject: [Numpy-discussion] Adding the new polynomial/chebyshev modules. In-Reply-To: <4B01F0C0.2050006@noaa.gov> References: <4B018F4F.7070701@noaa.gov> <4B01E8CF.7090507@noaa.gov> <3d375d730911161614p7808a898ic3750d0d5c535450@mail.gmail.com> <4B01F0C0.2050006@noaa.gov> Message-ID: <3d375d730911161649p578c722dvb32a8992dc6ee80a@mail.gmail.com> On Mon, Nov 16, 2009 at 18:39, Christopher Barker wrote: > Robert Kern wrote: >>> Is there a way to avoid importing everything when importing a module >>> deep in a big package? >> >> The package authors need to keep the __init__.py files clear. There is >> nothing you can do as a user. > > I figured. so, to bring this back on-topic: > > I recommend that no package imports anything it doesn't need to. It's > just not that hard to import what you need. > > numpy appears to be pretty bad in this regard: > > ?>>> len(sys.modules) > 29 > ?>>> import numpy > ?>>> len(sys.modules) > 174 It's not *quite* as bad as that. There are a lot of keys in sys.modules that point to None as an optimization (e.g. if foo/bar.py imports math, let's say, the import system will look up foo.math, which fails, so sys.modules['foo.math'] = None so it doesn't try that again). Here is the correct way to measure that: >>> import sys >>> old = set(sys.modules) >>> len(old) 30 >>> import numpy >>> new = set(sys.modules) >>> from_numpy = [m for m in (new - old) if sys.modules[m] is not None] >>> len(from_numpy) 93 >>> from_numpy ['numpy.lib._iotools', 'numpy.core.info', 'numpy.lib.getlimits', 'ctypes._endian', 'numpy.core.numerictypes', 'struct', 'numpy.random.info', 'numpy.linalg', 'numpy.testing', 'numpy.core.umath', 'numpy.core.scalarmath', 'string', 'numpy.lib.arraysetops', 'numpy.version', 'numpy.lib.type_check', 'numpy.lib._datasource', 'numpy.lib.io', 'numpy.ma.extras', 'token', 'numpy.fft.fftpack_lite', 'numpy.core.multiarray', 'dis', 'cStringIO', 'numpy.ma.core', 'numpy.add_newdocs', 'numpy.testing.decorators', 're', 'numpy.lib._compiled_base', 'numpy.random.mtrand', 'math', 'numpy.fft.helper', 'inspect', '_ctypes', 'numpy.lib.ufunclike', 'numpy.lib.info', 'ctypes', 'numpy.core._sort', 'numpy.core.memmap', 'traceback', 'itertools', 'numpy.fft.fftpack', 'opcode', 'numpy.linalg.lapack_lite', '__future__', '_sre', 'unittest', 'numpy.random', 'numpy.lib.twodim_base', 'operator', 'sre_constants', 'numpy.lib.arrayterator', 'numpy.lib.financial', 'imp', 'numpy.core.arrayprint', 'tokenize', 'numpy.lib.stride_tricks', 'numpy', 'numpy.core.defmatrix', 'cPickle', 'numpy.ma', 'numpy.testing.utils', 'gestalt', '_struct', 'numpy.core.fromnumeric', 'numpy.ctypeslib', 'numpy.lib.scimath', 'numpy.fft', 'numpy.lib', 'numpy.lib.function_base', 'sre_parse', 'sre_compile', 'numpy.lib.shape_base', 'numpy.lib.polynomial', 'numpy._import_tools', 'numpy.fft.info', 'numpy.core.records', 'numpy.core._dotblas', 'shutil', 'strop', 'numpy.testing.numpytest', 'numpy.core.numeric', 'numpy.linalg.info', 'numpy.core._internal', 'numpy.__config__', 'numpy.core', 'numpy.lib.index_tricks', 'numpy.lib.utils', 'numpy.core.defchararray', 'numpy.lib.format', 'numpy.testing.nosetester', 'time', 'numpy.lib.machar', 'numpy.linalg.linalg'] > with a simple "import numpy", I get all kinds of stuff I'm unlikely to > need in every app, and if I do, I import it anyway: > > numpy.testing > numpy.ma > numpy.random > numpy.linalg > numpy.lib.financial > numpy.lib.polynomial > numpy.fft > > > It's probably way too late from a backward compatibility perspective to > change any of this, but I'd love to see that cleaned up. Oh, good heavens, yes. > Maybe we can clean up scipy, though? Some of the parts checked in since 0.7, but otherwise not really, no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Mon Nov 16 22:52:22 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 17 Nov 2009 12:52:22 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> Message-ID: <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> On Tue, Nov 17, 2009 at 3:33 AM, wrote: > > Now, the numpy build runs for a while then breaks while building umath. > > Any ideas? The behavior of distutils with config files is mysterious, I gave up trying to understand it long ago :) I use scripts instead to control everything from command line. > > creating build\temp.win32-2.5\Release\numpy\core\src\umath > compile options: '-Ibuild\src.win32-2.5\numpy\core\src\umath -Inumpy\core\includ > e -Ibuild\src.win32-2.5\numpy\core\include/numpy -Inumpy\core\src\private -Inump > y\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -In > umpy\core\src\umath -Inumpy\core\include -IC:\Programs\Python25\include -IC:\Pro > grams\Python25\PC -Ibuild\src.win32-2.5\numpy\core\src\multiarray -Ibuild\src.wi > n32-2.5\numpy\core\src\umath -c' > gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\s > rc\umath -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core\include/numpy -In > umpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -In > umpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\include -IC:\Progra > ms\Python25\include -IC:\Programs\Python25\PC -Ibuild\src.win32-2.5\numpy\core\s > rc\multiarray -Ibuild\src.win32-2.5\numpy\core\src\umath -c numpy\core\src\umath > \umathmodule_onefile.c -o build\temp.win32-2.5\Release\numpy\core\src\umath\umat > hmodule_onefile.o > g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\src\umath\umathm > odule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python25\PCBuild -Lbui > ld\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.win32-2.5\numpy\co > re\umath.pyd > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xce07): undefined reference to `npy_spacingf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xcefc): undefined reference to `npy_nextafterf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xdeb7): undefined reference to `npy_spacing' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xdfae): undefined reference to `npy_nextafter' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xef75): undefined reference to `npy_spacingl' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0xf0b0): undefined reference to `npy_nextafterl' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1c3c6): undefined reference to `npy_csqrtf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1c436): undefined reference to `npy_clogf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1c4d6): undefined reference to `npy_cexpf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1c78c): undefined reference to `npy_cpowf' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1d28b): undefined reference to `npy_csqrt' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1d30b): undefined reference to `npy_clog' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1d3ab): undefined reference to `npy_cexp' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1d711): undefined reference to `npy_cpow' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1e379): undefined reference to `npy_csqrtl' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1e429): undefined reference to `npy_clogl' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1e4e9): undefined reference to `npy_cexpl' > build\temp.win32-2.5\Release\numpy\core\src\umath\umathmodule_onefile.o:umathmod > ule_onefile.c:(.text+0x1ea15): undefined reference to `npy_cpowl' > collect2: ld returned 1 exit status > error: Command "g++ -mno-cygwin -shared build\temp.win32-2.5\Release\numpy\core\ > src\umath\umathmodule_onefile.o -LC:\Programs\Python25\libs -LC:\Programs\Python > 25\PCBuild -Lbuild\temp.win32-2.5 -lnpymath -lpython25 -lmsvcr71 -o build\lib.wi > n32-2.5\numpy\core\umath.pyd" failed with exit status 1 All those refer to recently added functions, which suggest some old files/configuration are used. Be sure to clean the working tree and the build directory and try again. I have just tested it on both wine and a xp vm with mingw, and both build correctly. cheers, David From gokhansever at gmail.com Tue Nov 17 00:44:17 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 16 Nov 2009 23:44:17 -0600 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data Message-ID: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> Hello, I have a data which represents aerosol size distribution in between 0.1 to 3.0 micrometer ranges. I would like extrapolate the lower size down to 10 nm. The data in this context is log-normally distributed. Therefore I am looking a way to fit a log-normal curve onto my data. Could you please give me some pointers to solve this problem? Thank you. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Tue Nov 17 01:13:08 2009 From: geometrian at gmail.com (Ian Mallett) Date: Mon, 16 Nov 2009 22:13:08 -0800 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data In-Reply-To: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> References: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> Message-ID: Theory wise: -Do a linear regression on your data. -Apply a logrithmic transform to your data's dependent variable, and do another linear regression. -Apply a logrithmic transform to your data's independent variable, and do another linear regression. -Take the best regression (highest r^2 value) and execute a back transform. Then, to get your desired extrapolation, simply substitute in the size for the independent variable to get the expected value. If, however, you're looking for how to implement this in NumPy or SciPy, I can't really help :-P Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From fonnesbeck at gmail.com Tue Nov 17 02:07:11 2009 From: fonnesbeck at gmail.com (Chris) Date: Tue, 17 Nov 2009 07:07:11 +0000 (UTC) Subject: [Numpy-discussion] min bug Message-ID: I'm pretty sure this shouldn't happen: In [1]: from numpy import min In [2]: min(5000, 4) Out[2]: 5000 From sebastian at sipsolutions.net Tue Nov 17 02:13:14 2009 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 17 Nov 2009 08:13:14 +0100 Subject: [Numpy-discussion] min bug In-Reply-To: References: Message-ID: <1258441994.2860.2.camel@sebastian> Known issue, I think someone posted about it a while ago too. The numpy min is array aware, and it expects an array. The second argument is the axis, which in the case of a single number doesn't matter. On Tue, 2009-11-17 at 07:07 +0000, Chris wrote: > I'm pretty sure this shouldn't happen: > > In [1]: from numpy import min > > In [2]: min(5000, 4) > Out[2]: 5000 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From alan.mcintyre at gmail.com Tue Nov 17 02:13:34 2009 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Mon, 16 Nov 2009 23:13:34 -0800 Subject: [Numpy-discussion] min bug In-Reply-To: References: Message-ID: <1d36917a0911162313k4d82c2a9n78fbf706cd662d8b@mail.gmail.com> On Mon, Nov 16, 2009 at 11:07 PM, Chris wrote: > I'm pretty sure this shouldn't happen: > > In [1]: from numpy import min > > In [2]: min(5000, 4) > Out[2]: 5000 The way you're calling it is working like this: min((5000,) , axis=4) so you'd need to do this instead: min((5000,4)) From sole at esrf.fr Tue Nov 17 02:34:48 2009 From: sole at esrf.fr (=?ISO-8859-1?Q?=22V=2E_Armando_Sol=E9=22?=) Date: Tue, 17 Nov 2009 08:34:48 +0100 Subject: [Numpy-discussion] min bug In-Reply-To: <1258441994.2860.2.camel@sebastian> References: <1258441994.2860.2.camel@sebastian> Message-ID: <4B025218.5070909@esrf.fr> Sebastian Berg wrote: > Known issue, I think someone posted about it a while ago too. The numpy > min is array aware, and it expects an array. The second argument is the > axis, which in the case of a single number doesn't matter. > > On Tue, 2009-11-17 at 07:07 +0000, Chris wrote: > >> I'm pretty sure this shouldn't happen: >> >> In [1]: from numpy import min >> >> In [2]: min(5000, 4) >> Out[2]: 5000 >> I think I have to agree with the original poster. It would be more correct to rise an exception because the axis is beyond the number of axes than to return a confusing result. Armando From alan.mcintyre at gmail.com Tue Nov 17 02:49:00 2009 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Mon, 16 Nov 2009 23:49:00 -0800 Subject: [Numpy-discussion] min bug In-Reply-To: <4B025218.5070909@esrf.fr> References: <1258441994.2860.2.camel@sebastian> <4B025218.5070909@esrf.fr> Message-ID: <1d36917a0911162349o39a3fd29i30d177577d78eb22@mail.gmail.com> On Mon, Nov 16, 2009 at 11:34 PM, "V. Armando Sol?" wrote: > Sebastian Berg wrote: >> Known issue, I think someone posted about it a while ago too. The numpy >> min is array aware, and it expects an array. The second argument is the >> axis, which in the case of a single number doesn't matter. > > I think I have to agree with the original poster. > > It would be more correct to rise an exception because the axis is beyond > the number of axes than to return a confusing result. Hm, now that I actually try my example min((5000,), 4) it fails with an axis out of bounds error. I presume there's a reason why a 0-D array gets special treatment? From neilcrighton at gmail.com Tue Nov 17 04:47:43 2009 From: neilcrighton at gmail.com (Neil) Date: Tue, 17 Nov 2009 09:47:43 +0000 (UTC) Subject: [Numpy-discussion] min bug References: <1258441994.2860.2.camel@sebastian> <4B025218.5070909@esrf.fr> <1d36917a0911162349o39a3fd29i30d177577d78eb22@mail.gmail.com> Message-ID: Alan McIntyre gmail.com> writes: > > On Mon, Nov 16, 2009 at 11:34 PM, "V. Armando Sol?" esrf.fr> wrote: > > Sebastian Berg wrote: > >> Known issue, I think someone posted about it a while ago too. The numpy > >> min is array aware, and it expects an array. The second argument is the > >> axis, which in the case of a single number doesn't matter. > > > > I think I have to agree with the original poster. > > > > It would be more correct to rise an exception because the axis is beyond > > the number of axes than to return a confusing result. > > Hm, now that I actually try my example > > min((5000,), 4) > > it fails with an axis out of bounds error. I presume there's a reason > why a 0-D array gets special treatment? > In [16]: import numpy as np In [17]: np.__version__ Out[17]: '1.4.0.dev7746' In [18]: np.min(5000, 4) ... ValueError: axis(=4) out of bounds Neil From josef.pktd at gmail.com Tue Nov 17 10:38:00 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Nov 2009 10:38:00 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> Message-ID: <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> On Mon, Nov 16, 2009 at 10:52 PM, David Cournapeau wrote: > On Tue, Nov 17, 2009 at 3:33 AM, ? wrote: > >> >> Now, the numpy build runs for a while then breaks while building umath. >> >> Any ideas? > > The behavior of distutils with config files is mysterious, I gave up > trying to understand it long ago :) I use scripts instead to control > everything from command line. > > All those refer to recently added functions, which suggest some old > files/configuration are used. Be sure to clean the working tree and > the build directory and try again. I found the offending old file, checkout tree was clean, but last time I built scipy, I had copied libnpymath into the python include directory and forgot to remove it. My mistake. Now numpy builds without problems. When I run the tests I get 16 failures mostly nan related. I have no idea whether they are real or if there is still something screwed up in my setup. See below. Current scipy also builds without problems, but tests have 3 errors and 1 failure which I think are new. Thanks, Josef > > I have just tested it on both wine and a xp vm with mingw, and both > build correctly. > > cheers, > > David test results: C:\Josef\work-oth>python -c "import numpy; numpy.test()" Running unit tests for numpy NumPy version 1.4.0.dev7746 NumPy is installed in c:\programs\python25\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.11.1 ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ...............................................................................K .....................................................FF.......................FF FF....FF......FF.............FFFF....K......................K.F................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ..............................................c:\programs\python25\lib\site-pack ages\numpy\lib\io.py:1324: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 5) Line #12 (got 4 columns instead of 5) Line #22 (got 4 columns instead of 5) Line #32 (got 4 columns instead of 5) Line #42 (got 4 columns instead of 5) warnings.warn(errmsg, ConversionWarning) .c:\programs\python25\lib\site-packages\numpy\lib\io.py:1324: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 2) Line #12 (got 4 columns instead of 2) Line #22 (got 4 columns instead of 2) Line #32 (got 4 columns instead of 2) Line #42 (got 4 columns instead of 2) warnings.warn(errmsg, ConversionWarning) ..........................................K........K............F............... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ......S......................................................................... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................... ====================================================================== FAIL: test_umath.test_hypot_special_values(1.#QNAN, 1.#INF) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 213, in assert_hypot_isinf assert np.isinf(ncu.hypot(x, y)) AssertionError ====================================================================== FAIL: test_umath.test_hypot_special_values(1.#INF, 1.#QNAN) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 213, in assert_hypot_isinf assert np.isinf(ncu.hypot(x, y)) AssertionError ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, 2.3561944901923448) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 450, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: 2.3561944901923448 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, -2.3561944901923448) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 450, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: -2.3561944901923448 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, 0.78539816339744828) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 450, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: 0.78539816339744828 ====================================================================== FAIL: test_umath.test_arctan2_special_values(nan, -0.78539816339744828) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 450, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal ACTUAL: nan DESIRED: -0.78539816339744828 ====================================================================== FAIL: test_umath.test_nextafter ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 834, in test_nextafter assert np.nextafter(one, two) - one == eps AssertionError ====================================================================== FAIL: test_umath.test_spacing ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 846, in test_spacing assert np.spacing(one) == eps AssertionError ====================================================================== FAIL: test_umath_complex.TestCabs.test_cabs_inf_nan(, 1.#INF, 1.#QNAN, 1.#INF) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 437, in check_real_value assert_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 247, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 676, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(1.#INF) ====================================================================== FAIL: test_umath_complex.TestCabs.test_cabs_inf_nan(, -1.#INF, 1.#QNAN, 1.#INF) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 437, in check_real_value assert_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 247, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 676, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(1.#INF) ====================================================================== FAIL: test_umath_complex.TestCarg.test_special_values(, -1.#INF, 1 .#INF, 2.3561944901923448, False) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 439, in check_real_value assert_almost_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 440, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 764, in assert_array_almost_equal header='Arrays are not almost equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(2.3561944901923448) ====================================================================== FAIL: test_umath_complex.TestCarg.test_special_values(, -1.#INF, - 1.#INF, -2.3561944901923448, False) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 439, in check_real_value assert_almost_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 440, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 764, in assert_array_almost_equal header='Arrays are not almost equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(-2.3561944901923448) ====================================================================== FAIL: test_umath_complex.TestCarg.test_special_values(, 1.#INF, 1. #INF, 0.78539816339744828, False) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 439, in check_real_value assert_almost_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 440, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 764, in assert_array_almost_equal header='Arrays are not almost equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(0.78539816339744828) ====================================================================== FAIL: test_umath_complex.TestCarg.test_special_values(, 1.#INF, -1 .#INF, -0.78539816339744828, False) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 439, in check_real_value assert_almost_equal(f(z1), x) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 440, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 764, in assert_array_almost_equal header='Arrays are not almost equal') File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 586, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (x and y nan location mismatch [ True], False mismatch) x: array([ NaN]) y: array(-0.78539816339744828) ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 179, in test_special_values assert_almost_equal(np.log(x), y) File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 436, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [ Inf NaNj] DESIRED: (1.#INF+2.35619449019j) ====================================================================== FAIL: test_doctests (test_polynomial.TestDocs) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7746.win32\pr ograms\python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line 90, in test_doctests return rundocs() File "\programs\python25\Lib\site-packages\numpy\testing\utils.py", line 952, in rundocs raise AssertionError("Some doctests failed:\n%s" % "\n".join(msg)) AssertionError: Some doctests failed: ********************************************************************** File "c:\programs\python25\lib\site-packages\numpy\lib\tests\test_polynomial.py" , line 20, in test_polynomial Failed example: print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) Expected: 2 1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 Got: 2 1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 ---------------------------------------------------------------------- Ran 2338 tests in 17.719s FAILED (KNOWNFAIL=5, SKIP=1, failures=16) C:\Josef\work-oth> From boogaloojb at yahoo.fr Tue Nov 17 11:24:52 2009 From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant) Date: Tue, 17 Nov 2009 16:24:52 +0000 (GMT) Subject: [Numpy-discussion] Strange inversion in shape with some slices Message-ID: <418485.97808.qm@web28511.mail.ukl.yahoo.com> Hello, I think there's something strange with shape when a slice is given by an array. import numpy as N my_array = N.ones((2, 3, 6)) ind = N.arange(4) #you hope to find (3, 4) print my_array[0, :, ind].shape print my_array[0, :, 0:4].shape print my_array[0][:, ind].shape print my_array[0][:, 0:4].shape """ (4, 3) (3, 4) (3, 4) (3, 4) """ Jean-Baptiste Rudant -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 17 11:40:05 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 17 Nov 2009 11:40:05 -0500 Subject: [Numpy-discussion] Strange inversion in shape with some slices In-Reply-To: <418485.97808.qm@web28511.mail.ukl.yahoo.com> References: <418485.97808.qm@web28511.mail.ukl.yahoo.com> Message-ID: <1cd32cbb0911170840h25d7896ds65074bf6acd75de3@mail.gmail.com> On Tue, Nov 17, 2009 at 11:24 AM, Jean-Baptiste Rudant wrote: > Hello, > I think there's something strange with shape when a slice is given by an > array. > import numpy as N > my_array = N.ones((2, 3, 6)) > ind = N.arange(4) > #you hope to find (3, 4) > print my_array[0, :, ind].shape > print my_array[0, :, 0:4].shape > print my_array[0][:, ind].shape > print my_array[0][:, 0:4].shape > """ > (4, 3) > (3, 4) > (3, 4) > (3, 4) > """ > Jean-Baptiste Rudant there was a long thread on this with the explanation in March, title "is it a bug?" Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From dsdale24 at gmail.com Tue Nov 17 12:16:43 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 17 Nov 2009 12:16:43 -0500 Subject: [Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0 In-Reply-To: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> References: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> Message-ID: Please consider applying this patch before freezing, or you can't do "python setup.py develop" with Distribute (at least not with Enthought's Enable): ndex: numpy/distutils/command/build_ext.py =================================================================== --- numpy/distutils/command/build_ext.py (revision 7734) +++ numpy/distutils/command/build_ext.py (working copy) @@ -61,6 +61,7 @@ if self.distribution.have_run.get('build_clib'): log.warn('build_clib already run, it is too late to ' \ 'ensure in-place build of build_clib') + build_clib = self.distribution.get_command_obj('build_clib') else: build_clib = self.distribution.get_command_obj('build_clib') build_clib.inplace = 1 On Mon, Nov 16, 2009 at 4:29 AM, David Cournapeau wrote: > Hi, > > ? ?A quick remainder: the trunk will be closed for 1.4.0 changes within > a few hours. After that time, the trunk should only contain things which > will be in 1.5.0, and the 1.4.0 changes will be in the 1.4.0 branch, > which should contain only bug fixes. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Tue Nov 17 12:27:03 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 17 Nov 2009 11:27:03 -0600 Subject: [Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0 In-Reply-To: References: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911170927i746c6f21sd798e97e8e3a587@mail.gmail.com> On Tue, Nov 17, 2009 at 11:16, Darren Dale wrote: > Please consider applying this patch before freezing, or you can't do > "python setup.py develop" with Distribute (at least not with > Enthought's Enable): It's just a temporary complete freeze while David branches 1.4 off the trunk and then just a freeze on features. This is a bug fix that can be checked into the 1.4 branch once David is finished. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gokhansever at gmail.com Tue Nov 17 12:29:10 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 17 Nov 2009 11:29:10 -0600 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data In-Reply-To: References: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> Message-ID: <49d6b3500911170929rccaa0e7k75f1450a2f48e519@mail.gmail.com> On Tue, Nov 17, 2009 at 12:13 AM, Ian Mallett wrote: > Theory wise: > -Do a linear regression on your data. > -Apply a logrithmic transform to your data's dependent variable, and do > another linear regression. > -Apply a logrithmic transform to your data's independent variable, and do > another linear regression. > -Take the best regression (highest r^2 value) and execute a back transform. > > Then, to get your desired extrapolation, simply substitute in the size for > the independent variable to get the expected value. > > If, however, you're looking for how to implement this in NumPy or SciPy, I > can't really help :-P > Ian > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > OK, before applying your suggestions. I have a few more questions. Here is 1 real-sample data that I will use as a part of the log-normal fitting. There is 15 elements in this arrays each being a concentration for corresponding 0.1 - 3.0 um size ranges. I[74]: conc O[74]: array([ 119.7681, 118.546 , 146.6548, 96.5478, 109.9911, 32.9974, 20.7762, 6.1107, 12.2212, 3.6664, 3.6664, 1.2221, 2.4443, 2.4443, 3.6664]) For now not calibrated size range I just assume a linear array: I[78]: sizes = linspace(0.1, 3.0, 15) I[79]: sizes O[79]: array([ 0.1 , 0.30714286, 0.51428571, 0.72142857, 0.92857143, 1.13571429, 1.34285714, 1.55 , 1.75714286, 1.96428571, 2.17142857, 2.37857143, 2.58571429, 2.79285714, 3. ]) Not a very ideal looking log-normal, but so far I don't know what else besides a log-normal fit would give me a better estimate: I[80]: figure(); plot(sizes, conc) http://img406.imageshack.us/img406/156/sizeconc.png scipy.stats has the lognorm.fit lognorm.fit(data,s,loc=0,scale=1) - Parameter estimates for lognorm data and applying this to my data. However not sure the right way of calling it, and not sure if this could be applied to my case? I[81]: stats.lognorm.fit(conc) O[81]: array([ 2.31386066, 1.19126064, 9.5748391 ]) Lastly, what is the way to create a ideal log-normal sample using the stats.lognorm.rvs? Thanks -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Tue Nov 17 13:33:29 2009 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 17 Nov 2009 19:33:29 +0100 Subject: [Numpy-discussion] Strange inversion in shape with some slices In-Reply-To: <1cd32cbb0911170840h25d7896ds65074bf6acd75de3@mail.gmail.com> References: <418485.97808.qm@web28511.mail.ukl.yahoo.com> <1cd32cbb0911170840h25d7896ds65074bf6acd75de3@mail.gmail.com> Message-ID: On Tue, Nov 17, 2009 at 5:40 PM, wrote: > On Tue, Nov 17, 2009 at 11:24 AM, Jean-Baptiste Rudant > wrote: >> Hello, >> I think there's something strange with shape when a slice is given by an >> array. >> import numpy as N >> my_array = N.ones((2, 3, 6)) >> ind = N.arange(4) >> #you hope to find (3, 4) >> print my_array[0, :, ind].shape >> print my_array[0, :, 0:4].shape >> print my_array[0][:, ind].shape >> print my_array[0][:, 0:4].shape >> """ >> (4, 3) >> (3, 4) >> (3, 4) >> (3, 4) >> """ >> Jean-Baptiste Rudant > > there was a long thread on this with the explanation in March, title > "is it a bug?" > The thread is here: http://www.mail-archive.com/numpy-discussion at scipy.org/msg16300.html It looks to me like violating the principle of "least surprise". My reading of that thread is, that the observed "bug" is mostly a consequence coming from the way fancy indexing is implemented. How about deprecating this kind of index mixing !? Regards, Sebastian Haase From robert.kern at gmail.com Tue Nov 17 13:41:30 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 17 Nov 2009 12:41:30 -0600 Subject: [Numpy-discussion] Strange inversion in shape with some slices In-Reply-To: References: <418485.97808.qm@web28511.mail.ukl.yahoo.com> <1cd32cbb0911170840h25d7896ds65074bf6acd75de3@mail.gmail.com> Message-ID: <3d375d730911171041q7f8905ecy8365ad83300b7554@mail.gmail.com> On Tue, Nov 17, 2009 at 12:33, Sebastian Haase wrote: > On Tue, Nov 17, 2009 at 5:40 PM, ? wrote: >> On Tue, Nov 17, 2009 at 11:24 AM, Jean-Baptiste Rudant >> wrote: >>> Hello, >>> I think there's something strange with shape when a slice is given by an >>> array. >>> import numpy as N >>> my_array = N.ones((2, 3, 6)) >>> ind = N.arange(4) >>> #you hope to find (3, 4) >>> print my_array[0, :, ind].shape >>> print my_array[0, :, 0:4].shape >>> print my_array[0][:, ind].shape >>> print my_array[0][:, 0:4].shape >>> """ >>> (4, 3) >>> (3, 4) >>> (3, 4) >>> (3, 4) >>> """ >>> Jean-Baptiste Rudant >> >> there was a long thread on this with the explanation in March, title >> "is it a bug?" >> > The thread is here: > http://www.mail-archive.com/numpy-discussion at scipy.org/msg16300.html > > It looks to me like violating the principle of "least surprise". That "principle" is notoriously unreliable, and I am never convinced by it. > My reading of that thread is, that the observed "bug" is mostly a > consequence coming from the way fancy indexing is implemented. How > about deprecating this kind of index mixing !? No. When you need it, you need it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Tue Nov 17 14:24:21 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 17 Nov 2009 13:24:21 -0600 Subject: [Numpy-discussion] Strange inversion in shape with some slices In-Reply-To: References: <418485.97808.qm@web28511.mail.ukl.yahoo.com> <1cd32cbb0911170840h25d7896ds65074bf6acd75de3@mail.gmail.com> Message-ID: <777B7903-6BA1-4AA2-8024-008287352E93@enthought.com> On Nov 17, 2009, at 12:33 PM, Sebastian Haase wrote: > On Tue, Nov 17, 2009 at 5:40 PM, wrote: >> On Tue, Nov 17, 2009 at 11:24 AM, Jean-Baptiste Rudant >> wrote: >>> Hello, >>> I think there's something strange with shape when a slice is given >>> by an >>> array. >>> import numpy as N >>> my_array = N.ones((2, 3, 6)) >>> ind = N.arange(4) >>> #you hope to find (3, 4) >>> print my_array[0, :, ind].shape >>> print my_array[0, :, 0:4].shape >>> print my_array[0][:, ind].shape >>> print my_array[0][:, 0:4].shape >>> """ >>> (4, 3) >>> (3, 4) >>> (3, 4) >>> (3, 4) >>> """ >>> Jean-Baptiste Rudant >> >> there was a long thread on this with the explanation in March, title >> "is it a bug?" >> > The thread is here: > http://www.mail-archive.com/numpy-discussion at scipy.org/msg16300.html > > It looks to me like violating the principle of "least surprise". > My reading of that thread is, that the observed "bug" is mostly a > consequence coming from the way fancy indexing is implemented. How > about deprecating this kind of index mixing !? I don't think this is really a case of the way fancy indexing is implemented. It could have been done in many ways. The problem is one of real ambiguity in the general case and the fact that there wasn't any code written to handle the less ambiguous special-cases like this one. There are real applications of index mixing, so turning it off is not an option. We could perhaps add better handling of circumstances like this, however, with appropriate warnings and transitions. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 17 20:55:50 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 18 Nov 2009 10:55:50 +0900 Subject: [Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0 In-Reply-To: References: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> Message-ID: <5b8d13220911171755y2b1f306en38bbd40f789b4d80@mail.gmail.com> already done in r7743 :) Did you report it as a bug on trac, so that I close it as well, David From bergstrj at iro.umontreal.ca Tue Nov 17 21:48:47 2009 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Tue, 17 Nov 2009 21:48:47 -0500 Subject: [Numpy-discussion] Unexpected attribute error In-Reply-To: References: Message-ID: <7f1eaee30911171848m5cbf5a32wafcd135e5e36f5da@mail.gmail.com> Is it by design that ?"numpy.sqrt(None)" raises an "AttributeError: sqrt"? This was confusing because there was an attribute lookup of 'sqrt' in numpy right there in the expression I typed, but that was not the attribute that python was complaining about. ?I presume that numpy.sqrt didn't know what to do with None, so it looked up a .sqrt in it or something... but I presume this only in hindsight now that I figured out the problem--I didn't mean to take the sqrt of None in the first place. How about adding some information to the AttributeError, such as the object on which the lookup failed (here None, as opposed to the numpy module). Now I'm off to delete all the getattr(numpy, 'sqrt') calls I littered through a few files... James -- http://www-etud.iro.umontreal.ca/~bergstrj -- http://www-etud.iro.umontreal.ca/~bergstrj From robert.kern at gmail.com Tue Nov 17 21:53:46 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 17 Nov 2009 20:53:46 -0600 Subject: [Numpy-discussion] Unexpected attribute error In-Reply-To: <7f1eaee30911171848m5cbf5a32wafcd135e5e36f5da@mail.gmail.com> References: <7f1eaee30911171848m5cbf5a32wafcd135e5e36f5da@mail.gmail.com> Message-ID: <3d375d730911171853v1f914443t39b68153912921b0@mail.gmail.com> On Tue, Nov 17, 2009 at 20:48, James Bergstra wrote: > Is it by design that ?"numpy.sqrt(None)" raises an "AttributeError: sqrt"? Yes. numpy.sqrt() is a ufunc. Ufuncs take their arguments and try to convert them to numpy arrays; the manual equivalent is numpy.asarray(None). In this case, you get a rank-0 array with dtype=object. The way unary ufuncs work on object arrays is to call the method of the same name on the object. None.sqrt doesn't exist, so the AttributeError gets raised. > This was confusing because there was an attribute lookup of 'sqrt' in > numpy right there in the expression I typed, but that was not the > attribute that python was complaining about. ?I presume that numpy.sqrt > didn't know what to do with None, so it looked up a .sqrt in it or > something... but I presume this only in hindsight now that I figured > out the problem--I didn't mean to take the sqrt of None in the first place. > > How about adding some information to the AttributeError, such > as the object on which the lookup failed (here None, as opposed to the > numpy module). Patches welcome. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cheronetolivia at yahoo.com Wed Nov 18 04:08:47 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Wed, 18 Nov 2009 01:08:47 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin Message-ID: <13497.7968.qm@web51012.mail.re2.yahoo.com> Hello. I am currently trying to install the latest version of NumPy for my cygwin Python, and I am having problems... I have downloaded the source, and unzipped and untarred it in my home directory. Subsequently, I included the following in the site.cfg file: >[DEFAULT] >library_dirs = /cygdrive/c/cygwin/lib >include_dirs = /cygdrive/c/cygwin/usr/local/include >[blas_opt] >libraries = f77blas, cblas, atlas >[lapack_opt] >libraries = lapack, f77blas, cblas, atlas In the Cygwin bash shell, after going to my home directory, I have executed: python setup.py config --compiler=mingw32 build --compiler=mingw32 install as instructed in the "Installing SciPy/Windows" page. The result is the following: error: Command "gcc -mno-cygwin -mdll -O -Wall -Inumpy/core/include -Ibuild/src. cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/in clude -I/usr/include/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/sr c/npy_math.c -o build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2 .5/numpy/core/src/npy_math.o" failed with exit status 1 I have included below the complete output (and error) I get. What could be the problem? Thank you, Olivia ================================================================================== libraries lapack_atlas not found in /cygdrive/c/cygwin/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /cygdrive/c/cygwin/lib libraries lapack_atlas not found in /cygdrive/c/cygwin/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/cygdrive/c/cygwin/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/cygdrive/c/cygwin/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running config running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/include/numpy/config.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/inc lude/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__multiarray_api.h'] building extension "numpy.core.multiarray" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/src/scalartypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/ar raytypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config .h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__ufunc_api. h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/src/scalartypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/ar raytypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umath_funcs.inc' , 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umath_loops.inc', 'build/src. cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h', 'build/src.cygwin-1.5 .25-i686-2.5/numpy/core/include/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i 686-2.5/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__ufunc_api. h' to sources. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/include/numpy/config.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/inc lude/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__multiarray_api.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__ufunc_api.h'] building extension "numpy.core._dotblas" sources building extension "numpy.core.umath_tests" sources building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/python_xerbla.c' to sources. building extension "numpy.random.mtrand" sources /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/config.py:39: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of try_run is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) No module named cygwinccompiler in numpy.distutils; trying from distutils customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -mno-cygwin -mdll -O -Wall compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Inumpy/core/src -Inumpy/core/include -I/usr/incl ude/python2.5 -c _configtest.c -o _configtest.o gcc -mno-cygwin -s _configtest.o -o _configtest.exe _configtest.exe success! removing: _configtest.c _configtest.o _configtest.exe building data_files sources running build_py copying numpy/version.py -> build/lib.cygwin-1.5.25-i686-2.5/numpy copying build/src.cygwin-1.5.25-i686-2.5/numpy/__config__.py -> build/lib.cygwin -1.5.25-i686-2.5/numpy copying build/src.cygwin-1.5.25-i686-2.5/numpy/distutils/__config__.py -> build/ lib.cygwin-1.5.25-i686-2.5/numpy/distutils running build_clib No module named cygwinccompiler in numpy.distutils; trying from distutils customize Mingw32CCompiler customize Mingw32CCompiler using build_clib building 'npymath' library compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/ core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2. 5 -c' gcc -mno-cygwin -mdll -O -Wall -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i6 86-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/inc lude/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c -o build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/sr c/npy_math.o In file included from /usr/include/python2.5/Python.h:57, from numpy/core/src/npy_math.c.src:45: /usr/include/python2.5/pyport.h:257:24: sys/select.h: No such file or directory numpy/core/src/npy_math.c.src:187: warning: static declaration of 'log2' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sinl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'cosl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'tanl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sinhl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'coshl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'tanhl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'fabsl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'floorl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'ceill' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'rintl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'truncl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sqrtl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log10l' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'logl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'expl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'expm1l' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'asinl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'acosl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'atanl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'asinhl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'acoshl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'atanhl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log1pl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'exp2l' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log2l' follow s non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'atan2l' follo ws non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'hypotl' follo ws non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'powl' follows non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'fmodl' follow s non-static declaration numpy/core/src/npy_math.c.src:278: warning: static declaration of 'modfl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log2f' follow s non-static declaration In file included from /usr/include/python2.5/Python.h:57, from numpy/core/src/npy_math.c.src:45: /usr/include/python2.5/pyport.h:257:24: sys/select.h: No such file or directory numpy/core/src/npy_math.c.src:187: warning: static declaration of 'log2' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sinl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'cosl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'tanl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sinhl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'coshl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'tanhl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'fabsl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'floorl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'ceill' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'rintl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'truncl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'sqrtl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log10l' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'logl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'expl' follows non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'expm1l' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'asinl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'acosl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'atanl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'asinhl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'acoshl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'atanhl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log1pl' follo ws non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'exp2l' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log2l' follow s non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'atan2l' follo ws non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'hypotl' follo ws non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'powl' follows non-static declaration numpy/core/src/npy_math.c.src:267: warning: static declaration of 'fmodl' follow s non-static declaration numpy/core/src/npy_math.c.src:278: warning: static declaration of 'modfl' follow s non-static declaration numpy/core/src/npy_math.c.src:251: warning: static declaration of 'log2f' follow s non-static declaration error: Command "gcc -mno-cygwin -mdll -O -Wall -Inumpy/core/include -Ibuild/src. cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/in clude -I/usr/include/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/sr c/npy_math.c -o build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2 .5/numpy/core/src/npy_math.o" failed with exit status 1 From cournape at gmail.com Wed Nov 18 04:10:10 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 18 Nov 2009 18:10:10 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> Message-ID: <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> On Wed, Nov 18, 2009 at 12:38 AM, wrote: > > Now numpy builds without problems. > When I run the tests I get 16 failures mostly nan related. I have no idea > whether they are real or if there is still something screwed up in my > setup. See below. I can reproduce those. I did not see those because they are python 2.5 specific (which is puzzling). I will try to look at it - in the meantime, could you open a track ticket with 1.4.0 milestone so that I don't forget it ? cheers, David From david at ar.media.kyoto-u.ac.jp Wed Nov 18 04:01:06 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 18 Nov 2009 18:01:06 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <13497.7968.qm@web51012.mail.re2.yahoo.com> References: <13497.7968.qm@web51012.mail.re2.yahoo.com> Message-ID: <4B03B7D2.3080305@ar.media.kyoto-u.ac.jp> Olivia Cheronet wrote: > Hello. > > I am currently trying to install the latest version of NumPy for my cygwin Python, and I am having problems... > > I have downloaded the source, and unzipped and untarred it in my home directory. > Subsequently, I included the following in the site.cfg file: > >> [DEFAULT] >> library_dirs = /cygdrive/c/cygwin/lib >> include_dirs = /cygdrive/c/cygwin/usr/local/include >> [blas_opt] >> libraries = f77blas, cblas, atlas >> [lapack_opt] >> libraries = lapack, f77blas, cblas, atlas >> > > In the Cygwin bash shell, after going to my home directory, I have executed: > python setup.py config --compiler=mingw32 build --compiler=mingw32 install > as instructed in the "Installing SciPy/Windows" page. > If you use cygwin, you should not follow the windows instructions. For most purposes, cygwin works as unix. In particular, you don't think you should not use mingw32 build on cygwin. The -mno-cygwin flag appended to gcc is most likely coming from there, and this looks very wrong for a cygwin build. David From dsdale24 at gmail.com Wed Nov 18 07:34:27 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 18 Nov 2009 07:34:27 -0500 Subject: [Numpy-discussion] REMINDER: trunk is about to be frozen for 1.4.0 In-Reply-To: <5b8d13220911171755y2b1f306en38bbd40f789b4d80@mail.gmail.com> References: <4B011B7D.5000700@ar.media.kyoto-u.ac.jp> <5b8d13220911171755y2b1f306en38bbd40f789b4d80@mail.gmail.com> Message-ID: On Tue, Nov 17, 2009 at 8:55 PM, David Cournapeau wrote: > already done ?in r7743 :) Did you report it as a bug on trac, so that > I close it as well, Oh, thanks! No, I forgot to report it on trac, I'll try to remember that in the future. From renesd at gmail.com Wed Nov 18 10:09:45 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 18 Nov 2009 16:09:45 +0100 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> Message-ID: <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> On Fri, Nov 6, 2009 at 7:46 PM, David Cournapeau wrote: > On Sat, Nov 7, 2009 at 3:41 AM, David Cournapeau wrote: > >> I don't think LGPL has much meaning for >> python code, especially pure python code (and m4 for that matter) > > This is funny - besides pyqt, the only LGPL reference with python I > found was twisted, and they claim that: > > "... It would be a halfway accurate statement that > I selected the LGPL exactly because it doesn't make any sense." > > http://twistedmatrix.com/pipermail/twisted-python/2004-May/007946.html > > David pygame is also LGPL... as are a number of other libraries. (pyqt is GPL btw). LGPL basically means you can link to the library source, but if you make changes to the library you should give them back. Users should also be able to change the library if they want... either through source or relinking. That's the spirit of the LGPL in two sentences. From bergstrj at iro.umontreal.ca Wed Nov 18 12:17:57 2009 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Wed, 18 Nov 2009 12:17:57 -0500 Subject: [Numpy-discussion] Unexpected attribute error In-Reply-To: <3d375d730911171853v1f914443t39b68153912921b0@mail.gmail.com> References: <7f1eaee30911171848m5cbf5a32wafcd135e5e36f5da@mail.gmail.com> <3d375d730911171853v1f914443t39b68153912921b0@mail.gmail.com> Message-ID: <7f1eaee30911180917v44495c24nb937a7bd29366949@mail.gmail.com> On Tue, Nov 17, 2009 at 9:53 PM, Robert Kern wrote: > On Tue, Nov 17, 2009 at 20:48, James Bergstra wrote: >> Is it by design that ?"numpy.sqrt(None)" raises an "AttributeError: sqrt"? > > Yes. numpy.sqrt() is a ufunc. Ufuncs take their arguments and try to > convert them to numpy arrays; the manual equivalent is > numpy.asarray(None). In this case, you get a rank-0 array with > dtype=object. The way unary ufuncs work on object arrays is to call > the method of the same name on the object. None.sqrt doesn't exist, so > the AttributeError gets raised. > >> This was confusing because there was an attribute lookup of 'sqrt' in >> numpy right there in the expression I typed, but that was not the >> attribute that python was complaining about. ?I presume that numpy.sqrt >> didn't know what to do with None, so it looked up a .sqrt in it or >> something... but I presume this only in hindsight now that I figured >> out the problem--I didn't mean to take the sqrt of None in the first place. >> >> How about adding some information to the AttributeError, such >> as the object on which the lookup failed (here None, as opposed to the >> numpy module). > > Patches welcome. How about putting something like this into the scalarmathmodule.c.src file, around line 1177? static PyObject * _GetAttrString(PyObject * obj, const char * str) { PyObject * rval = PyObject_GetAttrString(obj, str); if (!rval) { PyExc_SetValue(PyExc_AttributeError, Py_BuildValue("(OO)", obj, PyString_FromString(str))) } return rval; } I'm not crystal on whether obj's refcount needs bumping, but this function would be used instead of PyObject_GetAttrString in the get_functions() body in that file. I know nothing about now numpy works internally... is this remotely correct? James -- http://www-etud.iro.umontreal.ca/~bergstrj From charlesr.harris at gmail.com Wed Nov 18 12:48:35 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Nov 2009 10:48:35 -0700 Subject: [Numpy-discussion] Unexpected attribute error In-Reply-To: <7f1eaee30911180917v44495c24nb937a7bd29366949@mail.gmail.com> References: <7f1eaee30911171848m5cbf5a32wafcd135e5e36f5da@mail.gmail.com> <3d375d730911171853v1f914443t39b68153912921b0@mail.gmail.com> <7f1eaee30911180917v44495c24nb937a7bd29366949@mail.gmail.com> Message-ID: On Wed, Nov 18, 2009 at 10:17 AM, James Bergstra wrote: > On Tue, Nov 17, 2009 at 9:53 PM, Robert Kern > wrote: > > On Tue, Nov 17, 2009 at 20:48, James Bergstra > wrote: > >> Is it by design that "numpy.sqrt(None)" raises an "AttributeError: > sqrt"? > > > > Yes. numpy.sqrt() is a ufunc. Ufuncs take their arguments and try to > > convert them to numpy arrays; the manual equivalent is > > numpy.asarray(None). In this case, you get a rank-0 array with > > dtype=object. The way unary ufuncs work on object arrays is to call > > the method of the same name on the object. None.sqrt doesn't exist, so > > the AttributeError gets raised. > > > >> This was confusing because there was an attribute lookup of 'sqrt' in > >> numpy right there in the expression I typed, but that was not the > >> attribute that python was complaining about. I presume that numpy.sqrt > >> didn't know what to do with None, so it looked up a .sqrt in it or > >> something... but I presume this only in hindsight now that I figured > >> out the problem--I didn't mean to take the sqrt of None in the first > place. > >> > >> How about adding some information to the AttributeError, such > >> as the object on which the lookup failed (here None, as opposed to the > >> numpy module). > > > > Patches welcome. > > How about putting something like this into the scalarmathmodule.c.src > file, around line 1177? > > static PyObject * _GetAttrString(PyObject * obj, const char * str) > { > PyObject * rval = PyObject_GetAttrString(obj, str); > if (!rval) > { > PyExc_SetValue(PyExc_AttributeError, Py_BuildValue("(OO)", > obj, PyString_FromString(str))) > } > return rval; > } > > I'm not crystal on whether obj's refcount needs bumping, but this > function would be used instead of PyObject_GetAttrString in the > get_functions() body in that file. > > I know nothing about now numpy works internally... is this remotely > correct? > > Looks about right to get a more informative error message. What happens when you compile with it and run the tests? What does the output of you example look like? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cheronetolivia at yahoo.com Wed Nov 18 13:55:52 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Wed, 18 Nov 2009 10:55:52 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin Message-ID: <443565.83539.qm@web51008.mail.re2.yahoo.com> Hi, I have started again, simply executing: >python setup.py install However, I now get the following: >File "numpy/core/setup.py", line 253, in check_mathlib > raise EnvironmentError("math library missing; rerun " >EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB >env variable I have a math library from cygwin (libm.a). How should I set this? Thank you, Olivia ----- Original Message ---- > From: David Cournapeau > To: Discussion of Numerical Python > Sent: Wed, November 18, 2009 9:01:06 AM > Subject: Re: [Numpy-discussion] Installing numpy under cygwin > > Olivia Cheronet wrote: > > Hello. > > > > I am currently trying to install the latest version of NumPy for my cygwin > Python, and I am having problems... > > > > I have downloaded the source, and unzipped and untarred it in my home > directory. > > Subsequently, I included the following in the site.cfg file: > > > >> [DEFAULT] > >> library_dirs = /cygdrive/c/cygwin/lib > >> include_dirs = /cygdrive/c/cygwin/usr/local/include > >> [blas_opt] > >> libraries = f77blas, cblas, atlas > >> [lapack_opt] > >> libraries = lapack, f77blas, cblas, atlas > >> > > > > In the Cygwin bash shell, after going to my home directory, I have executed: > > python setup.py config --compiler=mingw32 build --compiler=mingw32 install > > as instructed in the "Installing SciPy/Windows" page. > > > > If you use cygwin, you should not follow the windows instructions. For > most purposes, cygwin works as unix. In particular, you don't think you > should not use mingw32 build on cygwin. The -mno-cygwin flag appended to > gcc is most likely coming from there, and this looks very wrong for a > cygwin build. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From damienlmoore at gmail.com Wed Nov 18 14:17:02 2009 From: damienlmoore at gmail.com (Damien Moore) Date: Wed, 18 Nov 2009 14:17:02 -0500 Subject: [Numpy-discussion] combinatorial operations on a pair of arrays Message-ID: <2e04d3470911181117k107a6e92w7fe7cafd8b905457@mail.gmail.com> The title of this e-mail is probably misleading so let me just show some code: from numpy import * A=array([[1,2],[2,3],[3,4]]) B=array([[2,2],[3,3]]) C=zeros(A.shape) for i in xrange(len(A)): C[i]=sum(A[i]**B) print C What I want to do is eliminate the for loop and rely on numpy internals, but I'm not sure how to do this most efficiently. The fundamental aspect of the operation is that the array C is constructed by applying each subarray B to each subarray of A (i.e. all permutations) and then summing over a subset. From damienlmoore at gmail.com Wed Nov 18 15:43:09 2009 From: damienlmoore at gmail.com (Damien Moore) Date: Wed, 18 Nov 2009 15:43:09 -0500 Subject: [Numpy-discussion] combinatorial operations on a pair of arrays In-Reply-To: <2e04d3470911181117k107a6e92w7fe7cafd8b905457@mail.gmail.com> References: <2e04d3470911181117k107a6e92w7fe7cafd8b905457@mail.gmail.com> Message-ID: <2e04d3470911181243p1648c85k8714f14ada49c457@mail.gmail.com> ugh... I goofed. The code snippet should have read from numpy import * A=array([[1,2],[2,3],[3,4]]) B=array([[2,2],[3,3]]) C=zeros(A.shape) for i in xrange(len(A)): C[i]=(A[i]**B).sum(0) print C On Wed, Nov 18, 2009 at 2:17 PM, Damien Moore wrote: > The title of this e-mail is probably misleading so let me just show some code: > > from numpy import * > A=array([[1,2],[2,3],[3,4]]) > B=array([[2,2],[3,3]]) > C=zeros(A.shape) > for i in xrange(len(A)): > ?C[i]=sum(A[i]**B) > print C > > What I want to do is eliminate the for loop and rely on numpy > internals, but I'm not sure how to do this most efficiently. > > The fundamental aspect of the operation is that the array C is > constructed by applying each subarray B to each subarray of A (i.e. > all permutations) and then summing over a subset. > From emmanuelle.gouillart at normalesup.org Wed Nov 18 15:49:07 2009 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Wed, 18 Nov 2009 21:49:07 +0100 Subject: [Numpy-discussion] combinatorial operations on a pair of arrays In-Reply-To: <2e04d3470911181243p1648c85k8714f14ada49c457@mail.gmail.com> References: <2e04d3470911181117k107a6e92w7fe7cafd8b905457@mail.gmail.com> <2e04d3470911181243p1648c85k8714f14ada49c457@mail.gmail.com> Message-ID: <20091118204907.GA16502@phare.normalesup.org> Hello Damien, broadcasting can solve your problem (see http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html): (A[np.newaxis,:]**B[:,np.newaxis]).sum(axis=0) gives the result you want. (assuming "import numpy as np", which is considered as a better practice as "from numpy import *") Cheers, Emmanuelle On Wed, Nov 18, 2009 at 03:43:09PM -0500, Damien Moore wrote: > ugh... I goofed. The code snippet should have read > from numpy import * > A=array([[1,2],[2,3],[3,4]]) > B=array([[2,2],[3,3]]) > C=zeros(A.shape) > for i in xrange(len(A)): > C[i]=(A[i]**B).sum(0) > print C > On Wed, Nov 18, 2009 at 2:17 PM, Damien Moore wrote: > > The title of this e-mail is probably misleading so let me just show some code: > > from numpy import * > > A=array([[1,2],[2,3],[3,4]]) > > B=array([[2,2],[3,3]]) > > C=zeros(A.shape) > > for i in xrange(len(A)): > > ?C[i]=sum(A[i]**B) > > print C > > What I want to do is eliminate the for loop and rely on numpy > > internals, but I'm not sure how to do this most efficiently. > > The fundamental aspect of the operation is that the array C is > > constructed by applying each subarray B to each subarray of A (i.e. > > all permutations) and then summing over a subset. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From damienlmoore at gmail.com Wed Nov 18 15:55:48 2009 From: damienlmoore at gmail.com (Damien Moore) Date: Wed, 18 Nov 2009 15:55:48 -0500 Subject: [Numpy-discussion] combinatorial operations on a pair of arrays In-Reply-To: <20091118204907.GA16502@phare.normalesup.org> References: <2e04d3470911181117k107a6e92w7fe7cafd8b905457@mail.gmail.com> <2e04d3470911181243p1648c85k8714f14ada49c457@mail.gmail.com> <20091118204907.GA16502@phare.normalesup.org> Message-ID: <2e04d3470911181255lf5abebdrdbef01d9cbebeb2f@mail.gmail.com> Emanuelle, perfect! thanks. > (assuming "import numpy as np", which is considered as a better practice > as "from numpy import *") actually I normally just "import numpy" but I guess np is nicer to type and read... cheers, Damien From amcmorl at gmail.com Wed Nov 18 16:15:49 2009 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 18 Nov 2009 16:15:49 -0500 Subject: [Numpy-discussion] lstsq illegal instruction Message-ID: Hi all, Whenever I run numpy.linalg.lstsq with a xs parameter with both dimensions larger than 128, I get an "Illegal instruction" and python dies completely. It happens with both the Ubuntu jaunty standard numpy 1.2.1, and a recent svn 1.4.0.dev7727, but it doesn't seem to happen on any other machines. Anyone have any idea what might be causing this? It's most annoying. The crash can be recreated with the following four lines: import numpy as np xs = np.random.random(size=(129, 129)) y = np.random.random(size=(129,)) res = np.linalg.lstsq(xs,y) Thanks, Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From robert.kern at gmail.com Wed Nov 18 16:20:17 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Nov 2009 15:20:17 -0600 Subject: [Numpy-discussion] lstsq illegal instruction In-Reply-To: References: Message-ID: <3d375d730911181320k20a8ee5oac4bc2f0d5079cce@mail.gmail.com> On Wed, Nov 18, 2009 at 15:15, Angus McMorland wrote: > Hi all, > > Whenever I run numpy.linalg.lstsq with a xs parameter with both > dimensions larger than 128, I get an "Illegal instruction" and python > dies completely. It happens with both the Ubuntu jaunty standard numpy > 1.2.1, and a recent svn 1.4.0.dev7727, but it doesn't seem to happen > on any other machines. Anyone have any idea what might be causing > this? It's most annoying. This is almost always caused by using an ATLAS that is built for a different CPU than yours. Usually, the CPU that built the ATLAS binaries has SSE2 instructions while your CPU doesn't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mat.yeates at gmail.com Wed Nov 18 16:48:03 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 13:48:03 -0800 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError Message-ID: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> Hi I have a line of matplotlib code -self.ax.plot(plot_data,mif) that causes the line -self.data=numpy.zeros(shape=dims) to throw a MemoryError exception. (if I comment out the first line I get no error.) This is on a windows xp machine with latest numpy and the latest matplotlib. I have a feeling this may be a nightmare to figure out what matplotlib and/or numpy are doing wrong. Any ideas where I can start? Mathew -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Nov 18 16:51:32 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Nov 2009 15:51:32 -0600 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError In-Reply-To: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> References: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> Message-ID: <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> On Wed, Nov 18, 2009 at 15:48, Mathew Yeates wrote: > Hi > > I have a line of matplotlib code > > -self.ax.plot(plot_data,mif) > > > > that causes the line > > -self.data=numpy.zeros(shape=dims) > > > > to throw a MemoryError exception. > > (if I comment out the first line I get no error.) > > This is on a windows xp machine with latest numpy and the latest matplotlib. > > > > I have a feeling this may be a nightmare to figure out what matplotlib > and/or numpy are doing wrong. Any ideas where I can start? Print out dims just before the second line to make sure that it is reasonable. A MemoryError is raised when numpy cannot allocate enough memory on your system. If dims is too large for some reason, you could run into that limit. It might be because what you are trying to plot is simply too large or there might possibly (but unlikely) be a bug that is miscalculating dims. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From amcmorl at gmail.com Wed Nov 18 16:54:37 2009 From: amcmorl at gmail.com (Angus McMorland) Date: Wed, 18 Nov 2009 16:54:37 -0500 Subject: [Numpy-discussion] lstsq illegal instruction In-Reply-To: <3d375d730911181320k20a8ee5oac4bc2f0d5079cce@mail.gmail.com> References: <3d375d730911181320k20a8ee5oac4bc2f0d5079cce@mail.gmail.com> Message-ID: 2009/11/18 Robert Kern : > On Wed, Nov 18, 2009 at 15:15, Angus McMorland wrote: >> Hi all, >> >> Whenever I run numpy.linalg.lstsq with a xs parameter with both >> dimensions larger than 128, I get an "Illegal instruction" and python >> dies completely. It happens with both the Ubuntu jaunty standard numpy >> 1.2.1, and a recent svn 1.4.0.dev7727, but it doesn't seem to happen >> on any other machines. Anyone have any idea what might be causing >> this? It's most annoying. > > This is almost always caused by using an ATLAS that is built for a > different CPU than yours. Usually, the CPU that built the ATLAS > binaries has SSE2 instructions while your CPU doesn't. Many thanks for the pointer, Robert. In case anyone else comes across the same problem, installing libatlas3gf-sse2 made the problem go away; previously I only had the python-scipy dependency libatlas3gf-base installed. Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh From mat.yeates at gmail.com Wed Nov 18 17:13:45 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 14:13:45 -0800 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError In-Reply-To: <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> References: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> Message-ID: <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> The value of dims is constant and not particularly large. I also checked to make sure I wasn't running out of memory. Are there other reasons for this error? Mathew On Wed, Nov 18, 2009 at 1:51 PM, Robert Kern wrote: > On Wed, Nov 18, 2009 at 15:48, Mathew Yeates wrote: > > Hi > > > > I have a line of matplotlib code > > > > -self.ax.plot(plot_data,mif) > > > > > > > > that causes the line > > > > -self.data=numpy.zeros(shape=dims) > > > > > > > > to throw a MemoryError exception. > > > > (if I comment out the first line I get no error.) > > > > This is on a windows xp machine with latest numpy and the latest > matplotlib. > > > > > > > > I have a feeling this may be a nightmare to figure out what matplotlib > > and/or numpy are doing wrong. Any ideas where I can start? > > Print out dims just before the second line to make sure that it is > reasonable. A MemoryError is raised when numpy cannot allocate enough > memory on your system. If dims is too large for some reason, you could > run into that limit. It might be because what you are trying to plot > is simply too large or there might possibly (but unlikely) be a bug > that is miscalculating dims. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Wed Nov 18 17:22:59 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 14:22:59 -0800 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError In-Reply-To: <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> References: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> Message-ID: <7d0c05ac0911181422s9f1d6a6u2934ca0315f1186b@mail.gmail.com> also, the exception is only thrown when I plot something first. I wonder if matplotlib is messing something up. On Wed, Nov 18, 2009 at 2:13 PM, Mathew Yeates wrote: > The value of dims is constant and not particularly large. I also checked to > make sure I wasn't running out of memory. Are there other reasons for this > error? > > Mathew > > > On Wed, Nov 18, 2009 at 1:51 PM, Robert Kern wrote: > >> On Wed, Nov 18, 2009 at 15:48, Mathew Yeates >> wrote: >> > Hi >> > >> > I have a line of matplotlib code >> > >> > -self.ax.plot(plot_data,mif) >> > >> > >> > >> > that causes the line >> > >> > -self.data=numpy.zeros(shape=dims) >> > >> > >> > >> > to throw a MemoryError exception. >> > >> > (if I comment out the first line I get no error.) >> > >> > This is on a windows xp machine with latest numpy and the latest >> matplotlib. >> > >> > >> > >> > I have a feeling this may be a nightmare to figure out what matplotlib >> > and/or numpy are doing wrong. Any ideas where I can start? >> >> Print out dims just before the second line to make sure that it is >> reasonable. A MemoryError is raised when numpy cannot allocate enough >> memory on your system. If dims is too large for some reason, you could >> run into that limit. It might be because what you are trying to plot >> is simply too large or there might possibly (but unlikely) be a bug >> that is miscalculating dims. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> -- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Nov 18 17:43:47 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 Nov 2009 15:43:47 -0700 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError In-Reply-To: <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> References: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> Message-ID: On Wed, Nov 18, 2009 at 3:13 PM, Mathew Yeates wrote: > The value of dims is constant and not particularly large. Yes, but what are they? > I also checked to make sure I wasn't running out of memory. Are there other > reasons for this error? > > If there is a memory error, no memory is used. What versions of numpy/matplotlib are you using? Is xp 32 bit or 64 bits? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Wed Nov 18 17:58:32 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 14:58:32 -0800 Subject: [Numpy-discussion] matplotlib and numpy cause MemoryError In-Reply-To: References: <7d0c05ac0911181348l558d91a1vc164aac638d56723@mail.gmail.com> <3d375d730911181351h4a89d412n467c232a42e16e17@mail.gmail.com> <7d0c05ac0911181413j16648c67q4a70d3d698f5772f@mail.gmail.com> Message-ID: <7d0c05ac0911181458p5f1b3850ye7ca51a74e91d65e@mail.gmail.com> I turns out I *was* running out of memory. My dimensions would require 3.5 gig and my plot must have used up some memory. On Wed, Nov 18, 2009 at 2:43 PM, Charles R Harris wrote: > > > On Wed, Nov 18, 2009 at 3:13 PM, Mathew Yeates wrote: > >> The value of dims is constant and not particularly large. > > > Yes, but what are they? > > >> I also checked to make sure I wasn't running out of memory. Are there >> other reasons for this error? >> >> > If there is a memory error, no memory is used. > > What versions of numpy/matplotlib are you using? Is xp 32 bit or 64 bits? > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Wed Nov 18 18:43:33 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 15:43:33 -0800 Subject: [Numpy-discussion] memmap limits Message-ID: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> What limits are there on file size when using memmap? -Mathew -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Nov 18 18:48:13 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Nov 2009 17:48:13 -0600 Subject: [Numpy-discussion] memmap limits In-Reply-To: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> References: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> Message-ID: <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> On Wed, Nov 18, 2009 at 17:43, Mathew Yeates wrote: > What limits are there on file size when using memmap? With a modern filesystem, usually you are only limited to the amount of contiguous free space in your process's current address space. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mat.yeates at gmail.com Wed Nov 18 18:50:33 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Wed, 18 Nov 2009 15:50:33 -0800 Subject: [Numpy-discussion] memmap limits In-Reply-To: <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> References: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> Message-ID: <7d0c05ac0911181550v22f538a4s8f588e2e1de3899d@mail.gmail.com> for a 64 bit machine does this mean I am limited to 4 GB? -Mathew On Wed, Nov 18, 2009 at 3:48 PM, Robert Kern wrote: > On Wed, Nov 18, 2009 at 17:43, Mathew Yeates wrote: > > What limits are there on file size when using memmap? > > With a modern filesystem, usually you are only limited to the amount > of contiguous free space in your process's current address space. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Nov 18 18:53:13 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Nov 2009 17:53:13 -0600 Subject: [Numpy-discussion] memmap limits In-Reply-To: <7d0c05ac0911181550v22f538a4s8f588e2e1de3899d@mail.gmail.com> References: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> <7d0c05ac0911181550v22f538a4s8f588e2e1de3899d@mail.gmail.com> Message-ID: <3d375d730911181553i5687aef2v9889cb0fd389ed16@mail.gmail.com> On Wed, Nov 18, 2009 at 17:50, Mathew Yeates wrote: > for a 64 bit machine does this mean I am limited to 4 GB? It depends not just on what your CPU is capable of, but also your OS and how your Python was built. If all three of those are 64-bit capable, you should be able to address a lot more than 4 GB. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Wed Nov 18 23:52:26 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 19 Nov 2009 13:52:26 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <4AF3CC02.6070509@ar.media.kyoto-u.ac.jp> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> Message-ID: <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> Ren? Dudfield wrote: > pygame is also LGPL... as are a number of other libraries. (pyqt is GPL btw). > > LGPL basically means you can link to the library source, but if you > make changes to the library you should give them back. Users should > also be able to change the library if they want... either through > source or relinking. > It is far from being that simple, because the notion on whether you are allowed to use the library with proprietary software is based on derivative work, not "link to the library source". For compiled code, it is generally admitted that if you use the library as a shared library (dynamic linking), the program you link to is not derivative, and hence can be any license you want. But if you link statically, it is admitted that your work is derivative, with the associated constraints of the license (ability to reverse engineering, etc...). The fundamental issue is what constitutes derivative work with python code. David From robert.kern at gmail.com Thu Nov 19 00:47:25 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Nov 2009 23:47:25 -0600 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> On Wed, Nov 18, 2009 at 22:52, David Cournapeau wrote: > Ren? Dudfield wrote: >> pygame is also LGPL... as are a number of other libraries. ?(pyqt is GPL btw). >> >> LGPL basically means you can link to the library source, but if you >> make changes to the library you should give them back. ?Users should >> also be able to change the library if they want... either through >> source or relinking. >> > > It is far from being that simple, because the notion on whether you are > allowed to use the library with proprietary software is based on > derivative work, not "link to the library source". For compiled code, it > is generally admitted that if you use the library as a shared library > (dynamic linking), the program you link to is not derivative, and hence > can be any license you want. Not at all. The FSF and a large proportion of GPL users claim otherwise. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Nov 19 00:39:31 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 19 Nov 2009 14:39:31 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911060910y40a47333v3c774a64d47551ca@mail.gmail.com> <5b8d13220911060916o11838bbh61ae46451193fdc0@mail.gmail.com> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> Message-ID: <4B04DA13.6090000@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Wed, Nov 18, 2009 at 22:52, David Cournapeau > wrote: > >> Ren? Dudfield wrote: >> >>> pygame is also LGPL... as are a number of other libraries. (pyqt is GPL btw). >>> >>> LGPL basically means you can link to the library source, but if you >>> make changes to the library you should give them back. Users should >>> also be able to change the library if they want... either through >>> source or relinking. >>> >>> >> It is far from being that simple, because the notion on whether you are >> allowed to use the library with proprietary software is based on >> derivative work, not "link to the library source". For compiled code, it >> is generally admitted that if you use the library as a shared library >> (dynamic linking), the program you link to is not derivative, and hence >> can be any license you want. >> > > Not at all. The FSF and a large proportion of GPL users claim otherwise. > I thought we were discussing about the LGPL here. What I said previously only applies to the LGPL, and is certainly wrong for the GPL. But I don't think the GPL causes any issue for python code (I have encountered very little python LGPL code, but there is plenty under the GPL). David From robert.kern at gmail.com Thu Nov 19 02:29:35 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Nov 2009 01:29:35 -0600 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4B04DA13.6090000@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> <4B04DA13.6090000@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911182329n58a1613cu6c9dec28970d4c72@mail.gmail.com> On Wed, Nov 18, 2009 at 23:39, David Cournapeau wrote: > Robert Kern wrote: >> On Wed, Nov 18, 2009 at 22:52, David Cournapeau >> wrote: >> >>> Ren? Dudfield wrote: >>> >>>> pygame is also LGPL... as are a number of other libraries. ?(pyqt is GPL btw). >>>> >>>> LGPL basically means you can link to the library source, but if you >>>> make changes to the library you should give them back. ?Users should >>>> also be able to change the library if they want... either through >>>> source or relinking. >>>> >>>> >>> It is far from being that simple, because the notion on whether you are >>> allowed to use the library with proprietary software is based on >>> derivative work, not "link to the library source". For compiled code, it >>> is generally admitted that if you use the library as a shared library >>> (dynamic linking), the program you link to is not derivative, and hence >>> can be any license you want. >>> >> >> Not at all. The FSF and a large proportion of GPL users claim otherwise. > > I thought we were discussing about the LGPL here. What I said previously > only applies to the LGPL, and is certainly wrong for the GPL. But I > don't think the GPL causes any issue for python code (I have encountered > very little python LGPL code, but there is plenty under the GPL). Ah, I see the confusion. The term "derivative work" is a term from copyright law and is independent of any license under consideration. The FSF considers an application that links to a library in the same process space, either dynamically or statically, a derivative work of the library whether the library is GPLed or LGPLed. If you're talking about the distinction that the LGPL makes between linked code that can be proprietary and must not, you're still wrong, though. The LGPL does not make a distinction based on the linking technology. One can have a proprietary application statically linked with an LGPL library. The only detail there is that, in order to satisfy the "user must be able to relink the application with a modified Library" requirement, the distributor must provide object files for his proprietary code and the appropriate tools such the user can relink against a modified library. Rather, the distinction between code that can be proprietary and code that must be LGPLed is whether the code uses the interface exposed by the library or if it makes deeper modifications. The LGPLv3 makes things pretty explicit: """ An ?Application? is any work that makes use of an interface provided by the Library, but which is not otherwise based on the Library. Defining a subclass of a class defined by the Library is deemed a mode of using an interface provided by the Library. """ For Python libraries, I think the only gray area would be some deep, deep monkeypatching, but mostly I suspect that a reasonable line that most people would agree to is that if you import an unmodified LGPL module, you are just using its interface no matter what you do with the objects you get from it. If you edit the files of the LGPLed Python library, those modifications must be LGPLed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Nov 19 02:43:05 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 19 Nov 2009 16:43:05 +0900 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <3d375d730911182329n58a1613cu6c9dec28970d4c72@mail.gmail.com> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911061030n358dd421t72ba32fcea1a3040@mail.gmail.com> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> <4B04DA13.6090000@ar.media.kyoto-u.ac.jp> <3d375d730911182329n58a1613cu6c9dec28970d4c72@mail.gmail.com> Message-ID: <4B04F709.1010608@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > One can have a proprietary application statically linked > with an LGPL library. The only detail there is that, in order to > satisfy the "user must be able to relink the application with a > modified Library" requirement, the distributor must provide object > files for his proprietary code and the appropriate tools such the user > can relink against a modified library. > So the "static exception" which is often added to the LGPL is just to avoid having to distribute the object files ? Also, what does distributing the object files/sources for the proprietary part mean for python ? Is the bytecode enough ? cheers, David From robert.kern at gmail.com Thu Nov 19 12:46:44 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Nov 2009 11:46:44 -0600 Subject: [Numpy-discussion] Solaris Sparc build broken In-Reply-To: <4B04F709.1010608@ar.media.kyoto-u.ac.jp> References: <4AF1B6CA.1060608@stsci.edu> <5b8d13220911061041h1c2eb52en6cd721a427b1c114@mail.gmail.com> <5b8d13220911061046h5a091f21t3d61b32ae3d8a519@mail.gmail.com> <64ddb72c0911180709k3b865759p998c6bda1a486548@mail.gmail.com> <4B04CF0A.7070006@ar.media.kyoto-u.ac.jp> <3d375d730911182147t5fab6f30n6489e290b7d82097@mail.gmail.com> <4B04DA13.6090000@ar.media.kyoto-u.ac.jp> <3d375d730911182329n58a1613cu6c9dec28970d4c72@mail.gmail.com> <4B04F709.1010608@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730911190946k37ec3eeem8c013bcee242c68@mail.gmail.com> On Thu, Nov 19, 2009 at 01:43, David Cournapeau wrote: > Robert Kern wrote: > >> ?One can have a proprietary application statically linked >> with an LGPL library. The only detail there is that, in order to >> satisfy the "user must be able to relink the application with a >> modified Library" requirement, the distributor must provide object >> files for his proprietary code and the appropriate tools such the user >> can relink against a modified library. > > So the "static exception" which is often added to the LGPL is just to > avoid having to distribute the object files ? Yes. It *is* a hassle, and I've never actually seen it done in the wild. Everyone either dynamically links (making replacing the LGPLed Library trivial) or uses a library with one of these exceptions. > Also, what does > distributing the object files/sources for the proprietary part mean for > python ? Is the bytecode enough ? Ignore "object files". That's an example of satisfying the LGPL requirement using a popular technology, not the requirement itself. The requirement is that a user of the Combined Work must be able to replace the LGPLed Library with a modified version and combine it with the proprietary Application. For many Python deployments, satisfying this requirement is trivial, just like the dynamic linking case in other languages. The user can just drop in the new modules in the right places. The developer doesn't need to do anything special. For frozen/py2exe applications, where the Python modules are zipped up into the executable itself, you need to provide a way to rebuild the frozen executable with the modified Library. There are a couple of ways to do that, depending on the tool that is used. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mat.yeates at gmail.com Thu Nov 19 12:52:20 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 Nov 2009 09:52:20 -0800 Subject: [Numpy-discussion] matplotlib is breaking numpy Message-ID: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> There is definitely something wrong with matplotlib/numpy. Consider the following >from numpy import * >mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) > del mydata I can now remove the file map.dat with (from the command line) $rm map.dat However If I plot mydata before the line > del mydata I can't get rid of the file until I exit python!! Does matplotlib keep a reference to the data? How can I remove this reference? Mathew -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Nov 19 12:57:34 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Nov 2009 11:57:34 -0600 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> Message-ID: <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> On Thu, Nov 19, 2009 at 11:52, Mathew Yeates wrote: > There is definitely something wrong with matplotlib/numpy. Consider the > following >>from numpy import * >>mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) >> del mydata > > I can now remove the file map.dat with (from the command line) $rm map.dat > > However > If I plot? mydata before the line >> del mydata > > > I can't get rid of the file until I exit python!! > Does matplotlib keep a reference to the data? Almost certainly. > How can I remove this > reference? Probably by deleting the plot objects that were created and close all matplotlib windows referencing the data. If you are using IPython, you should know that many of the returned objects are kept in Out, so you will need to clear that. There might be some more places internal to matplotlib, I don't know. With some care, you can use gc.get_referrers() to find the objects that are holding direct references to your memmap. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jdh2358 at gmail.com Thu Nov 19 13:30:38 2009 From: jdh2358 at gmail.com (John Hunter) Date: Thu, 19 Nov 2009 12:30:38 -0600 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> Message-ID: <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> On Nov 19, 2009, at 11:57 AM, Robert Kern wrote: > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates > wrote: >> There is definitely something wrong with matplotlib/numpy. Consider >> the >> following >>> from numpy import * >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) >>> del mydata >> >> I can now remove the file map.dat with (from the command line) $rm >> map.dat >> >> However >> If I plot mydata before the line >>> del mydata >> >> >> I can't get rid of the file until I exit python!! >> Does matplotlib keep a reference to the data? > > Almost certainly. > >> How can I remove this >> reference? > > Probably by deleting the plot objects that were created and close all > matplotlib windows referencing the data. If you are using IPython, you > should know that many of the returned objects are kept in Out, so you > will need to clear that. There might be some more places internal to > matplotlib, I don't know. > Closing the figure window containg the data *should* be enough. In pylab/pyplot, this also triggers a call to gc.collect. > With some care, you can use gc.get_referrers() to find the objects > that are holding direct references to your memmap. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From mat.yeates at gmail.com Thu Nov 19 13:35:36 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 Nov 2009 10:35:36 -0800 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> Message-ID: <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> Yeah, I tried that. Here's what I'm doing. I have an application which displays different dataset which a user selects from a drop down list. I want to overwrite the existing plot with a new one. I've tried deleting just about everything get matplotlib to let go of my data! Mathew On Thu, Nov 19, 2009 at 10:30 AM, John Hunter wrote: > > > > > On Nov 19, 2009, at 11:57 AM, Robert Kern wrote: > > > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates > > wrote: > >> There is definitely something wrong with matplotlib/numpy. Consider > >> the > >> following > >>> from numpy import * > >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) > >>> del mydata > >> > >> I can now remove the file map.dat with (from the command line) $rm > >> map.dat > >> > >> However > >> If I plot mydata before the line > >>> del mydata > >> > >> > >> I can't get rid of the file until I exit python!! > >> Does matplotlib keep a reference to the data? > > > > Almost certainly. > > > >> How can I remove this > >> reference? > > > > Probably by deleting the plot objects that were created and close all > > matplotlib windows referencing the data. If you are using IPython, you > > should know that many of the returned objects are kept in Out, so you > > will need to clear that. There might be some more places internal to > > matplotlib, I don't know. > > > > Closing the figure window containg the data *should* be enough. In > pylab/pyplot, this also triggers a call to gc.collect. > > > > > > With some care, you can use gc.get_referrers() to find the objects > > that are holding direct references to your memmap. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Thu Nov 19 13:42:07 2009 From: jdh2358 at gmail.com (John Hunter) Date: Thu, 19 Nov 2009 12:42:07 -0600 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> Message-ID: <40034D0F-2236-4E98-9371-4414DE769DD5@gmail.com> On Nov 19, 2009, at 12:35 PM, Mathew Yeates wrote: > Yeah, I tried that. > > Here's what I'm doing. I have an application which displays > different dataset which a user selects from a drop down list. I want > to overwrite the existing plot with a new one. I've tried deleting > just about everything get matplotlib to let go of my data! What is everything? Are you using pyplot or are you embedding mpl in a GUI? If the latter, are you deleting the FigureCanvas? You will also need to call gc.collect after deleting the mpl objects because we use a lot of circular references. Pyplot close does this automatically, but this does not apply to embedding. How are you running you app? From the shell or IPython? > > Mathew > > On Thu, Nov 19, 2009 at 10:30 AM, John Hunter > wrote: > > > > > On Nov 19, 2009, at 11:57 AM, Robert Kern > wrote: > > > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates > > wrote: > >> There is definitely something wrong with matplotlib/numpy. Consider > >> the > >> following > >>> from numpy import * > >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) > >>> del mydata > >> > >> I can now remove the file map.dat with (from the command line) $rm > >> map.dat > >> > >> However > >> If I plot mydata before the line > >>> del mydata > >> > >> > >> I can't get rid of the file until I exit python!! > >> Does matplotlib keep a reference to the data? > > > > Almost certainly. > > > >> How can I remove this > >> reference? > > > > Probably by deleting the plot objects that were created and close > all > > matplotlib windows referencing the data. If you are using IPython, > you > > should know that many of the returned objects are kept in Out, so > you > > will need to clear that. There might be some more places internal to > > matplotlib, I don't know. > > > > Closing the figure window containg the data *should* be enough. In > pylab/pyplot, this also triggers a call to gc.collect. > > > > > > With some care, you can use gc.get_referrers() to find the objects > > that are holding direct references to your memmap. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a > harmless > > enigma that is made terrible by our own mad attempt to interpret > it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Thu Nov 19 13:53:28 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 Nov 2009 10:53:28 -0800 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <40034D0F-2236-4E98-9371-4414DE769DD5@gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> <40034D0F-2236-4E98-9371-4414DE769DD5@gmail.com> Message-ID: <7d0c05ac0911191053p6121a93av5db5dad1e99b685a@mail.gmail.com> I am running my gtk app from python. I am deleting the canvas and running gc.collect(). I still seem to have a reference to my memmapped data. Any other hints? -Mathew On Thu, Nov 19, 2009 at 10:42 AM, John Hunter wrote: > > > > > On Nov 19, 2009, at 12:35 PM, Mathew Yeates wrote: > > Yeah, I tried that. > > Here's what I'm doing. I have an application which displays different > dataset which a user selects from a drop down list. I want to overwrite the > existing plot with a new one. I've tried deleting just about everything get > matplotlib to let go of my data! > > > > What is everything? Are you using pyplot or are you embedding mpl in a > GUI? If the latter, are you deleting the FigureCanvas? You will also need > to call gc.collect after deleting the mpl objects because we use a lot of > circular references. Pyplot close does this automatically, but this does not > apply to embedding. > > How are you running you app? From the shell or IPython? > > > > > Mathew > > On Thu, Nov 19, 2009 at 10:30 AM, John Hunter < > jdh2358 at gmail.com> wrote: > >> >> >> >> >> On Nov 19, 2009, at 11:57 AM, Robert Kern < >> robert.kern at gmail.com> wrote: >> >> > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates < >> mat.yeates at gmail.com> >> > wrote: >> >> There is definitely something wrong with matplotlib/numpy. Consider >> >> the >> >> following >> >>> from numpy import * >> >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) >> >>> del mydata >> >> >> >> I can now remove the file map.dat with (from the command line) $rm >> >> map.dat >> >> >> >> However >> >> If I plot mydata before the line >> >>> del mydata >> >> >> >> >> >> I can't get rid of the file until I exit python!! >> >> Does matplotlib keep a reference to the data? >> > >> > Almost certainly. >> > >> >> How can I remove this >> >> reference? >> > >> > Probably by deleting the plot objects that were created and close all >> > matplotlib windows referencing the data. If you are using IPython, you >> > should know that many of the returned objects are kept in Out, so you >> > will need to clear that. There might be some more places internal to >> > matplotlib, I don't know. >> > >> >> Closing the figure window containg the data *should* be enough. In >> pylab/pyplot, this also triggers a call to gc.collect. >> >> >> >> >> > With some care, you can use gc.get_referrers() to find the objects >> > that are holding direct references to your memmap. >> > >> > -- >> > Robert Kern >> > >> > "I have come to believe that the whole world is an enigma, a harmless >> > enigma that is made terrible by our own mad attempt to interpret it as >> > though it had an underlying truth." >> > -- Umberto Eco >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Thu Nov 19 13:56:13 2009 From: jdh2358 at gmail.com (John Hunter) Date: Thu, 19 Nov 2009 12:56:13 -0600 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: <7d0c05ac0911191053p6121a93av5db5dad1e99b685a@mail.gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> <40034D0F-2236-4E98-9371-4414DE769DD5@gmail.com> <7d0c05ac0911191053p6121a93av5db5dad1e99b685a@mail.gmail.com> Message-ID: On Nov 19, 2009, at 12:53 PM, Mathew Yeates wrote: > I am running my gtk app from python. I am deleting the canvas and > running gc.collect(). I still seem to have a reference to my > memmapped data. > > Any other hints? Gtk app from the standard python shell? Are you using the mpl toolbar? It keeps a ref to the canvas. If you can create a small freestanding example, that would help > > -Mathew > > On Thu, Nov 19, 2009 at 10:42 AM, John Hunter > wrote: > > > > > On Nov 19, 2009, at 12:35 PM, Mathew Yeates > wrote: > >> Yeah, I tried that. >> >> Here's what I'm doing. I have an application which displays >> different dataset which a user selects from a drop down list. I >> want to overwrite the existing plot with a new one. I've tried >> deleting just about everything get matplotlib to let go of my data! > > > What is everything? Are you using pyplot or are you embedding mpl > in a GUI? If the latter, are you deleting the FigureCanvas? You > will also need to call gc.collect after deleting the mpl objects > because we use a lot of circular references. Pyplot close does this > automatically, but this does not apply to embedding. > > How are you running you app? From the shell or IPython? > > > >> >> Mathew >> >> On Thu, Nov 19, 2009 at 10:30 AM, John Hunter >> wrote: >> >> >> >> >> On Nov 19, 2009, at 11:57 AM, Robert Kern >> wrote: >> >> > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates >> > wrote: >> >> There is definitely something wrong with matplotlib/numpy. >> Consider >> >> the >> >> following >> >>> from numpy import * >> >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) >> >>> del mydata >> >> >> >> I can now remove the file map.dat with (from the command line) $rm >> >> map.dat >> >> >> >> However >> >> If I plot mydata before the line >> >>> del mydata >> >> >> >> >> >> I can't get rid of the file until I exit python!! >> >> Does matplotlib keep a reference to the data? >> > >> > Almost certainly. >> > >> >> How can I remove this >> >> reference? >> > >> > Probably by deleting the plot objects that were created and close >> all >> > matplotlib windows referencing the data. If you are using >> IPython, you >> > should know that many of the returned objects are kept in Out, so >> you >> > will need to clear that. There might be some more places internal >> to >> > matplotlib, I don't know. >> > >> >> Closing the figure window containg the data *should* be enough. In >> pylab/pyplot, this also triggers a call to gc.collect. >> >> >> >> >> > With some care, you can use gc.get_referrers() to find the objects >> > that are holding direct references to your memmap. >> > >> > -- >> > Robert Kern >> > >> > "I have come to believe that the whole world is an enigma, a >> harmless >> > enigma that is made terrible by our own mad attempt to interpret >> it as >> > though it had an underlying truth." >> > -- Umberto Eco >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Thu Nov 19 14:20:40 2009 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 Nov 2009 11:20:40 -0800 Subject: [Numpy-discussion] matplotlib is breaking numpy In-Reply-To: References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> <3d375d730911190957h4fe0642an3e3547fc0ba892c6@mail.gmail.com> <3EE91778-7622-43E7-B62E-C8DC940BB697@gmail.com> <7d0c05ac0911191035q1aaa0908wa1cd497f82024eb9@mail.gmail.com> <40034D0F-2236-4E98-9371-4414DE769DD5@gmail.com> <7d0c05ac0911191053p6121a93av5db5dad1e99b685a@mail.gmail.com> Message-ID: <7d0c05ac0911191120y7da8a9d3t291cc95a55614d40@mail.gmail.com> yes, a GTK app from the python shell. And not using the toolbar. I'll see if I can extract out a sample of code that demonstrates the problem I'm having. Thx Mathew On Thu, Nov 19, 2009 at 10:56 AM, John Hunter wrote: > > > > > On Nov 19, 2009, at 12:53 PM, Mathew Yeates wrote: > > I am running my gtk app from python. I am deleting the canvas and running > gc.collect(). I still seem to have a reference to my memmapped data. > > Any other hints? > > > Gtk app from the standard python shell? > > Are you using the mpl toolbar? It keeps a ref to the canvas. If you can > create a small freestanding example, that would help > > > > > -Mathew > > On Thu, Nov 19, 2009 at 10:42 AM, John Hunter < > jdh2358 at gmail.com> wrote: > >> >> >> >> >> On Nov 19, 2009, at 12:35 PM, Mathew Yeates < >> mat.yeates at gmail.com> wrote: >> >> Yeah, I tried that. >> >> Here's what I'm doing. I have an application which displays different >> dataset which a user selects from a drop down list. I want to overwrite the >> existing plot with a new one. I've tried deleting just about everything get >> matplotlib to let go of my data! >> >> >> >> What is everything? Are you using pyplot or are you embedding mpl in a >> GUI? If the latter, are you deleting the FigureCanvas? You will also need >> to call gc.collect after deleting the mpl objects because we use a lot of >> circular references. Pyplot close does this automatically, but this does not >> apply to embedding. >> >> How are you running you app? From the shell or IPython? >> >> >> >> >> Mathew >> >> On Thu, Nov 19, 2009 at 10:30 AM, John Hunter < >> jdh2358 at gmail.com> wrote: >> >>> >>> >>> >>> >>> On Nov 19, 2009, at 11:57 AM, Robert Kern < >>> robert.kern at gmail.com> wrote: >>> >>> > On Thu, Nov 19, 2009 at 11:52, Mathew Yeates < >>> mat.yeates at gmail.com> >>> > wrote: >>> >> There is definitely something wrong with matplotlib/numpy. Consider >>> >> the >>> >> following >>> >>> from numpy import * >>> >>> mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) >>> >>> del mydata >>> >> >>> >> I can now remove the file map.dat with (from the command line) $rm >>> >> map.dat >>> >> >>> >> However >>> >> If I plot mydata before the line >>> >>> del mydata >>> >> >>> >> >>> >> I can't get rid of the file until I exit python!! >>> >> Does matplotlib keep a reference to the data? >>> > >>> > Almost certainly. >>> > >>> >> How can I remove this >>> >> reference? >>> > >>> > Probably by deleting the plot objects that were created and close all >>> > matplotlib windows referencing the data. If you are using IPython, you >>> > should know that many of the returned objects are kept in Out, so you >>> > will need to clear that. There might be some more places internal to >>> > matplotlib, I don't know. >>> > >>> >>> Closing the figure window containg the data *should* be enough. In >>> pylab/pyplot, this also triggers a call to gc.collect. >>> >>> >>> >>> >>> > With some care, you can use gc.get_referrers() to find the objects >>> > that are holding direct references to your memmap. >>> > >>> > -- >>> > Robert Kern >>> > >>> > "I have come to believe that the whole world is an enigma, a harmless >>> > enigma that is made terrible by our own mad attempt to interpret it as >>> > though it had an underlying truth." >>> > -- Umberto Eco >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > >>> NumPy-Discussion at scipy.org >>> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> >>> NumPy-Discussion at scipy.org >>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Thu Nov 19 15:25:35 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Thu, 19 Nov 2009 15:25:35 -0500 Subject: [Numpy-discussion] [matplotlib-devel] matplotlib is breaking numpy In-Reply-To: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> References: <7d0c05ac0911190952k57ef3489rbec6db759206079b@mail.gmail.com> Message-ID: <4B05A9BF.9060107@stsci.edu> Yes, for efficiency reasons matplotlib will use a reference to original data whenever possible, rather than copying it. Also, when using the pylab/pyplot API, matplotlib figures also stay around until they are explicitly deleted. You may need an explicit call to "clf('all')" to remove the figures, and thus references to the memmap'd file. However, my advice would be to just not use memmap. I can't imagine matplotlib performs very well if the data can't all fit in memory, so you might as well just load it all into memory anyway. Mike Mathew Yeates wrote: > There is definitely something wrong with matplotlib/numpy. Consider > the following > >from numpy import * > >mydata=memmap('map.dat',dtype=float64,mode='w+',shape=56566500) > > del mydata > > I can now remove the file map.dat with (from the command line) $rm map.dat > > However > If I plot mydata before the line > > del mydata > > > I can't get rid of the file until I exit python!! > Does matplotlib keep a reference to the data? How can I remove this > reference? > > Mathew > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > ------------------------------------------------------------------------ > > _______________________________________________ > Matplotlib-devel mailing list > Matplotlib-devel at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From tsyu80 at gmail.com Thu Nov 19 15:30:44 2009 From: tsyu80 at gmail.com (Tony S Yu) Date: Thu, 19 Nov 2009 15:30:44 -0500 Subject: [Numpy-discussion] OverflowError using _nanop with unsigned integers Message-ID: <1E296EB4-0340-4496-8833-69C305E71E7E@gmail.com> Hi, Functions that call _nanop (i.e. nan[arg]min, nan[arg]max) currently fail with unsigned integers. For example: >>> np.nanmin(np.array([0, 1], dtype=np.uint8)) OverflowError: cannot convert float infinity to integer It seems that unsigned integers don't get identified as integers in the _nanop function. Checking against `np.integer` instead of the built-in `int` will catch signed and unsigned integers, as shown below. Cheers, -Tony Note: the diff below also contains a change I made to fix ticket #1177 Index: numpy/lib/function_base.py =================================================================== --- numpy/lib/function_base.py (revision 7749) +++ numpy/lib/function_base.py (working copy) @@ -1038,6 +1038,8 @@ """ if isinstance(x, (float, int, number)): return compiled_interp([x], xp, fp, left, right).item() + elif isinstance(x, np.ndarray) and x.ndim == 0: + return compiled_interp(x[np.newaxis], xp, fp, left, right)[0] else: return compiled_interp(x, xp, fp, left, right) @@ -1349,7 +1351,7 @@ mask = isnan(a) # We only need to take care of NaN's in floating point arrays - if not np.issubdtype(y.dtype, int): + if not np.issubdtype(y.dtype, np.integer): y[mask] = fill res = op(y, axis=axis) From scot.denhalter at gmail.com Thu Nov 19 16:34:49 2009 From: scot.denhalter at gmail.com (Scot Denhalter) Date: Thu, 19 Nov 2009 14:34:49 -0700 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: References: Message-ID: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> I am a beginning programmer who is reading Natural Language Processing with Python. The book provides tutorials for working with the NLTK, which needs numpy to run certain calculations. I have downloaded and installed Python 2.6. I have downloaded numpy 1.3.0, but I cannot figure out how to install it. I have read the Install.txt that is included, but I don't understand exactly what I need to do. The instructions are clear enough for those familiar with the programming world, but I am obviously not the document's target reader. I am working with a MacBook Pro and do not have either fortran compiler installed. Can someone give me some guidance? Thanks in advance for any help I might receive. Scot -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- .. -*- rest -*- .. vim:syntax=rest .. NB! Keep this document a valid restructured document. Building and installing NumPy +++++++++++++++++++++++++++++ :Authors: Numpy Developers :Discussions to: numpy-discussion at scipy.org .. Contents:: PREREQUISITES ============= Building NumPy requires the following software installed: 1) Python__ 2.4.x or newer On Debian and derivative (Ubuntu): python python-dev On Windows: the official python installer on Python__ is enough Make sure that the Python package distutils is installed before continuing. For example, in Debian GNU/Linux, distutils is included in the python-dev package. Python must also be compiled with the zlib module enabled. 2) nose__ (pptional) 0.10.3 or later This is required for testing numpy, but not for using it. Python__ http://www.python.org nose__ http://somethingaboutorange.com/mrl/projects/nose/ Fortran ABI mismatch ==================== The two most popular open source fortran compilers are g77 and gfortran. Unfortunately, they are not ABI compatible, which means that concretely you should avoid mixing libraries built with one with another. In particular, if your blas/lapack/atlas is built with g77, you *must* use g77 when building numpy and scipy; on the contrary, if your atlas is built with gfortran, you *must* build numpy/scipy with gfortran. Choosing the fortran compiler ----------------------------- To build with g77: python setup.py build --fcompiler=gnu To build with gfortran: python setup.py build --fcompiler=gnu95 How to check the ABI of blas/lapack/atlas ----------------------------------------- One relatively simple and reliable way to check for the compiler used to build a library is to use ldd on the library. If libg2c.so is a dependency, this means that g77 has been used. If libgfortran.so is a a dependency, gfortran has been used. If both are dependencies, this means both have been used, which is almost always a very bad idea. Building with ATLAS support =========================== Ubuntu 8.10 (Intrepid) ---------------------- You can install the necessary packages for optimized ATLAS with this command: sudo apt-get install libatlas-base-dev If you have a recent CPU with SIMD suppport (SSE, SSE2, etc...), you should also install the corresponding package for optimal performances. For example, for SSE2: sudo apt-get install libatlas3gf-sse2 *NOTE*: if you build your own atlas, Intrepid changed its default fortran compiler to gfortran. So you should rebuild everything from scratch, including lapack, to use it on Intrepid. Ubuntu 8.04 and lower --------------------- You can install the necessary packages for optimized ATLAS with this command: sudo apt-get install atlas3-base-dev If you have a recent CPU with SIMD suppport (SSE, SSE2, etc...), you should also install the corresponding package for optimal performances. For example, for SSE2: sudo apt-get install atlas3-sse2 Windows 64 bits notes ===================== Note: only AMD64 is supported (IA64 is not) - AMD64 is the version most people want. Free compilers (mingw-w64) -------------------------- http://mingw-w64.sourceforge.net/ To use the free compilers (mingw-w64), you need to build your own toolchain, as the mingw project only distribute cross-compilers (cross-compilation is not supported by numpy). Since this toolchain is still being worked on, serious compilers bugs can be expected. binutil 2.19 + gcc 4.3.3 + mingw-w64 runtime gives you a working C compiler (but the C++ is broken). gcc 4.4 will hopefully be able to run natively. This is the only tested way to get a numpy with a FULL blas/lapack (scipy does not work because of C++). MS compilers ------------ If you are familiar with MS tools, that's obviously the easiest path, and the compilers are hopefully more mature (although in my experience, they are quite fragile, and often segfault on invalid C code). The main drawback is that no fortran compiler + MS compiler combination has been tested - mingw-w64 gfortran + MS compiler does not work at all (it is unclear whether it ever will). For python 2.5, you need VS 2005 (MS compiler version 14) targetting AMD64 bits, or the Platform SDK v6.0 or below (which gives command line versions of 64 bits target compilers). The PSDK is free. For python 2.6, you need VS 2008. The freely available version does not contains 64 bits compilers (you also need the PSDK, v6.1). It is *crucial* to use the right version: python 2.5 -> version 14, python 2.6, version 15. You can check the compiler version with cl.exe /?. Note also that for python 2.5, 64 bits and 32 bits versions use a different compiler version. From efiring at hawaii.edu Thu Nov 19 16:46:03 2009 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 19 Nov 2009 11:46:03 -1000 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> Message-ID: <4B05BC9B.2030505@hawaii.edu> Scot Denhalter wrote: > I am a beginning programmer who is reading Natural Language Processing > with Python. The book provides tutorials for working with the NLTK, > which needs numpy to run certain calculations. I have downloaded and > installed Python 2.6. I have downloaded numpy 1.3.0, but I cannot > figure out how to install it. I have read the Install.txt that is > included, but I don't understand exactly what I need to do. The > instructions are clear enough for those familiar with the programming > world, but I am obviously not the document's target reader. > > I am working with a MacBook Pro and do not have either fortran compiler > installed. Can someone give me some guidance? You don't need a fortran compiler for numpy, even if you are building from source; and you probably don't need to build from source. Did you try the suggested binary packages for the Mac as listed on the nltk site? http://www.nltk.org/download Eric From pgmdevlist at gmail.com Thu Nov 19 16:57:03 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 19 Nov 2009 16:57:03 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> Message-ID: On Nov 19, 2009, at 4:34 PM, Scot Denhalter wrote: > I am a beginning programmer who is reading Natural Language Processing with Python. The book provides tutorials for working with the NLTK, which needs numpy to run certain calculations. I have downloaded and installed Python 2.6. I have downloaded numpy 1.3.0, but I cannot figure out how to install it. I have read the Install.txt that is included, but I don't understand exactly what I need to do. The instructions are clear enough for those familiar with the programming world, but I am obviously not the document's target reader. > > I am working with a MacBook Pro and do not have either fortran compiler installed. Can someone give me some guidance? Ciao Scott. First of all, don't panic... Mac OS 10.6.2 ? You have installed Xcode ? You're basically good to go. * Install nose (I use pip for the install), system-wide is fine. * Grab a fortran compiler here: http://r.research.att.com/tools/ * Unzip your numpy sources archive * In the directory that gets created, enter in Terminal (or iTerm): python setup.py install --user 2>&1| tee build.log The --user is to make sure that you install numpy in your Users directory and not mess with the system The 2>&1 | tee build.log sends the messages you get during the compilation to a file build.log (that you gonna need if anything goes wrong). That should be enough. If it doesn't work, then post your build.log so that we can try and check where it fails. Let me know how it goes P. From scot.denhalter at gmail.com Thu Nov 19 17:36:43 2009 From: scot.denhalter at gmail.com (Scot Denhalter) Date: Thu, 19 Nov 2009 15:36:43 -0700 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <4B05BC9B.2030505@hawaii.edu> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <4B05BC9B.2030505@hawaii.edu> Message-ID: <37d4d7f30911191436h7ccfce85gaa9723d4497258f0@mail.gmail.com> Eric, Yes, I downloaded the files recommended on that webpage. The tar.gz files unpack, but they do not install, and I am too much of a newbie to figure out how to install the files in the unpacked folder. No matter where I placed the numpy folder, I couldn't get Python 2.6 to find and install numpy. Nevertheless, I did finally find .dmg downloads for numpy and matplotlib and they installed without my having to get involved. But when I run a dispersion_plot function, I get an error message from Python, which I don't understand: Warning (from warnings module): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/dates.py", line 91 import pytz.zoneinfo ImportWarning: Not importing directory '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pytz/zoneinfo': missing __init__.py Python launches the dispersion plot in another window as it apparently should, so why am I getting an error message? Perhaps, those of you who program in Python can explain this to me. Scot On Thu, Nov 19, 2009 at 2:46 PM, Eric Firing wrote: > Scot Denhalter wrote: > > I am a beginning programmer who is reading Natural Language Processing > > with Python. The book provides tutorials for working with the NLTK, > > which needs numpy to run certain calculations. I have downloaded and > > installed Python 2.6. I have downloaded numpy 1.3.0, but I cannot > > figure out how to install it. I have read the Install.txt that is > > included, but I don't understand exactly what I need to do. The > > instructions are clear enough for those familiar with the programming > > world, but I am obviously not the document's target reader. > > > > I am working with a MacBook Pro and do not have either fortran compiler > > installed. Can someone give me some guidance? > > You don't need a fortran compiler for numpy, even if you are building > from source; and you probably don't need to build from source. Did you > try the suggested binary packages for the Mac as listed on the nltk site? > > http://www.nltk.org/download > > Eric > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Nov 19 17:40:04 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 19 Nov 2009 14:40:04 -0800 Subject: [Numpy-discussion] datetime64 In-Reply-To: <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> Message-ID: Hi all, On Thu, Oct 29, 2009 at 12:43 PM, Pierre GM wrote: > Oh yes, I saw that... Marty Fuhry, one of our GSoC students, had > written some pretty extensive series of tests to allocate datetime/ > strings to elements of a ndarray with datetime64 dtype. He also had > written some functions allowing conversion from one frequency to > another. Unfortunately, I don't think his work has been incorporated > yet. Maybe Jarrod M. and Travis O. will shed some light on that > matter. I for one would be quite interested into checking what's > happening on that front. we're starting to use these tools more and more, and with the 1.4 release coming out, we're a bit lost here... A few specific questions we have: - It seems the original spec that was written by Francesc with a lot of community feedback, whose final form (updated by Travis recently) is: http://projects.scipy.org/numpy/browser/trunk/doc/neps/datetime-proposal.rst was implemented differently in the final committed code. Are the differences and rationale for this documented somewhere? I'm not saying the current code is a bad idea, simply that we'd like to understand the differences and why they came about, I'm sure there were good reasons behind it... - Are there any tests for the current code, that will go in before 1.4 is out? We'd like to know if we can rely on 1.4 as a dependency for nitime, but for that we'd like to know what the code can actually do, and also to have checks for what appear to be segfault problems: >>> s = np.zeros((10),dtype=np.timedelta64) >>> s array([0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001, 0:00:00.000001], dtype=timedelta64[ns]) >>> s.dtype = 'timedelta64[s]' >>> s array([0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00, 0:01:00], dtype=timedelta64[s]) >>> s.dtype = 'timedelta64[m]' Segmentation fault - From our experimenting with the code it appears that the following code is not accepted: a = np.array([0.1, 0.5, 0.9, 1.4], dtype='some form of spelling seconds, nothing we've tried has worked') while it would appear to be a very natural way to use such an object: "here are a bunch of floats, treat them like seconds"... Is there a reason for this? This variant appears to work, but I'm not sure why the need for the nested array call: In [22]: a = np.array([0,0.1,0.5, 1.5]); a Out[22]: array([ 0. , 0.1, 0.5, 1.5]) In [23]: np.array(a, dtype='timedelta64[s]') Out[23]: array([0:00:00, 0:00:00, 0:00:00, 0:01:00], dtype=timedelta64[s]) and the input is not being interpreted in seconds as one would intuitively think, instead there's a truncation I'm not sure I understand... - I know that the name numpy.datetime() was in the spec above (Francesc's NEP) but we're wondering if having this is a good idea... This is a name clash with a stdlib python name that has existed for a long time; while I know that we all advocate against 'from import *' for many good reasons, but at least I think that staying clear of known stdlib name conflicts would be a good practice. It will reduce the surprise factor for someone using numpy interactively, for example (ipython -pylab) in situations like: dateime.foo() # this is np.datetime %run foo.py # script with 'import datetime' in it # now datetime at the top level is the one in the stdlib... - Sources of information? Docstrings and tests haven't gotten us very far, and googling points to the original spec as udpated by Travis: http://projects.scipy.org/numpy/browser/trunk/doc/neps/datetime-proposal.rst But even the examples in that document don't actually work. All of these copied from the spec, and run with: In [7]: np.__version__ Out[7]: '1.4.0.dev7421' In [2]: t = numpy.ones(3, dtype='M8[s]') --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/fperez/teach/code_review/ in () /usr/local/lib/python2.5/site-packages/numpy/core/numeric.pyc in ones(shape, dtype, order) 1656 a = empty(shape, dtype, order) 1657 try: -> 1658 a.fill(1) 1659 # Above is faster now after addition of fast loops. 1660 #a = zeros(shape, dtype, order) ValueError: Must be a datetime.date or datetime.datetime object In [5]: numpy.datetime64(42, 'us') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/fperez/teach/code_review/ in () TypeError: function takes at most 1 argument (2 given) In [6]: numpy.timedelta64(10, 'us') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/fperez/teach/code_review/ in () TypeError: function takes at most 1 argument (2 given) Thanks for any clarity on the status of this code... Sorry if we've missed the obvious, but we've honestly tried to find information and to understand this, and we're pretty lost. Any help will be appreciated. Cheers, f From scot.denhalter at gmail.com Thu Nov 19 17:41:00 2009 From: scot.denhalter at gmail.com (Scot Denhalter) Date: Thu, 19 Nov 2009 15:41:00 -0700 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> Message-ID: <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> Pierre, Yes, I am using the Snow Leopard OSX. Should I be coding through the Xcode interface and not Python's IDLE shell? You distinguish between system wide installation and user installation. You may have seen from my post to Eric, that I found and installed .dmg downloads for numpy and matplotlib. I have tried to figure out where these installed themselves, but I haven't found them in the user folder. I assume they have been installed system wide. Is this going to be a problem? Scot On Thu, Nov 19, 2009 at 2:57 PM, Pierre GM wrote: > > On Nov 19, 2009, at 4:34 PM, Scot Denhalter wrote: > > > I am a beginning programmer who is reading Natural Language Processing > with Python. The book provides tutorials for working with the NLTK, which > needs numpy to run certain calculations. I have downloaded and installed > Python 2.6. I have downloaded numpy 1.3.0, but I cannot figure out how to > install it. I have read the Install.txt that is included, but I don't > understand exactly what I need to do. The instructions are clear enough for > those familiar with the programming world, but I am obviously not the > document's target reader. > > > > I am working with a MacBook Pro and do not have either fortran compiler > installed. Can someone give me some guidance? > > Ciao Scott. > First of all, don't panic... > Mac OS 10.6.2 ? You have installed Xcode ? You're basically good to go. > * Install nose (I use pip for the install), system-wide is fine. > * Grab a fortran compiler here: http://r.research.att.com/tools/ > * Unzip your numpy sources archive > * In the directory that gets created, enter in Terminal (or iTerm): > python setup.py install --user 2>&1| tee build.log > > The --user is to make sure that you install numpy in your Users directory > and not mess with the system > The 2>&1 | tee build.log sends the messages you get during the compilation > to a file build.log (that you gonna need if anything goes wrong). > > That should be enough. If it doesn't work, then post your build.log so that > we can try and check where it fails. > Let me know how it goes > P. > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Nov 19 18:26:37 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 19 Nov 2009 18:26:37 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191436h7ccfce85gaa9723d4497258f0@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <4B05BC9B.2030505@hawaii.edu> <37d4d7f30911191436h7ccfce85gaa9723d4497258f0@mail.gmail.com> Message-ID: <6DDD17E9-A2FF-4D74-A500-F7465EFB19B2@cs.toronto.edu> On 19-Nov-09, at 5:36 PM, Scot Denhalter wrote: > On Thu, Nov 19, 2009 at 2:46 PM, Eric Firing > wrote: >> >> You don't need a fortran compiler for numpy, even if you are building >> from source; and you probably don't need to build from source. Did >> you >> try the suggested binary packages for the Mac as listed on the nltk >> site? >> >> http://www.nltk.org/download > Yes, I downloaded the files recommended on that webpage. The tar.gz > files > unpack, but they do not install, and I am too much of a newbie to > figure out > how to install the files in the unpacked folder. No matter where I > placed > the numpy folder, I couldn't get Python 2.6 to find and install numpy. > > Nevertheless, I did finally find .dmg downloads for numpy and > matplotlib and > they installed without my having to get involved. > > But when I run a dispersion_plot function, I get an error message from > Python, which I don't understand: > > Warning (from warnings module): > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ > site-packages/matplotlib/dates.py", > line 91 > import pytz.zoneinfo > ImportWarning: Not importing directory > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ > site-packages/pytz/zoneinfo': > missing __init__.py > > Python launches the dispersion plot in another window as it apparently > should, so why am I getting an error message? Perhaps, those of you > who > program in Python can explain this to me. I don't know why it would be trying to import zoneinfo since it doesn't seem to be a code directory, it just contains various data for timezone stuff. I'm CC'ing matplotlib-devel since that's really a matplotlib problem and not a Python or NumPy one. David P.S. please don't top-post replies, it makes things hard to follow. From pgmdevlist at gmail.com Thu Nov 19 18:25:12 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 19 Nov 2009 18:25:12 -0500 Subject: [Numpy-discussion] datetime64 In-Reply-To: References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> Message-ID: <3CF2046C-7463-408F-B671-5CDF5E613375@gmail.com> On Nov 19, 2009, at 5:40 PM, Fernando Perez wrote: > > we're starting to use these tools more and more, and with the 1.4 > release coming out, we're a bit lost here... Welcome to the club... Fernando, Ariel, I'm in the same spot as you are, I haven't been able to use it. I don't think it's that a good idea to advertise it in the upcoming release notes... In addition to your observations, here's something that surprised me: >>> x=np.array([datetime.date(2001,_,1) for _ in range(1,13)], dtype="M8[M]") array([2001-01-01 00:00:00, 2001-02-01 00:00:00, 2001-03-01 00:00:00, 2001-04-01 00:00:00, 2001-05-01 00:00:00, 2001-06-01 00:00:00, 2001-07-01 00:00:00, 2001-08-01 00:00:00, 2001-09-01 00:00:00, 2001-10-01 00:00:00, 2001-11-01 00:00:00, 2001-12-01 00:00:00], dtype=datetime64[M]) >>> x.dtype dtype('datetime64[M]') >>> x.astype("M8[Y]") array([2342-01-01 00:00:00, 2343-01-01 00:00:00, 2344-01-01 00:00:00, 2345-01-01 00:00:00, 2346-01-01 00:00:00, 2347-01-01 00:00:00, 2348-01-01 00:00:00, 2349-01-01 00:00:00, 2350-01-01 00:00:00, 2351-01-01 00:00:00, 2352-01-01 00:00:00, 2353-01-01 00:00:00], dtype=datetime64[Y]) >>> x.astype("M8[Y]") array([2342-01-01 00:00:00, 2343-01-01 00:00:00, 2344-01-01 00:00:00, 2345-01-01 00:00:00, 2346-01-01 00:00:00, 2347-01-01 00:00:00, 2348-01-01 00:00:00, 2349-01-01 00:00:00, 2350-01-01 00:00:00, 2351-01-01 00:00:00, 2352-01-01 00:00:00, 2353-01-01 00:00:00], dtype=datetime64[Y]) >>> x.dtype dtype('datetime64[Y]') Besides the fact that having to use a comprehension list is a bit of an hassle, I'm surprised that astype modifies my array in place... Mind you, I'm very grateful for Travis O's work on np.datetime64 and I'm quite excited to have time support in numpy directly... eventually. I'm sure we can get something ready for 1.5. Just let me know how I can help. From pgmdevlist at gmail.com Thu Nov 19 18:27:43 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 19 Nov 2009 18:27:43 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> Message-ID: <1AD11271-81B4-4EFA-AD9A-E65EB2CC20F3@gmail.com> On Nov 19, 2009, at 5:41 PM, Scot Denhalter wrote: > Pierre, > > Yes, I am using the Snow Leopard OSX. Should I be coding through the Xcode interface and not Python's IDLE shell? Oh no, that's not what I meant when I asked you if you had installed Xcode. It's just that you need Xcode for the C compiler. Stick with your IDLE shell, or whatever shell you prefer (I strongly advise you to use IPython...) > You distinguish between system wide installation and user installation. You may have seen from my post to Eric, that I found and installed .dmg downloads for numpy and matplotlib. I have tried to figure out where these installed themselves, but I haven't found them in the user folder. I assume they have been installed system wide. Is this going to be a problem? Nope. If you go the dmg way, then you're pretty much set. Please note that when reporting a problem (such as your dispersion_plot issue), you should try to give us more info (matplotlib version, the script that you were using, etc.) From dwf at cs.toronto.edu Thu Nov 19 18:32:46 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 19 Nov 2009 18:32:46 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> Message-ID: <05E5C6CC-D018-4A4F-8592-760775FFA922@cs.toronto.edu> On 19-Nov-09, at 5:41 PM, Scot Denhalter wrote: > Yes, I am using the Snow Leopard OSX. Should I be coding through > the Xcode > interface and not Python's IDLE shell? Using the Xcode IDE is not necessary and probably far from optimal for what you're doing; however Xcode installs all of the command line tools (gcc, g++, make) that are typically needed to build packages. > You distinguish between system wide installation and user > installation. You > may have seen from my post to Eric, that I found and installed .dmg > downloads for numpy and matplotlib. I have tried to figure out > where these > installed themselves, but I haven't found them in the user folder. > I assume > they have been installed system wide. Is this going to be a problem? The DMGs install in system-wide directories, /Library/Frameworks/ Python.framework/Versions/2.6/lib/python2.6/site-packages I'm guessing. The --user stuff should be used if you are installing against the Python that Apple ships with OS X. They also ship an old & crusty version of NumPy, I think. Unless Apple has relocated their Python install for 10.6 (it used to be in /System) it looks like you downloaded Python from Python.org since it's in /Library, in which case Pierre's fears about messing up the system install aren't a problem. David From scot.denhalter at gmail.com Thu Nov 19 19:00:48 2009 From: scot.denhalter at gmail.com (Scot Denhalter) Date: Thu, 19 Nov 2009 17:00:48 -0700 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <05E5C6CC-D018-4A4F-8592-760775FFA922@cs.toronto.edu> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> <05E5C6CC-D018-4A4F-8592-760775FFA922@cs.toronto.edu> Message-ID: <37d4d7f30911191600s6fd5aa93o713e0a8bb0a0e304@mail.gmail.com> On Thu, Nov 19, 2009 at 4:32 PM, David Warde-Farley wrote: > On 19-Nov-09, at 5:41 PM, Scot Denhalter wrote: > >> Yes, I am using the Snow Leopard OSX. ?Should I be coding through >> the Xcode >> interface and not Python's IDLE shell? > > Using the Xcode IDE is not necessary and probably far from optimal for > what you're doing; however Xcode installs all of the command line > tools (gcc, g++, make) that are typically needed to build packages. I am not building anything at the moment. I am simply trying to learn Python as it pertains to Natural Language Processing. The book I am using offers tutorials that sometimes require the installation of other program: nltk, numpy, matplotlib, Prover9, MaltParser, and MegaM. So far, I have only succeeded with installing nltk, numpy, and matplotlib. Nevertheless, I figure I should address this error message about matplotlib first. I am using matplotlib-0.99.1.1. with Python 2.6. > >> You distinguish between system wide installation and user >> installation. You >> may have seen from my post to Eric, that I found and installed .dmg >> downloads for numpy and matplotlib. ?I have tried to figure out >> where these >> installed themselves, but I haven't found them in the user folder. >> I assume >> they have been installed system wide. ?Is this going to be a problem? > > The DMGs install in system-wide directories, /Library/Frameworks/ > Python.framework/Versions/2.6/lib/python2.6/site-packages I'm > guessing. The --user stuff should be used if you are installing > against the Python that Apple ships with OS X. They also ship an old & > crusty version of NumPy, I think. Yes, my MacBook Pro came with Python 2.5. Pierre recommends iPython, but the book I am using to learn Python (Natural Language Processing with Python) says that the NLTK won't work with anything above version 2.6. I discovered that limitation after first installing Python 3.x. I don't know if iPython is compatible with the NLTK. What do you think, Pierre? > > Unless Apple has relocated their Python install for 10.6 (it used to > be in /System) it looks like you downloaded Python from Python.org > since it's in /Library, in which case Pierre's fears about messing up > the system install aren't a problem. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dwf at cs.toronto.edu Thu Nov 19 19:11:09 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 19 Nov 2009 19:11:09 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191600s6fd5aa93o713e0a8bb0a0e304@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> <05E5C6CC-D018-4A4F-8592-760775FFA922@cs.toronto.edu> <37d4d7f30911191600s6fd5aa93o713e0a8bb0a0e304@mail.gmail.com> Message-ID: <51278639-B7D1-4726-BE3D-3AA9C16E60D6@cs.toronto.edu> On 19-Nov-09, at 7:00 PM, Scot Denhalter wrote: > Yes, my MacBook Pro came with Python 2.5. Pierre recommends iPython, > but the book I am using to learn Python (Natural Language Processing > with Python) says that the NLTK won't work with anything above version > 2.6. I discovered that limitation after first installing Python 3.x. > I don't know if iPython is compatible with the NLTK. What do you > think, Pierre? IPython is not a version of Python; it is simply a front-end to the interactive Python prompt which makes it more pleasant to use (it adds tab completion, for example, and allows you to keep using the prompt while displaying a plot (e.g., to add things to an existing plot window, or to open multiple plots). IPython and NLTK will not interfere with one another. At any rate, the matplotlib error is not an error but a warning, and unless you are making plots that involve dates and/or time zones (which is improbable given the subject area), it can be safely ignored. David From pgmdevlist at gmail.com Thu Nov 19 19:50:32 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 19 Nov 2009 19:50:32 -0500 Subject: [Numpy-discussion] Beginner Needing Help with Installation In-Reply-To: <37d4d7f30911191600s6fd5aa93o713e0a8bb0a0e304@mail.gmail.com> References: <37d4d7f30911191334wfd97580t2a741348168c4226@mail.gmail.com> <37d4d7f30911191441v66b1be99ma61c707416c24bfb@mail.gmail.com> <05E5C6CC-D018-4A4F-8592-760775FFA922@cs.toronto.edu> <37d4d7f30911191600s6fd5aa93o713e0a8bb0a0e304@mail.gmail.com> Message-ID: On Nov 19, 2009, at 7:00 PM, Scot Denhalter wrote: > > I am not building anything at the moment. I am simply trying to learn > Python as it pertains to Natural Language Processing. And soon you'll get addicted and start installing stuffs ;) > Yes, my MacBook Pro came with Python 2.5. Pierre recommends iPython, > but the book I am using to learn Python (Natural Language Processing > with Python) says that the NLTK won't work with anything above version > 2.6. I discovered that limitation after first installing Python 3.x. > I don't know if iPython is compatible with the NLTK. What do you > think, Pierre? Like David said, IPython will not interfere. It's a replacement for IDLE, far far more convenient... Note that numpy won't work w/ Python 3.x From dyamins at gmail.com Thu Nov 19 20:45:21 2009 From: dyamins at gmail.com (Dan Yamins) Date: Thu, 19 Nov 2009 20:45:21 -0500 Subject: [Numpy-discussion] Numpy/Scipy for EC2 In-Reply-To: <15e4667e0910290845te2f9217gf03efa635ff82bd4@mail.gmail.com> References: <15e4667e0910282129t4ad7f5eble1de56c91e6cff25@mail.gmail.com> <3d375d730910290841u78343cd2v4715cd40001bb09a@mail.gmail.com> <15e4667e0910290845te2f9217gf03efa635ff82bd4@mail.gmail.com> Message-ID: <15e4667e0911191745y6e2b3e20pceaddd05aa3c0554@mail.gmail.com> Hi all: I'm just writing to report on my experience using Starcluster, which enables the use of NumPy and Scipy in the Amazon EC2 cloud computing environment. The purpose of my email is to extol Starcluster's qualities, and suggest that the NumPy community be aware of its development. I suspect there are others in the community who find cloud computing an attractive idea but a little daunting to get into, and would be pleasantly surprised out how easy Starcluster makes it to get started using NumPy on Amazon EC2. For those of you who aren't familiar with AMIs and the Amazon EC2 service, see e.g. http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud. Three of the basic concepts are "Amazon Machine Images" (AMIs), "machine instances" of AMIs, and the Elastic Block Storage (EBS) service. AMIs are disk images containing a virtual machine, including an operating system and other software you add on. Instances are temporarily allocated computers, booted with your chosen virtual machine, that you start up on demand, use for computations with software from the AMI, and then terminate. EBS is a persistent storage service, also from Amazon, that serves as permanent file-systems in the cloud. You allocate an EBS volume of a given size, attach the EBS volume(s) to a running machine instance just like any other hard-drive, and use it to store the files you use/create during computation, both during the computation and then for later use whenever you start up a new instance. A couple of weeks ago I wrote to this list asking for advice on finding a good Amazon Machine Instance (AMI) for using NumPy and Scipy on Amazon cloud. I didn't want to have to build a linux machine image with optimized blas and lapack myself, and I figured that there might be good existing publicly-available AMIs that I could use as a base. Robert Kern suggested that I look into the Starcluster project ( http://web.mit.edu/stardev/cluster/). I have found Starcluster extremely useful. It made it possible for me to, in the course of one day, go from knowing essentially nothing about cloud something, to being able to run large-scale parallel clusters with my favorite NumPy/SciPy-scripts. The basis of what Starcluster offers are two solidly-build AMIs. The operating system is Ubuntu Jaunty, and comes with prebuilt optimized blas and lapack, numpy, Scipy, matplotlib, ipython, and several other useful packages for scientific computing in python. It uses Python 2.6, and comes in both 32-bit and 64-bit flavors. The AMIs are based on AMIs from Alestic (http://alestic.com/), and are built with best-practices for ensuring stability and good interaction with Amazon's system. They have proved very stable and extensible. In addition to these AMIs, Starcluster has three extremely useful features: -- Built-in support for mounting EBS drives as NFS filesystems**, and then administering the shared drive across multiple machine instances. -- The Sun Grid Engine (SGE), a queuing system for scheduling jobs to be run in parallel across instances -- A python module with a few commands that give you an incredibly simple interface for automating the process of starting/terminating a cluster of instances, mounting the shared drive, starting the grid engine, &c -- and configuring your cluster needs (e.g. how many nodes it will contain, which AMIs to use, which EBS volumes to mount etc.). As a result, all you have to do to have a NumPy-enabled cluster-on-demand is: 1) Get an amazon EC2 account, and the accompanying security credentials (.501 certificates and PGP keypair) for your account. 2) Install starcluster ("easy_install starcluster") 3) Follow the installation procedure on the starcluster website for getting, attaching, and formatting an EBS volume as an NFS drive. 4) Set up your starcluster configuration file. 5) Start a 1-node cluster, modify the installation as you see fit, and re-bundle the result into a new AMI as described on the Amazon website http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/. (Don't forget to edit your starcluster configuration file to reflect your new AMI.) This step is optional -- If you don't need anything else special, you can just use Starcluster's base images. After that, starting a cluster is as easy as typing single command ("starcluster -s"). To submit parallel jobs on your cluster, you can learn to use the Sun Grid Engine "qsub" command ( http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/htmlman/htmlman1/qsub.html) or use the python bindings to the SGE interface ( http://code.google.com/p/drmaa-python/). Or, if you like Parallel Python, that works perfectly well on these clusters too. Overall, in my experience, Starcluster has been easy, stable and powerful, and I encourage anyone who is curious about cloud computing with Numpy to look into it. Starcluster is by no means a finished project. At the moment, you can only administer one cluster at a time from your given local machine, since starcluster has no notion of a "session" and it can't distinguish between different clusters you've started up (you can *start* multiple clusters, but then any starcluster commands that you type in your local terminal might get confused about which amazon machine instances you're referring to, so it has trouble administering them.) Also, there's no dynamic load balancing, so once you've started a cluster with a certain number of nodes, you're stuck with that number of computers while the cluster is running, even if you're only using a few of them or suddenly need more. The developer of the project (*Justin Riley)* says on his website that he's planning to add these features in the next release. Now, I'm not the creator or developer or maintainer of Starcluster, and I have no affiliation with Justin Riley or the project whatsoever, so I want to make it clear I don't speak for them in any way except as a satisfied user. I don't know what his commitment to his development plans are, either -- however, I hope he sticks to his timeline, as I think continuing the vigorous development of his project would be a real plus for the NumPy community. I'm hoping that if others in the NumPy community like his project and start using it, that will make add to the likelihood of continued development. (If anyone from the NumPy community is interesting in helping the developer out, perhaps you should consider shooting him an email.) Anyhow, I apologize for this long email, and hope it may be of use to somebody! Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Nov 19 20:47:23 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 20 Nov 2009 10:47:23 +0900 Subject: [Numpy-discussion] datetime64 In-Reply-To: <3CF2046C-7463-408F-B671-5CDF5E613375@gmail.com> References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> <3CF2046C-7463-408F-B671-5CDF5E613375@gmail.com> Message-ID: <5b8d13220911191747x80fdfb5u1c5b3ebee0e70d7f@mail.gmail.com> On Fri, Nov 20, 2009 at 8:25 AM, Pierre GM wrote: > > On Nov 19, 2009, at 5:40 PM, Fernando Perez wrote: >> >> we're starting to use these tools more and more, and with the 1.4 >> release coming out, we're a bit lost here... > > Welcome to the club... > Fernando, Ariel, I'm in the same spot as you are, I haven't been able to use it. I don't think it's that a good idea to advertise it in the upcoming release notes... I am inclined to simply remove the datetime code for 1.4.0. I think there has been plenty of time to document it, and I would prefer seeing the code in 1.5.0 with the rationale rather than changing something between 1.4.0 and 1.5.0 David From cournape at gmail.com Thu Nov 19 20:55:35 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 20 Nov 2009 10:55:35 +0900 Subject: [Numpy-discussion] Failure building pdf doc Message-ID: <5b8d13220911191755i5077ad00t6a5b419f4b10684d@mail.gmail.com> Hi, While checking everything builds for the 1.4.0 release, I noticed a problem with building the latex version: writing... done processing numpy-user.tex... user/index user/introduction user/whatisnumpy user/install user/howtofind user/basics user/basics.types user/basics.creation user/basics.io user/basics.io.genfromtxt user/basics.indexing user/basics.broadcasting user/basics.byteswapping user/basics.rec user/basics.subclassing user/performance user/misc user/c-info user/c-info.how-to-extend user/c-info.python-as-glue user/c-info.beyond-basics resolving references... writing... Markup is unsupported in LaTeX: user/basics.io.genfromtxt:: column or row spanning cells are not yet implemented. How can I fix this ? I noticed a similar bug reported on sphinx bug tracker 10 months ago by Pauli, is this indeed related ? I am using sphinx 0.6.2 David From oliphant at enthought.com Thu Nov 19 20:59:17 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Thu, 19 Nov 2009 19:59:17 -0600 Subject: [Numpy-discussion] datetime64 In-Reply-To: <5b8d13220911191747x80fdfb5u1c5b3ebee0e70d7f@mail.gmail.com> References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> <3CF2046C-7463-408F-B671-5CDF5E613375@gmail.com> <5b8d13220911191747x80fdfb5u1c5b3ebee0e70d7f@mail.gmail.com> Message-ID: -- (mobile phone of) Travis Oliphant Enthought, Inc. 1-512-536-1057 http://www.enthought.com On Nov 19, 2009, at 7:47 PM, David Cournapeau wrote: > On Fri, Nov 20, 2009 at 8:25 AM, Pierre GM > wrote: >> >> On Nov 19, 2009, at 5:40 PM, Fernando Perez wrote: >>> >>> we're starting to use these tools more and more, and with the 1.4 >>> release coming out, we're a bit lost here... >> >> Welcome to the club... >> Fernando, Ariel, I'm in the same spot as you are, I haven't been >> able to use it. I don't think it's that a good idea to advertise it >> in the upcoming release notes... > > I am inclined to simply remove the datetime code for 1.4.0. I think > there has been plenty of time to document it, and I would prefer > seeing the code in 1.5.0 with the rationale rather than changing > something between 1.4.0 and 1.5.0 > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Thu Nov 19 21:18:22 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 19 Nov 2009 19:18:22 -0700 Subject: [Numpy-discussion] datetime64 In-Reply-To: <5b8d13220911191747x80fdfb5u1c5b3ebee0e70d7f@mail.gmail.com> References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> <3CF2046C-7463-408F-B671-5CDF5E613375@gmail.com> <5b8d13220911191747x80fdfb5u1c5b3ebee0e70d7f@mail.gmail.com> Message-ID: On Thu, Nov 19, 2009 at 6:47 PM, David Cournapeau wrote: > On Fri, Nov 20, 2009 at 8:25 AM, Pierre GM wrote: > > > > On Nov 19, 2009, at 5:40 PM, Fernando Perez wrote: > >> > >> we're starting to use these tools more and more, and with the 1.4 > >> release coming out, we're a bit lost here... > > > > Welcome to the club... > > Fernando, Ariel, I'm in the same spot as you are, I haven't been able to > use it. I don't think it's that a good idea to advertise it in the upcoming > release notes... > > I am inclined to simply remove the datetime code for 1.4.0. I think > there has been plenty of time to document it, and I would prefer > seeing the code in 1.5.0 with the rationale rather than changing > something between 1.4.0 and 1.5.0 > > When I committed the Chebyshev stuff I was thinking it would be nice to have a "provisional" category. That is, since the code had seen little use and feedback some features, such as defaults/names might change before the next release. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Thu Nov 19 22:12:26 2009 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 19 Nov 2009 19:12:26 -0800 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data In-Reply-To: <49d6b3500911170929rccaa0e7k75f1450a2f48e519@mail.gmail.com> References: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> <49d6b3500911170929rccaa0e7k75f1450a2f48e519@mail.gmail.com> Message-ID: Hello, My analysis shows that the exponential regression gives the best result (r^2=87%)--power regression gives worse results (r^2=77%). Untransformed data gives r^2=76%. I don't think you want lognorm. If I'm not mistaken, that fits the data to a log(normal distribution random variable). So, take the logarithm (to any base) of all the "conc" values. Then do a linear regression on those values versus "sizes". Try (semi-pseudocode): slope, intercept, p, error = scipy.stats.linregress(sizes,log(conc)) Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Nov 20 01:03:09 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 20 Nov 2009 00:03:09 -0600 Subject: [Numpy-discussion] datetime64 In-Reply-To: References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> Message-ID: > > we're starting to use these tools more and more, and with the 1.4 > release coming out, we're a bit lost here... A few specific questions > we have: The short answer to all of the questions is that datetime is not done yet. Robert and I worked on the underlying framework so that it could be rolled in to the trunk without causing segfaults. I have been trying to find time to work on the code since then and just got some time this week. I made good progress on the code and fixed a few bugs and added a few tests. If someone else has tests they have written they could add that would be very helpful at this point. I am planning to have a description of what remains to be implemented and where the code is by the end of day tomorrow. This should provide a roadmap of where we can go from here. I don't think the code needs to be taken out of 1.4.0 (and in-fact I would strongly discourage it). However, it would be appropriate to keep the description of datetime out of the release notes until the code can get implemented. The framework is fairly mature and I don't see changes occurring, but there are un-implemented features at this point. I will provide a summary tomorrow evening of what remains to be done. The largest body of work that needs to be done is the code to coerce between date-time types and to and from other data-types. What works now is that you can specify a date-time data-type and convert to and from python date-time objects. > > - It seems the original spec that was written by Francesc with a lot > of community feedback, whose final form (updated by Travis recently) > is: > > http://projects.scipy.org/numpy/browser/trunk/doc/neps/datetime-proposal.rst > > was implemented differently in the final committed code. There are a couple of enhancements in the final proposal, but otherwise the original spec was followed. The biggest motivation for these enhancements was to satisfy a use-case that was brought up by the group that funded much of the development. Other changes are more implementation details or driven by implementation. > > - Are there any tests for the current code, that will go in before 1.4 > is out? We'd like to know if we can rely on 1.4 as a dependency for > nitime, but for that we'd like to know what the code can actually do, > and also to have checks for what appear to be segfault problems: I am planning to work hard on this over the next week or so. I don't know how far I will get, but would like to push it as far as possible. Help is appreciated. > > - From our experimenting with the code it appears that the following > code is not accepted: > > a = np.array([0.1, 0.5, 0.9, 1.4], dtype='some form of spelling > seconds, nothing we've tried has worked') A bug in the code that was fixed today left seconds out.... However, the biggest issue is that coercions are not yet implemented so I would not expect any of this to work yet. > > - I know that the name numpy.datetime() was in the spec above > (Francesc's NEP) but we're wondering if having this is a good idea... I'm not in love with numpy.datetime or numpy.timedelta either. If better names can be suggested, I'm all for it. > > - Sources of information? Docstrings and tests haven't gotten us very > far, and googling points to the original spec as udpated by Travis: > > http://projects.scipy.org/numpy/browser/trunk/doc/neps/datetime-proposal.rst The spec is the best resource right now beyond the code. i have worked to keep the spec consistent with what is in (or will be in) the code. However, as mentioned, it is not finished. > > But even the examples in that document don't actually work. All of > these copied from the spec, and run with: Again, the NEP has not been fully implemented yet. What is implemented works as far as I can tell, but could use more tests. I would like to finish the core functionality before 1.4.0 and will try to do that, but there is still quite a bit of work to be done. If it is not finished, then we can keep the code there, and just document what works and what remains to be completed. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From hspotter2002 at gmail.com Fri Nov 20 02:21:20 2009 From: hspotter2002 at gmail.com (-) Date: Fri, 20 Nov 2009 02:21:20 -0500 Subject: [Numpy-discussion] Applying argwhere to an array Message-ID: <892060e80911192321s3161259gae6da89863fef8bb@mail.gmail.com> So if I have a 2-dimensional array, and I want to return the indexes of all the elements that are larger than some value, then I just use np.argwhere(d > 1) or something, where d is the array. However, is there anything I can do that's similar, but return the indexes of the elements that are larger than a value, where the value depends on the column index? Like if the column index is less than 5 then do np.argwhere(d > 1) and np.argwhere(d > 2) otherwise. I know I can split up the array and then apply these on both arrays, but then I end up with an unsorted list if I want the indexes to be sorted by rows, so I'd have to deal with that also, which would take up a lot of time. So if anyone knows what I should do, that would be great. Thanks. From robert.kern at gmail.com Fri Nov 20 02:36:21 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 20 Nov 2009 01:36:21 -0600 Subject: [Numpy-discussion] Applying argwhere to an array In-Reply-To: <892060e80911192321s3161259gae6da89863fef8bb@mail.gmail.com> References: <892060e80911192321s3161259gae6da89863fef8bb@mail.gmail.com> Message-ID: <3d375d730911192336t67f8e0dfqc9c0045fb912d2ad@mail.gmail.com> On Fri, Nov 20, 2009 at 01:21, - wrote: > So if I have a 2-dimensional array, and I want to return the indexes > of all the elements that are larger than some value, then I just use > np.argwhere(d > 1) or something, where d is the array. ?However, is > there anything I can do that's similar, but return the indexes of the > elements that are larger than a value, where the value depends on the > column index? ?Like if the column index is less than 5 then do > np.argwhere(d > 1) and np.argwhere(d > 2) otherwise. ?I know I can > split up the array and then apply these on both arrays, but then I end > up with an unsorted list if I want the indexes to be sorted by rows, > so I'd have to deal with that also, which would take up a lot of time. > So if anyone knows what I should do, that would be great. ?Thanks. Comparisons are just operators like +-/* and array broadcasting works on them, too. So, if d.shape == (N, 10), let's say, then you could do: np.argwhere(d > [1,1,1,1,1,2,2,2,2,2]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From d.l.goldsmith at gmail.com Fri Nov 20 03:47:26 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 20 Nov 2009 00:47:26 -0800 Subject: [Numpy-discussion] Numpy/Scipy for EC2 In-Reply-To: <15e4667e0911191745y6e2b3e20pceaddd05aa3c0554@mail.gmail.com> References: <15e4667e0910282129t4ad7f5eble1de56c91e6cff25@mail.gmail.com> <3d375d730910290841u78343cd2v4715cd40001bb09a@mail.gmail.com> <15e4667e0910290845te2f9217gf03efa635ff82bd4@mail.gmail.com> <15e4667e0911191745y6e2b3e20pceaddd05aa3c0554@mail.gmail.com> Message-ID: <45d1ab480911200047o2a2cb230i4cf75ce8211a8c58@mail.gmail.com> On Thu, Nov 19, 2009 at 5:45 PM, Dan Yamins wrote: > Hi all: > > I'm just writing to report on my experience using Starcluster, which > enables the use of NumPy and Scipy in the Amazon EC2 cloud computing > environment. The purpose of my email is to extol Starcluster's qualities, > and suggest that the NumPy community be aware of its development. I > suspect there are others in the community who find cloud computing an > attractive idea but a little daunting to get into, > Thanks, Dan, this is me (for one), and I appreciate you making the time and effort to do this. If enough of us dive into this, perhaps we could/should start a numpig... DG > and would be pleasantly surprised out how easy Starcluster makes it to get > started using NumPy on Amazon EC2. > > For those of you who aren't familiar with AMIs and the Amazon EC2 service, > see e.g. http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud. > Three of the basic concepts are "Amazon Machine Images" (AMIs), "machine > instances" of AMIs, and the Elastic Block Storage (EBS) service. AMIs are > disk images containing a virtual machine, including an operating system and > other software you add on. Instances are temporarily allocated computers, > booted with your chosen virtual machine, that you start up on demand, use > for computations with software from the AMI, and then terminate. EBS is a > persistent storage service, also from Amazon, that serves as permanent > file-systems in the cloud. You allocate an EBS volume of a given size, > attach the EBS volume(s) to a running machine instance just like any other > hard-drive, and use it to store the files you use/create during > computation, both during the computation and then for later use whenever you > start up a new instance. > > A couple of weeks ago I wrote to this list asking for advice on finding a > good Amazon Machine Instance (AMI) for using NumPy and Scipy on Amazon > cloud. I didn't want to have to build a linux machine image with optimized > blas and lapack myself, and I figured that there might be good existing > publicly-available AMIs that I could use as a base. Robert Kern suggested > that I look into the Starcluster project ( > http://web.mit.edu/stardev/cluster/). > > I have found Starcluster extremely useful. It made it possible for me to, > in the course of one day, go from knowing essentially nothing about cloud > something, to being able to run large-scale parallel clusters with my > favorite NumPy/SciPy-scripts. > > The basis of what Starcluster offers are two solidly-build AMIs. The > operating system is Ubuntu Jaunty, and comes with prebuilt optimized blas > and lapack, numpy, Scipy, matplotlib, ipython, and several other useful > packages for scientific computing in python. It uses Python 2.6, and comes > in both 32-bit and 64-bit flavors. The AMIs are based on AMIs from Alestic > (http://alestic.com/), and are built with best-practices for ensuring > stability and good interaction with Amazon's system. They have proved > very stable and extensible. > > In addition to these AMIs, Starcluster has three extremely useful features: > > -- Built-in support for mounting EBS drives as NFS filesystems**, and > then administering the shared drive across multiple machine instances. > -- The Sun Grid Engine (SGE), a queuing system for scheduling jobs to > be run in parallel across instances > -- A python module with a few commands that give you an incredibly > simple interface for automating the process of starting/terminating a > cluster of instances, mounting the shared drive, starting the grid engine, > &c -- and configuring your cluster needs (e.g. how many nodes it will > contain, which AMIs to use, which EBS volumes to mount etc.). > > As a result, all you have to do to have a NumPy-enabled cluster-on-demand > is: > 1) Get an amazon EC2 account, and the accompanying security credentials > (.501 certificates and PGP keypair) for your account. > 2) Install starcluster ("easy_install starcluster") > 3) Follow the installation procedure on the starcluster website for > getting, attaching, and formatting an EBS volume as an NFS drive. > 4) Set up your starcluster configuration file. > 5) Start a 1-node cluster, modify the installation as you see fit, and > re-bundle the result into a new AMI as described on the Amazon website > http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/. > (Don't forget to edit your starcluster configuration file to reflect your > new AMI.) This step is optional -- If you don't need anything else > special, you can just use Starcluster's base images. > > After that, starting a cluster is as easy as typing single command > ("starcluster -s"). To submit parallel jobs on your cluster, you can learn > to use the Sun Grid Engine "qsub" command ( > http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/htmlman/htmlman1/qsub.html) > or use the python bindings to the SGE interface ( > http://code.google.com/p/drmaa-python/). Or, if you like Parallel > Python, that works perfectly well on these clusters too. > > Overall, in my experience, Starcluster has been easy, stable and powerful, > and I encourage anyone who is curious about cloud computing with Numpy to > look into it. > > Starcluster is by no means a finished project. At the moment, you can only > administer one cluster at a time from your given local machine, since > starcluster has no notion of a "session" and it can't distinguish between > different clusters you've started up (you can *start* multiple clusters, > but then any starcluster commands that you type in your local terminal might > get confused about which amazon machine instances you're referring to, so it > has trouble administering them.) Also, there's no dynamic load balancing, > so once you've started a cluster with a certain number of nodes, you're > stuck with that number of computers while the cluster is running, even if > you're only using a few of them or suddenly need more. > > The developer of the project (*Justin Riley)* says on his website that > he's planning to add these features in the next release. Now, I'm not the > creator or developer or maintainer of Starcluster, and I have no affiliation > with Justin Riley or the project whatsoever, so I want to make it clear I > don't speak for them in any way except as a satisfied user. I don't know > what his commitment to his development plans are, either -- however, I hope > he sticks to his timeline, as I think continuing the vigorous development of > his project would be a real plus for the NumPy community. I'm hoping that > if others in the NumPy community like his project and start using it, that > will make add to the likelihood of continued development. (If anyone from > the NumPy community is interesting in helping the developer out, perhaps you > should consider shooting him an email.) > > Anyhow, I apologize for this long email, and hope it may be of use to > somebody! > > Dan > > > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Fri Nov 20 03:47:23 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 20 Nov 2009 00:47:23 -0800 Subject: [Numpy-discussion] datetime64 In-Reply-To: References: <43958ee60910291218i322d94b0y2fe6e362b876ef8b@mail.gmail.com> <20091029202231.GA22121@virginia.edu> <41F46D98-37B4-4E88-BBEC-FCE2CE3C2E6A@gmail.com> Message-ID: Hi Travis, [...] On Thu, Nov 19, 2009 at 10:03 PM, Travis Oliphant wrote: > Again, the NEP has not been fully implemented yet. ? What is implemented > works as far as I can tell, but could use more tests. ? I would like to > finish the core functionality before 1.4.0 and will try to do that, but > there is still quite a bit of work to be done. > If it is not finished, then we can keep the code there, and just document > what works and what remains to be completed. thanks for your detailed response. It seems that you're on the ball for this (though I'm sure you can use any help available), which is great to hear. We don't need this *now*, I just wanted to get a better understanding of the situation before 1.4 was frozen, as there hadn't been much response. But the current plan sounds great, and from what I can see you have all the points we've seen so far already in mind. I hope the coercion work doesn't prove too difficult or time consuming, as right now the pure datetime.datetime() API feels rather unwieldy. Regarding naming: how about np.timestamp -> for what today is np.datetime np.timedelta -> stays the same I actually *don't* like the 'date' in datetime, it feels redundant and pointless. It's pretty clear to me that a date has to do with time in a library. The above names, at least to me, express the intent of representing absolute and relative time information fairly clearly, there's a certain nice symmetry to them, and they prevent stdlib name collisions. Just an idea... Thanks again for your detailed reply! Cheers, f From pav+sp at iki.fi Fri Nov 20 04:50:13 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Fri, 20 Nov 2009 09:50:13 +0000 (UTC) Subject: [Numpy-discussion] Failure building pdf doc References: <5b8d13220911191755i5077ad00t6a5b419f4b10684d@mail.gmail.com> Message-ID: Fri, 20 Nov 2009 10:55:35 +0900, David Cournapeau wrote: > While checking everything builds for the 1.4.0 release, I noticed a > problem with building the latex version: > > writing... done > processing numpy-user.tex... user/index user/introduction > user/whatisnumpy user/install user/howtofind user/basics > user/basics.types user/basics.creation user/basics.io > user/basics.io.genfromtxt user/basics.indexing user/basics.broadcasting > user/basics.byteswapping user/basics.rec user/basics.subclassing > user/performance user/misc user/c-info user/c-info.how-to-extend > user/c-info.python-as-glue user/c-info.beyond-basics > resolving references... > writing... > Markup is unsupported in LaTeX: > user/basics.io.genfromtxt:: column or row spanning cells are not yet > implemented. > > How can I fix this ? I noticed a similar bug reported on sphinx bug > tracker 10 months ago by Pauli, is this indeed related ? I think I fixed this in r7751. -- Pauli Virtanen From faltet at pytables.org Fri Nov 20 05:00:50 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 20 Nov 2009 11:00:50 +0100 Subject: [Numpy-discussion] memmap limits In-Reply-To: <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> References: <7d0c05ac0911181543l591e2d5i56e44e8a5172bdbd@mail.gmail.com> <3d375d730911181548n32a4af57i6c1e0fba2835d08@mail.gmail.com> Message-ID: <200911201100.51056.faltet@pytables.org> A Thursday 19 November 2009 00:48:13 Robert Kern escrigu?: > On Wed, Nov 18, 2009 at 17:43, Mathew Yeates wrote: > > What limits are there on file size when using memmap? > > With a modern filesystem, usually you are only limited to the amount > of contiguous free space in your process's current address space. This is usually the same than your available virtual memory, that is, your amount of RAM + the amount of SWAP space. -- Francesc Alted From kwgoodman at gmail.com Fri Nov 20 10:51:10 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 20 Nov 2009 07:51:10 -0800 Subject: [Numpy-discussion] Correlation filter Message-ID: I have a short 1d array x and a large 2d array y. I'd like to locate the places in the y array that are most like (correlated to) the x array. My first attempt, corr1, is too slow. My second attempt, corr2, is faster but still slow. I reuse the same y many times, so my third attempt will probably be to calculate the moving mean of y outside of the function. But before doing that, I was wondering if there is any existing code that could help me. It seems like this would be a common filter-type operation. Or could stacking several filter operations like mean and product do the trick? I don't need the actual correlation. I just need an output that preserves the ranking of the correlation. For benchmarking I am using x of shape (5,) and y of shape (500,500): x = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) y = np.random.randn(500, 500) def corr1(x, y): d = np.nan * np.ones_like(y) for i in range(y.shape[0]): yi = y[i,:] for j in range(x.shape[0]-1, y.shape[1]): yj = yi[j+1-x.shape[0]:j+1] d[i,j] = np.corrcoef(x, yj)[0,1] return d def corr2(x, y): dot = np.dot sqrt = np.sqrt d = np.nan * np.ones_like(y) x = x - x.mean() x /= x.std() x = np.tile(x, (y.shape[0],1)) nx = x.shape[1] one = np.ones(nx) / nx for i in range(nx-1, y.shape[1]): yi = y[:,i+1-nx:i+1] yi = yi - dot(yi, one).reshape(-1,1) yi /= sqrt(dot(yi*yi, one)).reshape(-1,1) d[:,i] = dot(x * yi, one) return d >> timeit corr1(x,y) 10 loops, best of 3: 13.3 s per loop >> timeit corr2(x,y) 10 loops, best of 3: 60.5 ms per loop From josef.pktd at gmail.com Fri Nov 20 11:53:51 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Nov 2009 11:53:51 -0500 Subject: [Numpy-discussion] Correlation filter In-Reply-To: References: Message-ID: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> On Fri, Nov 20, 2009 at 10:51 AM, Keith Goodman wrote: > I have a short 1d array x and a large 2d array y. I'd like to locate > the places in the y array that are most like (correlated to) the x > array. > > My first attempt, corr1, is too slow. My second attempt, corr2, is > faster but still slow. > > I reuse the same y many times, so my third attempt will probably be to > calculate the moving mean of y outside of the function. But before > doing that, I was wondering if there is any existing code that could > help me. It seems like this would be a common filter-type operation. > Or could stacking several filter operations like mean and product do > the trick? > > I don't need the actual correlation. I just need an output that > preserves the ranking of the correlation. For benchmarking I am using > x of shape (5,) and y of shape (500,500): > > x = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) > y = np.random.randn(500, 500) > > def corr1(x, y): > ? ?d = np.nan * np.ones_like(y) > ? ?for i in range(y.shape[0]): > ? ? ? ?yi = y[i,:] > ? ? ? ?for j in range(x.shape[0]-1, y.shape[1]): > ? ? ? ? ? ?yj = yi[j+1-x.shape[0]:j+1] > ? ? ? ? ? ?d[i,j] = np.corrcoef(x, yj)[0,1] > ? ?return d > > def corr2(x, y): > ? ?dot = np.dot > ? ?sqrt = np.sqrt > ? ?d = np.nan * np.ones_like(y) > ? ?x = x - x.mean() > ? ?x /= x.std() > ? ?x = np.tile(x, (y.shape[0],1)) > ? ?nx = x.shape[1] > ? ?one = np.ones(nx) / nx > ? ?for i in range(nx-1, y.shape[1]): > ? ? ? ?yi = y[:,i+1-nx:i+1] > ? ? ? ?yi = yi - dot(yi, one).reshape(-1,1) > ? ? ? ?yi /= sqrt(dot(yi*yi, one)).reshape(-1,1) > ? ? ? ?d[:,i] = dot(x * yi, one) > ? ?return d > >>> timeit corr1(x,y) > 10 loops, best of 3: 13.3 s per loop >>> timeit corr2(x,y) > 10 loops, best of 3: 60.5 ms per loop > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > scipy.signal.correlate would be fast, but it will not be easy to subtract the correct moving mean. Subtracting a standard moving mean would subtract different values for each observation in the window. One possibility would be to look at a moving regression and just take the estimated slope parameter. John D'Errico http://www.mathworks.com/matlabcentral/fileexchange/16997-movingslope (BSD licensed) uses a very nice trick with pinv to get the filter to calculate a moving slope coefficient. I read the source but didn't try it out and didn't try to figure out how the pinv trick exactly works. If this can be adapted to your case, then this would be the fastest I can think of (pinv and scipy.signal.correlate would do everything in C, or maybe one (500) loop might still be necessary) For just getting a ranking on a linear relationship, there might be other tricks possible, local linear regression, ... (?), but I never tried. Also, I think with one time loop, you can do all cross section regressions at the same time, without tricks. Josef From gokhansever at gmail.com Fri Nov 20 13:27:48 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 20 Nov 2009 12:27:48 -0600 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data In-Reply-To: References: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> <49d6b3500911170929rccaa0e7k75f1450a2f48e519@mail.gmail.com> Message-ID: <49d6b3500911201027v46a1dd7dm6072f0ee2092aa77@mail.gmail.com> On Thu, Nov 19, 2009 at 9:12 PM, Ian Mallett wrote: > Hello, > > My analysis shows that the exponential regression gives the best result > (r^2=87%)--power regression gives worse results (r^2=77%). Untransformed > data gives r^2=76%. > > I don't think you want lognorm. If I'm not mistaken, that fits the data to > a log(normal distribution random variable). > Lognormal fitting is the motivation behind my study since aerosol in the atmosphere typically log-normally size distributed. See for an example case : http://webmathematica.seas.harvard.edu/webMathematica/AerosolCalculator.jsp Of course this is just a simplification. There are other approaches to represent the size-distribution besides the lognormal. So my intention is not actually find the best fit but represent the actuality as much as I can. > > So, take the logarithm (to any base) of all the "conc" values. Then do a > linear regression on those values versus "sizes". > > Try (semi-pseudocode): > slope, intercept, p, error = scipy.stats.linregress(sizes,log(conc)) > linregress also returns the r_value which I am not sure if the documentation from the web-based editor checked-in completely to the scipy trunk yet. > > Ian > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Nov 20 13:51:01 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 20 Nov 2009 10:51:01 -0800 Subject: [Numpy-discussion] Correlation filter In-Reply-To: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> References: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> Message-ID: On Fri, Nov 20, 2009 at 8:53 AM, wrote: > scipy.signal.correlate ?would be fast, but it will not be easy to > subtract the correct moving mean. Subtracting a standard moving mean > would subtract different values for each observation in the window. > > One possibility would be to look at a moving regression and just take > the estimated slope parameter. John D'Errico > http://www.mathworks.com/matlabcentral/fileexchange/16997-movingslope > (BSD licensed) > uses a very nice trick with pinv to get the filter to calculate a > moving slope coefficient. I read the source but didn't try it out and > didn't try to figure out how the pinv trick exactly works. > If this can be adapted to your case, then this would be the fastest I > can think of (pinv and scipy.signal.correlate would do everything in > C, or maybe one (500) loop might still be necessary) > > For just getting a ranking on a linear relationship, there might be > other tricks possible, local linear regression, ... (?), but I never > tried. Also, I think with one time loop, you can do all cross section > regressions at the same time, without tricks. Those sound like good ideas. I was able to get rid of the for loop (and cut the time in half) by using lfilter from scipy: def corr3(x, y): x = x - x.mean() x /= x.std() nx = x.size one = np.ones(nx) xy = lfilter(x, 1, y) sy = lfilter(one, 1, y) sy2 = lfilter(one, 1, y*y) d = xy / np.sqrt(nx * sy2 - sy * sy) return d (Somehow I managed to flip the sign of the correlation in corr3.) I can trim a little more time by doing some of the operations in place but then the code becomes hard to read. This solution will work well for me since I can move much of the calculation outside of the function and reuse it across many calls (y does not change often compared to x). From josef.pktd at gmail.com Fri Nov 20 14:17:09 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Nov 2009 14:17:09 -0500 Subject: [Numpy-discussion] Correlation filter In-Reply-To: References: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> Message-ID: <1cd32cbb0911201117t5817da8dice8a4e822e390d07@mail.gmail.com> On Fri, Nov 20, 2009 at 1:51 PM, Keith Goodman wrote: > On Fri, Nov 20, 2009 at 8:53 AM, ? wrote: >> scipy.signal.correlate ?would be fast, but it will not be easy to >> subtract the correct moving mean. Subtracting a standard moving mean >> would subtract different values for each observation in the window. >> >> One possibility would be to look at a moving regression and just take >> the estimated slope parameter. John D'Errico >> http://www.mathworks.com/matlabcentral/fileexchange/16997-movingslope >> (BSD licensed) >> uses a very nice trick with pinv to get the filter to calculate a >> moving slope coefficient. I read the source but didn't try it out and >> didn't try to figure out how the pinv trick exactly works. >> If this can be adapted to your case, then this would be the fastest I >> can think of (pinv and scipy.signal.correlate would do everything in >> C, or maybe one (500) loop might still be necessary) >> >> For just getting a ranking on a linear relationship, there might be >> other tricks possible, local linear regression, ... (?), but I never >> tried. Also, I think with one time loop, you can do all cross section >> regressions at the same time, without tricks. > > Those sound like good ideas. > > I was able to get rid of the for loop (and cut the time in half) by > using lfilter from scipy: > > def corr3(x, y): > ? ?x = x - x.mean() > ? ?x /= x.std() > ? ?nx = x.size > ? ?one = np.ones(nx) > ? ?xy = lfilter(x, 1, y) > ? ?sy = lfilter(one, 1, y) > ? ?sy2 = lfilter(one, 1, y*y) > ? ?d = xy / np.sqrt(nx * sy2 - sy * sy) > ? ?return d Is this correct? xy uses the original y and not a demeaned y. I think your use of lfilter only uses features that are also available with correlate, and signal correlate can handle ndim arrays. I use correlate in my moving moment calculations http://bazaar.launchpad.net/~scipystats/statsmodels/trunk/annotate/head%3A/scikits/statsmodels/sandbox/tsa/movstat.py#L207 but I think I only tested 1d. Josef > > (Somehow I managed to flip the sign of the correlation in corr3.) > > I can trim a little more time by doing some of the operations in place > but then the code becomes hard to read. This solution will work well > for me since I can move much of the calculation outside of the > function and reuse it across many calls (y does not change often > compared to x). > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Fri Nov 20 14:28:49 2009 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 20 Nov 2009 11:28:49 -0800 Subject: [Numpy-discussion] Correlation filter In-Reply-To: <1cd32cbb0911201117t5817da8dice8a4e822e390d07@mail.gmail.com> References: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> <1cd32cbb0911201117t5817da8dice8a4e822e390d07@mail.gmail.com> Message-ID: On Fri, Nov 20, 2009 at 11:17 AM, wrote: > On Fri, Nov 20, 2009 at 1:51 PM, Keith Goodman wrote: >> def corr3(x, y): >> ? ?x = x - x.mean() >> ? ?x /= x.std() >> ? ?nx = x.size >> ? ?one = np.ones(nx) >> ? ?xy = lfilter(x, 1, y) >> ? ?sy = lfilter(one, 1, y) >> ? ?sy2 = lfilter(one, 1, y*y) >> ? ?d = xy / np.sqrt(nx * sy2 - sy * sy) >> ? ?return d > > Is this correct? xy uses the original y and not a demeaned y. I wouldn't be surprised if I made mistakes. But I don't think I need to demean y. One way to write the numerator of the correlation coefficent is N*sum(xy) - sum(x)*sum(y) but sum(x) is zero, so it becomes N*sum(xy) So I think I only need to demean x. But I'd better test the code. Plus I have no idea how lfilter will handle my missing values (NaN). From josef.pktd at gmail.com Fri Nov 20 16:27:47 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Nov 2009 16:27:47 -0500 Subject: [Numpy-discussion] Correlation filter In-Reply-To: References: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> <1cd32cbb0911201117t5817da8dice8a4e822e390d07@mail.gmail.com> Message-ID: <1cd32cbb0911201327j7b6bee04u6fc9cf637239a1cd@mail.gmail.com> On Fri, Nov 20, 2009 at 2:28 PM, Keith Goodman wrote: > On Fri, Nov 20, 2009 at 11:17 AM, ? wrote: >> On Fri, Nov 20, 2009 at 1:51 PM, Keith Goodman wrote: >>> def corr3(x, y): >>> ? ?x = x - x.mean() >>> ? ?x /= x.std() >>> ? ?nx = x.size >>> ? ?one = np.ones(nx) >>> ? ?xy = lfilter(x, 1, y) >>> ? ?sy = lfilter(one, 1, y) >>> ? ?sy2 = lfilter(one, 1, y*y) >>> ? ?d = xy / np.sqrt(nx * sy2 - sy * sy) >>> ? ?return d >> >> Is this correct? xy uses the original y and not a demeaned y. > > I wouldn't be surprised if I made mistakes. But I don't think I need > to demean y. One way to write the numerator of the correlation > coefficent is > > N*sum(xy) - sum(x)*sum(y) > > but sum(x) is zero, so it becomes > > N*sum(xy) > > So I think I only need to demean x. But I'd better test the code. Plus > I have no idea how lfilter will handle my missing values (NaN). I think nans fully propagate, trick might be to fill with zero and use another convolution/correlation with the non-nan mask to get the number of observations included (I think I saw it as a trick by Pierre for autocorrelation with missing observations). Here is a moving slope function which estimate the slope coefficient for a linear trend if x is arange() I compared the ranking with your moving correlation and it is quite different. def moving_slope(x,y) '''estimate moving slope coefficient of regression of y on x filters along axis=1, returns valid observations Todo: axis and lag options ''' xx = np.column_stack((np.ones(x.shape), x)) pinvxx = np.linalg.pinv(xx)[1:,:] windsize = len(x) lead = windsize//2 - 1 return signal.correlate(y, pinvxx, 'full' )[:,windsize-lead:-(windsize+1*lead-2)] Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dyamins at gmail.com Sat Nov 21 15:27:33 2009 From: dyamins at gmail.com (Dan Yamins) Date: Sat, 21 Nov 2009 15:27:33 -0500 Subject: [Numpy-discussion] Problems casting object arrays to string type on Ubuntu Message-ID: <15e4667e0911211227rd1cbe70g852702c51878ae26@mail.gmail.com> Hi all, I'm having some issues casting object arrays to string type, especially on my Ubuntu installation. (Ubuntu Jaunty, 9.04, with Numpy v. 1.3.) With small arrays, the conversion is just wrong. With large arrays, there seems to be some memory corruption. Conversion to int or float (when appropriate) seems to work. For instance, here are some examples of errors with small arrays: >>> X = numpy.array([13,14],'object') >>> X.astype('|S2') array(['13', '\xb2'], dtype='|S2') >>> X.astype('int') #conversion to int or float seems to work fine array([13, 14]) >>> X.astype(float) array([ 13., 14.]) >>> X = numpy.array(['cat','bat'],'object') >>> X.astype('|S3') array(['cat', '\x00ba'], dtype='|S3') Large arrays: In [24]: X = numpy.array(['cat','bat']*300000,'object') In [25]: Y = X.astype('|S3') *** glibc detected *** /usr/bin/python: munmap_chunk(): invalid pointer: 0xb7cd5008 *** I do NOT have this problem with Numpy 1.4.0.dev7746, installed on my OSX 10.5.8 machine. There, everything seems to work fine. What's going on? I feel like I've seem some traffic about related issues on the list before, but I couldn't quite tell from reading the threads what the "final upshot" of the discussion was ... Is this something that was fixed in recent NumPy 1.4 releases, or is something about my Ubuntu vs. OSX installations? Generally speaking can I / should I be relying on casting from object arrays to do the "right" (or "intuitive") thing? thanks, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Sat Nov 21 16:57:10 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sat, 21 Nov 2009 15:57:10 -0600 Subject: [Numpy-discussion] Fitting a curve on a log-normal distributed data In-Reply-To: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> References: <49d6b3500911162144x1193e04cj1a103776092c4471@mail.gmail.com> Message-ID: <49d6b3500911211357h4b4870c8q7355c2d32db55fe6@mail.gmail.com> One more update on this subject. I have been looking through some of the papers on this topic, and I have finally found exactly what I need in this paper: Hussein, T., Dal Maso, M., Petaja, T., Koponen, I. K., Paatero, P., Aalto, P. P., Hameri, K., and Kulmala, M.: Evaluation of an automatic algorithm for ?tting the particle number size distributions, Boreal Environ. Res., 10, 337?355, 2005. Here is the abstract: "The multi log-normal distribution function is widely in use to parameterize the aerosol particle size distributions. The main purpose of such a parameterization is to quantitatively describe size distributions and to allow straightforward comparisons between different aerosol particle data sets. In this study, we developed and evaluated an algorithm to parameterize aerosol particle number size distributions with the multi log-normal distribution function. The current algorithm is automatic and does not need a user decision for the initial input parameters; it requires only the maximum number of possible modes and then it reduces this number, if possible, without affecting the fitting quality. The reduction of the number of modes is based on an overlapping test between adjacent modes. The algorithm was evaluated against a previous algorithm that can be considered as a standard procedure. It was also evaluated against a long-term data set and different types of measured aerosol particle size distributions in the ambient atmosphere. The evaluation of the current algorithm showed the following advantages: (I) it is suitable for different types of aerosol particles observed in different environments and conditions, (2) it showed agreement with the previous standard algorithm in about 90% of long-term data set, (3) it is not time-consuming, particularly when long-term data sets are analyzed, and (4) it is a useful tool in the studies of atmospheric aerosol particle formation and transformation." The full-text is freely available at: http://www.borenv.net/BER/pdfs/ber10/ber10-337.pdf On Mon, Nov 16, 2009 at 11:44 PM, G?khan Sever wrote: > Hello, > > I have a data which represents aerosol size distribution in between 0.1 to > 3.0 micrometer ranges. I would like extrapolate the lower size down to 10 > nm. The data in this context is log-normally distributed. Therefore I am > looking a way to fit a log-normal curve onto my data. Could you please give > me some pointers to solve this problem? > > Thank you. > > -- > G?khan > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Sat Nov 21 20:00:33 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sat, 21 Nov 2009 20:00:33 -0500 Subject: [Numpy-discussion] 2to3c: 2to3 for C extensions Message-ID: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> Guido posted a link to this on Twitter, it might be of interest to people interested in the NumPy Python 3 transition: http://dmalcolm.livejournal.com/3935.html It's doubtful this is a magic bullet (at least, not yet) but it may still help reduce the amount of busywork involved. David From cournape at gmail.com Sun Nov 22 02:35:32 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 22 Nov 2009 16:35:32 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> Message-ID: <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> On Wed, Nov 18, 2009 at 6:10 PM, David Cournapeau wrote: > On Wed, Nov 18, 2009 at 12:38 AM, ? wrote: > >> >> Now numpy builds without problems. >> When I run the tests I get 16 failures mostly nan related. I have no idea >> whether they are real or if there is still something screwed up in my >> setup. See below. > > I can reproduce those. I did not see those because they are python 2.5 > specific (which is puzzling). I fix most of those, which were related to an atan2 bug in MS C library. Could you test whether this fixes it for you ? David From josef.pktd at gmail.com Sun Nov 22 08:28:22 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Nov 2009 08:28:22 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> Message-ID: <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> On Sun, Nov 22, 2009 at 2:35 AM, David Cournapeau wrote: > On Wed, Nov 18, 2009 at 6:10 PM, David Cournapeau wrote: >> On Wed, Nov 18, 2009 at 12:38 AM, ? wrote: >> >>> >>> Now numpy builds without problems. >>> When I run the tests I get 16 failures mostly nan related. I have no idea >>> whether they are real or if there is still something screwed up in my >>> setup. See below. >> >> I can reproduce those. I did not see those because they are python 2.5 >> specific (which is puzzling). > > I fix most of those, which were related to an atan2 bug in MS C > library. Could you test whether this fixes it for you ? > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Thanks, I have 4 failures remaining. full ouput below Josef >python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.4.0.dev7758 NumPy is installed in C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.de v7758.win32\Programs\Python25\Lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.11.1 ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ .......................K.....................................................FF. ............................K......................K.F.......................... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ .....................................C:\Josef\_progs\Subversion\numpy-trunk\dist \numpy-1.4.0.dev7758.win32\Programs\Python25\Lib\site-packages\numpy\lib\io.py:1 324: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 5) Line #12 (got 4 columns instead of 5) Line #22 (got 4 columns instead of 5) Line #32 (got 4 columns instead of 5) Line #42 (got 4 columns instead of 5) warnings.warn(errmsg, ConversionWarning) .C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Programs\ Python25\Lib\site-packages\numpy\lib\io.py:1324: ConversionWarning: Some errors were detected ! Line #2 (got 4 columns instead of 2) Line #12 (got 4 columns instead of 2) Line #22 (got 4 columns instead of 2) Line #32 (got 4 columns instead of 2) Line #42 (got 4 columns instead of 2) warnings.warn(errmsg, ConversionWarning) ..........................................K........K............F............... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ......S......................................................................... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................... ====================================================================== FAIL: test_umath.test_nextafter ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 851, in test_nextafter assert np.nextafter(one, two) - one == eps AssertionError ====================================================================== FAIL: test_umath.test_spacing ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 863, in test_spacing assert np.spacing(one) == eps AssertionError ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 179, in test_special_values assert_almost_equal(np.log(x), y) File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 436, in assert_almost_equal AssertionError: Items are not equal: ACTUAL: [ NaN+2.35619449j] DESIRED: (1.#INF+2.35619449019j) ====================================================================== FAIL: test_doctests (test_polynomial.TestDocs) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr ograms\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line 90, in test_doctests return rundocs() File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 952, in rundocs AssertionError: Some doctests failed: ********************************************************************** File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Prog rams\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line 20, in test_polynomial Failed example: print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) Expected: 2 1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 Got: 2 1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 ---------------------------------------------------------------------- Ran 2329 tests in 15.141s FAILED (KNOWNFAIL=5, SKIP=1, failures=4) >>> From charlesr.harris at gmail.com Sun Nov 22 09:45:39 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 22 Nov 2009 07:45:39 -0700 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> Message-ID: On Sun, Nov 22, 2009 at 6:28 AM, wrote: > On Sun, Nov 22, 2009 at 2:35 AM, David Cournapeau > wrote: > > On Wed, Nov 18, 2009 at 6:10 PM, David Cournapeau > wrote: > >> On Wed, Nov 18, 2009 at 12:38 AM, wrote: > >> > >>> > >>> Now numpy builds without problems. > >>> When I run the tests I get 16 failures mostly nan related. I have no > idea > >>> whether they are real or if there is still something screwed up in my > >>> setup. See below. > >> > >> I can reproduce those. I did not see those because they are python 2.5 > >> specific (which is puzzling). > > > > I fix most of those, which were related to an atan2 bug in MS C > > library. Could you test whether this fixes it for you ? > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > Thanks, > > I have 4 failures remaining. full ouput below > > Josef > > >python > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Running unit tests for numpy > NumPy version 1.4.0.dev7758 > NumPy is installed in C:\Josef\_progs\Subversion\numpy-trunk\dist\ > numpy-1.4.0.de > v7758.win32\Programs\Python25\Lib\site-packages\numpy > Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Int > el)] > nose version 0.11.1 > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > .......................K.....................................................FF. > > ............................K......................K.F.......................... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > .....................................C:\Josef\_progs\Subversion\numpy-trunk\dist > > \numpy-1.4.0.dev7758.win32\Programs\Python25\Lib\site-packages\numpy\lib\io.py:1 > 324: ConversionWarning: Some errors were detected ! > Line #2 (got 4 columns instead of 5) > Line #12 (got 4 columns instead of 5) > Line #22 (got 4 columns instead of 5) > Line #32 (got 4 columns instead of 5) > Line #42 (got 4 columns instead of 5) > warnings.warn(errmsg, ConversionWarning) > > .C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Programs\ > Python25\Lib\site-packages\numpy\lib\io.py:1324: ConversionWarning: Some > errors > were detected ! > Line #2 (got 4 columns instead of 2) > Line #12 (got 4 columns instead of 2) > Line #22 (got 4 columns instead of 2) > Line #32 (got 4 columns instead of 2) > Line #42 (got 4 columns instead of 2) > warnings.warn(errmsg, ConversionWarning) > > ..........................................K........K............F............... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ......S......................................................................... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > ................................................... > ====================================================================== > FAIL: test_umath.test_nextafter > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p > y", line 183, in runTest > self.test(*self.arg) > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line > 851, in > test_nextafter > assert np.nextafter(one, two) - one == eps > AssertionError > > ====================================================================== > FAIL: test_umath.test_spacing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p > y", line 183, in runTest > self.test(*self.arg) > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line > 863, in > test_spacing > assert np.spacing(one) == eps > AssertionError > > Might be nice to print out the actual values of np.spacing and np.nextafter here. > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", > line > 179, in test_special_values > assert_almost_equal(np.log(x), y) > File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line > 436, > in assert_almost_equal > AssertionError: Items are not equal: > ACTUAL: [ NaN+2.35619449j] > DESIRED: (1.#INF+2.35619449019j) > > > ====================================================================== > FAIL: test_doctests (test_polynomial.TestDocs) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line > 90, > in test_doctests > return rundocs() > File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line > 952, > in rundocs > AssertionError: Some doctests failed: > ********************************************************************** > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Prog > rams\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line > 20, in > test_polynomial > Failed example: > print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) > Expected: > 2 > 1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 > Got: > 2 > 1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 > > > I don't know what to do about the last, it comes from the poly1d pretty printing and relies on python for the formatting. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Nov 22 10:01:44 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 23 Nov 2009 00:01:44 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> Message-ID: <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris wrote: > > Might be nice to print out the actual values of np.spacing and np.nextafter > here. > Yes, I should add some utilities to print those for this kind of test. But in this case, I know the problem: mingw gcc use 80 bits long double but it uses the MS runtime which considers double and long double to be the same (8 bytes). I think the real fix is to force npy_longdouble to be double on mingw, but I don't want to make that change now for 1.4.0. David From josef.pktd at gmail.com Sun Nov 22 10:14:30 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Nov 2009 10:14:30 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> Message-ID: <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau wrote: > On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris > wrote: >> >> Might be nice to print out the actual values of np.spacing and np.nextafter >> here. >> > > Yes, I should add some utilities to print those for this kind of test. > But in this case, I know the problem: mingw gcc use 80 bits long > double but it uses the MS runtime which considers double and long > double to be the same (8 bytes). > > I think the real fix is to force npy_longdouble to be double on mingw, > but I don't want to make that change now for 1.4.0. adding the failing type in the test to the failure message would also be helpful from test_umath test_spacing(): >>> for t in [np.float32, np.float64, np.longdouble]: ... one = t(1) ... eps = np.finfo(t).eps ... nan = t(np.nan) ... inf = t(np.inf) ... print t ... print np.spacing(one), eps ... print np.isnan(np.spacing(nan)) ... print np.isnan(np.spacing(inf)) ... print np.isnan(np.spacing(-inf)) ... 1.19209e-007 1.19209e-007 True True True 2.22044604925e-016 2.22044604925e-016 True True True 1.08420217249e-019 2.22044604925e-016 True True True Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Sun Nov 22 10:25:32 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 23 Nov 2009 00:25:32 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> Message-ID: <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> On Mon, Nov 23, 2009 at 12:14 AM, wrote: > On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau wrote: >> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris >> wrote: >>> >>> Might be nice to print out the actual values of np.spacing and np.nextafter >>> here. >>> >> >> Yes, I should add some utilities to print those for this kind of test. >> But in this case, I know the problem: mingw gcc use 80 bits long >> double but it uses the MS runtime which considers double and long >> double to be the same (8 bytes). >> >> I think the real fix is to force npy_longdouble to be double on mingw, >> but I don't want to make that change now for 1.4.0. > > adding the failing type in the test to the failure message would also > be helpful Yes, you're right. I also used the nose facility for using generators for complex corner cases, but with retrospect, it is not so useful, because you don't get a name when you have a failure (or maybe I am using it wrong). David From josef.pktd at gmail.com Sun Nov 22 10:37:12 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Nov 2009 10:37:12 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> Message-ID: <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> On Sun, Nov 22, 2009 at 10:25 AM, David Cournapeau wrote: > On Mon, Nov 23, 2009 at 12:14 AM, ? wrote: >> On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau wrote: >>> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris >>> wrote: >>>> >>>> Might be nice to print out the actual values of np.spacing and np.nextafter >>>> here. >>>> >>> >>> Yes, I should add some utilities to print those for this kind of test. >>> But in this case, I know the problem: mingw gcc use 80 bits long >>> double but it uses the MS runtime which considers double and long >>> double to be the same (8 bytes). >>> >>> I think the real fix is to force npy_longdouble to be double on mingw, >>> but I don't want to make that change now for 1.4.0. >> >> adding the failing type in the test to the failure message would also >> be helpful > > Yes, you're right. I also used the nose facility for using generators > for complex corner cases, but with retrospect, it is not so useful, > because you don't get a name when you have a failure (or maybe I am > using it wrong). I don't know what the policy for the use of assert in numpy is, but if you use the function np.testing.assert_ then you can add a failure message with eg. repr(t) With "yield check_a_function arguments" nose prints the arguments in the test description, putting information into the arguments gets then displayed. This is also useful for arguments that the check function doesn't really need. Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Sun Nov 22 10:47:55 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Nov 2009 10:47:55 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> Message-ID: <1cd32cbb0911220747k7d8897b0q7b984fc779a8da7a@mail.gmail.com> On Sun, Nov 22, 2009 at 10:37 AM, wrote: > On Sun, Nov 22, 2009 at 10:25 AM, David Cournapeau wrote: >> On Mon, Nov 23, 2009 at 12:14 AM, ? wrote: >>> On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau wrote: >>>> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris >>>> wrote: >>>>> >>>>> Might be nice to print out the actual values of np.spacing and np.nextafter >>>>> here. >>>>> >>>> >>>> Yes, I should add some utilities to print those for this kind of test. >>>> But in this case, I know the problem: mingw gcc use 80 bits long >>>> double but it uses the MS runtime which considers double and long >>>> double to be the same (8 bytes). >>>> >>>> I think the real fix is to force npy_longdouble to be double on mingw, >>>> but I don't want to make that change now for 1.4.0. >>> >>> adding the failing type in the test to the failure message would also >>> be helpful >> >> Yes, you're right. I also used the nose facility for using generators >> for complex corner cases, but with retrospect, it is not so useful, >> because you don't get a name when you have a failure (or maybe I am >> using it wrong). > > I don't know what the policy for the use of assert in numpy is, > but if you use the function np.testing.assert_ then you can add a > failure message with eg. repr(t) > > With "yield check_a_function arguments" nose prints the arguments > ?in the test description, putting information into the arguments gets > then displayed. This is also useful for arguments that the check > function doesn't really need. example: I change the test to this, then the new test failure output is as below def test_spacing(): for t in [np.float32, np.float64, np.longdouble]: one = t(1) eps = np.finfo(t).eps nan = t(np.nan) inf = t(np.inf) assert_(np.spacing(one) == eps, repr(t)) assert_(np.isnan(np.spacing(nan)), repr(t)) assert_(np.isnan(np.spacing(inf)), repr(t)) assert_(np.isnan(np.spacing(-inf)), repr(t)) ====================================================================== FAIL: test_umath.test_spacing ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p y", line 183, in runTest self.test(*self.arg) File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 863, in test_spacing assert_(np.spacing(one) == eps, repr(t)) File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 33, i n assert_ AssertionError: Josef > Josef > >> >> David >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From josef.pktd at gmail.com Sun Nov 22 11:09:52 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Nov 2009 11:09:52 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220747k7d8897b0q7b984fc779a8da7a@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> <1cd32cbb0911220747k7d8897b0q7b984fc779a8da7a@mail.gmail.com> Message-ID: <1cd32cbb0911220809k329c7c3vf64eb7ea5cd343f6@mail.gmail.com> On Sun, Nov 22, 2009 at 10:47 AM, wrote: > On Sun, Nov 22, 2009 at 10:37 AM, ? wrote: >> On Sun, Nov 22, 2009 at 10:25 AM, David Cournapeau wrote: >>> On Mon, Nov 23, 2009 at 12:14 AM, ? wrote: >>>> On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau wrote: >>>>> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris >>>>> wrote: >>>>>> >>>>>> Might be nice to print out the actual values of np.spacing and np.nextafter >>>>>> here. >>>>>> >>>>> >>>>> Yes, I should add some utilities to print those for this kind of test. >>>>> But in this case, I know the problem: mingw gcc use 80 bits long >>>>> double but it uses the MS runtime which considers double and long >>>>> double to be the same (8 bytes). >>>>> >>>>> I think the real fix is to force npy_longdouble to be double on mingw, >>>>> but I don't want to make that change now for 1.4.0. >>>> >>>> adding the failing type in the test to the failure message would also >>>> be helpful >>> >>> Yes, you're right. I also used the nose facility for using generators >>> for complex corner cases, but with retrospect, it is not so useful, >>> because you don't get a name when you have a failure (or maybe I am >>> using it wrong). >> >> I don't know what the policy for the use of assert in numpy is, >> but if you use the function np.testing.assert_ then you can add a >> failure message with eg. repr(t) >> >> With "yield check_a_function arguments" nose prints the arguments >> ?in the test description, putting information into the arguments gets >> then displayed. This is also useful for arguments that the check >> function doesn't really need. > > example: > > I change the test to this, then the new test failure output is as below > > def test_spacing(): > ? ?for t in [np.float32, np.float64, np.longdouble]: > ? ? ? ?one = t(1) > ? ? ? ?eps = np.finfo(t).eps > ? ? ? ?nan = t(np.nan) > ? ? ? ?inf = t(np.inf) > ? ? ? ?assert_(np.spacing(one) == eps, repr(t)) > ? ? ? ?assert_(np.isnan(np.spacing(nan)), repr(t)) > ? ? ? ?assert_(np.isnan(np.spacing(inf)), repr(t)) > ? ? ? ?assert_(np.isnan(np.spacing(-inf)), repr(t)) > > > ====================================================================== > FAIL: test_umath.test_spacing > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p > y", line 183, in runTest > ? ?self.test(*self.arg) > ?File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line 863, in > test_spacing > ? ?assert_(np.spacing(one) == eps, repr(t)) > ?File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 33, i > n assert_ > AssertionError: or something like this to also get the values of the failing assertion def test_spacing(): for t in [np.float32, np.float64, np.longdouble]: one = t(1) eps = np.finfo(t).eps nan = t(np.nan) inf = t(np.inf) msg = '%r %r == %r' % (t, np.spacing(one), eps) assert_(np.spacing(one) == eps, msg) #assert_(np.spacing(one) == eps, repr(t)+ repr(np.spacing(one))+repr(eps)) assert_(np.isnan(np.spacing(nan)), repr(t)+ repr(np.spacing(nan))) assert_(np.isnan(np.spacing(inf)), repr(t)+ repr(np.spacing(inf))) assert_(np.isnan(np.spacing(-inf)), repr(t)+ repr(np.spacing(-inf))) Josef > > Josef > >> Josef >> >>> >>> David >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > From nadavh at visionsense.com Sun Nov 22 12:50:18 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 22 Nov 2009 19:50:18 +0200 Subject: [Numpy-discussion] neighborhood iterator Message-ID: <710F2847B0018641891D9A21602763605AD222@ex3.envision.co.il> I wonder if the neighbourhood iterator can be used as a more efficient replacement for ndimage.generic_filter. Is there a way to use it from cython? Nadav. From pav at iki.fi Sun Nov 22 13:16:55 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 22 Nov 2009 20:16:55 +0200 Subject: [Numpy-discussion] 2to3c: 2to3 for C extensions In-Reply-To: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> References: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> Message-ID: <1258913814.8816.11.camel@idol> la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: > Guido posted a link to this on Twitter, it might be of interest to > people interested in the NumPy Python 3 transition: > > http://dmalcolm.livejournal.com/3935.html > > It's doubtful this is a magic bullet (at least, not yet) but it may > still help reduce the amount of busywork involved. Interesting -- though at the moment it doesn't seem to solve very many of our problems. I did some cleanup of numpy.testing, distutils, and core so that using 2to3 the build_ext in numpy.core can at least be started: http://github.com/pv/numpy-work/tree/py3k python3 tools/py3test testing distutils core Of course, the build currently fails with zillion errors in multiarraymodule. Pauli From charlesr.harris at gmail.com Sun Nov 22 13:26:57 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 22 Nov 2009 11:26:57 -0700 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <5b8d13220911220701s7bd727a9g66a202c279346649@mail.gmail.com> <1cd32cbb0911220714t6b99fd50te232f4c6e48b2d9d@mail.gmail.com> <5b8d13220911220725r463da12fp20fdae2cf28349a2@mail.gmail.com> <1cd32cbb0911220737j70b2ec71rdee85e5580c93538@mail.gmail.com> Message-ID: On Sun, Nov 22, 2009 at 8:37 AM, wrote: > On Sun, Nov 22, 2009 at 10:25 AM, David Cournapeau > wrote: > > On Mon, Nov 23, 2009 at 12:14 AM, wrote: > >> On Sun, Nov 22, 2009 at 10:01 AM, David Cournapeau > wrote: > >>> On Sun, Nov 22, 2009 at 11:45 PM, Charles R Harris > >>> wrote: > >>>> > >>>> Might be nice to print out the actual values of np.spacing and > np.nextafter > >>>> here. > >>>> > >>> > >>> Yes, I should add some utilities to print those for this kind of test. > >>> But in this case, I know the problem: mingw gcc use 80 bits long > >>> double but it uses the MS runtime which considers double and long > >>> double to be the same (8 bytes). > >>> > >>> I think the real fix is to force npy_longdouble to be double on mingw, > >>> but I don't want to make that change now for 1.4.0. > >> > >> adding the failing type in the test to the failure message would also > >> be helpful > > > > Yes, you're right. I also used the nose facility for using generators > > for complex corner cases, but with retrospect, it is not so useful, > > because you don't get a name when you have a failure (or maybe I am > > using it wrong). > > I don't know what the policy for the use of assert in numpy is, > but if you use the function np.testing.assert_ then you can add a > failure message with eg. repr(t) > > They should all use assert_ as it is designed to work even in optimized environments. However, this is not enforced, official policy ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From renesd at gmail.com Sun Nov 22 13:28:10 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sun, 22 Nov 2009 19:28:10 +0100 Subject: [Numpy-discussion] 2to3c: 2to3 for C extensions In-Reply-To: <1258913814.8816.11.camel@idol> References: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> <1258913814.8816.11.camel@idol> Message-ID: <64ddb72c0911221028t7b15458era2c02503f70d9f75@mail.gmail.com> On Sun, Nov 22, 2009 at 7:16 PM, Pauli Virtanen wrote: > la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: >> Guido posted a link to this on Twitter, it might be of interest to >> people interested in the NumPy Python 3 transition: >> >> ? ? ? http://dmalcolm.livejournal.com/3935.html >> >> It's doubtful this is a magic bullet (at least, not yet) but it may >> still help reduce the amount of busywork involved. > > Interesting -- though at the moment it doesn't seem to solve very many > of our problems. > > > I did some cleanup of numpy.testing, distutils, and core so that using > 2to3 the build_ext in numpy.core can at least be started: > > ? ? ? ?http://github.com/pv/numpy-work/tree/py3k > > ? ? ? ?python3 tools/py3test testing distutils core > > Of course, the build currently fails with zillion errors in > multiarraymodule. > > ? ? ? ?Pauli > hi, I've been working on this too... see the previous py3k thread for details of my plan. See http://github.com/illume/numpy3k/tree/work for my (not very much) progress. I don't have time for about a week to do any more work on my tree. Maybe your tree is ahead anyway... I am taking the approach we used with pygame so that the tree can work with both python 2.x and 3.x. I got up to getting a fair amount of the setup code working... but there are still some issues. The 2to3 script for C code does look useful. However it will not work in a backwards compatible manner. However maybe it can be modified to work in such a way. anyway... I'll look at your tree again next time I have time to work on this in a week or so. cheers! From charlesr.harris at gmail.com Sun Nov 22 13:40:27 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 22 Nov 2009 11:40:27 -0700 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> Message-ID: On Sun, Nov 22, 2009 at 6:28 AM, wrote: > On Sun, Nov 22, 2009 at 2:35 AM, David Cournapeau > wrote: > > On Wed, Nov 18, 2009 at 6:10 PM, David Cournapeau > wrote: > >> On Wed, Nov 18, 2009 at 12:38 AM, wrote: > >> > >>> > >>> Now numpy builds without problems. > >>> When I run the tests I get 16 failures mostly nan related. I have no > idea > >>> whether they are real or if there is still something screwed up in my > >>> setup. See below. > >> > >> I can reproduce those. I did not see those because they are python 2.5 > >> specific (which is puzzling). > > > > I fix most of those, which were related to an atan2 bug in MS C > > library. Could you test whether this fixes it for you ? > > > > David > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > Thanks, > > I have 4 failures remaining. full ouput below > > Josef > > >python > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Running unit tests for numpy > NumPy version 1.4.0.dev7758 > NumPy is installed in C:\Josef\_progs\Subversion\numpy-trunk\dist\ > numpy-1.4.0.de > v7758.win32\Programs\Python25\Lib\site-packages\numpy > Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Int > el)] > nose version 0.11.1 > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > .......................K.....................................................FF. > > ............................K......................K.F.......................... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > .....................................C:\Josef\_progs\Subversion\numpy-trunk\dist > > \numpy-1.4.0.dev7758.win32\Programs\Python25\Lib\site-packages\numpy\lib\io.py:1 > 324: ConversionWarning: Some errors were detected ! > Line #2 (got 4 columns instead of 5) > Line #12 (got 4 columns instead of 5) > Line #22 (got 4 columns instead of 5) > Line #32 (got 4 columns instead of 5) > Line #42 (got 4 columns instead of 5) > warnings.warn(errmsg, ConversionWarning) > > .C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Programs\ > Python25\Lib\site-packages\numpy\lib\io.py:1324: ConversionWarning: Some > errors > were detected ! > Line #2 (got 4 columns instead of 2) > Line #12 (got 4 columns instead of 2) > Line #22 (got 4 columns instead of 2) > Line #32 (got 4 columns instead of 2) > Line #42 (got 4 columns instead of 2) > warnings.warn(errmsg, ConversionWarning) > > ..........................................K........K............F............... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ......S......................................................................... > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > > ................................................................................ > ................................................... > ====================================================================== > FAIL: test_umath.test_nextafter > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p > y", line 183, in runTest > self.test(*self.arg) > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line > 851, in > test_nextafter > assert np.nextafter(one, two) - one == eps > AssertionError > > ====================================================================== > FAIL: test_umath.test_spacing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p > y", line 183, in runTest > self.test(*self.arg) > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line > 863, in > test_spacing > assert np.spacing(one) == eps > AssertionError > > ====================================================================== > FAIL: test_special_values (test_umath_complex.TestClog) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", > line > 179, in test_special_values > assert_almost_equal(np.log(x), y) > File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line > 436, > in assert_almost_equal > AssertionError: Items are not equal: > ACTUAL: [ NaN+2.35619449j] > DESIRED: (1.#INF+2.35619449019j) > > > ====================================================================== > FAIL: test_doctests (test_polynomial.TestDocs) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr > ograms\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line > 90, > in test_doctests > return rundocs() > File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line > 952, > in rundocs > AssertionError: Some doctests failed: > ********************************************************************** > File > "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Prog > rams\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line > 20, in > test_polynomial > Failed example: > print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) > Expected: > 2 > 1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 > Got: > 2 > 1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 > > > I can maybe fix this, although I'm not sure it's worth the effort and I don't have a way to test it. There is another bug in the poly1d package that needs fixing. David, what is the preferred way to do that now? Also, there are fixes for the MS atan2 problems in the WTF_MatExtras file. I think I sent you a copy once upon a time. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From renesd at gmail.com Sun Nov 22 13:51:22 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sun, 22 Nov 2009 19:51:22 +0100 Subject: [Numpy-discussion] 2to3c: 2to3 for C extensions In-Reply-To: <64ddb72c0911221028t7b15458era2c02503f70d9f75@mail.gmail.com> References: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> <1258913814.8816.11.camel@idol> <64ddb72c0911221028t7b15458era2c02503f70d9f75@mail.gmail.com> Message-ID: <64ddb72c0911221051s5cecb0f2qdfe91b21de051a76@mail.gmail.com> On Sun, Nov 22, 2009 at 7:28 PM, Ren? Dudfield wrote: > On Sun, Nov 22, 2009 at 7:16 PM, Pauli Virtanen wrote: >> la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: >>> Guido posted a link to this on Twitter, it might be of interest to >>> people interested in the NumPy Python 3 transition: >>> >>> ? ? ? http://dmalcolm.livejournal.com/3935.html >>> >>> It's doubtful this is a magic bullet (at least, not yet) but it may >>> still help reduce the amount of busywork involved. >> >> Interesting -- though at the moment it doesn't seem to solve very many >> of our problems. >> >> >> I did some cleanup of numpy.testing, distutils, and core so that using >> 2to3 the build_ext in numpy.core can at least be started: >> >> ? ? ? ?http://github.com/pv/numpy-work/tree/py3k >> >> ? ? ? ?python3 tools/py3test testing distutils core >> >> Of course, the build currently fails with zillion errors in >> multiarraymodule. >> >> ? ? ? ?Pauli >> > > hi, > > I've been working on this too... see the previous py3k thread for > details of my plan. ?See http://github.com/illume/numpy3k/tree/work > for my (not very much) progress. > > I don't have time for about a week to do any more work on my tree. > Maybe your tree is ahead anyway... I am taking the approach we used > with pygame so that the tree can work with both python 2.x and 3.x. ?I > got up to getting a fair amount of the setup code working... but there > are still some issues. > > The 2to3 script for C code does look useful. ?However it will not work > in a backwards compatible manner. ?However maybe it can be modified to > work in such a way. > hi again, I was wrong about it not generating 2 and 3 compatible source... the tool does do things like this: #if PY_MAJOR_VERSION >= 3 foo(); #else bar(); #endif Very cool!!! cheers, From pav at iki.fi Sun Nov 22 14:10:30 2009 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 22 Nov 2009 21:10:30 +0200 Subject: [Numpy-discussion] 2to3c: 2to3 for C extensions In-Reply-To: <64ddb72c0911221028t7b15458era2c02503f70d9f75@mail.gmail.com> References: <67C0DE8B-141E-4BC9-A4D5-3993E83DE266@cs.toronto.edu> <1258913814.8816.11.camel@idol> <64ddb72c0911221028t7b15458era2c02503f70d9f75@mail.gmail.com> Message-ID: <1258917028.8816.29.camel@idol> su, 2009-11-22 kello 19:28 +0100, Ren? Dudfield kirjoitti: [clip] > I've been working on this too... see the previous py3k thread for > details of my plan. See http://github.com/illume/numpy3k/tree/work > for my (not very much) progress. > > I don't have time for about a week to do any more work on my tree. > Maybe your tree is ahead anyway... I am taking the approach we used > with pygame so that the tree can work with both python 2.x and 3.x. I > got up to getting a fair amount of the setup code working... but there > are still some issues. Ok, I forgot that you did also some work on this. I think my tree is currently a bit more along the way than yours -- it seems to have the same fixes as in yours, and it proceeds until it fails to compile multiarraymodule. The main difference seems to be that I'm using the 2to3 tool to automatically convert the Python code before the actual build -- this could be a major manual work saver. Pauli From cheronetolivia at yahoo.com Sun Nov 22 16:51:59 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Sun, 22 Nov 2009 13:51:59 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin Message-ID: <432197.42695.qm@web51001.mail.re2.yahoo.com> Hello, I attempted to install Numpy for my Cygwin python again, by simply executing: >python setup.py install However, I now get the following: >File "numpy/core/setup.py", line 253, in check_mathlib > raise EnvironmentError("math library missing; rerun " >EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB >env variable I have a math library from cygwin (libm.a), but I have not managed to set it. How should I set this? Thank you for the help, Olivia ----- Original Message ---- > From: David Cournapeau > To: Discussion of Numerical Python > Sent: Wed, November 18, 2009 9:01:06 AM > Subject: Re: [Numpy-discussion] Installing numpy under cygwin > > Olivia Cheronet wrote: > > Hello. > > > > I am currently trying to install the latest version of NumPy for my cygwin > Python, and I am having problems... > > > > I have downloaded the source, and unzipped and untarred it in my home > directory. > > Subsequently, I included the following in the site.cfg file: > > > >> [DEFAULT] > >> library_dirs = /cygdrive/c/cygwin/lib > >> include_dirs = /cygdrive/c/cygwin/usr/local/include > >> [blas_opt] > >> libraries = f77blas, cblas, atlas > >> [lapack_opt] > >> libraries = lapack, f77blas, cblas, atlas > >> > > > > In the Cygwin bash shell, after going to my home directory, I have executed: > > python setup.py config --compiler=mingw32 build --compiler=mingw32 install > > as instructed in the "Installing SciPy/Windows" page. > > > > If you use cygwin, you should not follow the windows instructions. For > most purposes, cygwin works as unix. In particular, you don't think you > should not use mingw32 build on cygwin. The -mno-cygwin flag appended to > gcc is most likely coming from there, and this looks very wrong for a > cygwin build. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From dwf at cs.toronto.edu Sun Nov 22 20:21:40 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sun, 22 Nov 2009 20:21:40 -0500 Subject: [Numpy-discussion] neighborhood iterator In-Reply-To: <710F2847B0018641891D9A21602763605AD222@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD222@ex3.envision.co.il> Message-ID: <26AEE5C3-49A6-47CD-89AC-D1D73DDBA272@cs.toronto.edu> On 22-Nov-09, at 12:50 PM, Nadav Horesh wrote: > > I wonder if the neighbourhood iterator can be used as a more > efficient replacement for ndimage.generic_filter. Is there a way to > use it from cython? Yes, using the NumPy C API, called like any other C function is from Cython. Something like: ###################### import numpy as np cimport numpy as np cdef extern from "numpy/arrayobject.h": object PyArray_NeighborhoodIterNew(object iter, np.npy_intp bounds, int mode, object, np.ndarray fill_value) int PyArrayNeighborhoodIter_Next(object iter) int PyArrayNeighborhoodIter_Reset(object iter) ###################### should do the trick. Note that you'll need to call np.import_array() before using any of these functions to initialize the C API, I think. David From pav at iki.fi Mon Nov 23 00:35:52 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Nov 2009 07:35:52 +0200 Subject: [Numpy-discussion] Numpy on Python3 Message-ID: <1258954552.8816.78.camel@idol> http://github.com/pv/numpy-work/tree/py3k $ mkdir -p $PWD/dist/lib/python3.1/site-packages $ python3 setup.py install --prefix=$PWD/dist $ cd $PWD/dist/lib/python3.1/site-packages && python3 Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: numpy.ma is disabled for now -- some issues >>> numpy.array([1., 2, 3, 4]) array([ 1., 2., 3., 4.]) >>> _ + 10 array([ 11., 12., 13., 14.]) >>> numpy.ones((4,), dtype=complex)/4 array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) >>> numpy.array([object(), object()]) array([, ], dtype=b'object') *** Things were fairly straightforward this far, just many tiny changes. What's left is then sorting out the bigger problems :) This is still far from being complete: - Most use of PyString_* needs auditing (note e.g. the b'object' in the dtype print above...). I simply added convenience wrappers for PyString -> PyBytes, but this is not the correct choice at all points. - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. - Whether to inherit Numpy ints from PyLong_* needs some thinking, as they are quite different objects. Now, I dropped the inheritance, but I wonder if this will break something. - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't seem to cut it -- don't know exactly what's wrong here. - Integer -> String formatting does not seem to work - Larger-than-long-long Python ints probably cause problems - The new buffer interface needs to be implemented -- currently there are just error-raising stubs. I remember Dag was working on this a bit: how far did it go? - Relative imports + 2to3 is a bit of a pain. A pity we can't have them in the mainline code because of python2.4. - I didn't check for semantic changes in tp_* interface functions. This we need still to do. - And probably many other issues lurking. Also, I didn't yet try checking how far the test suite passes on Python3. (It still passes completely on Python2, so at least I didn't break that part.) It might be nice to have this merged in at some point after 1.4.0 (after the most obvious glaring bugs have been fixed), so that we could perhaps start aiming for Python3 compatibility in Numpy 1.5.0. Cheers, Pauli From charlesr.harris at gmail.com Mon Nov 23 01:08:50 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 22 Nov 2009 23:08:50 -0700 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > > *** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > - Whether to inherit Numpy ints from PyLong_* needs some thinking, > as they are quite different objects. Now, I dropped the inheritance, > but I wonder if this will break something. > > Maybe. But it was always a hassle because it behaved differently than the other integer types. Now onto float ;) > - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't > seem to cut it -- don't know exactly what's wrong here. > > - Integer -> String formatting does not seem to work > > - Larger-than-long-long Python ints probably cause problems > > We used a python call which would raise an error if the number was too large. If that call is still valid, things should work. > - The new buffer interface needs to be implemented -- currently there > are just error-raising stubs. > > I remember Dag was working on this a bit: how far did it go? > > - Relative imports + 2to3 is a bit of a pain. A pity we can't have > them in the mainline code because of python2.4. > > - I didn't check for semantic changes in tp_* interface functions. > This we need still to do. > > I left some notes in the src folder. If you discover anything new put it in there. > - And probably many other issues lurking. > > We do need to look at the initialization of the type math stuff in the ufuncobject module. Yeah, its a bit of a circular dependency, one reason it would be nice to have ufuncs operate on buffer objects instead of ndarrays would be to break the mutual dependence. > > Also, I didn't yet try checking how far the test suite passes on > Python3. (It still passes completely on Python2, so at least I didn't > break that part.) > > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. > > If you want to see real suffering, look at the programmer notes in the hacked CRU files. Some poor sucker was stuck with fixing up the g*dawful code while also needing to determine what data was in undocumented binary files, some with the same names but containing different data. Folks, don't let that happen to you. The conversion to Py3k is going to be a breeze by comparison. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Nov 23 01:19:26 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 23 Nov 2009 15:19:26 +0900 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: <5b8d13220911222219j7b52d768u33f8130feaa001a5@mail.gmail.com> On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, David From charlesr.harris at gmail.com Mon Nov 23 01:33:31 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 22 Nov 2009 23:33:31 -0700 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > > *** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > Yeah, this needs deciding. I think compatibility with old files requires Bytes, we can't change file formats and keep compatibility. BTW, I made some changes for py3k that depend on the macro NPY_PY3K being defined, but didn't actually define it anywhere. It needs to go in one of the include files. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Nov 23 01:32:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 Nov 2009 01:32:03 -0500 Subject: [Numpy-discussion] Correlation filter In-Reply-To: <1cd32cbb0911201327j7b6bee04u6fc9cf637239a1cd@mail.gmail.com> References: <1cd32cbb0911200853ub26bd1dvce7b4681dd2796e1@mail.gmail.com> <1cd32cbb0911201117t5817da8dice8a4e822e390d07@mail.gmail.com> <1cd32cbb0911201327j7b6bee04u6fc9cf637239a1cd@mail.gmail.com> Message-ID: <1cd32cbb0911222232s6371da26v7b455cf4bc650eba@mail.gmail.com> On Fri, Nov 20, 2009 at 4:27 PM, wrote: > On Fri, Nov 20, 2009 at 2:28 PM, Keith Goodman wrote: >> On Fri, Nov 20, 2009 at 11:17 AM, ? wrote: >>> On Fri, Nov 20, 2009 at 1:51 PM, Keith Goodman wrote: >>>> def corr3(x, y): >>>> ? ?x = x - x.mean() >>>> ? ?x /= x.std() >>>> ? ?nx = x.size >>>> ? ?one = np.ones(nx) >>>> ? ?xy = lfilter(x, 1, y) >>>> ? ?sy = lfilter(one, 1, y) >>>> ? ?sy2 = lfilter(one, 1, y*y) >>>> ? ?d = xy / np.sqrt(nx * sy2 - sy * sy) >>>> ? ?return d >>> >>> Is this correct? xy uses the original y and not a demeaned y. >> >> I wouldn't be surprised if I made mistakes. But I don't think I need >> to demean y. One way to write the numerator of the correlation >> coefficent is >> >> N*sum(xy) - sum(x)*sum(y) >> >> but sum(x) is zero, so it becomes >> >> N*sum(xy) >> >> So I think I only need to demean x. But I'd better test the code. Plus >> I have no idea how lfilter will handle my missing values (NaN). > > I think nans fully propagate, trick might be to fill with zero and use > another convolution/correlation with the non-nan mask to get the > number of observations included (I think I saw it as a trick by Pierre > for autocorrelation with missing observations). Just an update, I used this for moving correlation between x and y of equal shape, just posted to the scipy-user mailing list. In the test case it works correctly. Josef > > Here is a moving slope function which estimate the slope coefficient > for a linear trend if x is arange() > I compared the ranking with your moving correlation and it is quite different. > > > def moving_slope(x,y) > ? ?'''estimate moving slope coefficient of regression of y on x > ? ?filters along axis=1, returns valid observations > ? ?Todo: axis and lag options > ? ?''' > ? ?xx = np.column_stack((np.ones(x.shape), x)) > ? ?pinvxx = np.linalg.pinv(xx)[1:,:] > ? ?windsize = len(x) > ? ?lead = windsize//2 - 1 > ? ?return signal.correlate(y, pinvxx, 'full' > )[:,windsize-lead:-(windsize+1*lead-2)] > > > Josef > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From pgmdevlist at gmail.com Mon Nov 23 01:40:00 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 23 Nov 2009 01:40:00 -0500 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: <0718316F-CF85-47C9-B0C6-BA2A8B19F5A2@gmail.com> On Nov 23, 2009, at 12:35 AM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues What are the issues ? From sturla at molden.no Mon Nov 23 02:58:47 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 23 Nov 2009 08:58:47 +0100 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: <4B0A40B7.90705@molden.no> Pauli Virtanen skrev: > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > I thought numpy.random uses Cython? Is it just a matter of recompiling the pyx-file? > I remember Dag was working on this a bit: how far did it go? > > Cython's include file numpy.pxd has an ndarray class that extend PyArrayObject with PEP 3118 buffer compatibility. From pav at iki.fi Mon Nov 23 04:08:03 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Nov 2009 11:08:03 +0200 Subject: [Numpy-discussion] Numpy on Python3 Message-ID: <0FgNcWJ3mGMP.Y4dhPryx@ltl.tkk.fi> Setup.py runs 2to3 automatically for all changed files. Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 L?hett?j?: David Cournapeau P?iv?m??r?: 23.11.2009 08:19 On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Mon Nov 23 04:13:48 2009 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Nov 2009 11:13:48 +0200 Subject: [Numpy-discussion] Numpy on Python3 Message-ID: <9l0XsFttJkpS.CBPrwBlm@ltl.tkk.fi> The issue with longs is that we wouldn't want array([1,2,3]) create object arrays -- so we need to decide on casting rules for longs. Currently, I think they're treated like python2 ints. -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 L?hett?j?: Charles R Harris P?iv?m??r?: 23.11.2009 08:08 On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > > *** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > - Whether to inherit Numpy ints from PyLong_* needs some thinking, > as they are quite different objects. Now, I dropped the inheritance, > but I wonder if this will break something. > > Maybe. But it was always a hassle because it behaved differently than the other integer types. Now onto float ;) > - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't > seem to cut it -- don't know exactly what's wrong here. > > - Integer -> String formatting does not seem to work > > - Larger-than-long-long Python ints probably cause problems > > We used a python call which would raise an error if the number was too large. If that call is still valid, things should work. > - The new buffer interface needs to be implemented -- currently there > are just error-raising stubs. > > I remember Dag was working on this a bit: how far did it go? > > - Relative imports + 2to3 is a bit of a pain. A pity we can't have > them in the mainline code because of python2.4. > > - I didn't check for semantic changes in tp_* interface functions. > This we need still to do. > > I left some notes in the src folder. If you discover anything new put it in there. > - And probably many other issues lurking. > > We do need to look at the initialization of the type math stuff in the ufuncobject module. Yeah, its a bit of a circular dependency, one reason it would be nice to have ufuncs operate on buffer objects instead of ndarrays would be to break the mutual dependence. > > Also, I didn't yet try checking how far the test suite passes on > Python3. (It still passes completely on Python2, so at least I didn't > break that part.) > > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. > > If you want to see real suffering, look at the programmer notes in the hacked CRU files. Some poor sucker was stuck with fixing up the g*dawful code while also needing to determine what data was in undocumented binary files, some with the same names but containing different data. Folks, don't let that happen to you. The conversion to Py3k is going to be a breeze by comparison. Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Mon Nov 23 04:02:53 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 23 Nov 2009 18:02:53 +0900 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <0FgNcWJ3mGMP.Y4dhPryx@ltl.tkk.fi> References: <0FgNcWJ3mGMP.Y4dhPryx@ltl.tkk.fi> Message-ID: <4B0A4FBD.8080304@ar.media.kyoto-u.ac.jp> Pauli Virtanen wrote: > Setup.py runs 2to3 automatically for all changed files. Yes, but I think it is more practical to have the build process to be 2 and 3-compatible. > Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? > For the exception catching, one can have a few utilities to walk through the stack - I don't remember how I did it for the relative import, but this was not a pb IIRC. I am quite disappointed that numscons will not be usable in the foreseable future for py3k, I hoped it would have been simpler than numpy.distutils, but porting scons to py3k is too big of a task. cheers, David From pav+sp at iki.fi Mon Nov 23 04:36:49 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Mon, 23 Nov 2009 09:36:49 +0000 (UTC) Subject: [Numpy-discussion] Numpy on Python3 References: <1258954552.8816.78.camel@idol> <0718316F-CF85-47C9-B0C6-BA2A8B19F5A2@gmail.com> Message-ID: Mon, 23 Nov 2009 01:40:00 -0500, Pierre GM wrote: [clip] >> XXX: 3K: numpy.ma is disabled for now -- some issues > > What are the issues ? Something resolving which would have taken more than 5 minutes :) Possibly because something that ma depends on is currently broken in numpy.core. I just wanted to breeze through and arrive as fast as possible at something that can be imported and works for simple things, so I didn't stop at anything that would take longer. I'll take a look at the rest later on. -- Pauli Virtanen From pav+sp at iki.fi Mon Nov 23 04:39:16 2009 From: pav+sp at iki.fi (Pauli Virtanen) Date: Mon, 23 Nov 2009 09:39:16 +0000 (UTC) Subject: [Numpy-discussion] Numpy on Python3 References: <1258954552.8816.78.camel@idol> <4B0A40B7.90705@molden.no> Message-ID: Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote: > Pauli Virtanen skrev: >> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: >> numpy.ma is disabled for now -- some issues >> > I thought numpy.random uses Cython? Is it just a matter of recompiling > the pyx-file? The Cython file uses the C-api directly, so we'll need a .h file with the necessary compile-time conditionals. >> I remember Dag was working on this a bit: how far did it go? > > Cython's include file numpy.pxd has an ndarray class that extend > PyArrayObject with PEP 3118 buffer compatibility. Great! I believe I will just steal whatever I can and rewrite it in C -- for now, it seems possible to keep Numpy's core in plain C. -- Pauli Virtanen From pgmdevlist at gmail.com Mon Nov 23 04:43:27 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 23 Nov 2009 04:43:27 -0500 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: References: <1258954552.8816.78.camel@idol> <0718316F-CF85-47C9-B0C6-BA2A8B19F5A2@gmail.com> Message-ID: <1D31F418-6AB2-4EE7-A5AB-A895B9D21702@gmail.com> On Nov 23, 2009, at 4:36 AM, Pauli Virtanen wrote: > Mon, 23 Nov 2009 01:40:00 -0500, Pierre GM wrote: > [clip] >>> XXX: 3K: numpy.ma is disabled for now -- some issues >> >> What are the issues ? > > Something resolving which would have taken more than 5 minutes :) > Possibly because something that ma depends on is currently broken in > numpy.core. Fair enough, fair enough... But y'all, please let me know potential issues (not that I'll be able to work on it anytime soon, but just in case...). P. From nadavh at visionsense.com Mon Nov 23 09:12:30 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 23 Nov 2009 16:12:30 +0200 Subject: [Numpy-discussion] neighborhood iterator References: <710F2847B0018641891D9A21602763605AD222@ex3.envision.co.il> <26AEE5C3-49A6-47CD-89AC-D1D73DDBA272@cs.toronto.edu> Message-ID: <710F2847B0018641891D9A21602763605AD224@ex3.envision.co.il> Thank you, this is a start. I seems that there are more issues to resolve. I am trying to make a general frame that would enable one to write filters using this iterator. Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of David Warde-Farley Sent: Mon 23-Nov-09 03:21 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] neighborhood iterator On 22-Nov-09, at 12:50 PM, Nadav Horesh wrote: > > I wonder if the neighbourhood iterator can be used as a more > efficient replacement for ndimage.generic_filter. Is there a way to > use it from cython? Yes, using the NumPy C API, called like any other C function is from Cython. Something like: ###################### import numpy as np cimport numpy as np cdef extern from "numpy/arrayobject.h": object PyArray_NeighborhoodIterNew(object iter, np.npy_intp bounds, int mode, object, np.ndarray fill_value) int PyArrayNeighborhoodIter_Next(object iter) int PyArrayNeighborhoodIter_Reset(object iter) ###################### should do the trick. Note that you'll need to call np.import_array() before using any of these functions to initialize the C API, I think. David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3261 bytes Desc: not available URL: From cjw at ncf.ca Mon Nov 23 09:35:05 2009 From: cjw at ncf.ca (Colin J. Williams) Date: Mon, 23 Nov 2009 09:35:05 -0500 Subject: [Numpy-discussion] Resize method Message-ID: <4B0A9D99.1010403@ncf.ca> Access by the interpreter prevents array resizing. Yes, one can use the function, in place of the method but this appears to require copying the whole array. If one sets b= a, then that reference can be deleted with del b. Is there any similar technique for the interpreter? Colin W. Python 2.6 (r26:66721, Oct 2 2008, 11:35:03) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a= array(7*[3]) >>> a.resize((3,7)) >>> a array([[3, 3, 3, 3, 3, 3, 3], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0]]) >>> a.resize((4,7)) Traceback (most recent call last): File "", line 1, in ValueError: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function >>> From josef.pktd at gmail.com Mon Nov 23 10:53:01 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 Nov 2009 10:53:01 -0500 Subject: [Numpy-discussion] recompiling everything ? Message-ID: <1cd32cbb0911230753g51142f79sbd67c5a6bc25b3a1@mail.gmail.com> I have several scikits with c extension compiled against an older trunk version of 1.4 but didn't recompile them with the numpy trunk version that I currently use. scikits.timeseries just crashed on me taking several hours worth of examples with it. (scipy crashed during testing, so I rebuilt it immediately with the matching numpy version) Is this only related to the temporary (?) ABI breaking during the datetime merge, or do we now have to recompile all packages with c-extension each time we update numpy trunk? (If scipy trunk didn't require numpy trunk, I would gladly just stick with the release version of numpy.) Josef From dagss at student.matnat.uio.no Mon Nov 23 13:49:38 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Nov 2009 10:49:38 -0800 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: References: <1258954552.8816.78.camel@idol> <4B0A40B7.90705@molden.no> Message-ID: <4B0AD942.3000302@student.matnat.uio.no> Pauli Virtanen wrote: > Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote: > >> Pauli Virtanen skrev: >> >>> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: >>> numpy.ma is disabled for now -- some issues >>> >>> >> I thought numpy.random uses Cython? Is it just a matter of recompiling >> the pyx-file? >> > > The Cython file uses the C-api directly, so we'll need a .h file with the > necessary compile-time conditionals. > > >>> I remember Dag was working on this a bit: how far did it go? >>> >> Cython's include file numpy.pxd has an ndarray class that extend >> PyArrayObject with PEP 3118 buffer compatibility. >> > > Great! I believe I will just steal whatever I can and rewrite it in C -- > for now, it seems possible to keep Numpy's core in plain C. > I did sit down with David to learn enough to do this and had a brief start on doing it properly for NumPy on SciPy 2009 (with seperate testcases and the buffer format string stored directly in the NumPy dtype structs on creation). I meant to come back to it in November but due to becoming sick etc. etc. that's no longer possible. If nothing happens by the mid/end of January I still hope to be able to do this then. Feel free to ask any questions about the buffer PEP if you do go forward with this as I've used it a lot in Cython (and wrote the "implementation on behalf of NumPy" there). The Cython numpy.pxd does: - Temporarily allocate buffers for the format string, this is inefficient as NumPy can store them directly in the dtype when the dtype is constructed - Not support non-native endian (and some other relevant packing formats I believe) - Not support the string types - Not support dtypes with nested sub-arrays within records What is done: David added some code (at least in some branch) that ensures that sizeof(npy_intp) == sizeof(Py_ssize_t) on Python 2.6+. (I.e. if that assumption is violated NumPy won't compile, so we're free to assume it until the issue of using npy_intp for indices is fixed on a more fundamental level in NumPy). This means that the shapes/strides in Py_buffer can be directed directly to the dimensions/strides in the NumPy array struct (whereas Cython's numpy.pxd has to make a copy on some platforms for Python 2.4). Dag Sverre From Chris.Barker at noaa.gov Mon Nov 23 15:10:53 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 23 Nov 2009 12:10:53 -0800 Subject: [Numpy-discussion] Resize method In-Reply-To: <4B0A9D99.1010403@ncf.ca> References: <4B0A9D99.1010403@ncf.ca> Message-ID: <4B0AEC4D.2040608@noaa.gov> Colin J. Williams wrote: > Access by the interpreter prevents array resizing. yup -- resize is really fragile for that reason. It really should be used quite sparingly. Personally, I think it should probably only be used when wrapped with a higher level layer. I've been working on an extendable array class, I call an accumulator (bad name...). The idea is that you can use it to accumulate values when you don't know how big it's going to end up, rather than using a list for this, which is the standard idiom. In [2]: import accumulator In [3]: a = accumulator.accumulator((1,2,3,4,)) In [4]: a Out[4]: accumulator([1, 2, 3, 4]) In [5]: a.append(5) In [6]: a Out[6]: accumulator([1, 2, 3, 4, 5]) In [8]: a.extend((6,7,8,9)) In [9]: a Out[9]: accumulator([1, 2, 3, 4, 5, 6, 7, 8, 9]) At the moment, it only support 1-d arrays, though I'd like to extend it to n-d, probably only allowing growing on the first axis. This has been discussed on this list a fair bit, with mixed reviews as to whether there is any point. It's slower than lists in common usage, but has other advantages -- I'd like to see a C version, but don't know if I'll ever have the time for that. I've enclosed to code for your viewing pleasure -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: Accumulator.zip Type: application/zip Size: 5184 bytes Desc: not available URL: From zachary.pincus at yale.edu Mon Nov 23 15:25:12 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 23 Nov 2009 15:25:12 -0500 Subject: [Numpy-discussion] neighborhood iterator In-Reply-To: <710F2847B0018641891D9A21602763605AD224@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD222@ex3.envision.co.il> <26AEE5C3-49A6-47CD-89AC-D1D73DDBA272@cs.toronto.edu> <710F2847B0018641891D9A21602763605AD224@ex3.envision.co.il> Message-ID: <71F749E4-247E-42BF-BD22-CB913F73BFDE@yale.edu> Hi, Having just written some cython code to iterate a neighborhood across an array, I have some ideas about features that would be useful for a general frame. Specifically, being able to pass in a "footprint" boolean array to define the neighborhood is really useful in many contexts. Also useful is the ability to query the offset of the current pixel from the center of the neighborhood. (These two features, plus very efficient handling of boundary conditions by breaking the image into regions where the conditions are and are not required, make the image iterators in ITK really nice to use.) Zach On Nov 23, 2009, at 9:12 AM, Nadav Horesh wrote: > > Thank you, this is a start. I seems that there are more issues to > resolve. I am trying to make a general frame that would enable one > to write filters using this iterator. > > > Nadav > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org on behalf of David Warde- > Farley > Sent: Mon 23-Nov-09 03:21 > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] neighborhood iterator > > > On 22-Nov-09, at 12:50 PM, Nadav Horesh wrote: > >> >> I wonder if the neighbourhood iterator can be used as a more >> efficient replacement for ndimage.generic_filter. Is there a way to >> use it from cython? > > Yes, using the NumPy C API, called like any other C function is from > Cython. Something like: > > ###################### > > import numpy as np > cimport numpy as np > > cdef extern from "numpy/arrayobject.h": > object PyArray_NeighborhoodIterNew(object iter, np.npy_intp > bounds, > int mode, object, np.ndarray > fill_value) > int PyArrayNeighborhoodIter_Next(object iter) > int PyArrayNeighborhoodIter_Reset(object iter) > > ###################### > > should do the trick. > > Note that you'll need to call np.import_array() before using any of > these functions to initialize the C API, I think. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From oliphant at enthought.com Mon Nov 23 18:42:38 2009 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 23 Nov 2009 17:42:38 -0600 Subject: [Numpy-discussion] datetime update Message-ID: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> I've made a few changes to datetime today and spent some time looking over what is there and what remains to be implemented. Basically, the biggest thing left to do is to implement the low-level casting functions to and from datetime types and other numpy types. In addition, the ufuncs need some auditing to make sure the right thing is being done when mixing different units. After that, lots and lots of additional tests need to be written. Once that is done, then most of the features should be available, but I suspect a few lingering issues might crop up and require fixing or fleshing out as well. I was hoping that someone would be able to contribute more tests for datetime. I will spend some time on the casting functions over the next few weeks and write a few tests. I fixed a problem today with the fact that PyArray_DescrFromScalar was not returning a data-type object with the correct frequency information stored when given a datetime64 or timedelta64 scalar (it was ignoring the date-time metadata on the scalar). This fixed a problem with the printing so that now a = arange(10).view('M8[Y]') shows something reasonable. I also removed numpy.datetime and numpy.timedelta from the namespace (replaced them with numpy.datetime_ and numpy.timedelta_). These were just short-hand for numpy.datetime64 and numpy.timedelta64 respectively. Avoiding the collision seemed like a good idea. Right now, what works is "viewing" arrays as datetime data-types and getting and setting date-time arrays using datetime objects. I would like to improve it so that setting with strings, integers, and other Python objects works as well. Also, adding simple integers works, but Dave C suggested removing the new C-API calls which sounds like a good idea to me for 1.4.0. Which functions get exported into the C-API for 1.5.0 could then receive some discussion. I apologize for the slow communication about where things are at. Best regards, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfeldman at verizon.net Mon Nov 23 19:43:23 2009 From: pfeldman at verizon.net (Dr. Phillip M. Feldman) Date: Mon, 23 Nov 2009 16:43:23 -0800 (PST) Subject: [Numpy-discussion] make hstack and vstack promote 1-D argument to 2-D when necessary Message-ID: <26488748.post@talk.nabble.com> I opened ticket #1302 to make the following enhancement request: I'd like to see hstack and vstack promote 1-D arguments to 2-D when this is necessary to make the dimensions match. In the following example, c_ works as expected while hstack does not: [~]|8> x <8> array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) [~]|9> y <9> array([10, 11, 12]) [~]|10> c_[x,y] <10> array([[ 1, 2, 3, 10], [ 4, 5, 6, 11], [ 7, 8, 9, 12]]) [~]|11> x <11> array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) [~]|12> y <12> array([10, 11, 12]) [~]|13> hstack((x,y)) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) \ in () C:\Program Files\Python25\lib\site-packages\numpy\lib\shape_base.pyc in hstack(tup) 503 504 """ --> 505 return _nx.concatenate(map(atleast_1d,tup),1) 506 507 row_stack = vstack ValueError: arrays must have same number of dimensions -- View this message in context: http://old.nabble.com/make-hstack-and-vstack-promote-1-D-argument-to-2-D-when-necessary-tp26488748p26488748.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From robert.kern at gmail.com Mon Nov 23 19:51:21 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 23 Nov 2009 18:51:21 -0600 Subject: [Numpy-discussion] make hstack and vstack promote 1-D argument to 2-D when necessary In-Reply-To: <26488748.post@talk.nabble.com> References: <26488748.post@talk.nabble.com> Message-ID: <3d375d730911231651g4463e686l9ee5b30d4fef8d7d@mail.gmail.com> On Mon, Nov 23, 2009 at 18:43, Dr. Phillip M. Feldman wrote: > > I opened ticket #1302 to make the following enhancement request: > > I'd like to see hstack and vstack promote 1-D arguments to 2-D when this is > necessary to make the dimensions match. ?In the following example, c_ works > as expected while hstack does not: This isn't going to change. It would be inconsistent with the way we promote 1D arrays to higher dimensions when broadcasting (i.e. we always prepend 1s to the shape tuple so 1D arrays are treated like 2D row vectors). For the hstack() case, you might want to use column_stack() which explicitly treats 1D arrays like columns. Otherwise, just use column vectors explicitly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Mon Nov 23 19:53:10 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 23 Nov 2009 19:53:10 -0500 Subject: [Numpy-discussion] datetime update In-Reply-To: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> References: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> Message-ID: <3188844F-75BD-4AE4-8CA6-D2859C69FBF4@gmail.com> On Nov 23, 2009, at 6:42 PM, Travis Oliphant wrote: > > > I've made a few changes to datetime today and spent some time looking over what is there and what remains to be implemented. As always, many thanks for your work !! > Basically, the biggest thing left to do is to implement the low-level casting functions to and from datetime types and other numpy types. > In addition, the ufuncs need some auditing to make sure the right thing is being done when mixing different units. After that, lots and lots of additional tests need to be written. Once that is done, then most of the features should be available, but I suspect a few lingering issues might crop up and require fixing or fleshing out as well. > > I was hoping that someone would be able to contribute more tests for datetime. I will spend some time on the casting functions over the next few weeks and write a few tests. Fortunately, the new modifications will make it easier to write such tests... But in any case, we can assume that what is proposed in the NEP should work, right ? > I also removed numpy.datetime and numpy.timedelta from the namespace (replaced them with numpy.datetime_ and numpy.timedelta_). These were just short-hand for numpy.datetime64 and numpy.timedelta64 respectively. Avoiding the collision seemed like a good idea. > > Right now, what works is "viewing" arrays as datetime data-types and getting and setting date-time arrays using datetime objects. I would like to improve it so that setting with strings, integers, and other Python objects works as well. Did you use any of Marty Fuhry's GSoC work ? What are the potential issues that could prevent an easy integration ? > Also, adding simple integers works, but > > Dave C suggested removing the new C-API calls which sounds like a good idea to me for 1.4.0. Which functions get exported into the C-API for 1.5.0 could then receive some discussion. Wouldn't it be easier to leave the C-APi as it is now, even for 1.4.0, but not to advertize it before 1.5.0 ? Thanks again for everything P. From cjw at ncf.ca Mon Nov 23 19:54:44 2009 From: cjw at ncf.ca (Colin J. Williams) Date: Mon, 23 Nov 2009 19:54:44 -0500 Subject: [Numpy-discussion] Resize method In-Reply-To: <4B0AEC4D.2040608@noaa.gov> References: <4B0A9D99.1010403@ncf.ca> <4B0AEC4D.2040608@noaa.gov> Message-ID: <4B0B2ED4.7030508@ncf.ca> Christopher Barker wrote: > Colin J. Williams wrote: >> Access by the interpreter prevents array resizing. > > yup -- resize is really fragile for that reason. It really should be > used quite sparingly. > > Personally, I think it should probably only be used when wrapped with > a higher level layer. > > I've been working on an extendable array class, I call an accumulator > (bad name...). The idea is that you can use it to accumulate values > when you don't know how big it's going to end up, rather than using a > list for this, which is the standard idiom. > > In [2]: import accumulator > > In [3]: a = accumulator.accumulator((1,2,3,4,)) > > In [4]: a > Out[4]: accumulator([1, 2, 3, 4]) > > In [5]: a.append(5) > > In [6]: a > Out[6]: accumulator([1, 2, 3, 4, 5]) > > In [8]: a.extend((6,7,8,9)) > In [9]: a > Out[9]: accumulator([1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > At the moment, it only support 1-d arrays, though I'd like to extend > it to n-d, probably only allowing growing on the first axis. > > This has been discussed on this list a fair bit, with mixed reviews as > to whether there is any point. It's slower than lists in common usage, > but has other advantages -- I'd like to see a C version, but don't > know if I'll ever have the time for that. > > I've enclosed to code for your viewing pleasure > > -Chris > Thanks for this. My aim is to extract a row of data from a line in a file and append it to an array. The number of columns is fixed but, at the start, the number of rows is unknown. I think that I have sorted out the resize approach but I need more tests before I share it. Your accumulator idea is interesting. Back in 2004, I worked on MyMatrix, based on numarray - abandoned when numpy came onto the scene. One of the capabilities there was an /append/ method, intended to add a conforming matrix to the right or below the given matrix. It was probably not efficient but it provided a means of joining together block matrices, The append signature, from a January 2005 backup is here: def append(self, other, toRight= False): ''' Return self, with other appended, to the Right or Below, default: Below. other - a matrix, a list of matrices, or objects which can be converted into matrices. ''' assert self.iscontiguous() assert self.rank == 2 if isinstance(other, _n.NumArray): ... Colin W. From cournape at gmail.com Mon Nov 23 19:55:49 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 24 Nov 2009 09:55:49 +0900 Subject: [Numpy-discussion] recompiling everything ? In-Reply-To: <1cd32cbb0911230753g51142f79sbd67c5a6bc25b3a1@mail.gmail.com> References: <1cd32cbb0911230753g51142f79sbd67c5a6bc25b3a1@mail.gmail.com> Message-ID: <5b8d13220911231655g32f785d9leed2dab7a441d853@mail.gmail.com> On Tue, Nov 24, 2009 at 12:53 AM, wrote: > > Is this only related to the temporary (?) ABI breaking during the > datetime merge, or do we now have to recompile all packages with > c-extension each time we update numpy trunk? Hopefully that's due to the temporary breakage. There is unfortunately no easy way to fix this, because of the way C extensions work in python: we can't rely on the linker to help use, in particular. David From Chris.Barker at noaa.gov Mon Nov 23 20:04:56 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 23 Nov 2009 17:04:56 -0800 Subject: [Numpy-discussion] Resize method In-Reply-To: <4B0B2ED4.7030508@ncf.ca> References: <4B0A9D99.1010403@ncf.ca> <4B0AEC4D.2040608@noaa.gov> <4B0B2ED4.7030508@ncf.ca> Message-ID: <4B0B3138.9050004@noaa.gov> Colin J. Williams wrote: > Thanks for this. My aim is to extract a row of data from a line in a > file and append it to an array. The number of columns is fixed but, at > the start, the number of rows is unknown. That is exactly the kind of use-case I had in mind. In fact, you can use it now if you use a custom dtype to hold your row of data, rather than using a 2-d array. > I think that I have sorted out the resize approach but I need more tests > before I share it. Please do, and consider adding to and/or making suggestions to accumulator -- why should we all re-invent this wheel? To be far, you can do what you need by accumulating in a python list, then making an array out of it when you are done -- and it will probably be faster, but *may* take more memory. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rkdelisle at gmail.com Mon Nov 23 20:07:53 2009 From: rkdelisle at gmail.com (rkdelisle at gmail.com) Date: Tue, 24 Nov 2009 01:07:53 +0000 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 Message-ID: <0015176f0a825794ed047913921e@google.com> An application package that I have requires Python 2.6 and NumPy. I've installed Python 2.6 in a parallel manner as follows: NO modification of the core Python2.4 in /usr/bin has been done. Rather, I installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root) environment variables appropriately. The directory /opt/Python_2.6.4 was modified with chown to give me rwx access. To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to build. When I attempt to configure/build I receive various errors related to blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64, /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas libraries. Within /usr/lib64 I do find a few lapack and blas and lapack libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure is not finding them. No atlas libraries are found, but my understanding is that these are deprecated anyway. As an alternative, I tried to install NumPy using the standard Python 2.4.3 using yum install NumPy but I receive an error saying that NumPy is obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even in the most basic instance, I cannot install NumPy because a deprecated library is seen as higher priority? Even given the generally out of date nature of CentOS this is unrealistic. Finally, I could try to build blas and lapack myself, but this seems to border on insanity. Any help is appreciated. -Kirk -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Nov 23 23:53:59 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 24 Nov 2009 13:53:59 +0900 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: <0015176f0a825794ed047913921e@google.com> References: <0015176f0a825794ed047913921e@google.com> Message-ID: <4B0B66E7.1060905@ar.media.kyoto-u.ac.jp> rkdelisle at gmail.com wrote: > An application package that I have requires Python 2.6 and NumPy. > > I've installed Python 2.6 in a parallel manner as follows: > > NO modification of the core Python2.4 in /usr/bin has been done. > Rather, I installed Python 2.6 under /opt/Python_2.6.4 and modified my > user (not root) environment variables appropriately. The directory > /opt/Python_2.6.4 was modified with chown to give me rwx access. > > To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to > build. When I attempt to configure/build I receive various errors > related to blas and lapack. The NumPy configuration is searching > /usr/lib, /usr/lib64, /usr/local/lib, and /usr/local/lib64 for various > blas, lapack, and atlas libraries. Within /usr/lib64 I do find a few > lapack and blas and lapack libraries installed (libblas.so.3.1.1 and > liblapack.so.3.1.1), but configure is not finding them. You need the *.so and not the *.so.3.1.1. The latter are enough to *run* applications linked against the library, the former is necessary to link against them. IOW, you need the devel packages for blas, lapack (and python). If you want to do it without admin rights, there is no other solution than building them by yourself. David From renesd at gmail.com Tue Nov 24 07:51:32 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 24 Nov 2009 13:51:32 +0100 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <5b8d13220911222219j7b52d768u33f8130feaa001a5@mail.gmail.com> References: <1258954552.8816.78.camel@idol> <5b8d13220911222219j7b52d768u33f8130feaa001a5@mail.gmail.com> Message-ID: <64ddb72c0911240451m1787b92fx6ad80d283fb3843a@mail.gmail.com> On Mon, Nov 23, 2009 at 7:19 AM, David Cournapeau wrote: > On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > >> It might be nice to have this merged in at some point after 1.4.0 (after >> the most obvious glaring bugs have been fixed), so that we could perhaps >> start aiming for Python3 compatibility in Numpy 1.5.0. > > One thing I have on my end is a numpy.distutils which runs under both > python 2 and 3, so that you don't have to run 2to3 everytime you make > a change. > > I did not put it in the trunk because I did not want to modify > numpy.distutils for 1.4.0 at this point, but I will include the > changes as soon as I branch the trunk into 1.4.0, > hi, my tree has this fairly far along (2 and 3 compatible). cheers, From renesd at gmail.com Tue Nov 24 07:53:12 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 24 Nov 2009 13:53:12 +0100 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <1258954552.8816.78.camel@idol> References: <1258954552.8816.78.camel@idol> Message-ID: <64ddb72c0911240453p30a7a8d3v1ade06c423dbebab@mail.gmail.com> awesome work Pauli! From renesd at gmail.com Tue Nov 24 09:09:39 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 24 Nov 2009 15:09:39 +0100 Subject: [Numpy-discussion] Numpy on Python3 In-Reply-To: <0FgNcWJ3mGMP.Y4dhPryx@ltl.tkk.fi> References: <0FgNcWJ3mGMP.Y4dhPryx@ltl.tkk.fi> Message-ID: <64ddb72c0911240609r78eb2611y8d1cc197fb3abc21@mail.gmail.com> On Mon, Nov 23, 2009 at 10:08 AM, Pauli Virtanen wrote: > Setup.py runs 2to3 automatically for all changed files. Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? > hi, see my tree for how I did these things. http://github.com/illume/numpy3k I'm not sure I like how I handled the changed relative imports. However the exception handling is fairly easy/clean to do. cheers, From dyamins at gmail.com Tue Nov 24 09:43:21 2009 From: dyamins at gmail.com (Dan Yamins) Date: Tue, 24 Nov 2009 09:43:21 -0500 Subject: [Numpy-discussion] Problems casting object arrays to string type on Ubuntu In-Reply-To: <15e4667e0911211227rd1cbe70g852702c51878ae26@mail.gmail.com> References: <15e4667e0911211227rd1cbe70g852702c51878ae26@mail.gmail.com> Message-ID: <15e4667e0911240643y396cf452pad4a0ddbad22762e@mail.gmail.com> Really, no idea about this? (Sorry if my original email was unclear.) On Sat, Nov 21, 2009 at 3:27 PM, Dan Yamins wrote: > Hi all, > > I'm having some issues casting object arrays to string type, especially on > my Ubuntu installation. (Ubuntu Jaunty, 9.04, with Numpy v. 1.3.) > > With small arrays, the conversion is just wrong. With large arrays, there > seems to be some memory corruption. Conversion to int or float (when > appropriate) seems to work. > > For instance, here are some examples of errors with small arrays: > > >>> X = numpy.array([13,14],'object') > >>> X.astype('|S2') > array(['13', '\xb2'], > dtype='|S2') > >>> X.astype('int') #conversion to int or float seems to work fine > array([13, 14]) > >>> X.astype(float) > array([ 13., 14.]) > >>> X = numpy.array(['cat','bat'],'object') > >>> X.astype('|S3') > array(['cat', '\x00ba'], > dtype='|S3') > > Large arrays: > > In [24]: X = numpy.array(['cat','bat']*300000,'object') > In [25]: Y = X.astype('|S3') > *** glibc detected *** /usr/bin/python: munmap_chunk(): invalid pointer: > 0xb7cd5008 *** > > > I do NOT have this problem with Numpy 1.4.0.dev7746, installed on my OSX > 10.5.8 machine. There, everything seems to work fine. > > What's going on? I feel like I've seem some traffic about related issues > on the list before, but I couldn't quite tell from reading the threads what > the "final upshot" of the discussion was ... Is this something that was > fixed in recent NumPy 1.4 releases, or is something about my Ubuntu vs. OSX > installations? Generally speaking can I / should I be relying on casting > from object arrays to do the "right" (or "intuitive") thing? > > thanks, > Dan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Nov 24 10:15:12 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Nov 2009 10:15:12 -0500 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> Message-ID: <1cd32cbb0911240715s6ad839ccq36d964f22c567fa1@mail.gmail.com> On Sun, Nov 22, 2009 at 1:40 PM, Charles R Harris wrote: > > > On Sun, Nov 22, 2009 at 6:28 AM, wrote: >> >> On Sun, Nov 22, 2009 at 2:35 AM, David Cournapeau >> wrote: >> > On Wed, Nov 18, 2009 at 6:10 PM, David Cournapeau >> > wrote: >> >> On Wed, Nov 18, 2009 at 12:38 AM, ? wrote: >> >> >> >>> >> >>> Now numpy builds without problems. >> >>> When I run the tests I get 16 failures mostly nan related. I have no >> >>> idea >> >>> whether they are real or if there is still something screwed up in my >> >>> setup. See below. >> >> >> >> I can reproduce those. I did not see those because they are python 2.5 >> >> specific (which is puzzling). >> > >> > I fix most of those, which were related to an atan2 bug in MS C >> > library. Could you test whether this fixes it for you ? >> > >> > David >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> Thanks, >> >> I have 4 failures remaining. full ouput below >> >> Josef >> >> >python >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit >> (Intel)] on >> win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numpy >> >>> numpy.test() >> Running unit tests for numpy >> NumPy version 1.4.0.dev7758 >> NumPy is installed in >> C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.de >> v7758.win32\Programs\Python25\Lib\site-packages\numpy >> Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 >> bit (Int >> el)] >> nose version 0.11.1 >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> .......................K.....................................................FF. >> >> ............................K......................K.F.......................... >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> .....................................C:\Josef\_progs\Subversion\numpy-trunk\dist >> >> \numpy-1.4.0.dev7758.win32\Programs\Python25\Lib\site-packages\numpy\lib\io.py:1 >> 324: ConversionWarning: Some errors were detected ! >> ? ?Line #2 (got 4 columns instead of 5) >> ? ?Line #12 (got 4 columns instead of 5) >> ? ?Line #22 (got 4 columns instead of 5) >> ? ?Line #32 (got 4 columns instead of 5) >> ? ?Line #42 (got 4 columns instead of 5) >> ?warnings.warn(errmsg, ConversionWarning) >> >> .C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Programs\ >> Python25\Lib\site-packages\numpy\lib\io.py:1324: ConversionWarning: Some >> errors >> were detected ! >> ? ?Line #2 (got 4 columns instead of 2) >> ? ?Line #12 (got 4 columns instead of 2) >> ? ?Line #22 (got 4 columns instead of 2) >> ? ?Line #32 (got 4 columns instead of 2) >> ? ?Line #42 (got 4 columns instead of 2) >> ?warnings.warn(errmsg, ConversionWarning) >> >> ..........................................K........K............F............... >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ......S......................................................................... >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> >> ................................................................................ >> ................................................... >> ====================================================================== >> FAIL: test_umath.test_nextafter >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File >> "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p >> y", line 183, in runTest >> ? ?self.test(*self.arg) >> ?File >> "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr >> ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line >> 851, in >> test_nextafter >> ? ?assert np.nextafter(one, two) - one == eps >> AssertionError >> >> ====================================================================== >> FAIL: test_umath.test_spacing >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File >> "c:\programs\python25\lib\site-packages\nose-0.11.1-py2.5.egg\nose\case.p >> y", line 183, in runTest >> ? ?self.test(*self.arg) >> ?File >> "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr >> ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath.py", line >> 863, in >> test_spacing >> ? ?assert np.spacing(one) == eps >> AssertionError >> >> ====================================================================== >> FAIL: test_special_values (test_umath_complex.TestClog) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File >> "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr >> ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", >> line >> 179, in test_special_values >> ? ?assert_almost_equal(np.log(x), y) >> ?File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line >> 436, >> in assert_almost_equal >> AssertionError: Items are not equal: >> ACTUAL: [ NaN+2.35619449j] >> DESIRED: (1.#INF+2.35619449019j) >> >> >> ====================================================================== >> FAIL: test_doctests (test_polynomial.TestDocs) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File >> "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Pr >> ograms\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", >> line 90, >> in test_doctests >> ? ?return rundocs() >> ?File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line >> 952, >> in rundocs >> AssertionError: Some doctests failed: >> ********************************************************************** >> File >> "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7758.win32\Prog >> rams\Python25\Lib\site-packages\numpy\lib\tests\test_polynomial.py", line >> 20, in >> ?test_polynomial >> Failed example: >> ? ?print poly1d([100e-90, 1.234567e-9j+3, -1234.999e8]) >> Expected: >> ? ? ? ? ? 2 >> ? ?1e-88 x + (3 + 1.235e-09j) x - 1.235e+11 >> Got: >> ? ? ? ? ? ?2 >> ? ?1e-088 x + (3 + 1.235e-009j) x - 1.235e+011 >> >> > > I can maybe fix this, although I'm not sure it's worth the effort and I > don't have a way to test it. There is another bug in the poly1d package that > needs fixing. David, what is the preferred way to do that now? > > > Also, there are fixes for the MS atan2 problems in the WTF_MatExtras file. > I think I sent you a copy once upon a time. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > update, with revision 7769: Ran 2335 tests in 13.250s FAILED (KNOWNFAIL=7, SKIP=1, failures=2) ====================================================================== FAIL: test_special_values (test_umath_complex.TestClog) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Josef\_progs\Subversion\numpy-trunk\dist\numpy-1.4.0.dev7769.win32\Pr ograms\Python25\Lib\site-packages\numpy\core\tests\test_umath_complex.py", line 179, in test_special_values assert_almost_equal(np.log(x), y) File "\Programs\Python25\Lib\site-packages\numpy\testing\utils.py", line 437, in assert_almost_equal AssertionError: Items are not equal: ACTUAL: [ NaN+2.35619449j] DESIRED: (1.#INF+2.35619449019j) some examples based on test_umath_complex.TestClog show differences between complex128 and complex64 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.version.version '1.4.0.dev7769' >>> x = np.array([complex(-np.inf, np.inf),complex(np.inf, -np.inf),complex(-0,np.inf),complex(-1e300, np.inf)], dtype=np.complex) >>> np.log(x) array([ NaN+2.35619449j, NaN-0.78539816j, Inf Infj, Inf+1.57079633j]) >>> x64 = np.array([complex(-np.inf, np.inf),complex(np.inf, -np.inf),complex(-0, np.inf),complex(-1e300, np.inf)], dtype=np.complex64) >>> np.log(x64) array([ Inf+2.3561945j , Inf-0.78539819j, Inf+1.57079637j, Inf+2.3561945j ], dtype=complex64) I also ran the test of some old scipy against the trunk numpy: scipy.0.7.0('4826') finishes tests with some display errors scipy0.7.1rc3 test suite crashes both scipy versions build by David, according to __config__.py (scipy build against trunk a week ago has no problems) Josef From mdroe at stsci.edu Tue Nov 24 10:58:29 2009 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 24 Nov 2009 10:58:29 -0500 Subject: [Numpy-discussion] Problems casting object arrays to string type on Ubuntu In-Reply-To: <15e4667e0911240643y396cf452pad4a0ddbad22762e@mail.gmail.com> References: <15e4667e0911211227rd1cbe70g852702c51878ae26@mail.gmail.com> <15e4667e0911240643y396cf452pad4a0ddbad22762e@mail.gmail.com> Message-ID: <4B0C02A5.2000308@stsci.edu> Sorry I missed your message the first time around. I suspect you're running into this bug: http://projects.scipy.org/numpy/ticket/1235 This has been fixed in SVN r7514, (and thus 1.4 pre releases), so it's not a platform difference. The workaround (at the expense of performance, of course) is to cast to a list then cast to a string array. I'm not sure if there's a better (i.e. more performant) workaround. But you could (in your client code) check for the version of Numpy and act accordingly. Mike Dan Yamins wrote: > Really, no idea about this? (Sorry if my original email was unclear.) > > On Sat, Nov 21, 2009 at 3:27 PM, Dan Yamins > wrote: > > Hi all, > > I'm having some issues casting object arrays to string type, > especially on my Ubuntu installation. (Ubuntu Jaunty, 9.04, with > Numpy v. 1.3.) > > With small arrays, the conversion is just wrong. With large > arrays, there seems to be some memory corruption. Conversion to > int or float (when appropriate) seems to work. > > For instance, here are some examples of errors with small arrays: > > >>> X = numpy.array([13,14],'object') > >>> X.astype('|S2') > array(['13', '\xb2'], > dtype='|S2') > >>> X.astype('int') #conversion to int or float seems to > work fine > array([13, 14]) > >>> X.astype(float) > array([ 13., 14.]) > >>> X = numpy.array(['cat','bat'],'object') > >>> X.astype('|S3') > array(['cat', '\x00ba'], > dtype='|S3') > > Large arrays: > > In [24]: X = numpy.array(['cat','bat']*300000,'object') > In [25]: Y = X.astype('|S3') > *** glibc detected *** /usr/bin/python: munmap_chunk(): invalid > pointer: 0xb7cd5008 *** > > > I do NOT have this problem with Numpy 1.4.0.dev7746, installed on > my OSX 10.5.8 machine. There, everything seems to work fine. > > What's going on? I feel like I've seem some traffic about related > issues on the list before, but I couldn't quite tell from reading > the threads what the "final upshot" of the discussion was ... > Is this something that was fixed in recent NumPy 1.4 releases, or > is something about my Ubuntu vs. OSX installations? Generally > speaking can I / should I be relying on casting from object arrays > to do the "right" (or "intuitive") thing? > > thanks, > Dan > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From gokhansever at gmail.com Tue Nov 24 11:33:14 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 24 Nov 2009 10:33:14 -0600 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: <0015176f0a825794ed047913921e@google.com> References: <0015176f0a825794ed047913921e@google.com> Message-ID: <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> On Mon, Nov 23, 2009 at 7:07 PM, wrote: > An application package that I have requires Python 2.6 and NumPy. > > I've installed Python 2.6 in a parallel manner as follows: > > NO modification of the core Python2.4 in /usr/bin has been done. Rather, I > installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root) > environment variables appropriately. The directory /opt/Python_2.6.4 was > modified with chown to give me rwx access. > > To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to > build. When I attempt to configure/build I receive various errors related to > blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64, > /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas > libraries. Within /usr/lib64 I do find a few lapack and blas and lapack > libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure > is not finding them. No atlas libraries are found, but my understanding is > that these are deprecated anyway. > > As an alternative, I tried to install NumPy using the standard Python 2.4.3 > using yum install NumPy but I receive an error saying that NumPy is > obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even > in the most basic instance, I cannot install NumPy because a deprecated > library is seen as higher priority? Even given the generally out of date > nature of CentOS this is unrealistic. > > Finally, I could try to build blas and lapack myself, but this seems to > border on insanity. > > Any help is appreciated. > > -Kirk > I had success installing NumPy 1.3 to two of our RHEL 5.x (x86) machines using the pre-built packages from this repository: http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ RHEL 5.x serie has Python 2.4.3 by default as you mentioned. Installation of NumPy et. al is far easier in Fedora (mine is still at version 11) from the source check-out plus manual installation of dependencies. I haven't tried installing a Python 2.6.x to RHEL machines. I would like to see an update from if you succeed with the mentioned repository. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcenerney1 at llnl.gov Tue Nov 24 11:57:26 2009 From: mcenerney1 at llnl.gov (James McEnerney) Date: Tue, 24 Nov 2009 08:57:26 -0800 Subject: [Numpy-discussion] f2py callback bug? Message-ID: <893bk3$1qhn5c@nspiron-1.llnl.gov> An HTML attachment was scrubbed... URL: From rkdelisle at gmail.com Tue Nov 24 12:11:42 2009 From: rkdelisle at gmail.com (Robert DeLisle) Date: Tue, 24 Nov 2009 10:11:42 -0700 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> References: <0015176f0a825794ed047913921e@google.com> <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> Message-ID: David Cournapeau wrote: You need the *.so and not the *.so.3.1.1. The latter are enough to *run* applications linked against the library, the former is necessary to link against them. IOW, you need the devel packages for blas, lapack (and python). If you want to do it without admin rights, there is no other solution than building them by yourself. David, Thank you so much! That helps a lot and I don't know why I didn't think of it myself. As root I'm able to install blas-devel and lapack-devel which gives me new libraries in /usr/lib64. Specifically, libblas.a, liblapack.a, and liblapack_pic.a. Now when I run setup.py from the numpy directory, it says that it finds the libraries in the appropriate locations. A few more issues: The configuration complains that there are no mkl, vml, guide libraries. Also, atlas libraries are not found. I cannot seem to find these at this point. The configuration also complains about no Fortran 90 compiler. I do have f77 and f95 as part of the GCC package. Is this enough. The build and install step seems to go OK despite the above warnings but then it fails with the error could not create /usr/lib/python2.6. So, I did an in place build and I see some libraries in the /build directory but there is no python2.6 file that I can find. What are the steps that I need to take in order to manually install these, and which libraries should I be paying attention to? Just to clarify, I'm running this NOT as root in order to be able to use Python 2.6. I have installed Python 2.6 under /opt/Python_2.6.4 and adjusted my own environment variables to target the appropriate version. (This should keep CentOS from breaking by retaining the system python, v.2.4.3.) I've also made the /opt/Python_2.6.4 directory writable by my user account to facilitate automated steps. Any help you can supply is very greatly appreciated. -Kirk On Tue, Nov 24, 2009 at 9:33 AM, G?khan Sever wrote: > On Mon, Nov 23, 2009 at 7:07 PM, wrote: > >> An application package that I have requires Python 2.6 and NumPy. >> >> I've installed Python 2.6 in a parallel manner as follows: >> >> NO modification of the core Python2.4 in /usr/bin has been done. Rather, I >> installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root) >> environment variables appropriately. The directory /opt/Python_2.6.4 was >> modified with chown to give me rwx access. >> >> To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to >> build. When I attempt to configure/build I receive various errors related to >> blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64, >> /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas >> libraries. Within /usr/lib64 I do find a few lapack and blas and lapack >> libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure >> is not finding them. No atlas libraries are found, but my understanding is >> that these are deprecated anyway. >> >> As an alternative, I tried to install NumPy using the standard Python >> 2.4.3 using yum install NumPy but I receive an error saying that NumPy is >> obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even >> in the most basic instance, I cannot install NumPy because a deprecated >> library is seen as higher priority? Even given the generally out of date >> nature of CentOS this is unrealistic. >> >> Finally, I could try to build blas and lapack myself, but this seems to >> border on insanity. >> >> Any help is appreciated. >> >> -Kirk >> > > I had success installing NumPy 1.3 to two of our RHEL 5.x (x86) machines > using the pre-built packages from this repository: > http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ > > RHEL 5.x serie has Python 2.4.3 by default as you mentioned. Installation > of NumPy et. al is far easier in Fedora (mine is still at version 11) from > the source check-out plus manual installation of dependencies. I haven't > tried installing a Python 2.6.x to RHEL machines. I would like to see an > update from if you succeed with the mentioned repository. > > > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkdelisle at gmail.com Tue Nov 24 12:13:44 2009 From: rkdelisle at gmail.com (Robert DeLisle) Date: Tue, 24 Nov 2009 10:13:44 -0700 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> References: <0015176f0a825794ed047913921e@google.com> <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> Message-ID: Gokhan, Thank you for the referred repo. I'll keep this as a reference for future endeavours should I fail given my current approach. As for installing Python 2.6.4, that was very easily done as a parallel installation as I described. I'm more than happy to give you more details if you need them. -Kirk On Tue, Nov 24, 2009 at 9:33 AM, G?khan Sever wrote: > On Mon, Nov 23, 2009 at 7:07 PM, wrote: > >> An application package that I have requires Python 2.6 and NumPy. >> >> I've installed Python 2.6 in a parallel manner as follows: >> >> NO modification of the core Python2.4 in /usr/bin has been done. Rather, I >> installed Python 2.6 under /opt/Python_2.6.4 and modified my user (not root) >> environment variables appropriately. The directory /opt/Python_2.6.4 was >> modified with chown to give me rwx access. >> >> To install NumPy, I've downloaded the latest .tgz sources (v1.3.0) to >> build. When I attempt to configure/build I receive various errors related to >> blas and lapack. The NumPy configuration is searching /usr/lib, /usr/lib64, >> /usr/local/lib, and /usr/local/lib64 for various blas, lapack, and atlas >> libraries. Within /usr/lib64 I do find a few lapack and blas and lapack >> libraries installed (libblas.so.3.1.1 and liblapack.so.3.1.1), but configure >> is not finding them. No atlas libraries are found, but my understanding is >> that these are deprecated anyway. >> >> As an alternative, I tried to install NumPy using the standard Python >> 2.4.3 using yum install NumPy but I receive an error saying that NumPy is >> obsoleted by PyNumeric. What?? PyNumeric is the precursor to NumPy. So even >> in the most basic instance, I cannot install NumPy because a deprecated >> library is seen as higher priority? Even given the generally out of date >> nature of CentOS this is unrealistic. >> >> Finally, I could try to build blas and lapack myself, but this seems to >> border on insanity. >> >> Any help is appreciated. >> >> -Kirk >> > > I had success installing NumPy 1.3 to two of our RHEL 5.x (x86) machines > using the pre-built packages from this repository: > http://download.opensuse.org/repositories/home:/ashigabou/CentOS_5/ > > RHEL 5.x serie has Python 2.4.3 by default as you mentioned. Installation > of NumPy et. al is far easier in Fedora (mine is still at version 11) from > the source check-out plus manual installation of dependencies. I haven't > tried installing a Python 2.6.x to RHEL machines. I would like to see an > update from if you succeed with the mentioned repository. > > > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyamins at gmail.com Tue Nov 24 14:46:43 2009 From: dyamins at gmail.com (Dan Yamins) Date: Tue, 24 Nov 2009 14:46:43 -0500 Subject: [Numpy-discussion] Problems casting object arrays to string type on Ubuntu In-Reply-To: <4B0C02A5.2000308@stsci.edu> References: <15e4667e0911211227rd1cbe70g852702c51878ae26@mail.gmail.com> <15e4667e0911240643y396cf452pad4a0ddbad22762e@mail.gmail.com> <4B0C02A5.2000308@stsci.edu> Message-ID: <15e4667e0911241146i3d9dd77dxd598c80aa56746b6@mail.gmail.com> > I suspect you're running into this bug: > > http://projects.scipy.org/numpy/ticket/1235 > > This has been fixed in SVN r7514, (and thus 1.4 pre releases), so it's > not a platform difference. > Great, thanks for the reply! best, Dan > > > wrote: > > > > Hi all, > > > > I'm having some issues casting object arrays to string type, > > especially on my Ubuntu installation. (Ubuntu Jaunty, 9.04, with > > Numpy v. 1.3.) > > > > With small arrays, the conversion is just wrong. With large > > arrays, there seems to be some memory corruption. Conversion to > > int or float (when appropriate) seems to work. > > > > For instance, here are some examples of errors with small arrays: > > > > >>> X = numpy.array([13,14],'object') > > >>> X.astype('|S2') > > array(['13', '\xb2'], > > dtype='|S2') > > >>> X.astype('int') #conversion to int or float seems to > > work fine > > array([13, 14]) > > >>> X.astype(float) > > array([ 13., 14.]) > > >>> X = numpy.array(['cat','bat'],'object') > > >>> X.astype('|S3') > > array(['cat', '\x00ba'], > > dtype='|S3') > > > > Large arrays: > > > > In [24]: X = numpy.array(['cat','bat']*300000,'object') > > In [25]: Y = X.astype('|S3') > > *** glibc detected *** /usr/bin/python: munmap_chunk(): invalid > > pointer: 0xb7cd5008 *** > > > > > > I do NOT have this problem with Numpy 1.4.0.dev7746, installed on > > my OSX 10.5.8 machine. There, everything seems to work fine. > > > > What's going on? I feel like I've seem some traffic about related > > issues on the list before, but I couldn't quite tell from reading > > the threads what the "final upshot" of the discussion was ... > > Is this something that was fixed in recent NumPy 1.4 releases, or > > is something about my Ubuntu vs. OSX installations? Generally > > speaking can I / should I be relying on casting from object arrays > > to do the "right" (or "intuitive") thing? > > > > thanks, > > Dan > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Michael Droettboom > Science Software Branch > Operations and Engineering Division > Space Telescope Science Institute > Operated by AURA for NASA > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Tue Nov 24 16:33:55 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Tue, 24 Nov 2009 13:33:55 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions Message-ID: <26503657.post@talk.nabble.com> Hi I have written some code and I would appreciate any suggestions to make better use of the numpy arrays functions to make it a bit more efficient and less of a port from C. Any tricks are thoughts would be much appreciated. The code reads in a series of images, collects a running total if the value is valid and averages everything every 8 images. Code below... Thanks in advance... #!/usr/bin/env python """ Average the daily LST values to make an 8 day product... Martin De Kauwe 18th November 2009 """ import sys, os, glob import numpy as np import matplotlib.pyplot as plt def averageEightDays(files, lst, count, numrows, numcols): day_count = 0 # starting day - tag for output file doy = 122 # loop over all the images and average all the information # every 8 days... for fname in glob.glob(os.path.join(path, files)): current_file = fname.split('/')[8] year = current_file.split('_')[2][0:4] try: f = open(fname, 'rb') except IOError: print "Cannot open outfile for read", fname sys.exit(1) data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) f.close() # if it is day 1 then we need to fill up the holding array... # copy the first file into the array... if day_count == 0: lst = data for i in xrange(numrows): for j in xrange(numcols): # increase the pixel count if we got vaild data (undefined = -999.0 if lst[i,j] > -900.: count[i,j] = 1 day_count += 1 # keep a running total of valid lst values in an 8 day sequence elif day_count > 0 and day_count <= 7: for i in xrange(numrows): for j in xrange(numcols): # lst valid pixel? if data[i,j] > -900.: # was the existing pixel valid? if lst[i,j] < -900.: lst[i,j] = data[i,j] count[i,j] = 1 else: lst[i,j] += data[i,j] count[i,j] += 1 day_count += 1 # need to average previous 8 days and write to a file... if day_count == 8: for i in xrange(numrows): for j in xrange(numcols): if count[i,j] > 0: lst[i,j] = lst[i,j] / count[i,j] else: lst[i,j] = -999.0 day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) # it is possible we didn't have enough slices to do the last 8day avg... # but if we have more than one day we shall use it # need to average previous 8 days and write to a file... if day_count > 1: for i in xrange(numrows): for j in xrange(numcols): if count[i,j] > 0: lst[i,j] = lst[i,j] / count[i,j] else: lst[i,j] = -999.0 day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) def write_outputs(lst, count, year, doy): path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" try: of = open(os.path.join(path, outfile), 'wb') except IOError: print "Cannot open outfile for write", outfile sys.exit(1) outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" try: of2 = open(os.path.join(path, outfile2), 'wb') except IOError: print "Cannot open outfile for write", outfile2 sys.exit(1) # empty stuff and zero 8day counts lst.tofile(of) count.tofile(of2) of.close() of2.close() lst = 0.0 count = 0.0 return lst, count if __name__ == "__main__": numrows = 332 numcols = 667 path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" lst = np.zeros((numrows, numcols), dtype=np.float32) count = np.zeros((numrows, numcols), dtype=np.int) averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) lst = 0.0 count = 0.0 averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26503657.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From teoliphant at gmail.com Tue Nov 24 18:24:25 2009 From: teoliphant at gmail.com (Travis Oliphant) Date: Tue, 24 Nov 2009 17:24:25 -0600 Subject: [Numpy-discussion] datetime update In-Reply-To: <3188844F-75BD-4AE4-8CA6-D2859C69FBF4@gmail.com> References: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> <3188844F-75BD-4AE4-8CA6-D2859C69FBF4@gmail.com> Message-ID: <5FB0A71F-C3AE-4E5B-A5DE-6ABDD0FDAF73@enthought.com> On Nov 23, 2009, at 6:53 PM, Pierre GM wrote: > On Nov 23, 2009, at 6:42 PM, Travis Oliphant wrote: >> >> >> I've made a few changes to datetime today and spent some time >> looking over what is there and what remains to be implemented. > > As always, many thanks for your work !! > >> Basically, the biggest thing left to do is to implement the low- >> level casting functions to and from datetime types and other numpy >> types. > > >> In addition, the ufuncs need some auditing to make sure the right >> thing is being done when mixing different units. After that, lots >> and lots of additional tests need to be written. Once that is >> done, then most of the features should be available, but I suspect >> a few lingering issues might crop up and require fixing or fleshing >> out as well. >> >> I was hoping that someone would be able to contribute more tests >> for datetime. I will spend some time on the casting functions >> over the next few weeks and write a few tests. > > Fortunately, the new modifications will make it easier to write such > tests... But in any case, we can assume that what is proposed in the > NEP should work, right ? Yes, that is correct. > >> I also removed numpy.datetime and numpy.timedelta from the >> namespace (replaced them with numpy.datetime_ and >> numpy.timedelta_). These were just short-hand for >> numpy.datetime64 and numpy.timedelta64 respectively. Avoiding the >> collision seemed like a good idea. >> >> Right now, what works is "viewing" arrays as datetime data-types >> and getting and setting date-time arrays using datetime objects. >> I would like to improve it so that setting with strings, integers, >> and other Python objects works as well. > > > Did you use any of Marty Fuhry's GSoC work ? What are the potential > issues that could prevent an easy integration ? Yes, I did actually. His work was quite helpful in converting from date-time objects. The major issues were the approach taken to a few of the functions was not quite right, but quite a bit of the raw code I just used. Marty deserves kudos for his work here. It was very useful and helpful. Please pass that on to him. > > >> Also, adding simple integers works, but >> >> Dave C suggested removing the new C-API calls which sounds like a >> good idea to me for 1.4.0. Which functions get exported into the >> C-API for 1.5.0 could then receive some discussion. > > Wouldn't it be easier to leave the C-APi as it is now, even for > 1.4.0, but not to advertize it before 1.5.0 ? Not necessarily. My understanding is that we just have to turn-off exposure of the API and leave everything else the same. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Nov 24 21:55:23 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 11:55:23 +0900 Subject: [Numpy-discussion] datetime update In-Reply-To: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> References: <5A71C3BA-E8CC-47FD-BB58-6AFD3E1425C6@enthought.com> Message-ID: <5b8d13220911241855vc8ede22h26d73f9cb9010a06@mail.gmail.com> On Tue, Nov 24, 2009 at 8:42 AM, Travis Oliphant wrote: > > I've made a few changes to datetime today and spent some time looking over > what is there and what remains to be implemented. > Basically, the biggest thing left to do is to implement the low-level > casting functions to and from datetime types and other numpy types. ?In > addition, the ufuncs need some auditing to make sure the right thing is > being done when mixing different units. ?After that, lots and lots of > additional tests need to be written. ? ? ?Once that is done, then most of > the features should be available, but I suspect a few lingering issues might > crop up and require fixing or fleshing out as well. > I was hoping that someone would be able to contribute more tests for > datetime. ? ?I will spend some time on the casting functions over the next > few weeks and write a few tests. > I fixed a problem today with the fact that PyArray_DescrFromScalar was not > returning a data-type object with the correct frequency information stored > when given a datetime64 or timedelta64 scalar (it was ignoring the date-time > metadata on the scalar). ? This fixed a problem with the printing so that > now?a = arange(10).view('M8[Y]')?shows something reasonable. > I also removed numpy.datetime and numpy.timedelta from the namespace > (replaced them with numpy.datetime_ and numpy.timedelta_). ? These were just > short-hand for numpy.datetime64 and numpy.timedelta64 respectively. > Avoiding the collision seemed like a good idea. > Right now, what works is "viewing" arrays as datetime data-types and getting > and setting date-time arrays using datetime objects. ? I would like to > improve it so that setting with strings, integers, and other Python objects > works as well. Thanks for the update, Travis, that's really helpful. >? ?Also, adding simple integers works, but > Dave C suggested removing the new C-API calls which sounds like a good idea > to me for 1.4.0. So that's clear for everyone, this just means that the functions will not be exported (i.e. 3rd party C extensions cannot use those functions), the code will still be there. The rationale is that if the functions need to be changed for any reason, it will break backward compatibility for *any* package using numpy C api (scipy, matplotlib, etc...). By not exporting those functions, we allow ourselves to change them in the 1.5.0 timeframe, at the much cheaper cost of preventing 3rd party extensions to use datetime stuff from C for 1.4.0 (but it can still be used from python). David From cournape at gmail.com Tue Nov 24 21:57:19 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 11:57:19 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <432197.42695.qm@web51001.mail.re2.yahoo.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> Message-ID: <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> Hi Olivia, On Mon, Nov 23, 2009 at 6:51 AM, Olivia Cheronet wrote: > Hello, > > ?I attempted to install Numpy for my Cygwin python again, by simply executing: > ?>python setup.py install > > ?However, I now get the following: > ?>File "numpy/core/setup.py", line 253, in check_mathlib > ?> ?raise EnvironmentError("math library missing; rerun " > ?>EnvironmentError: math library missing; rerun setup.py after setting the > ?MATHLIB > ?>env variable > > > ?I have a math library from cygwin (libm.a), but I have not managed to set it. The message is misleading (the message is hopefully improved with the last trunk), and it is almost always caused by a broken toolchain (that is you are missing gcc, or g++, etc...). It would be helpful to have the full build log to help you better, David From cournape at gmail.com Tue Nov 24 22:02:53 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 12:02:53 +0900 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: References: <0015176f0a825794ed047913921e@google.com> <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> Message-ID: <5b8d13220911241902n47597042ke4e53cae5948aab5@mail.gmail.com> On Wed, Nov 25, 2009 at 2:11 AM, Robert DeLisle wrote: > David Cournapeau wrote: > > You need the *.so and not the *.so.3.1.1. The latter are enough to *run* > applications linked against the library, the former is necessary to link > against them. IOW, you need the devel packages for blas, lapack (and > python). If you want to do it without admin rights, there is no other > solution than building them by yourself. > > > > David, > > Thank you so much!? That helps a lot and I don't know why I didn't think of > it myself.? As root I'm able to install blas-devel and lapack-devel which > gives me new libraries in /usr/lib64.? Specifically, libblas.a, liblapack.a, > and liblapack_pic.a.? Now when I run setup.py from the numpy directory, it > says that it finds the libraries in the appropriate locations. > > A few more issues: > > The configuration complains that there are no mkl, vml, guide libraries. > Also, atlas libraries are not found.? I cannot seem to find these at this > point. That's ok. Our build process is too verbose, and is a bit confusing. MKL and Atlas are just optimized BLAS/LAPACK, but they are not necessary to build scipy. They can significantly improve scipy speed for linear algebra stuff, but building Atlas or linking against the Intel MKL is complicated. cheers, David From cournape at gmail.com Tue Nov 24 22:16:52 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 12:16:52 +0900 Subject: [Numpy-discussion] failure building trunk with mingw In-Reply-To: <1cd32cbb0911240715s6ad839ccq36d964f22c567fa1@mail.gmail.com> References: <1cd32cbb0911151226t423585a6x246c6d6f7221b040@mail.gmail.com> <4B00E7B7.9010608@ar.media.kyoto-u.ac.jp> <1cd32cbb0911161033v76aa629ck33194a668634c38b@mail.gmail.com> <5b8d13220911161952m3fd013b5re6a079f5775f05a1@mail.gmail.com> <1cd32cbb0911170738l27375508xefca2bc876c97aed@mail.gmail.com> <5b8d13220911180110i5931b9b1j748c983518417cda@mail.gmail.com> <5b8d13220911212335l226b912dt4cfc03ac357284e5@mail.gmail.com> <1cd32cbb0911220528t20893a0awc100d608eb68b59@mail.gmail.com> <1cd32cbb0911240715s6ad839ccq36d964f22c567fa1@mail.gmail.com> Message-ID: <5b8d13220911241916q5d4ae1a0x313d87af7de4a232@mail.gmail.com> On Wed, Nov 25, 2009 at 12:15 AM, wrote: > > > I also ran the test of some old scipy against the trunk numpy: > > scipy.0.7.0('4826') finishes tests with some display errors > scipy0.7.1rc3 test suite crashes > both scipy versions build by David, according to ?__config__.py I have build scipy 0.7.1 against numpy 1.3.0, and then tested the installed scipy when run against numpy trunk, and I did not get any crash. It would be nice to have a way to test this automatically - the recent changes I have made to track API changes should hopefully be more robust, but there is no way to guarantee that we did not change more subtle problems like struct changing their layout, etc... without a thorough test of our C API. cheers, David From cheronetolivia at yahoo.com Wed Nov 25 02:40:47 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Tue, 24 Nov 2009 23:40:47 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> Message-ID: <544434.51110.qm@web51003.mail.re2.yahoo.com> Hello David, Below is the full log. Thanks very much for the help. Olivia Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/system_info.py:1383: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources Generating build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c success! removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prot otypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype _configtest.c: In function `main': _configtest.c:5: error: size of array `test_array' is negative _configtest.c:4: warning: function declaration isn't a prototype _configtest.c: In function `main': _configtest.c:5: error: size of array `test_array' is negative C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype _configtest.c: In function `main': _configtest.c:5: error: size of array `test_array' is negative _configtest.c:4: warning: function declaration isn't a prototype _configtest.c: In function `main': _configtest.c:5: error: size of array `test_array' is negative C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:6: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:6: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:5: warning: function declaration isn't a prototype success! removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:6: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:6: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:4: warning: function declaration isn't a prototype removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c _configtest.c:5: warning: function declaration isn't a prototype success! removing: _configtest.c _configtest.o /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/config.py:39: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of try_run is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c gcc _configtest.o -llibm.a -o _configtest.exe /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c gcc _configtest.o -o _configtest.exe /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c gcc _configtest.o -lm -o _configtest.exe /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c gcc _configtest.o -lcpml -o _configtest.exe /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 172, in setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/usr/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/install.py", line 49, in run r = old_install.run(self) File "/usr/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/usr/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 289, in generate_config_h mathlibs = check_mathlib(config_cmd) File "numpy/core/setup.py", line 253, in check_mathlib raise EnvironmentError("math library missing; rerun " EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB env variable ----- Original Message ---- > Hi Olivia, > > On Mon, Nov 23, 2009 at 6:51 AM, Olivia Cheronet > wrote: > > Hello, > > > > I attempted to install Numpy for my Cygwin python again, by simply executing: > > >python setup.py install > > > > However, I now get the following: > > >File "numpy/core/setup.py", line 253, in check_mathlib > > > raise EnvironmentError("math library missing; rerun " > > >EnvironmentError: math library missing; rerun setup.py after setting the > > MATHLIB > > >env variable > > > > > > I have a math library from cygwin (libm.a), but I have not managed to set it. > > The message is misleading (the message is hopefully improved with the > last trunk), and it is almost always caused by a broken toolchain > (that is you are missing gcc, or g++, etc...). > > It would be helpful to have the full build log to help you better, > > David From pearu.peterson at gmail.com Wed Nov 25 02:58:46 2009 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Wed, 25 Nov 2009 09:58:46 +0200 Subject: [Numpy-discussion] f2py callback bug? In-Reply-To: <893bk3$1qhn5c@nspiron-1.llnl.gov> References: <893bk3$1qhn5c@nspiron-1.llnl.gov> Message-ID: <4B0CE3B6.304@cens.ioc.ee> Hi, It is not really a bug what you are seeing.. In pycalc when assigning j = 20 * j you create a new object `j` and the argument object `j`, that links back to Fortran data, gets discarded. So, you can change j inplace, for example: j[:] = 20*j The first argument `i` is int object that in Python is immutable, and hence cannot be changed inplace and the corresponding Fortran scalar object cannot be changed in the callback (in fact, `i` is a copy of the corresponding Fortran data value). To change `i` in Python callback, define it as an array (similar to `j`) and do inplace operations with it. Regarding pyreturn and assuming Fortran 77: you cannot return arrays or multiple object in Fortran functions. Fortran functions may return only scalars. So, the pyreturn trick will never work. The solution is to change arguments in place as with `j`. Hmm, regarding `intent(in, out) j`, this should work. I'll check what is going on.. HTH, Pearu James McEnerney wrote: > While using the call-back feature of f2py I stumbled across what appears > to be a bug and I'm asking the community to look into this. > > Background: I'm in the middle of converting some legacy fortran to python. > There is one routine that is particulary thorny that calls more easily > convertible service routines and my intention is to convert the latter > and use the callback feature of f2py to execute them within the fortran > followed by a systematic conversion of what remains. This seems to be > doable from what I've read on callback. I have not seen an example > of using callback where the python actually changes parameters that are > returned to the fortran; this is a requirement for me. While setting up > an example to illustrate this I came across a syntactically correct > situation(this means it compiles & executes) but gives the wrong answer. > Here's the code: > In fortran, source foo.f > subroutine calc(i, j) > Cf2py intent(callback) pycalc > external pycalc > Cf2py integer intent(in,out,copy):: i > Cf2py integer dimension(1), intent(in,out):: j > integer pyreturn > > integer i, j(1) > print *, 'in fortran before pycalc ','i=',i, ' j=', j(1) > pyreturn = pycalc(i, j) > print *, 'in fortran after pycalc ','i=', i, ' j=', j(1) > > end > > Standard build: f2py -c -m foo foo.f > > In python, execute > import foo,numpy > > def pycalc(i, j): > print ' in pycalc ', 'i=',i, 'j=', j > i=10*i > j = 20*j > return i, j > > print foo.calc.__doc__ > i=2 > j = 1+numpy.array([i]) > print foo.calc(i,j, pycalc) > > Here's the output: > calc - Function signature: > i,j = calc(i,j,pycalc,[pycalc_extra_args]) > Required arguments: > i : input int > j : input rank-1 array('i') with bounds (1) > pycalc : call-back function > Optional arguments: > pycalc_extra_args := () input tuple > Return objects: > i : int > j : rank-1 array('i') with bounds (1) > Call-back functions: > def pycalc(i,j): return pyreturn,i,j > Required arguments: > i : input int > j : input rank-1 array('i') with bounds (1) > Return objects: > pyreturn : int > i : int > j : rank-1 array('i') with bounds (1) > > > in fortran before pycalc i= 2 j= 3 > in pycalc i= 2 j= [3] > in fortran after pycalc i= 60 j= 3 > (60, array([3])) > > The bug: > on return to the fortran why is i=60 & j=3? > shouldn't it be i=10 & j=60 > > While that's what I expect, I might not be defining the > interface properly; but this compiles & executes. If this > is incorrect, what is? In the fortran, pyreturn appears > to be an address; how do I get the retuned values? > > I'm running > Redhat Linux > python version 2.5 > f2py version 2_3979 > numpy version 1.0.3.1 > Thanks > > Jim McEnerney > Lawrence Livermore National Laboratory > 7000 East Ave. > Livermore, Ca. 94550-9234 > > USA > > 925-422-1963 > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pearu.peterson at gmail.com Wed Nov 25 03:06:39 2009 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Wed, 25 Nov 2009 10:06:39 +0200 Subject: [Numpy-discussion] f2py callback bug? In-Reply-To: <4B0CE3B6.304@cens.ioc.ee> References: <893bk3$1qhn5c@nspiron-1.llnl.gov> <4B0CE3B6.304@cens.ioc.ee> Message-ID: <4B0CE58F.6040402@cens.ioc.ee> Pearu Peterson wrote: > Hmm, regarding `intent(in, out) j`, this should work. I'll check what > is going on.. The `intent(in, out) j` works when pycalc is defined as subroutine: call pycalc(i, j) instead of pyreturn = pycalc(i, j) Pearu From david at ar.media.kyoto-u.ac.jp Wed Nov 25 03:09:50 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 25 Nov 2009 17:09:50 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <544434.51110.qm@web51003.mail.re2.yahoo.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> Message-ID: <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> Olivia Cheronet wrote: > compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' > gcc: _configtest.c > gcc _configtest.o -llibm.a -o _configtest.exe > /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory > collect2: ld returned 1 exit status > /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: crt0.o: No such file: No such file or directory > collect2: ld returned 1 exit status This is your problem: crt0.o is a file part of the C runtime, necessary to create any executable. This should have been installed at the same time as gcc, I would suggest reinstalling gcc on your cygwin. If this works, you should be able to compile a hello world from the command line, cheers, David From cheronetolivia at yahoo.com Wed Nov 25 04:42:00 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Wed, 25 Nov 2009 01:42:00 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> Message-ID: <196130.37019.qm@web51009.mail.re2.yahoo.com> The crt0.o file was indeed missing. I have reinstalled cygwin from the cygwin setup.exe (as it seemed to be included therein), and it seems to have solved that. However, I now get the error below. Thanks, Olivia _____________________________________________________________________ Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/system_info.py:1383: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/include/numpy/config.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/inc lude/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__multiarray_api.h'] building extension "numpy.core.multiarray" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/src/scalartypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/ar raytypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config .h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__ufunc_api. h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/src/scalartypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/ar raytypes.inc', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umath_funcs.inc' , 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/umath_loops.inc', 'build/src. cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h', 'build/src.cygwin-1.5 .25-i686-2.5/numpy/core/include/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i 686-2.5/numpy/core/include/numpy/__ufunc_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/config.h' to sources. adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/numpyconfig. h' to sources. executing numpy/core/code_generators/generate_numpy_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__multiarray _api.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy/__ufunc_api. h' to sources. numpy.core - nothing done with h_files = ['build/src.cygwin-1.5.25-i686-2.5/nump y/core/include/numpy/config.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/inc lude/numpy/numpyconfig.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__multiarray_api.h', 'build/src.cygwin-1.5.25-i686-2.5/numpy/core/include/ numpy/__ufunc_api.h'] building extension "numpy.core._dotblas" sources building extension "numpy.core.umath_tests" sources building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/python_xerbla.c' to sources. building extension "numpy.random.mtrand" sources /cygdrive/c/cygwin/home/Global/numpy-1.3.0/numpy/distutils/command/config.py:39: DeprecationWarning: +++++++++++++++++++++++++++++++++++++++++++++++++ Usage of try_run is deprecated: please do not use it anymore, and avoid configuration checks involving running executable on the target machine. +++++++++++++++++++++++++++++++++++++++++++++++++ DeprecationWarning) customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prot otypes compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: _configtest.c gcc _configtest.o -o _configtest.exe _configtest.exe failure. removing: _configtest.c _configtest.o _configtest.exe building data_files sources running build_py copying numpy/version.py -> build/lib.cygwin-1.5.25-i686-2.5/numpy copying build/src.cygwin-1.5.25-i686-2.5/numpy/__config__.py -> build/lib.cygwin -1.5.25-i686-2.5/numpy copying build/src.cygwin-1.5.25-i686-2.5/numpy/distutils/__config__.py -> build/ lib.cygwin-1.5.25-i686-2.5/numpy/distutils running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib building 'npymath' library compiling C sources C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prot otypes compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/ core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2. 5 -c' gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c numpy/core/src/npy_math.c.src:186: error: parse error before '/' token numpy/core/src/npy_math.c.src:186: error: parse error before '/' token error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/in clude/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c bu ild/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c -o build/temp.cygwin-1. 5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.o" failed with exit status 1 ----- Original Message ---- > This is your problem: crt0.o is a file part of the C runtime, necessary > to create any executable. This should have been installed at the same > time as gcc, I would suggest reinstalling gcc on your cygwin. If this > works, you should be able to compile a hello world from the command line, > > cheers, > > David From cournape at gmail.com Wed Nov 25 04:59:48 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 18:59:48 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <196130.37019.qm@web51009.mail.re2.yahoo.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> Message-ID: <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> On Wed, Nov 25, 2009 at 6:42 PM, Olivia Cheronet wrote: > The crt0.o file was indeed missing. I have reinstalled cygwin from the cygwin setup.exe (as it seemed to be included therein), and it seems to have solved that. > > compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/ > core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2. > 5 -c' > gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c > numpy/core/src/npy_math.c.src:186: error: parse error before '/' token > numpy/core/src/npy_math.c.src:186: error: parse error before '/' token > error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- > prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/in > clude/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c bu > ild/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c -o build/temp.cygwin-1. > 5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.o" failed > ?with exit status 1 Which version of the trunk are you using ? From the error, it looks like a C99-style // comment (which should not be there), but I don't see it in the last revision. Could you put the content of the file build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c as well ? thanks, David From cheronetolivia at yahoo.com Wed Nov 25 05:07:33 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Wed, 25 Nov 2009 02:07:33 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> Message-ID: <305713.71840.qm@web51008.mail.re2.yahoo.com> The npy_math.c is attached here. Cheers, Olivia ----- Original Message ---- > On Wed, Nov 25, 2009 at 6:42 PM, Olivia Cheronet > wrote: > > The crt0.o file was indeed missing. I have reinstalled cygwin from the cygwin > setup.exe (as it seemed to be included therein), and it seems to have solved > that. > > > > compile options: '-Inumpy/core/include > -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/ > > core/include/numpy -Inumpy/core/src -Inumpy/core/include > -I/usr/include/python2. > > 5 -c' > > gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c > > numpy/core/src/npy_math.c.src:186: error: parse error before '/' token > > numpy/core/src/npy_math.c.src:186: error: parse error before '/' token > > error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall > -Wstrict- > > prototypes -Inumpy/core/include > -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/in > > clude/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c > bu > > ild/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c -o > build/temp.cygwin-1. > > 5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.o" > failed > > with exit status 1 > > Which version of the trunk are you using ? From the error, it looks > like a C99-style // comment (which should not be there), but I don't > see it in the last revision. > > Could you put the content of the file > build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c as well ? > > thanks, > > David -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: npy_math.c URL: From cournape at gmail.com Wed Nov 25 05:56:51 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Nov 2009 19:56:51 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <305713.71840.qm@web51008.mail.re2.yahoo.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> <305713.71840.qm@web51008.mail.re2.yahoo.com> Message-ID: <5b8d13220911250256k62a3c13ej98ac04e89ccacfeb@mail.gmail.com> On Wed, Nov 25, 2009 at 7:07 PM, Olivia Cheronet wrote: > The npy_math.c is attached here. > I have just tested a fresh svn checkout, and could built numpy correctly on cygwin. I would suggest you update your sources, and build from scratch (i.e. remove the entire build directory and start from scratch). David From dyamins at gmail.com Wed Nov 25 09:48:50 2009 From: dyamins at gmail.com (Dan Yamins) Date: Wed, 25 Nov 2009 09:48:50 -0500 Subject: [Numpy-discussion] Bug in rec.fromarrays ; plus one other possible bug Message-ID: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> Hi, I'm writing to report what looks like a two bugs in the handling of strings of length 0. (I'm using 1.4.0.dev7746, on Mac OSX 10.5.8. The problems below occur both for python 2.5 compiled 32-bit as well as python2.6 compiled 64-bit). Bug #1: A problem arises when you try to create a record array passing a type of '|S0'. >>> Cols = [['test']*10,['']*10] When not passing any dtype, this is created into a recarray with no problem: >>> np.rec.fromarrays(Cols) rec.array([('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', ''), ('test', '')], dtype=[('f0', '|S4'), ('f1', '|S1')]) However, trouble arises when I try to pass a length-0 dtype explicitly. >>> d = np.dtype([('A', '|S4'), ('B', '|S')]) >>> np.rec.fromarrays(Cols,dtype=d) rec.array([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', '')], dtype=[('A', '|S4'), ('B', '|S0')]) The same thing occurs if I cast to np arrays before passing to np.rec.fromarrays: >>> _arrays = [np.array(Cols[0],'|S4'),np.array(Cols[1],'|S')] [array(['test', 'test', 'test', 'test', 'test', 'test', 'test', 'test', 'test', 'test'], dtype='|S4'), array(['', '', '', '', '', '', '', '', '', ''], dtype='|S1')] >>> np.rec.fromarrays(_arrays,dtype=d) rec.array([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', '')], dtype=[('A', '|S4'), ('B', '|S0')]) (Btw, why does np.array(['',''],'|S')) return an array with dtype '|S1'? Why not '|S0'? Are length-0 arrays being avoided explicitly? If so, why?) Bug #2: I'm not sure this is a bug, but it is annoying: np.dtype won't accept '|S0' as a type argument. >>> np.dtype('|S0') TypeError: data type not understood I have to do something like this: >>> d = np.dtype('|S') >>> d dtype('|S0') to get what I want. Is this intended? Regardless, this inconsistency also means that things like: >>> np.dtype(d.descr) can fail even when d is a properly constructed dtype object with a '|S0' type, which seems a little perverse. Am I just not supposed to be working with length-0 string columns, period? Thanks, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdh2358 at gmail.com Wed Nov 25 09:55:33 2009 From: jdh2358 at gmail.com (John Hunter) Date: Wed, 25 Nov 2009 08:55:33 -0600 Subject: [Numpy-discussion] Bug in rec.fromarrays ; plus one other possible bug In-Reply-To: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> References: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> Message-ID: <88e473830911250655i3f5aec14s3ad42900b15c16c6@mail.gmail.com> On Wed, Nov 25, 2009 at 8:48 AM, Dan Yamins wrote: > Am I just not supposed to be working with length-0 string columns, period? But why would you want to? array dtypes are immutable, so you are saying: I want this field to be only empty strings now and forever. So you can't initialize them to be empty and then fill them later. If by construction it is always empty, why have it at all? Then again, numpy allows me to create empty arrays.... x = np.array([]) JDH From pav at iki.fi Wed Nov 25 10:05:21 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 25 Nov 2009 17:05:21 +0200 Subject: [Numpy-discussion] Bug in rec.fromarrays ; plus one other possible bug In-Reply-To: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> References: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> Message-ID: <1259161521.4110.333.camel@talisman> ke, 2009-11-25 kello 09:48 -0500, Dan Yamins kirjoitti: > Hi, I'm writing to report what looks like a two bugs in the handling > of strings of length 0. (I'm using 1.4.0.dev7746, on Mac OSX 10.5.8. > The problems below occur both for python 2.5 compiled 32-bit as well > as python2.6 compiled 64-bit). [clip] > >>> Cols = [['test']*10,['']*10] > > When not passing any dtype, this is created into a recarray with no > problem: > > >>> np.rec.fromarrays(Cols) > rec.array([('test', ''), ('test', ''), ('test', ''), ('test', ''), > ('test', ''), ('test', ''), ('test', ''), ('test', ''), > ('test', ''), ('test', '')], > dtype=[('f0', '|S4'), ('f1', '|S1')]) > > However, trouble arises when I try to pass a length-0 dtype > explicitly. > > >>> d = np.dtype([('A', '|S4'), ('B', '|S')]) > >>> np.rec.fromarrays(Cols,dtype=d) > recarray([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', > ''), > ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', > ''), > ('\x00est', ''), ('\x00est', '')], > dtype=[('A', '|S4'), ('B', '|S0')]) That certainly looks like a bug -- where does the \0 appear in front of all but the first string? File a ticket in Trac, http://projects.scipy.org/numpy/ -- small bugs like this tend to get lost and forgotten in mailing list traffic. -- Pauli Virtanen From dyamins at gmail.com Wed Nov 25 10:19:35 2009 From: dyamins at gmail.com (Dan Yamins) Date: Wed, 25 Nov 2009 10:19:35 -0500 Subject: [Numpy-discussion] Bug in rec.fromarrays ; plus one other possible bug In-Reply-To: <1259161521.4110.333.camel@talisman> References: <15e4667e0911250648h7edee9b4k8126c1ba4f5ce392@mail.gmail.com> <1259161521.4110.333.camel@talisman> Message-ID: <15e4667e0911250719j408d7baco2a0ba47c195c8979@mail.gmail.com> > ('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est', > > ''), > > ('\x00est', ''), ('\x00est', '')], > > dtype=[('A', '|S4'), ('B', '|S0')]) > > That certainly looks like a bug -- where does the \0 appear in front of > all but the first string? > Sorry, I'm not sure what you mean precisely ... > File a ticket in Trac, http://projects.scipy.org/numpy/ -- small bugs > like this tend to get lost and forgotten in mailing list traffic. > > Will do. > -- > Pauli Virtanen > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkdelisle at gmail.com Wed Nov 25 13:12:15 2009 From: rkdelisle at gmail.com (Robert DeLisle) Date: Wed, 25 Nov 2009 11:12:15 -0700 Subject: [Numpy-discussion] Python 2.6, NumPy on CentOS 5.3 In-Reply-To: <5b8d13220911241902n47597042ke4e53cae5948aab5@mail.gmail.com> References: <0015176f0a825794ed047913921e@google.com> <49d6b3500911240833w77be7421k8dab02f5c68a1d08@mail.gmail.com> <5b8d13220911241902n47597042ke4e53cae5948aab5@mail.gmail.com> Message-ID: David, It does indeed work now. I also was able to find a repo package with the atlas libraries, so I installed them as well. It appears that everything went well. Thank you again for your assistance. -Kirk On Tue, Nov 24, 2009 at 8:02 PM, David Cournapeau wrote: > On Wed, Nov 25, 2009 at 2:11 AM, Robert DeLisle > wrote: > > David Cournapeau wrote: > > > > You need the *.so and not the *.so.3.1.1. The latter are enough to *run* > > applications linked against the library, the former is necessary to link > > against them. IOW, you need the devel packages for blas, lapack (and > > python). If you want to do it without admin rights, there is no other > > solution than building them by yourself. > > > > > > > > David, > > > > Thank you so much! That helps a lot and I don't know why I didn't think > of > > it myself. As root I'm able to install blas-devel and lapack-devel which > > gives me new libraries in /usr/lib64. Specifically, libblas.a, > liblapack.a, > > and liblapack_pic.a. Now when I run setup.py from the numpy directory, > it > > says that it finds the libraries in the appropriate locations. > > > > A few more issues: > > > > The configuration complains that there are no mkl, vml, guide libraries. > > Also, atlas libraries are not found. I cannot seem to find these at this > > point. > > That's ok. Our build process is too verbose, and is a bit confusing. > MKL and Atlas are just optimized BLAS/LAPACK, but they are not > necessary to build scipy. They can significantly improve scipy speed > for linear algebra stuff, but building Atlas or linking against the > Intel MKL is complicated. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qubax at gmx.at Wed Nov 25 13:23:20 2009 From: qubax at gmx.at (qubax at gmx.at) Date: Wed, 25 Nov 2009 19:23:20 +0100 Subject: [Numpy-discussion] Correlation function about a factor of 100 slower than matlab/mathcad ... but with fft even worse ? Message-ID: <20091125182320.GA3596@localhost.hotze.com> The correlation of a large data (about 250k points) v can be checked via correlate(v,v,mode='full') and ought to give the same result as the matlab function xcorr(v) FFT might speed up the evaluation ... In my specific case: xcorr takes about 0.2 seconds. correlate takes about 70 seconds. fftconvolve takes about 400 seconds. on the irc channel a second person happened to run into the same problem using mathcad and data consisting of about 300k points: correl takes about 1 second correlate takes 127 seconds fftconvolve was aborded because it took to long These tests were checked and confirmed by two other persons on the irc channel. The computers involved were 32bit as well as 64bit machines. All 4 persons are sure that lapack/atlas libraries are properly installed. Could someone please investigate why correlate and especially fftconvolve are orders of magnitude slower? Should more details / sample data be required, please let me know. Thanks, q ---------------------- executed code: tic=time.time() cor_c1=correlate(c1data[:,1],c1data[:,1],mode='full') toc=time.time() tic=time.time() cor_c1=fftconvolve(c1data[:,1],c1data[:,1],mode='full') toc=time.time() xcorr(data) -- The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity. From mcenerney1 at llnl.gov Wed Nov 25 13:42:25 2009 From: mcenerney1 at llnl.gov (James McEnerney) Date: Wed, 25 Nov 2009 10:42:25 -0800 Subject: [Numpy-discussion] f2py callback bug? In-Reply-To: <4B0CE58F.6040402@cens.ioc.ee> References: <893bk3$1qhn5c@nspiron-1.llnl.gov> <4B0CE3B6.304@cens.ioc.ee> <4B0CE58F.6040402@cens.ioc.ee> Message-ID: <80955b$2bdfmn@smtp.llnl.gov> An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Nov 25 14:09:50 2009 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 25 Nov 2009 21:09:50 +0200 Subject: [Numpy-discussion] Correlation function about a factor of 100 slower than matlab/mathcad ... but with fft even worse ? In-Reply-To: <20091125182320.GA3596@localhost.hotze.com> References: <20091125182320.GA3596@localhost.hotze.com> Message-ID: <1259176190.4110.335.camel@talisman> ke, 2009-11-25 kello 19:23 +0100, qubax at gmx.at kirjoitti: [clip] > Could someone please investigate why correlate and especially > fftconvolve are orders of magnitude slower? Read http://projects.scipy.org/numpy/ticket/1260 From dyamins at gmail.com Wed Nov 25 14:34:36 2009 From: dyamins at gmail.com (Dan Yamins) Date: Wed, 25 Nov 2009 14:34:36 -0500 Subject: [Numpy-discussion] Type Inference Message-ID: <15e4667e0911251134q75f40a5fw8aafdecb6a28b08d@mail.gmail.com> Sometimes I need to convert object-type arrays to their "natural, real" type, without a priori knowing what that type is, e.g. the equivalent of: >>> Y = np.array(X.tolist()) where X is the object array. If X is naturally an array of ints, Y will be an int array, if X is naturally strings, then Y will be '|Sn' where n is the right string length, the right casting will be done if there are mixed types, etc... My question is: is there a faster way to do this, given that X is already an object array. One possible (though maybe not the only) way this might work is if there is a faster lower-level numpy type-inference function that doesn't actually do the conversion, but just reports the "right" type, the type that np.array() *would* convert to. Does such a thing exist? And if so, (let's call it 'np.typeinferer', hypothetically), would it then be faster to do something like: >>> Y = np.array(X,np.typeinferer(X)) by-passing the step of converting the list-ified version of X to the new type? Or is the bulk of the work being done by np.array() the type inference anyway, so that it just makes sense to use np.array() in the first place? Sorry if this question is unclear -- and thanks for all the help recently. best, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Wed Nov 25 14:42:12 2009 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Wed, 25 Nov 2009 21:42:12 +0200 Subject: [Numpy-discussion] f2py callback bug? In-Reply-To: <80955b$2bdfmn@smtp.llnl.gov> References: <893bk3$1qhn5c@nspiron-1.llnl.gov> <4B0CE3B6.304@cens.ioc.ee> <4B0CE58F.6040402@cens.ioc.ee> <80955b$2bdfmn@smtp.llnl.gov> Message-ID: <4B0D8894.8090804@cens.ioc.ee> Hi James, To answer the second question, use: j = 1+numpy.array([2], numpy.int32) The answer to the first question is that the type of 1+numpy.array([2]) is numpy.int64 but Fortran function expects an array of type numpy.int32 and hence the wrapper makes a copy of the input array (which is also returned by the wrapper) before passing it to Fortran. Regards, Pearu James McEnerney wrote: > Pearu, > Thanks. a follow question. > Using fortran > > subroutine calc(j) > Cf2py intent(callback) pycalc > external pycalc > Cf2py integer dimension(1), intent(in,out):: j > > integer j(1) > print *, 'in fortran before pycalc ', 'j=', j(1) > call pycalc(j) > print *, 'in fortran after pycalc ', ' j=', j(1) > end > > in python > > import foo,numpy > def pycalc(j): > > print ' in pycalc ', 'j=', j > j[:] = 20*j > return > > print foo.calc.__doc__ > j = 1+numpy.array([2]) > print foo.calc(j, pycalc) > print j > > the output is > > calc - Function signature: > j = calc(j,pycalc,[pycalc_extra_args]) > Required arguments: > j : input rank-1 array('i') with bounds (1) > pycalc : call-back function > Optional arguments: > pycalc_extra_args := () input tuple > Return objects: > j : rank-1 array('i') with bounds (1) > Call-back functions: > def pycalc(j): return j > Required arguments: > j : input rank-1 array('i') with bounds (1) > Return objects: > j : rank-1 array('i') with bounds (1) > > in fortran before pycalc j= 3 > in pycalc j= [3] > in fortran after pycalc j= 60 > [60] > [3] > > Why is the return from foo.calc different from j? > How do I make them the same? > "return j" in pycalc doesn't change things. > > Thanks again! > > At 12:06 AM 11/25/2009, you wrote: > > >> Pearu Peterson wrote: >> >> > Hmm, regarding `intent(in, out) j`, this should work. I'll check what >> > is going on.. >> >> The `intent(in, out) j` works when pycalc is defined as subroutine: >> >> call pycalc(i, j) >> >> instead of >> >> pyreturn = pycalc(i, j) >> >> Pearu >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://*mail.scipy.org/mailman/listinfo/numpy-discussion > > Jim McEnerney > Lawrence Livermore National Laboratory > 7000 East Ave. > Livermore, Ca. 94550-9234 > > USA > > 925-422-1963 > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From mdekauwe at gmail.com Wed Nov 25 16:13:18 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Wed, 25 Nov 2009 13:13:18 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26503657.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> Message-ID: <26520383.post@talk.nabble.com> I tried redoing the internal logic for example by using the where function but I can't seem to work out how to match up the logic. For example (note slightly different from above). If I change the main loop to lst = np.where((data > -900.0) & (lst < -900.0), data, lst) lst = np.where((data > -900.0) & (lst > -900.0), 5.0, lst) If I then had a data array value of 10.0 and lst array value of -999.0. In this scenario the first statement would set the LST value to 10.0. However when you run the second statement, data is still bigger than the undefined (-999.0) but now so is the LST, hence the lst is set to 5.0 This doesn't match with the logic of if data[i,j] > -900.: if lst[i,j] < -900.: lst[i,j] = data[i,j] elif lst[i,j] > -900.: lst[i,j] = 5.0 as it would never get to the second branch under this logic. Does anyone have any suggestions on how to match up the logic? mdekauwe wrote: > > Hi I have written some code and I would appreciate any suggestions to make > better use of the numpy arrays functions to make it a bit more efficient > and less of a port from C. Any tricks are thoughts would be much > appreciated. > > The code reads in a series of images, collects a running total if the > value is valid and averages everything every 8 images. > > Code below... > > Thanks in advance... > > #!/usr/bin/env python > > """ > Average the daily LST values to make an 8 day product... > > Martin De Kauwe > 18th November 2009 > """ > import sys, os, glob > import numpy as np > import matplotlib.pyplot as plt > > > def averageEightDays(files, lst, count, numrows, numcols): > > day_count = 0 > # starting day - tag for output file > doy = 122 > > # loop over all the images and average all the information > # every 8 days... > for fname in glob.glob(os.path.join(path, files)): > current_file = fname.split('/')[8] > year = current_file.split('_')[2][0:4] > > try: > f = open(fname, 'rb') > except IOError: > print "Cannot open outfile for read", fname > sys.exit(1) > > data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) > f.close() > > # if it is day 1 then we need to fill up the holding array... > # copy the first file into the array... > if day_count == 0: > lst = data > for i in xrange(numrows): > for j in xrange(numcols): > # increase the pixel count if we got vaild data > (undefined = -999.0 > if lst[i,j] > -900.: > count[i,j] = 1 > day_count += 1 > > # keep a running total of valid lst values in an 8 day sequence > elif day_count > 0 and day_count <= 7: > for i in xrange(numrows): > for j in xrange(numcols): > # lst valid pixel? > if data[i,j] > -900.: > # was the existing pixel valid? > if lst[i,j] < -900.: > lst[i,j] = data[i,j] > count[i,j] = 1 > else: > lst[i,j] += data[i,j] > count[i,j] += 1 > day_count += 1 > > # need to average previous 8 days and write to a file... > if day_count == 8: > for i in xrange(numrows): > for j in xrange(numcols): > if count[i,j] > 0: > lst[i,j] = lst[i,j] / count[i,j] > else: > lst[i,j] = -999.0 > > day_count = 0 > doy += 8 > > lst, count = write_outputs(lst, count, year, doy) > > # it is possible we didn't have enough slices to do the last 8day > avg... > # but if we have more than one day we shall use it > # need to average previous 8 days and write to a file... > if day_count > 1: > for i in xrange(numrows): > for j in xrange(numcols): > if count[i,j] > 0: > lst[i,j] = lst[i,j] / count[i,j] > else: > lst[i,j] = -999.0 > > day_count = 0 > doy += 8 > > lst, count = write_outputs(lst, count, year, doy) > > def write_outputs(lst, count, year, doy): > path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" > outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" > > try: > of = open(os.path.join(path, outfile), 'wb') > except IOError: > print "Cannot open outfile for write", outfile > sys.exit(1) > > outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" > try: > of2 = open(os.path.join(path, outfile2), 'wb') > except IOError: > print "Cannot open outfile for write", outfile2 > sys.exit(1) > > # empty stuff and zero 8day counts > lst.tofile(of) > count.tofile(of2) > of.close() > of2.close() > lst = 0.0 > count = 0.0 > > return lst, count > > > if __name__ == "__main__": > > numrows = 332 > numcols = 667 > > path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" > lst = np.zeros((numrows, numcols), dtype=np.float32) > count = np.zeros((numrows, numcols), dtype=np.int) > averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) > > lst = 0.0 > count = 0.0 > averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) > > > > -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26520383.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From josef.pktd at gmail.com Wed Nov 25 16:22:34 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Nov 2009 16:22:34 -0500 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26520383.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> <26520383.post@talk.nabble.com> Message-ID: <1cd32cbb0911251322q37ce9137t97a97c08bdb675f@mail.gmail.com> On Wed, Nov 25, 2009 at 4:13 PM, mdekauwe wrote: > > I tried redoing the internal logic for example by using the where function > but I can't seem to work out how to match up the logic. For example (note > slightly different from above). If I change the main loop to > > lst = np.where((data > -900.0) & (lst < -900.0), data, lst) > lst = np.where((data > -900.0) & (lst > -900.0), 5.0, lst) does an intermediate array help? lst2 = np.where((data > -900.0) & (lst < -900.0), data, lst) lst = np.where((data > -900.0) & (lst > -900.0), 5.0, lst2) I didn't read through all the loops to see what would be a better way to do it. Josef > > If I then had a data array value of 10.0 and lst array value of -999.0. In > this scenario the first statement would set the LST value to 10.0. However > when you run the second statement, data is still bigger than the undefined > (-999.0) but now so is the LST, hence the lst is set to 5.0 > > This doesn't match with the logic of > > if data[i,j] > -900.: > ? ?if lst[i,j] < -900.: > ? ? ? ?lst[i,j] = data[i,j] > ? ?elif lst[i,j] > -900.: > ? ? ? ?lst[i,j] = 5.0 > > as it would never get to the second branch under this logic. > > Does anyone have any suggestions on how to match up the logic? > > > > mdekauwe wrote: >> >> Hi I have written some code and I would appreciate any suggestions to make >> better use of the numpy arrays functions to make it a bit more efficient >> and less of a port from C. Any tricks are thoughts would be much >> appreciated. >> >> The code reads in a series of images, collects a running total if the >> value is valid and averages everything every 8 images. >> >> Code below... >> >> Thanks in advance... >> >> #!/usr/bin/env python >> >> """ >> Average the daily LST values to make an 8 day product... >> >> Martin De Kauwe >> 18th November 2009 >> """ >> import sys, os, glob >> import numpy as np >> import matplotlib.pyplot as plt >> >> >> def averageEightDays(files, lst, count, numrows, numcols): >> >> ? ? day_count = 0 >> ? ? # starting day - tag for output file >> ? ? doy = 122 >> >> ? ? # loop over all the images and average all the information >> ? ? # every 8 days... >> ? ? for fname in glob.glob(os.path.join(path, files)): >> ? ? ? ? current_file = fname.split('/')[8] >> ? ? ? ? year = current_file.split('_')[2][0:4] >> >> ? ? ? ? try: >> ? ? ? ? ? ? f = open(fname, 'rb') >> ? ? ? ? except IOError: >> ? ? ? ? ? ? print "Cannot open outfile for read", fname >> ? ? ? ? ? ? sys.exit(1) >> >> ? ? ? ? data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) >> ? ? ? ? f.close() >> >> ? ? ? ? # if it is day 1 then we need to fill up the holding array... >> ? ? ? ? # copy the first file into the array... >> ? ? ? ? if day_count == 0: >> ? ? ? ? ? ? lst = data >> ? ? ? ? ? ? for i in xrange(numrows): >> ? ? ? ? ? ? ? ? ?for j in xrange(numcols): >> ? ? ? ? ? ? ? ? ? ? ?# increase the pixel count if we got vaild data >> (undefined = -999.0 >> ? ? ? ? ? ? ? ? ? ? ?if lst[i,j] > -900.: >> ? ? ? ? ? ? ? ? ? ? ? ? ?count[i,j] = 1 >> ? ? ? ? ? ? ? ? ? ? ? ? ?day_count += 1 >> >> ? ? ? ? # keep a running total of valid lst values in an 8 day sequence >> ? ? ? ? elif day_count > 0 and day_count <= 7: >> ? ? ? ? ? ? for i in xrange(numrows): >> ? ? ? ? ? ? ? ? for j in xrange(numcols): >> ? ? ? ? ? ? ? ? ? ? # lst valid pixel? >> ? ? ? ? ? ? ? ? ? ? if data[i,j] > -900.: >> ? ? ? ? ? ? ? ? ? ? ? ? # was the existing pixel valid? >> ? ? ? ? ? ? ? ? ? ? ? ? if lst[i,j] < -900.: >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? lst[i,j] = data[i,j] >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? count[i,j] = 1 >> ? ? ? ? ? ? ? ? ? ? ? ? else: >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? lst[i,j] += data[i,j] >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? count[i,j] += 1 >> ? ? ? ? ? ? ? ? day_count += 1 >> >> ? ? ? ? # need to average previous 8 days and write to a file... >> ? ? ? ? if day_count == 8: >> ? ? ? ? ? ? for i in xrange(numrows): >> ? ? ? ? ? ? ? ? for j in xrange(numcols): >> ? ? ? ? ? ? ? ? ? ? if count[i,j] > 0: >> ? ? ? ? ? ? ? ? ? ? ? ? lst[i,j] = lst[i,j] / count[i,j] >> ? ? ? ? ? ? ? ? ? ? else: >> ? ? ? ? ? ? ? ? ? ? ? ? lst[i,j] = -999.0 >> >> ? ? ? ? ? ? day_count = 0 >> ? ? ? ? ? ? doy += 8 >> >> ? ? ? ? ? ? lst, count = write_outputs(lst, count, year, doy) >> >> ? ? # it is possible we didn't have enough slices to do the last 8day >> avg... >> ? ? # but if we have more than one day we shall use it >> ? ? # need to average previous 8 days and write to a file... >> ? ? if day_count > 1: >> ? ? ? ? for i in xrange(numrows): >> ? ? ? ? ? ? for j in xrange(numcols): >> ? ? ? ? ? ? ? ? if count[i,j] > 0: >> ? ? ? ? ? ? ? ? ? ? lst[i,j] = lst[i,j] / count[i,j] >> ? ? ? ? ? ? ? ? else: >> ? ? ? ? ? ? ? ? ? ? lst[i,j] = -999.0 >> >> ? ? ? ? day_count = 0 >> ? ? ? ? doy += 8 >> >> ? ? ? ? lst, count = write_outputs(lst, count, year, doy) >> >> def write_outputs(lst, count, year, doy): >> ? ? path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" >> ? ? outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" >> >> ? ? try: >> ? ? ? ? of = open(os.path.join(path, outfile), 'wb') >> ? ? except IOError: >> ? ? ? ? print "Cannot open outfile for write", outfile >> ? ? ? ? sys.exit(1) >> >> ? ? outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" >> ? ? try: >> ? ? ? ? of2 = open(os.path.join(path, outfile2), 'wb') >> ? ? except IOError: >> ? ? ? ? print "Cannot open outfile for write", outfile2 >> ? ? ? ? sys.exit(1) >> >> ? ? # empty stuff and zero 8day counts >> ? ? lst.tofile(of) >> ? ? count.tofile(of2) >> ? ? of.close() >> ? ? of2.close() >> ? ? lst = 0.0 >> ? ? count = 0.0 >> >> ? ? return lst, count >> >> >> if __name__ == "__main__": >> >> ? ? numrows = 332 >> ? ? numcols = 667 >> >> ? ? path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" >> ? ? lst = np.zeros((numrows, numcols), dtype=np.float32) >> ? ? count = np.zeros((numrows, numcols), dtype=np.int) >> ? ? averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) >> >> ? ? lst = 0.0 >> ? ? count = 0.0 >> ? ? averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) >> >> >> >> > > -- > View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26520383.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Wed Nov 25 16:41:06 2009 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 25 Nov 2009 16:41:06 -0500 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26520383.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> <26520383.post@talk.nabble.com> Message-ID: <0B8A83A1-7E01-4936-A6B6-D69685D26DCF@gmail.com> On Nov 25, 2009, at 4:13 PM, mdekauwe wrote: > I tried redoing the internal logic for example by using the where function > but I can't seem to work out how to match up the logic. For example (note > slightly different from above). If I change the main loop to > > lst = np.where((data > -900.0) & (lst < -900.0), data, lst) > lst = np.where((data > -900.0) & (lst > -900.0), 5.0, lst) > > If I then had a data array value of 10.0 and lst array value of -999.0. In > this scenario the first statement would set the LST value to 10.0. However > when you run the second statement, data is still bigger than the undefined > (-999.0) but now so is the LST, hence the lst is set to 5.0 > > This doesn't match with the logic of > > if data[i,j] > -900.: > if lst[i,j] < -900.: > lst[i,j] = data[i,j] > elif lst[i,j] > -900.: > lst[i,j] = 5.0 > > as it would never get to the second branch under this logic. > > Does anyone have any suggestions on how to match up the logic? > np.where(data>-900,np.where(lst<-900,data,5.),lst) Note that this should fail if lst=-900 From qubax at gmx.at Wed Nov 25 17:50:36 2009 From: qubax at gmx.at (qubax at gmx.at) Date: Wed, 25 Nov 2009 23:50:36 +0100 Subject: [Numpy-discussion] Correlation function about a factor of 100 slower than matlab/mathcad ... but with fft even worse ? In-Reply-To: <1259176190.4110.335.camel@talisman> References: <20091125182320.GA3596@localhost.hotze.com> <1259176190.4110.335.camel@talisman> Message-ID: <20091125225036.GC3856@localhost.hotze.com> My own solution (i just heard that a very similar fix is (about to be) included in the new svn version) - for those who need a quickfix: *) This quick and dirty solution is about a factor of 300 faster for an input of 10^5 random numbers. Probably alot more for larger vectors. *) The deviation from correlate(v,v,mode='full') is max. about 10^-10 and similar to the difference between conv() and xcorr() in matlab. The trick: please notice the fft(x,2**_nextPow2()) part. ------------------------------------------ def myXcorrfun(x): len_x=len(a) maxlag = len_x - 1 c = ifft(abs(fft(x,2**__nextPowOf2(2*len_x-1)))**2) # force it to be real c = real(c) c_final = append(c[-maxlag:],c[0:maxlag+1]) return c_final def __nextPowOf2(x): # round up what remains after log2 ... that should be # the next highest power of 2 for the given number return ceil(log2(x)) ---------------------------------------- you can give it a quick try via: expon = range(5) for k in expon: a = random.rand(10**k) print 'testing with 10^',k,' random numbers' tic=time.time() myc = myXcorrfun(a) mytime = time.time()-tic print mytime,'seconds for myXcorr' tic=time.time() pyc = correlate(a,a,mode='full') pytime = time.time()-tic print pytime,'seconds for scipy correlate' print print 'estimated speedup:', int(pytime/mytime) print max(myc-pyc),' max deviation between the two' print I hope it helps one or the other out there. With best regards, q On Wed, Nov 25, 2009 at 09:09:50PM +0200, Pauli Virtanen wrote: > ke, 2009-11-25 kello 19:23 +0100, qubax at gmx.at kirjoitti: > [clip] > > Could someone please investigate why correlate and especially > > fftconvolve are orders of magnitude slower? > > Read http://projects.scipy.org/numpy/ticket/1260 > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- The king who needs to remind his people of his rank, is no king. A beggar's mistake harms no one but the beggar. A king's mistake, however, harms everyone but the king. Too often, the measure of power lies not in the number who obey your will, but in the number who suffer your stupidity. From david at ar.media.kyoto-u.ac.jp Thu Nov 26 00:03:40 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 26 Nov 2009 14:03:40 +0900 Subject: [Numpy-discussion] The trunk has been branched for 1.4.0 Message-ID: <4B0E0C2C.60707@ar.media.kyoto-u.ac.jp> Hi, I have finally branched the trunk into the 1.4.x branch. I have disabled the C API for datetime, and fixed the C API/ABI numbers. At this point, you should avoid committing anything which is not a high priority bug in the branch. I will prepare a first rc1 based on the branch, cheers, David From charlesr.harris at gmail.com Thu Nov 26 00:51:29 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 Nov 2009 22:51:29 -0700 Subject: [Numpy-discussion] Type Inference In-Reply-To: <15e4667e0911251134q75f40a5fw8aafdecb6a28b08d@mail.gmail.com> References: <15e4667e0911251134q75f40a5fw8aafdecb6a28b08d@mail.gmail.com> Message-ID: On Wed, Nov 25, 2009 at 12:34 PM, Dan Yamins wrote: > Sometimes I need to convert object-type arrays to their "natural, real" > type, without a priori knowing what that type is, e.g. the equivalent of: > > >>> Y = np.array(X.tolist()) > > where X is the object array. If X is naturally an array of ints, Y will be > an int array, if X is naturally strings, then Y will be '|Sn' where n is the > right string length, the right casting will be done if there are mixed > types, etc... > > My question is: is there a faster way to do this, given that X is already > an object array. > > One possible (though maybe not the only) way this might work is if there is > a faster lower-level numpy type-inference function that doesn't actually do > the conversion, but just reports the "right" type, the type that np.array() > *would* convert to. Does such a thing exist? And if so, (let's call it > 'np.typeinferer', hypothetically), would it then be faster to do something > like: > > >>> Y = np.array(X,np.typeinferer(X)) > > by-passing the step of converting the list-ified version of X to the new > type? Or is the bulk of the work being done by np.array() the type > inference anyway, so that it just makes sense to use np.array() in the first > place? > > Sorry if this question is unclear -- and thanks for all the help recently. > > This seems to work: In [5]: x = ones(2, dtype=object) In [6]: x Out[6]: array([1, 1], dtype=object) In [7]: array(x, dtype=int) Out[7]: array([1, 1]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdekauwe at gmail.com Thu Nov 26 04:29:34 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Thu, 26 Nov 2009 01:29:34 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <0B8A83A1-7E01-4936-A6B6-D69685D26DCF@gmail.com> References: <26503657.post@talk.nabble.com> <26520383.post@talk.nabble.com> <0B8A83A1-7E01-4936-A6B6-D69685D26DCF@gmail.com> Message-ID: <26525315.post@talk.nabble.com> Thanks. Agreed it will break down under that scenario but that shouldn't be encountered as I am simply checking the value is greater than what I have set the undefined to be (-999.0). Here is the refjigged logic using numpy functionality for anyone who it might help. Would appreciate any suggestions how I can further improve this. The code is significantly more efficient now (thankfully ;P)! #!/usr/bin/env python """ Average the daily LST values to make an 8 day product... Martin De Kauwe 26th November 2009 """ import sys, os, glob import numpy as np def averageEightDays(files, lst, count, numrows, numcols): day_count = 0 doy = 122 for fname in glob.glob(os.path.join(path, files)): current_file = fname.split('/')[8] year = current_file.split('_')[2][0:4] try: f = open(fname, 'rb') except IOError: print "Cannot open outfile for read", fname sys.exit(1) data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) f.close() # copy the first file into the array... # increase the pixel count if we have some valid data at timestep 1 if day_count == 0: lst = data count = np.where(lst > -900.0, 1, 0) day_count += 1 # keep a running total of valid lst values in an 8 day sequence # need to check whether the lst pixel is valid elif day_count > 0 and day_count <= 7: lst = np.where(data > -900.0, np.where(lst < -900.0, data, lst + data), lst) count = np.where(data > -900.0, np.where(lst < -900.0, 1, count + 1), count) day_count += 1 # need to average previous 8 days and write to a file... if day_count == 8: print year, ':', doy lst = np.where(count > 0, lst / np.float32(count), -999.0) day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) # it is possible we didn't have enough slices to do the last 8day avg... # but if we have more than one day we shall use it # need to average previous 8 days and write to a file... if day_count > 1: lst = np.where(count > 0, lst / np.float32(count), -999.0) day_count = 0 doy += 8 lst, count = write_outputs(lst, count, year, doy) def write_outputs(lst, count, year, doy): path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" try: of = open(os.path.join(path, outfile), 'wb') except IOError: print "Cannot open outfile for write", outfile sys.exit(1) outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" try: of2 = open(os.path.join(path, outfile2), 'wb') except IOError: print "Cannot open outfile for write", outfile2 sys.exit(1) # empty stuff lst.tofile(of) count.tofile(of2) of.close() of2.close() lst = np.zeros((numrows, numcols), dtype=np.float32) count = np.zeros((numrows, numcols), dtype=np.int) return lst, count if __name__ == "__main__": numrows = 332 numcols = 667 path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" lst = np.zeros((numrows, numcols), dtype=np.float32) count = np.zeros((numrows, numcols), dtype=np.int) averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) lst = np.zeros((numrows, numcols), dtype=np.float32) count = np.zeros((numrows, numcols), dtype=np.int) averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) Pierre GM-2 wrote: > > On Nov 25, 2009, at 4:13 PM, mdekauwe wrote: >> I tried redoing the internal logic for example by using the where >> function >> but I can't seem to work out how to match up the logic. For example (note >> slightly different from above). If I change the main loop to >> >> lst = np.where((data > -900.0) & (lst < -900.0), data, lst) >> lst = np.where((data > -900.0) & (lst > -900.0), 5.0, lst) >> >> If I then had a data array value of 10.0 and lst array value of -999.0. >> In >> this scenario the first statement would set the LST value to 10.0. >> However >> when you run the second statement, data is still bigger than the >> undefined >> (-999.0) but now so is the LST, hence the lst is set to 5.0 >> >> This doesn't match with the logic of >> >> if data[i,j] > -900.: >> if lst[i,j] < -900.: >> lst[i,j] = data[i,j] >> elif lst[i,j] > -900.: >> lst[i,j] = 5.0 >> >> as it would never get to the second branch under this logic. >> >> Does anyone have any suggestions on how to match up the logic? >> > > > np.where(data>-900,np.where(lst<-900,data,5.),lst) > > Note that this should fail if lst=-900 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26525315.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From schut at sarvision.nl Thu Nov 26 04:55:51 2009 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 26 Nov 2009 10:55:51 +0100 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26503657.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> Message-ID: <4B0E50A7.6090109@sarvision.nl> Hi mdekauwe, as long as your data size is small enough to fit a 8x array in memory and use it, why not just skip the running total and average 8 data arrays each 8day period? And skip the x and y loops; these are real performance killers. As a bonus, your code gets a lot smaller and more readeable... :) Please correct me is I'm wrong: the ultimate goal is to have a average of the valid pixels 8 days of data, each 8 days, correct? I'd do it this way (more or less pythonic pseudocode, untested, but your should get the idea): # read filenames filenames = glob.glob..... filenames.sort() # to be sure they are in proper order filenamesChunked = [filenames[n:n+8] for n in range(0, len(filenames, 8)] # this will produce a list of lists, with each sublist containing the # next 8 filenames nodatavalue = -999 for fchunk in filenamesChunked: data8days = numpy.ones((8, ncols, nrows)) * -999 # fill with nodata values, in case there are less than 8 days for di in range(len(fchunk)): f = fchunk[di] data8days[di] = weights = data8days > nodatavalue # this way all nodata values get a weight of 0 avg8days = numpy.average(data8days, axis=0, weights=weights) uhm... well, that's it, really :) best regards, Vincent. mdekauwe wrote: > Hi I have written some code and I would appreciate any suggestions to make > better use of the numpy arrays functions to make it a bit more efficient and > less of a port from C. Any tricks are thoughts would be much appreciated. > > The code reads in a series of images, collects a running total if the value > is valid and averages everything every 8 images. > > Code below... > > Thanks in advance... > > #!/usr/bin/env python > > """ > Average the daily LST values to make an 8 day product... > > Martin De Kauwe > 18th November 2009 > """ > import sys, os, glob > import numpy as np > import matplotlib.pyplot as plt > > > def averageEightDays(files, lst, count, numrows, numcols): > > day_count = 0 > # starting day - tag for output file > doy = 122 > > # loop over all the images and average all the information > # every 8 days... > for fname in glob.glob(os.path.join(path, files)): > current_file = fname.split('/')[8] > year = current_file.split('_')[2][0:4] > > try: > f = open(fname, 'rb') > except IOError: > print "Cannot open outfile for read", fname > sys.exit(1) > > data = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) > f.close() > > # if it is day 1 then we need to fill up the holding array... > # copy the first file into the array... > if day_count == 0: > lst = data > for i in xrange(numrows): > for j in xrange(numcols): > # increase the pixel count if we got vaild data > (undefined = -999.0 > if lst[i,j] > -900.: > count[i,j] = 1 > day_count += 1 > > # keep a running total of valid lst values in an 8 day sequence > elif day_count > 0 and day_count <= 7: > for i in xrange(numrows): > for j in xrange(numcols): > # lst valid pixel? > if data[i,j] > -900.: > # was the existing pixel valid? > if lst[i,j] < -900.: > lst[i,j] = data[i,j] > count[i,j] = 1 > else: > lst[i,j] += data[i,j] > count[i,j] += 1 > day_count += 1 > > # need to average previous 8 days and write to a file... > if day_count == 8: > for i in xrange(numrows): > for j in xrange(numcols): > if count[i,j] > 0: > lst[i,j] = lst[i,j] / count[i,j] > else: > lst[i,j] = -999.0 > > day_count = 0 > doy += 8 > > lst, count = write_outputs(lst, count, year, doy) > > # it is possible we didn't have enough slices to do the last 8day avg... > # but if we have more than one day we shall use it > # need to average previous 8 days and write to a file... > if day_count > 1: > for i in xrange(numrows): > for j in xrange(numcols): > if count[i,j] > 0: > lst[i,j] = lst[i,j] / count[i,j] > else: > lst[i,j] = -999.0 > > day_count = 0 > doy += 8 > > lst, count = write_outputs(lst, count, year, doy) > > def write_outputs(lst, count, year, doy): > path = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" > outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" > > try: > of = open(os.path.join(path, outfile), 'wb') > except IOError: > print "Cannot open outfile for write", outfile > sys.exit(1) > > outfile2 = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" > try: > of2 = open(os.path.join(path, outfile2), 'wb') > except IOError: > print "Cannot open outfile for write", outfile2 > sys.exit(1) > > # empty stuff and zero 8day counts > lst.tofile(of) > count.tofile(of2) > of.close() > of2.close() > lst = 0.0 > count = 0.0 > > return lst, count > > > if __name__ == "__main__": > > numrows = 332 > numcols = 667 > > path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" > lst = np.zeros((numrows, numcols), dtype=np.float32) > count = np.zeros((numrows, numcols), dtype=np.int) > averageEightDays('lst_scr_2006*.gra', lst, count, numrows, numcols) > > lst = 0.0 > count = 0.0 > averageEightDays('lst_scr_2007*.gra', lst, count, numrows, numcols) > > > From mdekauwe at gmail.com Thu Nov 26 06:53:02 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Thu, 26 Nov 2009 03:53:02 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <4B0E50A7.6090109@sarvision.nl> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> Message-ID: <26528176.post@talk.nabble.com> Thanks...I have adopted that and as you said it is a lot neater! Though I need to keep the pixel count for a weighting in another piece of code so have amended your logic slightly. #!/usr/bin/env python import sys, os, glob import numpy as np def averageEightDays(filenamesList, numrows, numcols, year): doy = 122 nodatavalue = -999.0 for fchunk in filenamesList: # fill with nodata values, in case there are less than 8 days data8days = np.zeros((8, numrows, numcols), dtype=np.float32) * -999.0 pixCount = np.zeros((numrows, numcols), dtype=np.int) avg8days = np.zeros((numrows, numcols), dtype=np.float32) for day in xrange(len(fchunk)): f = fchunk[day] data8days[day] = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) avg8days = np.where(data8days[day] > nodatavalue, avg8days+data8days[day], avg8days) pixCount = np.where(data8days[day] > nodatavalue, pixCount+1, pixCount) avg8days = np.where(pixCount > 0, avg8days / np.float32(pixCount), -999.0) doy += 8 print year,':',doy outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" write_outputs(outfile, avg8days) outfile = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" write_outputs(outfile, pixCount) def write_outputs(outfile, data): opath = "/users/eow/mgdk/research/HOFF_plots/LST/tmp" try: of = open(os.path.join(opath, outfile), 'wb') except IOError: print "Cannot open outfile for write", outfile sys.exit(1) # empty stuff data.tofile(of) of.close() if __name__ == "__main__": numrows = 332 numcols = 667 path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" files = 'lst_scr_2006*.gra' filenames = glob.glob(os.path.join(path, files)) filenames.sort() filenamesList = [filenames[n:n+8] for n in xrange(0, len(filenames), 8)] averageEightDays(filenamesList, numrows, numcols, year=2006) files = 'lst_scr_2007*.gra' filenames = glob.glob(os.path.join(path, files)) filenames.sort() filenamesList = [filenames[n:n+8] for n in xrange(0, len(filenames), 8)] averageEightDays(filenamesList, numrows, numcols, year=2007) -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26528176.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From nwagner at iam.uni-stuttgart.de Thu Nov 26 08:20:40 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 26 Nov 2009 14:20:40 +0100 Subject: [Numpy-discussion] boolean arrays Message-ID: Hi all, is the following behaviour correct >>> a = array(([True,True],[True,True])) >>> b = array(([False,False],[False,False])) >>> a+b array([[ True, True], [ True, True]]) I have expected False. Nils From schut at sarvision.nl Thu Nov 26 08:27:44 2009 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 26 Nov 2009 14:27:44 +0100 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26528176.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> <26528176.post@talk.nabble.com> Message-ID: <4B0E8250.8040303@sarvision.nl> mdekauwe wrote: > Thanks...I have adopted that and as you said it is a lot neater! Though I > need to keep the pixel count for a weighting in another piece of code so > have amended your logic slightly. Alternatively, you could simply take the sum over axis=0 of the weight array to get the pixel count (e.g. "pixelcount=weight.sum(axis=0)"). I don't know if performance is an issue, but I'd say that skipping these numpy.where's and increments of count and running total, and use numpy.average instead, should give you a speedup. Though the size of your arrays might be too small for it to become very noticeable. However, in your current version, it makes no sense to allocate an 8.nx.ny array to read the data, as you anyways calculate a running total. The 8,nx,ny array only makes sense to postpone calculations till you've read all 8 days, and only then calculate the average (and pixel count). Currently you preserve the previous days of data, but you don't use them anywhere anymore. Either way will work, though, I guess. Choose what feels most naturally for you, as that will help you maintain and understand your code later on. Oh, and minor issue: creating a array of zeros and then multiplying with -999 still makes an array of zeros... I'd incorporated an array of *ones* multiplied with -999, because for the last chunk of days you could end up with a 8day array only partly filled with real data. E.g. if you'd have only 3 days of data left in your last chunk, 8dayData[0:3] would be data, and to prevent the rest ([3:8]) to be incorporated into the average calculation, these need to be -999. Currently these pixels will be 0, which is > nodatavalue, and thus infuencing your average (the pixelcount will be incremented for these 0 values). Vincent. > > #!/usr/bin/env python > > > import sys, os, glob > import numpy as np > > def averageEightDays(filenamesList, numrows, numcols, year): > doy = 122 > nodatavalue = -999.0 > > for fchunk in filenamesList: > # fill with nodata values, in case there are less than 8 days > data8days = np.zeros((8, numrows, numcols), dtype=np.float32) * -999.0 > pixCount = np.zeros((numrows, numcols), dtype=np.int) > avg8days = np.zeros((numrows, numcols), dtype=np.float32) > for day in xrange(len(fchunk)): > f = fchunk[day] > data8days[day] = np.fromfile(f, dtype=np.float32).reshape(numrows, > numcols) > avg8days = np.where(data8days[day] > nodatavalue, > avg8days+data8days[day], avg8days) > pixCount = np.where(data8days[day] > nodatavalue, pixCount+1, pixCount) > > avg8days = np.where(pixCount > 0, avg8days / np.float32(pixCount), -999.0) > doy += 8 > print year,':',doy > > outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" > write_outputs(outfile, avg8days) > outfile = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" > write_outputs(outfile, pixCount) > > def write_outputs(outfile, data): > opath = "/users/eow/mgdk/research/HOFF_plots/LST/tmp" > > try: > of = open(os.path.join(opath, outfile), 'wb') > except IOError: > print "Cannot open outfile for write", outfile > sys.exit(1) > > # empty stuff > data.tofile(of) > of.close() > > > if __name__ == "__main__": > > numrows = 332 > numcols = 667 > path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" > > files = 'lst_scr_2006*.gra' > filenames = glob.glob(os.path.join(path, files)) > filenames.sort() > filenamesList = [filenames[n:n+8] for n in xrange(0, len(filenames), 8)] > averageEightDays(filenamesList, numrows, numcols, year=2006) > > files = 'lst_scr_2007*.gra' > filenames = glob.glob(os.path.join(path, files)) > filenames.sort() > filenamesList = [filenames[n:n+8] for n in xrange(0, len(filenames), 8)] > averageEightDays(filenamesList, numrows, numcols, year=2007) > From nadavh at visionsense.com Thu Nov 26 11:26:19 2009 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 26 Nov 2009 18:26:19 +0200 Subject: [Numpy-discussion] boolean arrays In-Reply-To: References: Message-ID: <1259252779.26782.67.camel@nadav.envision.co.il> It is obvious to me that True+False == True,. Why do you think it should be False? Nadav On Thu, 2009-11-26 at 14:20 +0100, Nils Wagner wrote: > Hi all, > > is the following behaviour correct > > >>> a = array(([True,True],[True,True])) > >>> b = array(([False,False],[False,False])) > > >>> a+b > array([[ True, True], > [ True, True]]) > > I have expected False. > > Nils > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From silva at lma.cnrs-mrs.fr Thu Nov 26 08:43:14 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Thu, 26 Nov 2009 14:43:14 +0100 Subject: [Numpy-discussion] boolean arrays In-Reply-To: <1259252779.26782.67.camel@nadav.envision.co.il> References: <1259252779.26782.67.camel@nadav.envision.co.il> Message-ID: <1259242994.6875.0.camel@localhost.localdomain> Le jeudi 26 novembre 2009 ? 18:26 +0200, Nadav Horesh a ?crit : > It is obvious to me that True+False == True,. Why do you think it should > be False? > I would understand it is not obvious that '+' stands for logical 'or', and '*' for logical 'and'... -- Fabrice Silva LMA UPR CNRS 7051 From gael.varoquaux at normalesup.org Thu Nov 26 08:44:07 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 26 Nov 2009 14:44:07 +0100 Subject: [Numpy-discussion] boolean arrays In-Reply-To: <1259242994.6875.0.camel@localhost.localdomain> References: <1259252779.26782.67.camel@nadav.envision.co.il> <1259242994.6875.0.camel@localhost.localdomain> Message-ID: <20091126134407.GB30492@phare.normalesup.org> On Thu, Nov 26, 2009 at 02:43:14PM +0100, Fabrice Silva wrote: > Le jeudi 26 novembre 2009 ? 18:26 +0200, Nadav Horesh a ?crit : > > It is obvious to me that True+False == True,. Why do you think it should > > be False? > I would understand it is not obvious that '+' stands for logical 'or', > and '*' for logical 'and'... In Bool's algebra, this is the common convention. The reason being that only 'or' can correspond to the additive law of an algebra: its null element is absorbant for 'and'. In other words, if you map '+' and '*' to the opposite, you'll get suprising behaviors. Ga?l From mdekauwe at gmail.com Thu Nov 26 08:49:18 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Thu, 26 Nov 2009 05:49:18 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <4B0E8250.8040303@sarvision.nl> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> <26528176.post@talk.nabble.com> <4B0E8250.8040303@sarvision.nl> Message-ID: <26529681.post@talk.nabble.com> Vincent Schut-2 wrote: > > Oh, and minor issue: creating a array of zeros and then multiplying with > -999 still makes an array of zeros... I'd incorporated an array of > *ones* multiplied with -999, because for the last chunk of days you > could end up with a 8day array only partly filled with real data. E.g. > if you'd have only 3 days of data left in your last chunk, 8dayData[0:3] > would be data, and to prevent the rest ([3:8]) to be incorporated into > the average calculation, these need to be -999. Currently these pixels > will be 0, which is > nodatavalue, and thus infuencing your average (the > pixelcount will be incremented for these 0 values). > Ok I hadn't thought about it in that way but you are of course right! I have amended it. Vincent Schut-2 wrote: > > Alternatively, you could simply take the sum over axis=0 of the weight > array to get the pixel count (e.g. "pixelcount=weight.sum(axis=0)"). > Ok I see your point here as well. So I tried implementing your suggestion, as I understand it weights = data8days > nodatavalue will make and 8, nrows, ncols array containing true and false. as you said I can get the pixel count I was after by using weights.sum(axis=0). However when I try to do the averaging step: avg8days = np.average(data8days, axis=0, weights=weights) I get the error msg " in average raise ZeroDivisionError, "Weights sum to zero, can't be normalized" ZeroDivisionError: Weights sum to zero, can't be normalized" Which I guess (but don't know) comes from the trying do weight by a pixel count of zero. So not sure what has happened here? Thanks -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26529681.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From aisaac at american.edu Thu Nov 26 08:55:04 2009 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 26 Nov 2009 08:55:04 -0500 Subject: [Numpy-discussion] boolean arrays In-Reply-To: References: Message-ID: <4B0E88B8.1070406@american.edu> On 11/26/2009 8:20 AM, Nils Wagner wrote: > a = array(([True,True],[True,True])) > b = array(([False,False],[False,False])) > a+b NumPy's boolean operations are very well behaved. >>> a = np.array(([True,True],[True,True])) >>> a+a array([[ True, True], [ True, True]], dtype=bool) Compare Python: >>> True + True 2 Ugh! Not fixing this in Python 3 was a big mistake, imo. Alan Isaac From silva at lma.cnrs-mrs.fr Thu Nov 26 09:14:04 2009 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Thu, 26 Nov 2009 15:14:04 +0100 Subject: [Numpy-discussion] boolean arrays In-Reply-To: <20091126134407.GB30492@phare.normalesup.org> References: <1259252779.26782.67.camel@nadav.envision.co.il> <1259242994.6875.0.camel@localhost.localdomain> <20091126134407.GB30492@phare.normalesup.org> Message-ID: <1259244844.6875.3.camel@localhost.localdomain> Le jeudi 26 novembre 2009 ? 14:44 +0100, Gael Varoquaux a ?crit : > On Thu, Nov 26, 2009 at 02:43:14PM +0100, Fabrice Silva wrote: > > Le jeudi 26 novembre 2009 ? 18:26 +0200, Nadav Horesh a ?crit : > > > It is obvious to me that True+False == True,. Why do you think it should > > > be False? > > > I would understand it is not obvious that '+' stands for logical 'or', > > and '*' for logical 'and'... > > In Bool's algebra, this is the common convention. The reason being that > only 'or' can correspond to the additive law of an algebra: its null > element is absorbant for 'and'. > > In other words, if you map '+' and '*' to the opposite, you'll get > suprising behaviors. I fully agree with you. My point was to complete Nadav's comment with potentially missing information, trying to figrue why Nils was expected False... -- Fabrice Silva LMA UPR CNRS 7051 From schut at sarvision.nl Thu Nov 26 09:36:46 2009 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 26 Nov 2009 15:36:46 +0100 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26529681.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> <26528176.post@talk.nabble.com> <4B0E8250.8040303@sarvision.nl> <26529681.post@talk.nabble.com> Message-ID: <4B0E927E.3030706@sarvision.nl> mdekauwe wrote: > > Vincent Schut-2 wrote: >> Oh, and minor issue: creating a array of zeros and then multiplying with >> -999 still makes an array of zeros... I'd incorporated an array of >> *ones* multiplied with -999, because for the last chunk of days you >> could end up with a 8day array only partly filled with real data. E.g. >> if you'd have only 3 days of data left in your last chunk, 8dayData[0:3] >> would be data, and to prevent the rest ([3:8]) to be incorporated into >> the average calculation, these need to be -999. Currently these pixels >> will be 0, which is > nodatavalue, and thus infuencing your average (the >> pixelcount will be incremented for these 0 values). >> > > Ok I hadn't thought about it in that way but you are of course right! I have > amended it. > > > Vincent Schut-2 wrote: >> Alternatively, you could simply take the sum over axis=0 of the weight >> array to get the pixel count (e.g. "pixelcount=weight.sum(axis=0)"). >> > > Ok I see your point here as well. So I tried implementing your suggestion, > as I understand it > > weights = data8days > nodatavalue > > will make and 8, nrows, ncols array containing true and false. > > as you said I can get the pixel count I was after by using > weights.sum(axis=0). > > However when I try to do the averaging step: > > avg8days = np.average(data8days, axis=0, weights=weights) > > I get the error msg " in average raise ZeroDivisionError, "Weights sum to > zero, can't be normalized" > ZeroDivisionError: Weights sum to zero, can't be normalized" > > Which I guess (but don't know) comes from the trying do weight by a pixel > count of zero. So not sure what has happened here? > > Thanks > Ah... apparently numpy.average can't handle the situation when all weights are 0... didn't know that. Hmm... what would you want to have in your average array when for a certain pixel there are only nodata values? If you'd like to have -999 in your average result then, the solution is simple: in the weight array, for those pixels where weight is always 0, set 1 dayweight to 1. this way you'll get a nice -999 in your average result for those pixels. e.g.: weights[0] |= (weights.sum(axis=0)==0) will set (boolean OR assign) all pixels with weight==0 for all days to True (==1). Hope this helps. Vincent. From schut at sarvision.nl Thu Nov 26 09:36:46 2009 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 26 Nov 2009 15:36:46 +0100 Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <26529681.post@talk.nabble.com> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> <26528176.post@talk.nabble.com> <4B0E8250.8040303@sarvision.nl> <26529681.post@talk.nabble.com> Message-ID: <4B0E927E.3030706@sarvision.nl> mdekauwe wrote: > > Vincent Schut-2 wrote: >> Oh, and minor issue: creating a array of zeros and then multiplying with >> -999 still makes an array of zeros... I'd incorporated an array of >> *ones* multiplied with -999, because for the last chunk of days you >> could end up with a 8day array only partly filled with real data. E.g. >> if you'd have only 3 days of data left in your last chunk, 8dayData[0:3] >> would be data, and to prevent the rest ([3:8]) to be incorporated into >> the average calculation, these need to be -999. Currently these pixels >> will be 0, which is > nodatavalue, and thus infuencing your average (the >> pixelcount will be incremented for these 0 values). >> > > Ok I hadn't thought about it in that way but you are of course right! I have > amended it. > > > Vincent Schut-2 wrote: >> Alternatively, you could simply take the sum over axis=0 of the weight >> array to get the pixel count (e.g. "pixelcount=weight.sum(axis=0)"). >> > > Ok I see your point here as well. So I tried implementing your suggestion, > as I understand it > > weights = data8days > nodatavalue > > will make and 8, nrows, ncols array containing true and false. > > as you said I can get the pixel count I was after by using > weights.sum(axis=0). > > However when I try to do the averaging step: > > avg8days = np.average(data8days, axis=0, weights=weights) > > I get the error msg " in average raise ZeroDivisionError, "Weights sum to > zero, can't be normalized" > ZeroDivisionError: Weights sum to zero, can't be normalized" > > Which I guess (but don't know) comes from the trying do weight by a pixel > count of zero. So not sure what has happened here? > > Thanks > Ah... apparently numpy.average can't handle the situation when all weights are 0... didn't know that. Hmm... what would you want to have in your average array when for a certain pixel there are only nodata values? If you'd like to have -999 in your average result then, the solution is simple: in the weight array, for those pixels where weight is always 0, set 1 dayweight to 1. this way you'll get a nice -999 in your average result for those pixels. e.g.: weights[0] |= (weights.sum(axis=0)==0) will set (boolean OR assign) all pixels with weight==0 for all days to True (==1). Hope this helps. Vincent. From mdekauwe at gmail.com Thu Nov 26 11:33:03 2009 From: mdekauwe at gmail.com (mdekauwe) Date: Thu, 26 Nov 2009 08:33:03 -0800 (PST) Subject: [Numpy-discussion] Help making better use of numpy array functions In-Reply-To: <4B0E927E.3030706@sarvision.nl> References: <26503657.post@talk.nabble.com> <4B0E50A7.6090109@sarvision.nl> <26528176.post@talk.nabble.com> <4B0E8250.8040303@sarvision.nl> <26529681.post@talk.nabble.com> <4B0E927E.3030706@sarvision.nl> Message-ID: <26531909.post@talk.nabble.com> Yep that will do nicely, code becomes import sys, os, glob import numpy as np def averageEightDays(files, numrows, numcols, year, doy): """ Read in 8 files at a time, sum the valid LST, keep a count of the valid pixels and average the result every 8days. """ nodatavalue = -999.0 # break the files up into chunks of 8 filenames = glob.glob(os.path.join(path, files)) filenames.sort() filenamesList = [filenames[n:n+8] for n in xrange(0, len(filenames), 8)] for fchunk in filenamesList: # fill with nodata values, in case there are less than 8 days data8days = np.ones((8, numrows, numcols), dtype=np.float32) * -999.0 avg8days = np.zeros((numrows, numcols), dtype=np.float32) for day in xrange(len(fchunk)): fname = fchunk[day] try: f = open(fname, 'rb') except IOError: print "Cannot open outfile for read", fname sys.exit(1) data8days[day] = np.fromfile(f, dtype=np.float32).reshape(numrows, numcols) # build an array (ndays, nrow, ncols) of True and False for the pixel count # when these are summed we get the relative contributions weights = data8days > nodatavalue # np.average doesn't accept a weight of zero, COMMENT ME weights[0] |= (weights.sum(axis=0) == 0) pixelCount = weights.sum(axis=0) avg8days = np.average(data8days, axis=0, weights=weights) doy += 8 #print year,':',doy outfile = "lst_8day1030am_" + str(year) + str(doy) + ".gra" write_outputs(outfile, avg8days) outfile = "pixelcount_8day1030am_" + str(year) + str(doy) + ".gra" write_outputs(outfile, pixelCount) def write_outputs(outfile, data): opath = "/users/eow/mgdk/research/HOFF_plots/LST/8dayLST" try: of = open(os.path.join(opath, outfile), 'wb') except IOError: print "Cannot open outfile for write", outfile sys.exit(1) # empty stuff data.tofile(of) of.close() if __name__ == "__main__": numrows = 332 numcols = 667 path = "/users/eow/mgdk/research/HOFF_plots/LST/gridded_03/" averageEightDays('lst_scr_2006*.gra', numrows, numcols, year=2006, doy=122) averageEightDays('lst_scr_2007*.gra', numrows, numcols, year=2007, doy=122) -- View this message in context: http://old.nabble.com/Help-making-better-use-of-numpy-array-functions-tp26503657p26531909.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From josef.pktd at gmail.com Thu Nov 26 11:34:46 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 11:34:46 -0500 Subject: [Numpy-discussion] matrix inverse Message-ID: <1cd32cbb0911260834l4e12c86w9ec4fca264dab3ec@mail.gmail.com> why is http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.I/ classified as unimportant? .T ? I was looking at http://projects.scipy.org/numpy/ticket/1093 and didn't find any docs for the matrix inverse. Josef From josef.pktd at gmail.com Thu Nov 26 11:49:18 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 11:49:18 -0500 Subject: [Numpy-discussion] matrix inverse In-Reply-To: <1cd32cbb0911260834l4e12c86w9ec4fca264dab3ec@mail.gmail.com> References: <1cd32cbb0911260834l4e12c86w9ec4fca264dab3ec@mail.gmail.com> Message-ID: <1cd32cbb0911260849p45c327d6g674c243876a3ea7e@mail.gmail.com> On Thu, Nov 26, 2009 at 11:34 AM, wrote: > why is http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.I/ > classified as unimportant? ? .T ? > > I was looking at http://projects.scipy.org/numpy/ticket/1093 and > didn't find any docs for the matrix inverse. Ok looking some more, I found http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.getI/ I thought it might be problem with generating docs for attributes, but the docstring for .I seems editable. Aren't A.I and A.T the more common use than A.getI, ..? Josef > > Josef > From josef.pktd at gmail.com Thu Nov 26 11:55:52 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 11:55:52 -0500 Subject: [Numpy-discussion] matrix inverse In-Reply-To: <1cd32cbb0911260849p45c327d6g674c243876a3ea7e@mail.gmail.com> References: <1cd32cbb0911260834l4e12c86w9ec4fca264dab3ec@mail.gmail.com> <1cd32cbb0911260849p45c327d6g674c243876a3ea7e@mail.gmail.com> Message-ID: <1cd32cbb0911260855o6de80cabte37f0365103e0083@mail.gmail.com> On Thu, Nov 26, 2009 at 11:49 AM, wrote: > On Thu, Nov 26, 2009 at 11:34 AM, ? wrote: >> why is http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.I/ >> classified as unimportant? ? .T ? >> >> I was looking at http://projects.scipy.org/numpy/ticket/1093 and >> didn't find any docs for the matrix inverse. > > Ok looking some more, I found > http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.getI/ > > I thought it might be problem with generating docs for attributes, but > the docstring for .I seems editable. > > Aren't A.I and A.T the more common use than A.getI, ..? in defmatrix.py: T = property(getT, None, doc="transpose") A = property(getA, None, doc="base array") A1 = property(getA1, None, doc="1-d base array") H = property(getH, None, doc="hermitian (conjugate) transpose") I = property(getI, None, doc="inverse") could take the docstrings from the get? function instead ? Josef > > Josef > >> >> Josef >> > From nwagner at iam.uni-stuttgart.de Thu Nov 26 13:35:26 2009 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 26 Nov 2009 19:35:26 +0100 Subject: [Numpy-discussion] boolean arrays In-Reply-To: <1259244844.6875.3.camel@localhost.localdomain> References: <1259252779.26782.67.camel@nadav.envision.co.il> <1259242994.6875.0.camel@localhost.localdomain> <20091126134407.GB30492@phare.normalesup.org> <1259244844.6875.3.camel@localhost.localdomain> Message-ID: On Thu, 26 Nov 2009 15:14:04 +0100 Fabrice Silva wrote: > Le jeudi 26 novembre 2009 ? 14:44 +0100, Gael Varoquaux >a ?crit : >> On Thu, Nov 26, 2009 at 02:43:14PM +0100, Fabrice Silva >>wrote: >> > Le jeudi 26 novembre 2009 ? 18:26 +0200, Nadav Horesh >>a ?crit : >> > > It is obvious to me that True+False == True,. Why do >>you think it should >> > > be False? >> >> > I would understand it is not obvious that '+' stands >>for logical 'or', >> > and '*' for logical 'and'... >> >> In Bool's algebra, this is the common convention. The >>reason being that >> only 'or' can correspond to the additive law of an >>algebra: its null >> element is absorbant for 'and'. >> >> In other words, if you map '+' and '*' to the opposite, >>you'll get >> suprising behaviors. > > I fully agree with you. My point was to complete Nadav's >comment with > potentially missing information, trying to figrue why >Nils was expected >False... > > -- >Fabrice Silva > LMA UPR CNRS 7051 Sorry, I mixed up '+' and '&' >>> a = array(([True,True],[True,True])) >>> b = array(([False,False],[False,False])) >>> a & b array([[False, False], [False, False]], dtype=bool) Cheers, Nils From sierra_mtnview at sbcglobal.net Thu Nov 26 14:44:47 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 11:44:47 -0800 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions Message-ID: <4B0EDAAF.5020101@sbcglobal.net> I decided to try some example code from Subject. import numpy import pylab # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 mu, sigma = 2, 0.5 v = numpy.random.normal(mu,sigma,10000) # Plot a normalized histogram with 50 bins pylab.hist(v, bins=50, normed=1) # matplotlib version (plot) pylab.show() # Compute the histogram with numpy and then plot it (n, bins) = numpy.histogram(v, bins=50, normed=1) # NumPy version (no plot) pylab.plot(.5*(bins[1:]+bins[:-1]), n) pylab.show() After the histogram is displayed how do I get to the plot? Where is histogram described in some detail? Normalized? The histogram x-axis goes from 0 to 4.5. How does that happen? Is v is two dimensional? What if it's one dimensional? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From josef.pktd at gmail.com Thu Nov 26 15:23:06 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 15:23:06 -0500 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <4B0EDAAF.5020101@sbcglobal.net> References: <4B0EDAAF.5020101@sbcglobal.net> Message-ID: <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> On Thu, Nov 26, 2009 at 2:44 PM, Wayne Watson wrote: > I decided to try some example code from Subject. > > import numpy > import pylab > # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 > mu, sigma = 2, 0.5 > v = numpy.random.normal(mu,sigma,10000) > # Plot a normalized histogram with 50 bins > pylab.hist(v, bins=50, normed=1) ? ? ? # matplotlib version (plot) > pylab.show() > # Compute the histogram with numpy and then plot it > (n, bins) = numpy.histogram(v, bins=50, normed=1) ?# NumPy version (no plot) > pylab.plot(.5*(bins[1:]+bins[:-1]), n) > pylab.show() > > After the histogram is displayed how do I get to the plot? > Where is histogram described in some detail? Normalized? > The histogram x-axis goes from 0 to 4.5. How does that happen? > Is v is two dimensional? What if it's one dimensional? some quick answers: matlplotlib's histogram uses numpy histogram for the calculations, options are pretty well explained in the numpy docs, matplotlib has docs and examples for the display. If I use numpy.histogram, then, I think, I used bar plot for the display (scipy.stats.tutorial might also have an example where I had taken the pattern from somewhere else) numpy also has 2d and multidimensional histogram, but I don't know if the new 3d features of matplotlib can display them. Josef > > > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sierra_mtnview at sbcglobal.net Thu Nov 26 16:15:43 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 13:15:43 -0800 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> Message-ID: <4B0EEFFF.3090405@sbcglobal.net> josef.pktd at gmail.com wrote: > On Thu, Nov 26, 2009 at 2:44 PM, Wayne Watson > wrote: > >> I decided to try some example code from Subject. >> >> import numpy >> import pylab >> # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 >> mu, sigma = 2, 0.5 >> v = numpy.random.normal(mu,sigma,10000) >> # Plot a normalized histogram with 50 bins >> pylab.hist(v, bins=50, normed=1) # matplotlib version (plot) >> pylab.show() >> # Compute the histogram with numpy and then plot it >> (n, bins) = numpy.histogram(v, bins=50, normed=1) # NumPy version (no plot) >> pylab.plot(.5*(bins[1:]+bins[:-1]), n) >> pylab.show() >> >> After the histogram is displayed how do I get to the plot? >> Where is histogram described in some detail? Normalized? >> The histogram x-axis goes from 0 to 4.5. How does that happen? >> Is v is two dimensional? What if it's one dimensional? >> > > some quick answers: > > matlplotlib's histogram uses numpy histogram for the calculations, > options are pretty well explained in the numpy docs, matplotlib has > docs and examples for the display. > > If I use numpy.histogram, then, I think, I used bar plot for the > display (scipy.stats.tutorial might also have an example where I had > taken the pattern from somewhere else) > > numpy also has 2d and multidimensional histogram, but I don't know if > the new 3d features of matplotlib can display them. > > Josef > > Thanks. That link doesn't seem to exist. When you say docs, do you mean some specific Python-like documentation? It looks like matplotlib has some decent descriptions of their hist program. I'd still like to know how one closes a graph and allows the program to continue. The above program is supposed to do a plot after the histogram, but I do not see one appear. If I press X to exit. It exits. -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From sierra_mtnview at sbcglobal.net Thu Nov 26 18:08:04 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 15:08:04 -0800 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <4B0EEFFF.3090405@sbcglobal.net> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> Message-ID: <4B0F0A54.1090606@sbcglobal.net> I guess the answer is easy about why a plot is not produced. The remark in the histogram line says this will not work in numpy. Oh, well. Wayne Watson wrote: > josef.pktd at gmail.com wrote: > >> On Thu, Nov 26, 2009 at 2:44 PM, Wayne Watson >> wrote: >> >> >>> I decided to try some example code from Subject. >>> >>> import numpy >>> import pylab >>> # Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2 >>> mu, sigma = 2, 0.5 >>> v = numpy.random.normal(mu,sigma,10000) >>> # Plot a normalized histogram with 50 bins >>> pylab.hist(v, bins=50, normed=1) # matplotlib version (plot) >>> pylab.show() >>> # Compute the histogram with numpy and then plot it >>> (n, bins) = numpy.histogram(v, bins=50, normed=1) # NumPy version (no ... From pav at iki.fi Thu Nov 26 18:08:18 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 01:08:18 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 Message-ID: <1259276898.8494.18.camel@idol> Hi, The Python 3 porting needs some decisions on what is Bytes and what is Unicode. I'm currently taking the following approach. Comments? *** dtype field names Either Bytes or Unicode. But 'a' and b'a' are *different* fields. The issue is that: Python 2: {'a': 2}[u'a'] == 2, {u'a': 2}['a'] == 2 Python 3: {'a': 2}[b'a'], {b'a': 2}['a'] raise exceptions so the current assumptions in the C code of u'a' == b'a' cease to hold. dtype titles If Bytes or Unicode, work similarly as field names. dtype format strings, datetime tuple, and any other "protocol" strings Bytes. User can pass in Unicode, but it's converted using UTF8 codec. This will likely change repr() of various objects. Acceptable? -- Pauli Virtanen From pav at iki.fi Thu Nov 26 18:57:59 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 01:57:59 +0200 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <4B0F0A54.1090606@sbcglobal.net> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> <4B0F0A54.1090606@sbcglobal.net> Message-ID: <1259279879.8494.22.camel@idol> to, 2009-11-26 kello 15:08 -0800, Wayne Watson kirjoitti: > I guess the answer is easy about why a plot is not produced. The remark > in the histogram line says this will not work in numpy. Oh, well. It works as it is intended to work. Numpy's histogram function just computes the histogram -- you have to plot it yourself. Indeed, the next two lines in the example use Matplotlib (ie. pylab) to plot the histogram. -- Pauli Virtanen From charlesr.harris at gmail.com Thu Nov 26 19:37:43 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 26 Nov 2009 17:37:43 -0700 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259276898.8494.18.camel@idol> References: <1259276898.8494.18.camel@idol> Message-ID: Hi Pauli, On Thu, Nov 26, 2009 at 4:08 PM, Pauli Virtanen wrote: > Hi, > > The Python 3 porting needs some decisions on what is Bytes and > what is Unicode. > > I'm currently taking the following approach. Comments? > > *** > > dtype field names > > Either Bytes or Unicode. > But 'a' and b'a' are *different* fields. > > The issue is that: > Python 2: {'a': 2}[u'a'] == 2, {u'a': 2}['a'] == 2 > Python 3: {'a': 2}[b'a'], {b'a': 2}['a'] raise exceptions > so the current assumptions in the C code of u'a' == b'a' > cease to hold. > > dtype titles > > If Bytes or Unicode, work similarly as field names. > > dtype format strings, datetime tuple, and any other "protocol" strings > > Bytes. User can pass in Unicode, but it's converted using > UTF8 codec. > > This will likely change repr() of various objects. Acceptable? > > I'm not clear on your recommendation here, is it that we should use bytes, with unicode converted to UTF8? Will that support arrays that have been pickled and such? Or will we just have a minimum of code to fix up? And could you expand on the changes that repr() might undergo? Mind, I think using bytes sounds best, but I haven't looked into the whole strings part of the transition and don't have an informed opinion on the matter. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sierra_mtnview at sbcglobal.net Thu Nov 26 20:18:39 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 17:18:39 -0800 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <1259279879.8494.22.camel@idol> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> <4B0F0A54.1090606@sbcglobal.net> <1259279879.8494.22.camel@idol> Message-ID: <4B0F28EF.8050100@sbcglobal.net> Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and SciPy. They all have seemed part of one another, but I think I see how they've divided up the game. NumPy has no graphics. PyLab has them, basically under matplotlib. histogram seems a bit odd to me. Here's what I mean. I chopped some lines out of the other program: import numpy import pylab #v = [5,5,2,2,2,6,4,4,3,3] v=[1,2,3,4] # Plot a normalized histogram pylab.hist(v, bins=8, normed=1) pylab.show() If it's run, I would have expected 4 bars separated by the same distance. The commented version of v is worse. It gives bar heights that are of an even height. Using more bins gives some odd length bars. Maybe an x row would help. While I'm at it what's with the two arrows at the bottom and the cross? I see the cross moves the drawing area, but does anyone really want to do that? The arrows seem to only move the drawing area slightly. Zoom, pencil, seems pretty offbeat. Pauli Virtanen wrote: > to, 2009-11-26 kello 15:08 -0800, Wayne Watson kirjoitti: > >> I guess the answer is easy about why a plot is not produced. The remark >> in the histogram line says this will not work in numpy. Oh, well. >> > > It works as it is intended to work. Numpy's histogram function just > computes the histogram -- you have to plot it yourself. > > Indeed, the next two lines in the example use Matplotlib (ie. pylab) to > plot the histogram. > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From dwf at cs.toronto.edu Thu Nov 26 20:53:58 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 26 Nov 2009 20:53:58 -0500 Subject: [Numpy-discussion] another numpy/ATLAS problem Message-ID: It seems like I've done this enough times on enough machines but I'm still running into problems... After installing what I'm pretty sure is a complete ATLAS+LAPACK and building numpy against it (appropriately editing site.cfg, etc.) I'm faced with this when building numpy-1.3.0: dwf at mirage:~/tmp$ python Python 2.6.4 (r264:75706, Nov 2 2009, 14:44:17) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ __init__.py", line 130, in import add_newdocs File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ add_newdocs.py", line 9, in from lib import add_newdoc File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ lib/__init__.py", line 13, in from polynomial import * File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ lib/polynomial.py", line 18, in from numpy.linalg import eigvals, lstsq File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ linalg/__init__.py", line 47, in from linalg import * File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ linalg/linalg.py", line 22, in from numpy.linalg import lapack_lite ImportError: /home/dwf/virtualenv/stable/lib/python2.6/site-packages/ numpy/linalg/lapack_lite.so: undefined symbol: dlamc3_ FWIW these are the files that ATLAS has installed: dwf at mirage:~/Downloads$ ls /opt/sw/ATLAS/lib/ -al total 23472 drwxr-xr-x 2 root root 4096 2009-11-26 20:36 . drwxr-xr-x 4 root root 4096 2009-11-26 20:36 .. -rw-r--r-- 1 root root 12963138 2009-11-26 20:36 libatlas.a -rw-r--r-- 1 root root 466244 2009-11-26 20:36 libcblas.a -rw-r--r-- 1 root root 572812 2009-11-26 20:36 libf77blas.a -rw-r--r-- 1 root root 8979016 2009-11-26 20:36 liblapack.a -rw-r--r-- 1 root root 466678 2009-11-26 20:36 libptcblas.a -rw-r--r-- 1 root root 573196 2009-11-26 20:36 libptf77blas.a liblapack.a looks big enough for it to be a complete LAPACK. ATLAS (3.9.17) was configured with the following: ../configure -Ss kern /usr/bin/gcc-4.2 --prefix=/opt/sw/ATLAS --with- netlib-lapack-tarfile=/home/dwf/tmp/lapack-lite-3.1.1.tgz -Fa alg - fPIC -D c -DPentiumCPS=2668 -t 4 -b 64 I tried a few different things, including building dylibs, in which case I get an undefined symbol error for ATL__zttrsm. Any help would be appreciated. To be honest I'm not sure why lapack_lite is getting involved at all; I thought that was what got used in case of no detected LAPACK, but config definitely does detect ATLAS/LAPACK. David From charlesr.harris at gmail.com Thu Nov 26 20:56:22 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 26 Nov 2009 18:56:22 -0700 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: Message-ID: On Thu, Nov 26, 2009 at 6:53 PM, David Warde-Farley wrote: > It seems like I've done this enough times on enough machines but I'm > still running into problems... > > After installing what I'm pretty sure is a complete ATLAS+LAPACK and > building numpy against it (appropriately editing site.cfg, etc.) I'm > faced with this when building numpy-1.3.0: > > dwf at mirage:~/tmp$ python > Python 2.6.4 (r264:75706, Nov 2 2009, 14:44:17) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > Traceback (most recent call last): > File "", line 1, in > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > __init__.py", line 130, in > import add_newdocs > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > add_newdocs.py", line 9, in > from lib import add_newdoc > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > lib/__init__.py", line 13, in > from polynomial import * > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > lib/polynomial.py", line 18, in > from numpy.linalg import eigvals, lstsq > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > linalg/__init__.py", line 47, in > from linalg import * > File "/home/dwf/virtualenv/stable/lib/python2.6/site-packages/numpy/ > linalg/linalg.py", line 22, in > from numpy.linalg import lapack_lite > ImportError: /home/dwf/virtualenv/stable/lib/python2.6/site-packages/ > numpy/linalg/lapack_lite.so: undefined symbol: dlamc3_ > > FWIW these are the files that ATLAS has installed: > > dwf at mirage:~/Downloads$ ls /opt/sw/ATLAS/lib/ -al > total 23472 > drwxr-xr-x 2 root root 4096 2009-11-26 20:36 . > drwxr-xr-x 4 root root 4096 2009-11-26 20:36 .. > -rw-r--r-- 1 root root 12963138 2009-11-26 20:36 libatlas.a > -rw-r--r-- 1 root root 466244 2009-11-26 20:36 libcblas.a > -rw-r--r-- 1 root root 572812 2009-11-26 20:36 libf77blas.a > -rw-r--r-- 1 root root 8979016 2009-11-26 20:36 liblapack.a > -rw-r--r-- 1 root root 466678 2009-11-26 20:36 libptcblas.a > -rw-r--r-- 1 root root 573196 2009-11-26 20:36 libptf77blas.a > > liblapack.a looks big enough for it to be a complete LAPACK. > > ATLAS (3.9.17) was configured with the following: > > ../configure -Ss kern /usr/bin/gcc-4.2 --prefix=/opt/sw/ATLAS --with- > netlib-lapack-tarfile=/home/dwf/tmp/lapack-lite-3.1.1.tgz -Fa alg - > fPIC -D c -DPentiumCPS=2668 -t 4 -b 64 > > I never had luck with the netlib-lapack-tarfile= option, it didn't build lapack correctly. Try doing them separately. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Nov 26 21:48:28 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 21:48:28 -0500 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <4B0F28EF.8050100@sbcglobal.net> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> <4B0F0A54.1090606@sbcglobal.net> <1259279879.8494.22.camel@idol> <4B0F28EF.8050100@sbcglobal.net> Message-ID: <1cd32cbb0911261848g1619186bv9bf49861b8f12fa0@mail.gmail.com> On Thu, Nov 26, 2009 at 8:18 PM, Wayne Watson wrote: > Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and > SciPy. They all have seemed part of one another, but I think I see how > they've divided up the game. NumPy has no graphics. PyLab has them, > basically under matplotlib. histogram seems a bit odd to me. ?Here's > what I mean. I chopped some lines out of the other program: > import numpy > import pylab > #v = [5,5,2,2,2,6,4,4,3,3] > v=[1,2,3,4] > # Plot a normalized histogram > pylab.hist(v, bins=8, normed=1) > pylab.show() > > If it's run, I would have expected 4 bars separated by the same > distance. The commented version of v is worse. It gives bar heights that > are of an even height. Using more bins gives some odd length bars. Maybe > an x row would help. > > While I'm at it what's with the two arrows at the bottom and the cross? > I see the cross moves the drawing area, but does anyone really want to > do that? The arrows seem to only move the drawing area slightly. Zoom, > pencil, ?seems pretty offbeat. Do you mean this http://matplotlib.sourceforge.net/users/navigation_toolbar.html Zoom is convenient when you have for example a thousand points with lots of variation in one graph. Browsing some examples in the matplotlib docs might be useful to get a basic idea of it e.g http://matplotlib.sourceforge.net/gallery.html The main docs for numpy scipy start here http://docs.scipy.org/doc/ many/most of the numpy functions have by now examples in the reference guide and docstrings and might be more up to date than the tutorial. It might be helpful to browse them in parallel to the tutorial. Josef > > Pauli Virtanen wrote: >> to, 2009-11-26 kello 15:08 -0800, Wayne Watson kirjoitti: >> >>> I guess the answer is easy about why a plot is not produced. The remark >>> in the histogram line says this will not work in numpy. Oh, well. >>> >> >> It works as it is intended to work. Numpy's histogram function just >> computes the histogram -- you have to plot it yourself. >> >> Indeed, the next two lines in the example use Matplotlib (ie. pylab) to >> plot the histogram. >> >> > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Thu Nov 26 21:49:38 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Nov 2009 21:49:38 -0500 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <1cd32cbb0911261848g1619186bv9bf49861b8f12fa0@mail.gmail.com> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> <4B0F0A54.1090606@sbcglobal.net> <1259279879.8494.22.camel@idol> <4B0F28EF.8050100@sbcglobal.net> <1cd32cbb0911261848g1619186bv9bf49861b8f12fa0@mail.gmail.com> Message-ID: <1cd32cbb0911261849n4c507ebes91fdccd38b6b948a@mail.gmail.com> On Thu, Nov 26, 2009 at 9:48 PM, wrote: > On Thu, Nov 26, 2009 at 8:18 PM, Wayne Watson > wrote: >> Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and >> SciPy. They all have seemed part of one another, but I think I see how >> they've divided up the game. NumPy has no graphics. PyLab has them, >> basically under matplotlib. histogram seems a bit odd to me. ?Here's >> what I mean. I chopped some lines out of the other program: >> import numpy >> import pylab >> #v = [5,5,2,2,2,6,4,4,3,3] >> v=[1,2,3,4] >> # Plot a normalized histogram >> pylab.hist(v, bins=8, normed=1) >> pylab.show() >> >> If it's run, I would have expected 4 bars separated by the same >> distance. The commented version of v is worse. It gives bar heights that >> are of an even height. Using more bins gives some odd length bars. Maybe >> an x row would help. >> >> While I'm at it what's with the two arrows at the bottom and the cross? >> I see the cross moves the drawing area, but does anyone really want to >> do that? The arrows seem to only move the drawing area slightly. Zoom, >> pencil, ?seems pretty offbeat. > > Do you mean this http://matplotlib.sourceforge.net/users/navigation_toolbar.html > > Zoom is convenient when you have for example a thousand points with > lots of variation in one graph. > Browsing some examples in the matplotlib docs might be useful to get a > basic idea of it > e.g http://matplotlib.sourceforge.net/gallery.html > > The main docs for numpy scipy start here http://docs.scipy.org/doc/ http://docs.scipy.org/doc/ I guess a linebreak got wrongly interpreted Josef > many/most of the numpy functions have by now examples in the reference > guide and docstrings and might be more up to date than the tutorial. > It might be helpful to browse them in parallel to the tutorial. > > Josef > >> >> Pauli Virtanen wrote: >>> to, 2009-11-26 kello 15:08 -0800, Wayne Watson kirjoitti: >>> >>>> I guess the answer is easy about why a plot is not produced. The remark >>>> in the histogram line says this will not work in numpy. Oh, well. >>>> >>> >>> It works as it is intended to work. Numpy's histogram function just >>> computes the histogram -- you have to plot it yourself. >>> >>> Indeed, the next two lines in the example use Matplotlib (ie. pylab) to >>> plot the histogram. >>> >>> >> >> -- >> ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 >> ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org >> ? ? ? ? ? ?The major event has passed, but keep the number alive. >> >> ? ? ? ? ? ? ? ? ? ?Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From sierra_mtnview at sbcglobal.net Fri Nov 27 00:16:21 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 21:16:21 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known Message-ID: <4B0F60A5.7040302@sbcglobal.net> I have a list that already has the frequencies from 0 to 255. However, I'd like to make a histogram that has say 32 bins whose ranges are 0-7, 8-15, ... 248-255. Is it possible? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From sierra_mtnview at sbcglobal.net Fri Nov 27 00:20:05 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Thu, 26 Nov 2009 21:20:05 -0800 Subject: [Numpy-discussion] NumPy Histogram for Tentative NumPy Tutorial Questions In-Reply-To: <1cd32cbb0911261849n4c507ebes91fdccd38b6b948a@mail.gmail.com> References: <4B0EDAAF.5020101@sbcglobal.net> <1cd32cbb0911261223i6037080dt945d581918f67bca@mail.gmail.com> <4B0EEFFF.3090405@sbcglobal.net> <4B0F0A54.1090606@sbcglobal.net> <1259279879.8494.22.camel@idol> <4B0F28EF.8050100@sbcglobal.net> <1cd32cbb0911261848g1619186bv9bf49861b8f12fa0@mail.gmail.com> <1cd32cbb0911261849n4c507ebes91fdccd38b6b948a@mail.gmail.com> Message-ID: <4B0F6185.70902@sbcglobal.net> josef.pktd at gmail.com wrote: > On Thu, Nov 26, 2009 at 9:48 PM, wrote: > >> On Thu, Nov 26, 2009 at 8:18 PM, Wayne Watson >> wrote: >> >>> Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and >>> SciPy. They all have seemed part of one another, but I think I see how >>> they've divided up the game. NumPy has no graphics. PyLab has them, >>> basically under matplotlib. histogram seems a bit odd to me. Here's >>> what I mean. I chopped some lines out of the other program: >>> import numpy >>> import pylab >>> #v = [5,5,2,2,2,6,4,4,3,3] >>> v=[1,2,3,4] >>> # Plot a normalized histogram >>> pylab.hist(v, bins=8, normed=1) >>> pylab.show() >>> >>> If it's run, I would have expected 4 bars separated by the same >>> distance. The commented version of v is worse. It gives bar heights that >>> are of an even height. Using more bins gives some odd length bars. Maybe >>> an x row would help. >>> >>> While I'm at it what's with the two arrows at the bottom and the cross? >>> I see the cross moves the drawing area, but does anyone really want to >>> do that? The arrows seem to only move the drawing area slightly. Zoom, >>> pencil, seems pretty offbeat. >>> >> Do you mean this http://matplotlib.sourceforge.net/users/navigation_toolbar.html >> >> Zoom is convenient when you have for example a thousand points with >> lots of variation in one graph. >> Browsing some examples in the matplotlib docs might be useful to get a >> basic idea of it >> e.g http://matplotlib.sourceforge.net/gallery.html >> >> The main docs for numpy scipy start here http://docs.scipy.org/doc/ >> > > http://docs.scipy.org/doc/ > > I guess a linebreak got wrongly interpreted > > Josef > Ah, navigation. Thanks for the links From richard.beare at sci.monash.edu.au Fri Nov 27 00:30:07 2009 From: richard.beare at sci.monash.edu.au (Richared Beare) Date: Fri, 27 Nov 2009 16:30:07 +1100 Subject: [Numpy-discussion] using numpy.savetxt to save columns of numerical values and columns of text values Message-ID: <4B0F63DF.2040306@sci.monash.edu.au> I have been unable to find a way of doing a very simple thing: saving data that contains both arrays of numerical values and arrays of string values, using savetxt in numpy. As a very simple example, suppose a is a numpy array of integers and b is one containing strings, e.g.: a = np.array([1,2,3]) b = np.array(['one', 'two', 'three']) The following call to savetxt: savetxt('test.txt', transpose((a,b)), fmt='%i %s') produces the following error: float argument required, not numpy.string_ I don't have any problems if I mix integers and floats in the format string, or save all values as strings, but I am completely unable to save formatted numerical values and strings in different columns using savetxt. There must be an easy solution! Thank you. Richard Beare From josef.pktd at gmail.com Fri Nov 27 00:39:29 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 00:39:29 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B0F60A5.7040302@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> Message-ID: <1cd32cbb0911262139m680d8039qb6126adf432e1964@mail.gmail.com> On Fri, Nov 27, 2009 at 12:16 AM, Wayne Watson wrote: > I have a list that already has the frequencies from 0 to 255. However, > I'd like to make a histogram ?that has say 32 bins whose ranges are 0-7, > 8-15, ... 248-255. Is it possible? If they have equal sized (width) bins, then you should be able to reshape and sum up frequ.reshape((-1,8)).sum(1) or something like this, If the bins have different interval widths, then it might be a bit trickier to do it without loop. Josef > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gokhansever at gmail.com Fri Nov 27 00:55:26 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Thu, 26 Nov 2009 23:55:26 -0600 Subject: [Numpy-discussion] using numpy.savetxt to save columns of numerical values and columns of text values In-Reply-To: <4B0F63DF.2040306@sci.monash.edu.au> References: <4B0F63DF.2040306@sci.monash.edu.au> Message-ID: <49d6b3500911262155p50f202eemb1bd43ac73f461ae@mail.gmail.com> On Thu, Nov 26, 2009 at 11:30 PM, Richared Beare < richard.beare at sci.monash.edu.au> wrote: > I have been unable to find a way of doing a very simple thing: saving > data that contains both arrays of numerical values and arrays of string > values, using savetxt in numpy. > > As a very simple example, suppose a is a numpy array of integers and b > is one containing strings, e.g.: > > a = np.array([1,2,3]) > > b = np.array(['one', 'two', 'three']) > > The following call to savetxt: > > savetxt('test.txt', transpose((a,b)), fmt='%i %s') > > produces the following error: > > float argument required, not numpy.string_ > > I don't have any problems if I mix integers and floats in the format > string, or save all values as strings, but I am completely unable to > save formatted numerical values and strings in different columns using > savetxt. > > There must be an easy solution! > Seemingly savetxt('test.txt', transpose((a,b)), fmt='%s %s') ; replacing the %i with %s works. However you can't control the output precision while saving the data into a file. > > Thank you. > > Richard Beare > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Fri Nov 27 01:44:49 2009 From: jsseabold at gmail.com (Skipper Seabold) Date: Fri, 27 Nov 2009 01:44:49 -0500 Subject: [Numpy-discussion] using numpy.savetxt to save columns of numerical values and columns of text values In-Reply-To: <49d6b3500911262155p50f202eemb1bd43ac73f461ae@mail.gmail.com> References: <4B0F63DF.2040306@sci.monash.edu.au> <49d6b3500911262155p50f202eemb1bd43ac73f461ae@mail.gmail.com> Message-ID: On Fri, Nov 27, 2009 at 12:55 AM, G?khan Sever wrote: > > > On Thu, Nov 26, 2009 at 11:30 PM, Richared Beare > wrote: >> >> I have been unable to find a way of doing a very simple thing: saving >> data that contains both arrays of numerical values and arrays of string >> values, using savetxt in numpy. >> >> As a very simple example, suppose a is a numpy array of integers and b >> is one containing strings, e.g.: >> >> ? ?a = np.array([1,2,3]) >> >> ? ?b = np.array(['one', 'two', 'three']) >> >> The following call to savetxt: >> >> ? ?savetxt('test.txt', transpose((a,b)), fmt='%i %s') >> >> produces the following error: >> >> ? ?float argument required, not numpy.string_ >> In [14]: np.transpose((a,b)) Out[14]: array([['1', 'one'], ['2', 'two'], ['3', 'three']], dtype='|S8') The ints get cast to strings by transpose. You need a structured array. This works but there may be an easier way to build the array. ab = np.zeros(3, dtype=[('var1',float),('var2','a5')]) ab['var1'] = a ab['var2'] = b np.savetxt('test.txt', ab, fmt="%i %s") hth, Skipper >> I don't have any problems if I mix integers and floats in the format >> string, or save all values as strings, but I am completely unable to >> save formatted numerical values and strings in different columns using >> savetxt. >> >> There must be an easy solution! > > Seemingly savetxt('test.txt', transpose((a,b)), fmt='%s %s')? ; replacing > the %i with %s works. However you can't control the output precision while > saving the data into a file. > >> >> Thank you. >> >> Richard Beare >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From renesd at gmail.com Fri Nov 27 02:05:10 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 08:05:10 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: References: <1259276898.8494.18.camel@idol> Message-ID: <64ddb72c0911262305w1654fa98t897ea2ffffa3edec@mail.gmail.com> On Fri, Nov 27, 2009 at 1:37 AM, Charles R Harris wrote: > Hi Pauli, > > On Thu, Nov 26, 2009 at 4:08 PM, Pauli Virtanen wrote: >> >> Hi, >> >> The Python 3 porting needs some decisions on what is Bytes and >> what is Unicode. >> >> I'm currently taking the following approach. Comments? >> >> ? ? ? ?*** >> >> dtype field names >> >> ? ? ? ?Either Bytes or Unicode. >> ? ? ? ?But 'a' and b'a' are *different* fields. >> >> ? ? ? ?The issue is that: >> ? ? ? ? ? ?Python 2: {'a': 2}[u'a'] == 2, {u'a': 2}['a'] == 2 >> ? ? ? ? ? ?Python 3: {'a': 2}[b'a'], {b'a': 2}['a'] raise exceptions >> ? ? ? ?so the current assumptions in the C code of u'a' == b'a' >> ? ? ? ?cease to hold. >> >> dtype titles >> >> ? ? ? ?If Bytes or Unicode, work similarly as field names. >> >> dtype format strings, datetime tuple, and any other "protocol" strings >> >> ? ? ? ?Bytes. User can pass in Unicode, but it's converted using >> ? ? ? ?UTF8 codec. >> >> ? ? ? ?This will likely change repr() of various objects. Acceptable? >> > > I'm not clear on your recommendation here, is it that we should use bytes, > with unicode converted to UTF8? Will that support arrays that have been > pickled and such? Or will we just have a minimum of code to fix up? And > could you expand on the changes that repr() might undergo? > > Mind, I think using bytes sounds best, but I haven't looked into the whole > strings part of the transition and don't have an informed opinion on the > matter. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > To help clarify for people who are not familiar with python 3... To put it simply... in py3, str == unicode utf-8, there is no 'unicode' anymore. bytes == raw data... but kind of like a str type with less methods. bytes + encoding is how you do non utf-8 strings. 'array' exists in both py2 and py3 with a very similar interface on both. There's a more precise description of strings in python3 on these pages: http://diveintopython3.org/strings.html http://diveintopython3.org/porting-code-to-python-3-with-2to3.html It depends on the use cases for each thing which will depend on how it should work imho. Mostly if you are using the str type, then keep using the str type. Many functions take both bytes and strings. Since it is sane to work on both bytes and strings from a users perspective. There have been some methods in the stdlib that have not consumed both, and they have been treated as bugs, and are being fixed (eg, some urllib methods). For dtype, using the python 'str' by default seems ok. Since all of those characters come out in the same manner on both pythons for the data used by numpy. eg. 'float32' is shown the same as a py3 string as a py2 string. Internally it is unicode data however. Within py2, we save a pickle with the str: >>> import pickle >>> pickle.dump(s, open('/tmp/p.pickle', 'wb')) >>> pickle.dump(s, open('/tmp/p.pickle', 'wb')) >>> pickle.dump('float32', open('/tmp/p.pickle', 'wb')) Within py3 we open the pickle with the str: >>> import pickle >>> pickle.load(open('/tmp/p.pickle', 'rb')) 'float32' cheers, From renesd at gmail.com Fri Nov 27 02:20:59 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 08:20:59 +0100 Subject: [Numpy-discussion] boolean arrays In-Reply-To: References: <1259252779.26782.67.camel@nadav.envision.co.il> <1259242994.6875.0.camel@localhost.localdomain> <20091126134407.GB30492@phare.normalesup.org> <1259244844.6875.3.camel@localhost.localdomain> Message-ID: <64ddb72c0911262320s2a186dabs3173499004cc1dbe@mail.gmail.com> On Thu, Nov 26, 2009 at 7:35 PM, Nils Wagner wrote: > > > Sorry, I mixed up '+' and '&' > >>>> a = array(([True,True],[True,True])) >>>> b = array(([False,False],[False,False])) >>>> a & b > array([[False, False], > ? ? ? ?[False, False]], dtype=bool) > > Cheers, > > ? ? ? ? ? ? ?Nils hey, this is a classical problem with + (sometimes pronounced 'and') and & (also pronounced 'and'). Happens to all of us sometimes. cu, From schut at sarvision.nl Fri Nov 27 03:33:26 2009 From: schut at sarvision.nl (Vincent Schut) Date: Fri, 27 Nov 2009 09:33:26 +0100 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B0F60A5.7040302@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> Message-ID: Wayne Watson wrote: > I have a list that already has the frequencies from 0 to 255. However, > I'd like to make a histogram that has say 32 bins whose ranges are 0-7, > 8-15, ... 248-255. Is it possible? > Wayne, you might find the 'numpy example list with doc' webpage quite informative... http://www.scipy.org/Numpy_Example_List_With_Doc (give it some time to load, it's pretty large...) For new users (I was one once...) it usually takes some time to find the usual suspects in numpy/scipy help and docs... This one page has really become unvaluable for me. It gives you the docstrings for numpy functions, often including some example code. If you check out the histogram() function, you'll see it takes a 'bins=' argument: bins : int or sequence of scalars, optional If `bins` is an int, it defines the number of equal-width bins in the given range (10, by default). If `bins` is a sequence, it defines the bin edges, including the rightmost edge, allowing for non-uniform bin widths. So, if your bins are known, you can pass it to numpy.histogram, either as number-of-bins (if equal width), if necessary combined with the 'range=' parameter to specify the range to divide into equal bins, or as bin edges (e.g. in your case: (0, 8, 16, ... 256) or numpy.linspace(0,256,33) which will give you this range nicely. If you don't specify the 'range=' parameter, it will check the min and max from your input data and use that as lower and upper bounds. Good luck learning numpy! :) Vincent. From pav at iki.fi Fri Nov 27 04:47:53 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 11:47:53 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: References: <1259276898.8494.18.camel@idol> Message-ID: <1259315273.4110.804.camel@talisman> to, 2009-11-26 kello 17:37 -0700, Charles R Harris kirjoitti: [clip] > I'm not clear on your recommendation here, is it that we should use > bytes, with unicode converted to UTF8? The point is that I don't think we can just decide to use Unicode or Bytes in all places where PyString was used earlier. Which one it will be should depend on the use. Users will expect that eg. array([1,2,3], dtype='f4') still works, and they don't have to do e.g. array([1,2,3], dtype=b'f4'). To summarize the use cases I've ran across so far: 1) For 'S' dtype, I believe we use Bytes for the raw data and the interface. Maybe we want to introduce a separate "bytes" dtype that's an alias for 'S'? 2) The field names: a = array([], dtype=[('a', int)]) a = array([], dtype=[(b'a', int)]) This is somewhat of an internal issue. We need to decide whether we internally coerce input to Unicode or Bytes. Or whether we allow for both Unicode and Bytes (but preserving previous semantics in this case requires extra work, due to semantic changes in PyDict). Currently, there's some code in Numpy to allow for Unicode field names, but it's not been coherently implemented in all places, so e.g. direct creation of dtypes with unicode field names fails. This has also implications on field titles, as also those are stored in the fields dict. 3) Format strings a = array([], dtype=b'i4') I don't think it makes sense to handle format strings in Unicode internally -- they should always be coerced to bytes. This will make it easier at many points, since it will be enought to do PyBytes_AS_STRING(str) to get the char* pointer, rather than having to encode to utf-8 first. Same for all other similar uses of string, e.g. protocol descriptors. User input should just be coerced to ASCII on input, I believe. The problem here is that preserving repr() in this case requires some extra work. But maybe that has to be done. > Will that support arrays that have been pickled and such? Are the pickles backward compatible between Python 2 and 3 at all? I think using Bytes for format strings will be backward-compatible. Field names are then a bit more difficult. Actually, we'll probably just have to coerce them to either Bytes or Unicode internally, since we'll need to do that on unpickling if we want to be backward-compatible. > Or will we just have a minimum of code to fix up? I think we will need in any case to replace all use of PyString in Numpy by PyBytes or PyUnicode, depending on context, and #define PyString PyBytes for Python 2. This seems to be the easiest way to make sure we have fixed all points that need fixing. Currently, 193 of 800 numpy.core tests don't pass, and this seems largely due to Bytes vs. Unicode issues. > And could you expand on the changes that repr() might undergo? The main thing is that dtype('i4') dtype([('a', 'i4')]) may become dtype(b'i4') dtype([(b'a', b'i4')]) Of course, we can write and #ifdef separate repr formatting code for Py3, but this is a bit of extra work. > Mind, I think using bytes sounds best, but I haven't looked into the > whole strings part of the transition and don't have an informed > opinion on the matter. *** By the way, should I commit this stuff (after factoring the commits to logical chunks) to SVN? It does not break anything for Python 2, at least as far as the test suite is concerned. Pauli From david at ar.media.kyoto-u.ac.jp Fri Nov 27 04:30:25 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 27 Nov 2009 18:30:25 +0900 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259315273.4110.804.camel@talisman> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> Message-ID: <4B0F9C31.2060006@ar.media.kyoto-u.ac.jp> Pauli Virtanen wrote: > By the way, should I commit this stuff (after factoring the commits to > logical chunks) to SVN? > I would prefer getting at least one py3 buildbot before doing anything significant, cheers, David From pav at iki.fi Fri Nov 27 05:02:54 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 12:02:54 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <4B0F9C31.2060006@ar.media.kyoto-u.ac.jp> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> <4B0F9C31.2060006@ar.media.kyoto-u.ac.jp> Message-ID: <1259316175.4110.806.camel@talisman> pe, 2009-11-27 kello 18:30 +0900, David Cournapeau kirjoitti: > Pauli Virtanen wrote: > > By the way, should I commit this stuff (after factoring the commits to > > logical chunks) to SVN? > > I would prefer getting at least one py3 buildbot before doing anything > significant, I can add it to mine: http://buildbot.scipy.org/builders/Linux_x86_Ubuntu/builds/279/steps/shell_1/logs/stdio It already does 2.4, 2.5 and 2.6. Pauli From cheronetolivia at yahoo.com Fri Nov 27 05:11:55 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Fri, 27 Nov 2009 02:11:55 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <5b8d13220911250256k62a3c13ej98ac04e89ccacfeb@mail.gmail.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> <305713.71840.qm@web51008.mail.re2.yahoo.com> <5b8d13220911250256k62a3c13ej98ac04e89ccacfeb@mail.gmail.com> Message-ID: <887521.34374.qm@web51012.mail.re2.yahoo.com> Hi, I have tried to remove my entire numpy directory and starting to build it again from a newly downloaded source (numpy-1.3.0.tar.gz), but it has made no difference. I still get the output below. Thank you for the suggestions, Olivia ... ... ... creating build/temp.cygwin-1.5.25-i686-2.5 creating build/temp.cygwin-1.5.25-i686-2.5/build creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5 creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core creating build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c numpy/core/src/npy_math.c.src:186: error: parse error before '/' token numpy/core/src/npy_math.c.src:186: error: parse error before '/' token error: Command "gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.5.25-i686-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.c -o build/temp.cygwin-1.5.25-i686-2.5/build/src.cygwin-1.5.25-i686-2.5/numpy/core/src/npy_math.o" failed with exit status 1 ----- Original Message ---- > > I have just tested a fresh svn checkout, and could built numpy > correctly on cygwin. I would suggest you update your sources, and > build from scratch (i.e. remove the entire build directory and start > from scratch). > > David From faltet at pytables.org Fri Nov 27 05:17:15 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Nov 2009 11:17:15 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259315273.4110.804.camel@talisman> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> Message-ID: <200911271117.16002.faltet@pytables.org> A Friday 27 November 2009 10:47:53 Pauli Virtanen escrigu?: > 1) For 'S' dtype, I believe we use Bytes for the raw data and the > interface. > > Maybe we want to introduce a separate "bytes" dtype that's an alias > for 'S'? Yeah. As regular strings in Python 3 are Unicode, I think that introducing separate "bytes" dtype would help doing the transition. Meanwhile, the next should still work: In [2]: s = np.array(['asa'], dtype="S10") In [3]: s[0] Out[3]: 'asa' # will become b'asa' in Python 3 In [4]: s.dtype.itemsize Out[4]: 10 # still 1-byte per element Also, I suppose that there will be issues with the current Unicode support in NumPy: In [5]: u = np.array(['asa'], dtype="U10") In [6]: u[0] Out[6]: u'asa' # will become 'asa' in Python 3 In [7]: u.dtype.itemsize Out[7]: 40 # not sure about the size in Python 3 For example, if it is true that internal strings in Python 3 and Unicode UTF-8 (as Ren? seems to suggest), I suppose that the internal conversions from 2- bytes or 4-bytes (depending on how the Python interpreter has been compiled) in NumPy Unicode dtype to the new Python string should have to be reworked (perhaps you have dealt with that already). Cheers, -- Francesc Alted From david at ar.media.kyoto-u.ac.jp Fri Nov 27 05:04:23 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 27 Nov 2009 19:04:23 +0900 Subject: [Numpy-discussion] New behavior for correlate w.r.t. swapped input: deprecate current default for 1.4.0 ? Message-ID: <4B0FA427.3010607@ar.media.kyoto-u.ac.jp> Hi, The function correlate can now implement what is generally accepted as the definition of correlation, but you need to request this behavior ATM (by default, it still does the wrong thing). Is it ok to deprecate the default ? I.e. the behavior is exactly the same, but we warn users about the change, and in 1.5.0, the default is to use the new behavior. cheers, David From pav at iki.fi Fri Nov 27 05:27:00 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 12:27:00 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <200911271117.16002.faltet@pytables.org> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> <200911271117.16002.faltet@pytables.org> Message-ID: <1259317620.4110.813.camel@talisman> pe, 2009-11-27 kello 11:17 +0100, Francesc Alted kirjoitti: > A Friday 27 November 2009 10:47:53 Pauli Virtanen escrigu?: > > 1) For 'S' dtype, I believe we use Bytes for the raw data and the > > interface. > > > > Maybe we want to introduce a separate "bytes" dtype that's an alias > > for 'S'? > > Yeah. As regular strings in Python 3 are Unicode, I think that introducing > separate "bytes" dtype would help doing the transition. Meanwhile, the next > should still work: > > In [2]: s = np.array(['asa'], dtype="S10") > > In [3]: s[0] > Out[3]: 'asa' # will become b'asa' in Python 3 > > In [4]: s.dtype.itemsize > Out[4]: 10 # still 1-byte per element Yes. But now I wonder, should array(['foo'], str) array(['foo']) be of dtype 'S' or 'U' in Python 3? I think I'm leaning towards 'U', which will mean unavoidable code breakage -- there's probably no avoiding it. [clip] > Also, I suppose that there will be issues with the current Unicode support in > NumPy: > > In [5]: u = np.array(['asa'], dtype="U10") > > In [6]: u[0] > Out[6]: u'asa' # will become 'asa' in Python 3 > > In [7]: u.dtype.itemsize > Out[7]: 40 # not sure about the size in Python 3 I suspect the Unicode stuff will keep working without major changes, except maybe dropping the u in repr. It is difficult to believe the CPython guys would have significantly changed the current Unicode implementation, if they didn't bother changing the names of the functions :) > For example, if it is true that internal strings in Python 3 and Unicode UTF-8 > (as Ren? seems to suggest), I suppose that the internal conversions from 2- > bytes or 4-bytes (depending on how the Python interpreter has been compiled) > in NumPy Unicode dtype to the new Python string should have to be reworked > (perhaps you have dealt with that already). I don't think they are internally UTF-8: http://docs.python.org/3.1/c-api/unicode.html """Python?s default builds use a 16-bit type for Py_UNICODE and store Unicode values internally as UCS2.""" -- Pauli Virtanen From faltet at pytables.org Fri Nov 27 05:50:03 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Nov 2009 11:50:03 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259317620.4110.813.camel@talisman> References: <1259276898.8494.18.camel@idol> <200911271117.16002.faltet@pytables.org> <1259317620.4110.813.camel@talisman> Message-ID: <200911271150.03234.faltet@pytables.org> A Friday 27 November 2009 11:27:00 Pauli Virtanen escrigu?: > Yes. But now I wonder, should > > array(['foo'], str) > array(['foo']) > > be of dtype 'S' or 'U' in Python 3? I think I'm leaning towards 'U', > which will mean unavoidable code breakage -- there's probably no > avoiding it. Mmh, you are right. Yes, this seems to be difficult to solve. Well, I'm changing my mind and think that both 'str' and 'S' should stand for Unicode in NumPy for Python 3. If people is aware of the change for Python 3, they should be expecting the same change happening in NumPy too, I guess. Then, I suppose that a new dtype "bytes" that replaces the existing "string" would be absolutely necessary. > > Also, I suppose that there will be issues with the current Unicode > > support in NumPy: > > > > In [5]: u = np.array(['asa'], dtype="U10") > > > > In [6]: u[0] > > Out[6]: u'asa' # will become 'asa' in Python 3 > > > > In [7]: u.dtype.itemsize > > Out[7]: 40 # not sure about the size in Python 3 > > I suspect the Unicode stuff will keep working without major changes, > except maybe dropping the u in repr. It is difficult to believe the > CPython guys would have significantly changed the current Unicode > implementation, if they didn't bother changing the names of the > functions :) > > > For example, if it is true that internal strings in Python 3 and Unicode > > UTF-8 (as Ren? seems to suggest), I suppose that the internal conversions > > from 2- bytes or 4-bytes (depending on how the Python interpreter has > > been compiled) in NumPy Unicode dtype to the new Python string should > > have to be reworked (perhaps you have dealt with that already). > > I don't think they are internally UTF-8: > http://docs.python.org/3.1/c-api/unicode.html > > """Python?s default builds use a 16-bit type for Py_UNICODE and store > Unicode values internally as UCS2.""" Ah! No changes for that matter. Much better then. -- Francesc Alted From cournape at gmail.com Fri Nov 27 06:32:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 27 Nov 2009 20:32:09 +0900 Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <887521.34374.qm@web51012.mail.re2.yahoo.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> <305713.71840.qm@web51008.mail.re2.yahoo.com> <5b8d13220911250256k62a3c13ej98ac04e89ccacfeb@mail.gmail.com> <887521.34374.qm@web51012.mail.re2.yahoo.com> Message-ID: <5b8d13220911270332n5f208024p272eeee00f77b4fd@mail.gmail.com> On Fri, Nov 27, 2009 at 7:11 PM, Olivia Cheronet wrote: > Hi, > > I have tried to remove my entire numpy directory and starting to build it again from a newly downloaded source (numpy-1.3.0.tar.gz), but it has made no difference. Please update to the trunk - I can see the error as well for 1.3.0, and the trunk does build correctly on cygwin. I don't understand where the error is coming from in 1.3.0, it almost look like a cpp bug. cheers, David From renesd at gmail.com Fri Nov 27 07:23:10 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 13:23:10 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <200911271150.03234.faltet@pytables.org> References: <1259276898.8494.18.camel@idol> <200911271117.16002.faltet@pytables.org> <1259317620.4110.813.camel@talisman> <200911271150.03234.faltet@pytables.org> Message-ID: <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> On Fri, Nov 27, 2009 at 11:50 AM, Francesc Alted wrote: > A Friday 27 November 2009 11:27:00 Pauli Virtanen escrigu?: >> Yes. But now I wonder, should >> >> ? ? ? array(['foo'], str) >> ? ? ? array(['foo']) >> >> be of dtype 'S' or 'U' in Python 3? I think I'm leaning towards 'U', >> which will mean unavoidable code breakage -- there's probably no >> avoiding it. > > Mmh, you are right. ?Yes, this seems to be difficult to solve. ?Well, I'm > changing my mind and think that both 'str' and 'S' should stand for Unicode in > NumPy for Python 3. ?If people is aware of the change for Python 3, they > should be expecting the same change happening in NumPy too, I guess. ?Then, I > suppose that a new dtype "bytes" that replaces the existing "string" would be > absolutely necessary. > >> > Also, I suppose that there will be issues with the current Unicode >> > support in NumPy: >> > >> > In [5]: u = np.array(['asa'], dtype="U10") >> > >> > In [6]: u[0] >> > Out[6]: u'asa' ?# will become 'asa' in Python 3 >> > >> > In [7]: u.dtype.itemsize >> > Out[7]: 40 ? ? ?# not sure about the size in Python 3 >> >> I suspect the Unicode stuff will keep working without major changes, >> except maybe dropping the u in repr. It is difficult to believe the >> CPython guys would have significantly changed the current Unicode >> implementation, if they didn't bother changing the names of the >> functions :) >> >> > For example, if it is true that internal strings in Python 3 and Unicode >> > UTF-8 (as Ren? seems to suggest), I suppose that the internal conversions >> > from 2- bytes or 4-bytes (depending on how the Python interpreter has >> > been compiled) in NumPy Unicode dtype to the new Python string should >> > have to be reworked (perhaps you have dealt with that already). >> >> I don't think they are internally UTF-8: >> http://docs.python.org/3.1/c-api/unicode.html >> >> """Python?s default builds use a 16-bit type for Py_UNICODE and store >> Unicode values internally as UCS2.""" > > Ah! ?No changes for that matter. ?Much better then. > Hello, in py3... >>> 'Hello\u0020World !'.encode() b'Hello World !' >>> "?pfel".encode('utf-8') b'\xc3\x84pfel' >>> "?pfel".encode() b'\xc3\x84pfel' The default encoding does appear to be utf-8 in py3. Although it is compiled with something different, and stores it as something different, that is UCS2 or UCS4. I imagine dtype 'S' and 'U' need more clarification. As it misses the concept of encodings it seems? Currently, S appears to mean 8bit characters no encoding, and U appears to mean 16bit characters no encoding? Or are some sort of default encodings assumed? 2to3/3to2 fixers will probably have to be written for users code here... whatever is decided. At least warnings should be generated I'm guessing. btw, in my numpy tree there is a unicode_() alias to str in py3, and to unicode in py2 (inside the compat.py file). This helped us in many cases with compatible string code in the pygame port. This allows you to create unicode strings on both platforms with the same code. cheers, From pav at iki.fi Fri Nov 27 07:41:54 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 14:41:54 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> References: <1259276898.8494.18.camel@idol> <200911271117.16002.faltet@pytables.org> <1259317620.4110.813.camel@talisman> <200911271150.03234.faltet@pytables.org> <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> Message-ID: <1259325714.4110.822.camel@talisman> pe, 2009-11-27 kello 13:23 +0100, Ren? Dudfield kirjoitti: [clip] > I imagine dtype 'S' and 'U' need more clarification. As it misses the > concept of encodings it seems? Currently, S appears to mean 8bit > characters no encoding, and U appears to mean 16bit characters no > encoding? Or are some sort of default encodings assumed? Currently in Numpy in Python 2, 'S' is the same as Python 3 bytes, 'U' is same as Python 3 unicode and probably in same internal representation (need to check). Neither is associated with encoding info. We need probably to change the meaning of 'S', as Francesc noted, and add a separate bytes dtype. > 2to3/3to2 fixers will probably have to be written for users code > here... whatever is decided. At least warnings should be generated > I'm guessing. Possibly. Does 2to3 support plugins? If yes, it could be possible to write one. > btw, in my numpy tree there is a unicode_() alias to str in py3, and > to unicode in py2 (inside the compat.py file). This helped us in many > cases with compatible string code in the pygame port. This allows you > to create unicode strings on both platforms with the same code. Yes, I saw that. The name unicode_ is however already taken by the Numpy scalar type, so we need to think of a different name for it. asstring, maybe. Btw, do you want to rebase your distutils changes on top of my tree? I tried yours out quickly, but there were some issues there that prevented distutils from working. (Also, you can use absolute imports both for Python 2 and 3 -- there's probably no need to use relative imports.) Pauli From faltet at pytables.org Fri Nov 27 07:49:21 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Nov 2009 13:49:21 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> References: <1259276898.8494.18.camel@idol> <200911271150.03234.faltet@pytables.org> <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> Message-ID: <200911271349.21492.faltet@pytables.org> A Friday 27 November 2009 13:23:10 Ren? Dudfield escrigu?: > >> I don't think they are internally UTF-8: > >> http://docs.python.org/3.1/c-api/unicode.html > >> > >> """Python?s default builds use a 16-bit type for Py_UNICODE and store > >> Unicode values internally as UCS2.""" > > > > Ah! No changes for that matter. Much better then. > > Hello, > > > in py3... > > >>> 'Hello\u0020World !'.encode() > > b'Hello World !' > > >>> "?pfel".encode('utf-8') > > b'\xc3\x84pfel' > > >>> "?pfel".encode() > > b'\xc3\x84pfel' > > The default encoding does appear to be utf-8 in py3. > > Although it is compiled with something different, and stores it as > something different, that is UCS2 or UCS4. OK. One thing is which is the default encoding for Unicode and another is how Python keeps Unicode internally. And internally Python 3 is still using UCS2 or UCS4, i.e. the same thing than in Python 2, so no worries here. > I imagine dtype 'S' and 'U' need more clarification. As it misses the > concept of encodings it seems? Currently, S appears to mean 8bit > characters no encoding, and U appears to mean 16bit characters no > encoding? Or are some sort of default encodings assumed? [clip] You only need encoding if you are going to represent Unicode strings with other types (for example bytes). Currently, NumPy can transparently import/export native Python Unicode strings (UCS2 or UCS4) into its own Unicode type (always UCS4). So, we don't have to worry here either. > btw, in my numpy tree there is a unicode_() alias to str in py3, and > to unicode in py2 (inside the compat.py file). This helped us in many > cases with compatible string code in the pygame port. This allows you > to create unicode strings on both platforms with the same code. Correct. But, in addition, we are going to need a new 'bytes' dtype for NumPy for Python 3, right? -- Francesc Alted From charlesr.harris at gmail.com Fri Nov 27 08:29:49 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 27 Nov 2009 06:29:49 -0700 Subject: [Numpy-discussion] New behavior for correlate w.r.t. swapped input: deprecate current default for 1.4.0 ? In-Reply-To: <4B0FA427.3010607@ar.media.kyoto-u.ac.jp> References: <4B0FA427.3010607@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Nov 27, 2009 at 3:04 AM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Hi, > > The function correlate can now implement what is generally accepted > as the definition of correlation, but you need to request this behavior > ATM (by default, it still does the wrong thing). Is it ok to deprecate > the default ? I.e. the behavior is exactly the same, but we warn users > about the change, and in 1.5.0, the default is to use the new behavior. > > What exactly are the differences? IIRC, there is a change for complex arguements and what looks like a bugfix for real arguments. Is there a way to warn *only* when the behavior will be different? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sierra_mtnview at sbcglobal.net Fri Nov 27 08:43:51 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 05:43:51 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> Message-ID: <4B0FD797.70506@sbcglobal.net> Thanks. That sounds like it should help a lot. Finding meaningful examples anywhere hasn't been easy. I thought I'd look through Amazon for books on Python and scientific uses. I found almost all were written by authors outside the US, and none seemed to talk about items like matplotlib. Ezdraw or something like that was often cited. I'm definitely in a learning stage, and much of what I need is in graphics to support some data analysis that I'm doing. Glad to hear it can gather bins into groups. It would be very disappointing if such a mechanism did not exist. In the distant past, I've all too often had to write my own histogram programs for this, FORTRAN, etc. My data is from a 640x480 collection of b/w pixels, which a processor has binned from 0-255, so I don't want repeat doing a histogram on 307K data points. Vincent Schut wrote: > Wayne Watson wrote: > >> I have a list that already has the frequencies from 0 to 255. However, >> I'd like to make a histogram that has say 32 bins whose ranges are 0-7, >> 8-15, ... 248-255. Is it possible? >> >> > Wayne, > > you might find the 'numpy example list with doc' webpage quite > informative... http://www.scipy.org/Numpy_Example_List_With_Doc (give it > some time to load, it's pretty large...) > For new users (I was one once...) it usually takes some time to find the > usual suspects in numpy/scipy help and docs... This one page has really > become unvaluable for me. > > It gives you the docstrings for numpy functions, often including some > example code. > > If you check out the histogram() function, you'll see it takes a 'bins=' > argument: > > bins : int or sequence of scalars, optional > If `bins` is an int, it defines the number of equal-width > bins in the given range (10, by default). If `bins` is a sequence, > it defines the bin edges, including the rightmost edge, allowing > for non-uniform bin widths. > > So, if your bins are known, you can pass it to numpy.histogram, either > as number-of-bins (if equal width), if necessary combined with the > 'range=' parameter to specify the range to divide into equal bins, or as > bin edges (e.g. in your case: (0, 8, 16, ... 256) or > numpy.linspace(0,256,33) which will give you this range nicely. > > If you don't specify the 'range=' parameter, it will check the min and > max from your input data and use that as lower and upper bounds. > > Good luck learning numpy! :) > > Vincent. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From eg at fft.be Fri Nov 27 09:01:16 2009 From: eg at fft.be (Eloi Gaudry) Date: Fri, 27 Nov 2009 15:01:16 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? References: fp4buv$cin$1@ger.gmane.org Message-ID: <4B0FDBAC.3030108@fft.be> David, I know this discussion first took place months ago, but I'd like to know whether or not you find a solution to the SxS assemblies issues using MSVC9.0. In case you haven't (the binaries package for windows is built using mingw), I'd like to know if this would be possible to relocate all numpy *.pyd from their original location (i.e. site-packages/numpy/core/, etc.) to the main executable one (i.e. python.exe). AFAIK, this would eventually solve the redistributing issue we are facing with python extensions built with SxS policy. Indeed, both Microsoft.VC90.CRT.manifest and msvcr90.dll files are located next to the python.exe executable in the standard distribution (http://www.python.org/ftp/python/2.6.4/python-2.6.4.msi). This way, all numpy *.pyd extension would be able to use these files (considering that python and numpy are built using the same revision of the crt library). IIRC, the SxS look-up sequence starts with the dirname of the executable... (i.e. ./python.exe/.) Regards, Eloi -- Eloi Gaudry Free Field Technologies Axis Park Louvain-la-Neuve Rue Emile Francqui, 1 B-1435 Mont-Saint Guibert BELGIUM Company Phone: +32 10 487 959 Company Fax: +32 10 454 626 From renesd at gmail.com Fri Nov 27 09:07:00 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 15:07:00 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259325714.4110.822.camel@talisman> References: <1259276898.8494.18.camel@idol> <200911271117.16002.faltet@pytables.org> <1259317620.4110.813.camel@talisman> <200911271150.03234.faltet@pytables.org> <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> <1259325714.4110.822.camel@talisman> Message-ID: <64ddb72c0911270607o7479bcfci9e8e5fe7bdd8c0b0@mail.gmail.com> On Fri, Nov 27, 2009 at 1:41 PM, Pauli Virtanen wrote: >> 2to3/3to2 fixers will probably have to be written for users code >> here... whatever is decided. ?At least warnings should be generated >> I'm guessing. > > Possibly. Does 2to3 support plugins? If yes, it could be possible to > write one. You can put them in here: [lib_dir]lib2to3/fixes/fix_*.py I'm not sure about how to use custom ones without just copying them in... need to research that. There's no documentation about how to write custom ones here: http://docs.python.org/library/2to3.html You can pass lib2to3 a package to try import fixers from. However I'm not sure how to make that appear from the command line, other than copying the fixer into place. I guess the numpy setup script could copy the fixer into place. >> btw, in my numpy tree there is a unicode_() alias to str in py3, and >> to unicode in py2 (inside the compat.py file). ?This helped us in many >> cases with compatible string code in the pygame port. ?This allows you >> to create unicode strings on both platforms with the same code. > > Yes, I saw that. The name unicode_ is however already taken by the Numpy > scalar type, so we need to think of a different name for it. asstring, > maybe. something like numpy.compat.unicode_ ? > > Btw, do you want to rebase your distutils changes on top of my tree? I > tried yours out quickly, but there were some issues there that prevented > distutils from working. (Also, you can use absolute imports both for > Python 2 and 3 -- there's probably no need to use relative imports.) > > ? ? ? ?Pauli > hey, yeah I definitely would :) I don't have much time for the next week or so though. cu, From renesd at gmail.com Fri Nov 27 09:09:00 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 15:09:00 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <200911271349.21492.faltet@pytables.org> References: <1259276898.8494.18.camel@idol> <200911271150.03234.faltet@pytables.org> <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> <200911271349.21492.faltet@pytables.org> Message-ID: <64ddb72c0911270609y5c17555dk95f17e12a0140faa@mail.gmail.com> On Fri, Nov 27, 2009 at 1:49 PM, Francesc Alted wrote: > Correct. ?But, in addition, we are going to need a new 'bytes' dtype for NumPy > for Python 3, right? I think so. However, I think S is probably closest to bytes... and maybe S can be reused for bytes... I'm not sure though. Also, what will a bytes dtype mean within a py2 program context? Does it matter if the bytes dtype just fails somehow if used in a py2 program? cheers, From renesd at gmail.com Fri Nov 27 09:10:14 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 27 Nov 2009 15:10:14 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <64ddb72c0911270607o7479bcfci9e8e5fe7bdd8c0b0@mail.gmail.com> References: <1259276898.8494.18.camel@idol> <200911271117.16002.faltet@pytables.org> <1259317620.4110.813.camel@talisman> <200911271150.03234.faltet@pytables.org> <64ddb72c0911270423g1b4ecc5fy10048a5c1b321b1d@mail.gmail.com> <1259325714.4110.822.camel@talisman> <64ddb72c0911270607o7479bcfci9e8e5fe7bdd8c0b0@mail.gmail.com> Message-ID: <64ddb72c0911270610r3525c7bdp95eea7beba970e4c@mail.gmail.com> On Fri, Nov 27, 2009 at 3:07 PM, Ren? Dudfield wrote: > > hey, > > yeah I definitely would :) ? I don't have much time for the next week > or so though. > btw, feel free to just copy whatever you like from there into your tree. cheers, From josef.pktd at gmail.com Fri Nov 27 09:33:32 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 09:33:32 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B0FD797.70506@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> Message-ID: <1cd32cbb0911270633w1ea48cdcmb87e1d8f65399aba@mail.gmail.com> On Fri, Nov 27, 2009 at 8:43 AM, Wayne Watson wrote: > Thanks. That sounds like it should help a lot. Finding meaningful > examples anywhere hasn't been easy. I thought I'd look through Amazon > for books on Python and scientific uses. I found almost all were written > by authors outside the US, and none seemed to talk about items like > matplotlib. Ezdraw or something like that was often cited. I'm > definitely in a learning stage, and much of what I need is in graphics > to support some data analysis that I'm doing. > > Glad to hear it can gather bins into groups. It would be very > disappointing if such a mechanism did not ?exist. In the distant past, > I've all too often had to write my own histogram programs for this, > FORTRAN, etc. ?My data is from a 640x480 collection of b/w pixels, which > a processor has binned from 0-255, so I don't want repeat doing a > histogram on 307K data points. > > Vincent Schut wrote: >> Wayne Watson wrote: >> >>> I have a list that already has the frequencies from 0 to 255. However, >>> I'd like to make a histogram ?that has say 32 bins whose ranges are 0-7, >>> 8-15, ... 248-255. Is it possible? >>> >>> >> Wayne, >> >> you might find the 'numpy example list with doc' webpage quite >> informative... http://www.scipy.org/Numpy_Example_List_With_Doc (give it >> some time to load, it's pretty large...) >> For new users (I was one once...) it usually takes some time to find the >> usual suspects in numpy/scipy help and docs... This one page has really >> become unvaluable for me. >> >> It gives you the docstrings for numpy functions, often including some >> example code. Numpy_Example_List_With_Doc is for an older version of numpy and hasn't been kept up to date. So if your results don't match up, then the function might have changed and the official docs will have the current description. from numpy import * is not recommended anymore, it messes up the global namespace too much Besides the the example list, I found http://scipy.org/Numpy_Functions_by_Category very helpful, because it gave a better overview of which functions do similar things. In the current docs, "See Also" can be used now in a similar way. Good luck, Josef >> >> If you check out the histogram() function, you'll see it takes a 'bins=' >> argument: >> >> bins : int or sequence of scalars, optional >> ? ? ?If `bins` is an int, it defines the number of equal-width >> ? ? ?bins in the given range (10, by default). If `bins` is a sequence, >> ? ? ?it defines the bin edges, including the rightmost edge, allowing >> ? ? ?for non-uniform bin widths. >> >> So, if your bins are known, you can pass it to numpy.histogram, either >> as number-of-bins (if equal width), if necessary combined with the >> 'range=' parameter to specify the range to divide into equal bins, or as >> bin edges (e.g. in your case: (0, 8, 16, ... 256) or >> numpy.linspace(0,256,33) which will give you this range nicely. >> >> If you don't specify the 'range=' parameter, it will check the min and >> max from your input data and use that as lower and upper bounds. >> >> Good luck learning numpy! :) >> >> Vincent. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From faltet at pytables.org Fri Nov 27 10:33:47 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Nov 2009 16:33:47 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <64ddb72c0911270609y5c17555dk95f17e12a0140faa@mail.gmail.com> References: <1259276898.8494.18.camel@idol> <200911271349.21492.faltet@pytables.org> <64ddb72c0911270609y5c17555dk95f17e12a0140faa@mail.gmail.com> Message-ID: <200911271633.47281.faltet@pytables.org> A Friday 27 November 2009 15:09:00 Ren? Dudfield escrigu?: > On Fri, Nov 27, 2009 at 1:49 PM, Francesc Alted wrote: > > Correct. But, in addition, we are going to need a new 'bytes' dtype for > > NumPy for Python 3, right? > > I think so. However, I think S is probably closest to bytes... and > maybe S can be reused for bytes... I'm not sure though. That could be a good idea because that would ensure compatibility with existing NumPy scripts (i.e. old 'string' dtypes are mapped to 'bytes', as it should). The only thing that I don't like is that that 'S' seems to be the initial letter for 'string', which is actually 'unicode' in Python 3 :-/ But, for the sake of compatibility, we can probably live with that. > Also, what will a bytes dtype mean within a py2 program context? Does > it matter if the bytes dtype just fails somehow if used in a py2 > program? Mmh, I'm of the opinion that the new 'bytes' type should be available only with NumPy for Python 3. Would that be possible? -- Francesc Alted From pav at iki.fi Fri Nov 27 10:41:04 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 17:41:04 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <200911271633.47281.faltet@pytables.org> References: <1259276898.8494.18.camel@idol> <200911271349.21492.faltet@pytables.org> <64ddb72c0911270609y5c17555dk95f17e12a0140faa@mail.gmail.com> <200911271633.47281.faltet@pytables.org> Message-ID: <1259336464.4110.825.camel@talisman> pe, 2009-11-27 kello 16:33 +0100, Francesc Alted kirjoitti: > A Friday 27 November 2009 15:09:00 Ren? Dudfield escrigu?: > > On Fri, Nov 27, 2009 at 1:49 PM, Francesc Alted wrote: > > > Correct. But, in addition, we are going to need a new 'bytes' dtype for > > > NumPy for Python 3, right? > > > > I think so. However, I think S is probably closest to bytes... and > > maybe S can be reused for bytes... I'm not sure though. > > That could be a good idea because that would ensure compatibility with > existing NumPy scripts (i.e. old 'string' dtypes are mapped to 'bytes', as it > should). The only thing that I don't like is that that 'S' seems to be the > initial letter for 'string', which is actually 'unicode' in Python 3 :-/ > But, for the sake of compatibility, we can probably live with that. Well, we can "deprecate" 'S' (ie. never show it in repr, always only 'B' or 'U'). > > Also, what will a bytes dtype mean within a py2 program context? Does > > it matter if the bytes dtype just fails somehow if used in a py2 > > program? > > Mmh, I'm of the opinion that the new 'bytes' type should be available only > with NumPy for Python 3. Would that be possible? I don't see a problem in making a bytes_ scalar type available for Python2. In fact, it would be useful for making upgrading to Py3 easier. -- Pauli Virtanen From faltet at pytables.org Fri Nov 27 11:04:57 2009 From: faltet at pytables.org (Francesc Alted) Date: Fri, 27 Nov 2009 17:04:57 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259336464.4110.825.camel@talisman> References: <1259276898.8494.18.camel@idol> <200911271633.47281.faltet@pytables.org> <1259336464.4110.825.camel@talisman> Message-ID: <200911271704.57927.faltet@pytables.org> A Friday 27 November 2009 16:41:04 Pauli Virtanen escrigu?: > > > I think so. However, I think S is probably closest to bytes... and > > > maybe S can be reused for bytes... I'm not sure though. > > > > That could be a good idea because that would ensure compatibility with > > existing NumPy scripts (i.e. old 'string' dtypes are mapped to 'bytes', > > as it should). The only thing that I don't like is that that 'S' seems > > to be the initial letter for 'string', which is actually 'unicode' in > > Python 3 :-/ But, for the sake of compatibility, we can probably live > > with that. > > Well, we can "deprecate" 'S' (ie. never show it in repr, always only 'B' > or 'U'). Well, deprecating 'S' seems a sensible option too. But why only avoiding showing it in repr? Why not issue a DeprecationWarning too? > > > Also, what will a bytes dtype mean within a py2 program context? Does > > > it matter if the bytes dtype just fails somehow if used in a py2 > > > program? > > > > Mmh, I'm of the opinion that the new 'bytes' type should be available > > only with NumPy for Python 3. Would that be possible? > > I don't see a problem in making a bytes_ scalar type available for > Python2. In fact, it would be useful for making upgrading to Py3 easier. I think introducing a bytes_ scalar dtype can be somewhat confusing for Python 2 users. But if the 'S' typecode is to be deprecated also for NumPy for Python 2, then it makes perfect sense to introduce bytes_ there too. -- Francesc Alted From sierra_mtnview at sbcglobal.net Fri Nov 27 12:14:54 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 09:14:54 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <1cd32cbb0911270633w1ea48cdcmb87e1d8f65399aba@mail.gmail.com> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <1cd32cbb0911270633w1ea48cdcmb87e1d8f65399aba@mail.gmail.com> Message-ID: <4B10090E.4000002@sbcglobal.net> It's good to have some extra references for NumPy. Actually, it looks like exercising histogram in NunPy has gotten me past the difficulties with hist in matplotlib. I Is there a matplotlib or Pylab mailing list. It uses hist and looks very much like histogram, but has some parameters that I need to understand better. . -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From cournape at gmail.com Fri Nov 27 12:21:34 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 28 Nov 2009 02:21:34 +0900 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B0FDBAC.3030108@fft.be> References: <4B0FDBAC.3030108@fft.be> Message-ID: <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> On Fri, Nov 27, 2009 at 11:01 PM, Eloi Gaudry wrote: > David, > > I know this discussion first took place months ago, but I'd like to know > whether or not you find a solution to the SxS assemblies issues using > MSVC9.0. Which issue exactly are you talking about ? When python is installed for one user (when installed for all users, the C runtime is installed in the SxS) ? > > In case you haven't (the binaries package for windows is built using > mingw), I'd like to know if this would be possible to relocate all numpy > *.pyd from their original location (i.e. site-packages/numpy/core/, > etc.) to the main executable one (i.e. python.exe). This sounds like a very bad idea: if packages start doing that, there will quickly be clashes between extensions. You would need to describe the exact scenario which is failing: how python is installed, how you build numpy, what is not working with which message, etc... David From jsseabold at gmail.com Fri Nov 27 12:26:39 2009 From: jsseabold at gmail.com (Skipper Seabold) Date: Fri, 27 Nov 2009 12:26:39 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B10090E.4000002@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <1cd32cbb0911270633w1ea48cdcmb87e1d8f65399aba@mail.gmail.com> <4B10090E.4000002@sbcglobal.net> Message-ID: On Fri, Nov 27, 2009 at 12:14 PM, Wayne Watson wrote: > It's good to have some extra references for NumPy. > > Actually, it looks like exercising histogram in NunPy has gotten me past > the difficulties with hist in matplotlib. I Is there a matplotlib or > Pylab mailing list. It uses hist and looks very much like histogram, but > has some parameters that I need to understand better. . > I don't know if this has come up yet in your questions, but matplotlib has a mailing list that can be found here: http://matplotlib.sourceforge.net/ There are also *numerous* examples that I have found indispensable in getting over the initial learning curve if you click on examples from their sourceforge docs. It is my (quite possibly incorrect) understanding that matplotlib.pylab has many of the same functions of numpy (and leverages numpy in most(?) instances when possible). I think that is why it is recommended that for numpy users who only wish to use the plotting functionality of matplotlib that you do from matplotlib import pyplot as plt # or whatever In any case, have a look through the examples in the matplotlib docs. There may also be more examples installed with matplotlib itself. I don't know if they're all in the online docs. Skipper From Chris.Barker at noaa.gov Fri Nov 27 12:41:50 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 Nov 2009 09:41:50 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B0FD797.70506@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> Message-ID: <4B100F5E.7080409@noaa.gov> Wayne Watson wrote: > Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and > SciPy. They all have seemed part of one another, but I think I see how > they've divided up the game. For the record: I know this is a bit confusing, particularly for someone used to an integrated package like Matlab, etc, but there is a lot of power an flexibility gained by the divisions: Python: is a general-purpose, extensible programming language Numpy: is a set of package of classes, functions, etc that provide facilities for numeric computation -- primarily a n-d array class and the utilities to use it. Matplotlib (MPL): is a plotting package, built on top of numpy -- it was originally designed to somewhat mimic the plotting interface of Matlab. MPL is the most commonly used plotting package for numpy, but by no means the only one. Pylab: Is a package that integrates matplotlib and numpy and an assortment of other utilities into one namespace, making it more like Matlab -- personally, I think you should avoid using it, it makes it a bit easier to type code, but harder to know where the heck what you are doing is coming from. SciPy: Is a broad collection of assorted utilities that facilitate scientific computing, built on Numpy -- it is also sometimes used as an umbrella term for anything connected to scientific computing with Python (i.e. the SciPy conferences) These distinctions are a bit confusing (particularly MPL-numpy), because MPL includes a number of utility functions that combine computation and plotting: like "hist", which both computes a histogram, and plots it as bar chart in one call -- it's a convenient way to perform a common operation, but it does blur the lines a bit! By the way -- there is also potentially a bit of confusion as to how MPL uses/interacts with the command line and GUI toolkits. This is because MPL can be used with a number of different GUI front-ends (or none), and they tend to take over control from the command line. Which brings up to: iPython: an enhanced python interactive interpreter command line system. It adds many nice features that make using python in interactive mode nicer. IN particularly, it adds a "--pylab" mode that helps it play well with MPL. You won't regret using it! > I thought I'd look through Amazon > for books on Python and scientific uses. I found almost all were written > by authors outside the US, and none seemed to talk about items like > matplotlib. FWIW, a book about MPL has just been published -- I don't know any more about it, but I'm sure google will tell you. > Is there a matplotlib or Pylab mailing list? There certainly is: https://lists.sourceforge.net/lists/listinfo/matplotlib-users And yes, that is the place for such questions. HTH, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jsseabold at gmail.com Fri Nov 27 12:57:45 2009 From: jsseabold at gmail.com (Skipper Seabold) Date: Fri, 27 Nov 2009 12:57:45 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B100F5E.7080409@noaa.gov> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> Message-ID: On Fri, Nov 27, 2009 at 12:41 PM, Christopher Barker wrote: > Wayne Watson wrote: >> Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and >> SciPy. They all have seemed part of one another, but I think I see how >> they've divided up the game. > > For the record: > > I know this is a bit confusing, particularly for someone used to an > integrated package like Matlab, etc, but there is a lot of power an > flexibility gained by the divisions: > > Python: is a general-purpose, extensible programming language > > Numpy: is a set of package of classes, functions, etc that provide > facilities for numeric computation -- primarily a n-d array class and > the utilities to use it. > > Matplotlib (MPL): is a plotting package, built on top of numpy -- it was > originally designed to somewhat mimic the plotting interface of Matlab. > MPL is the most commonly used plotting package for numpy, but by no > means the only one. > > Pylab: Is a package that integrates matplotlib and numpy and an > assortment of other utilities into one namespace, making it more like > Matlab -- personally, I think you should avoid using it, it makes it a > bit easier to type code, but harder to know where the heck what you are > doing is coming from. > > SciPy: Is a broad collection of assorted utilities that facilitate > scientific computing, built on Numpy -- it is also sometimes used as an > umbrella term for anything connected to scientific computing with Python > (i.e. the SciPy conferences) > > > These distinctions are a bit confusing (particularly MPL-numpy), because > MPL includes a number of utility functions that combine computation and > plotting: like "hist", which both computes a histogram, and plots it as > bar chart in one call -- it's a convenient way to perform a common > operation, but it does blur the lines a bit! > > By the way -- there is also potentially a bit of confusion as to how MPL > uses/interacts with the command line and GUI toolkits. This is because > MPL can be used with a number of different GUI front-ends (or none), and > they tend to take over control from the command line. Which brings up to: > > iPython: an enhanced python interactive interpreter command line system. > It adds many nice features that make using python in interactive mode > nicer. IN particularly, it adds a "--pylab" mode that helps it play well > with MPL. You won't regret using it! > > >> I thought I'd look through Amazon >> for books on Python and scientific uses. I found almost all were written >> by authors outside the US, and none seemed to talk about items like >> matplotlib. > > FWIW, a book about MPL has just been published -- I don't know any more > about it, but I'm sure google will tell you. > >> Is there a matplotlib or Pylab mailing list? > > There certainly is: > > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > And yes, that is the place for such questions. > > > HTH, > > -Chris > Well put, Chris. It took me a long time get my head around these distinctions, and then only when others pointed out my errors in understanding. This kind of info might be useful to other newcomers somewhere... ? Thoughts on posting this on the wiki here? Skipper From josef.pktd at gmail.com Fri Nov 27 13:15:50 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 13:15:50 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> Message-ID: <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold wrote: > On Fri, Nov 27, 2009 at 12:41 PM, Christopher Barker > wrote: >> Wayne Watson wrote: >>> Yes, I'm just beginning to deal with the contents of NumPy, SciLab, and >>> SciPy. They all have seemed part of one another, but I think I see how >>> they've divided up the game. >> >> For the record: >> >> I know this is a bit confusing, particularly for someone used to an >> integrated package like Matlab, etc, but there is a lot of power an >> flexibility gained by the divisions: >> >> Python: is a general-purpose, extensible programming language >> >> Numpy: is a set of package of classes, functions, etc that provide >> facilities for numeric computation -- primarily a n-d array class and >> the utilities to use it. >> >> Matplotlib (MPL): is a plotting package, built on top of numpy -- it was >> originally designed to somewhat mimic the plotting interface of Matlab. >> MPL is the most commonly used plotting package for numpy, but by no >> means the only one. >> >> Pylab: Is a package that integrates matplotlib and numpy and an >> assortment of other utilities into one namespace, making it more like >> Matlab -- personally, I think you should avoid using it, it makes it a >> bit easier to type code, but harder to know where the heck what you are >> doing is coming from. >> >> SciPy: Is a broad collection of assorted utilities that facilitate >> scientific computing, built on Numpy -- it is also sometimes used as an >> umbrella term for anything connected to scientific computing with Python >> (i.e. the SciPy conferences) >> >> >> These distinctions are a bit confusing (particularly MPL-numpy), because >> MPL includes a number of utility functions that combine computation and >> plotting: like "hist", which both computes a histogram, and plots it as >> bar chart in one call -- it's a convenient way to perform a common >> operation, but it does blur the lines a bit! >> >> By the way -- there is also potentially a bit of confusion as to how MPL >> uses/interacts with the command line and GUI toolkits. This is because >> MPL can be used with a number of different GUI front-ends (or none), and >> they tend to take over control from the command line. Which brings up to: >> >> iPython: an enhanced python interactive interpreter command line system. >> It adds many nice features that make using python in interactive mode >> nicer. IN particularly, it adds a "--pylab" mode that helps it play well >> with MPL. You won't regret using it! >> >> >>> I thought I'd look through Amazon >>> for books on Python and scientific uses. I found almost all were written >>> by authors outside the US, and none seemed to talk about items like >>> matplotlib. >> >> FWIW, a book about MPL has just been published -- I don't know any more >> about it, but I'm sure google will tell you. >> >>> Is there a matplotlib or Pylab mailing list? >> >> There certainly is: >> >> https://lists.sourceforge.net/lists/listinfo/matplotlib-users >> >> And yes, that is the place for such questions. >> >> >> HTH, >> >> -Chris >> > > Well put, Chris. ?It took me a long time get my head around these > distinctions, and then only when others pointed out my errors in > understanding. ?This kind of info might be useful to other newcomers > somewhere... ?? ?Thoughts on > posting this on the wiki here? I also agree. It will improve with the newly redesigned website for scipy.org However, I cannot find the link right now for the development version of the new website. Josef > > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Fri Nov 27 13:30:53 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 27 Nov 2009 19:30:53 +0100 Subject: [Numpy-discussion] matrix inverse In-Reply-To: <1cd32cbb0911260855o6de80cabte37f0365103e0083@mail.gmail.com> References: <1cd32cbb0911260834l4e12c86w9ec4fca264dab3ec@mail.gmail.com> <1cd32cbb0911260849p45c327d6g674c243876a3ea7e@mail.gmail.com> <1cd32cbb0911260855o6de80cabte37f0365103e0083@mail.gmail.com> Message-ID: On Thu, Nov 26, 2009 at 5:55 PM, wrote: > On Thu, Nov 26, 2009 at 11:49 AM, wrote: > > On Thu, Nov 26, 2009 at 11:34 AM, wrote: > >> why is > http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.I/ > >> classified as unimportant? .T ? > >> > >> I was looking at http://projects.scipy.org/numpy/ticket/1093 and > >> didn't find any docs for the matrix inverse. > > > > Ok looking some more, I found > > http://docs.scipy.org/numpy/docs/numpy.matrixlib.defmatrix.matrix.getI/ > > > > I thought it might be problem with generating docs for attributes, but > > the docstring for .I seems editable. > > > > Aren't A.I and A.T the more common use than A.getI, ..? > > in defmatrix.py: > > T = property(getT, None, doc="transpose") > A = property(getA, None, doc="base array") > A1 = property(getA1, None, doc="1-d base array") > H = property(getH, None, doc="hermitian (conjugate) transpose") > I = property(getI, None, doc="inverse") > > could take the docstrings from the get? function instead ? > That seems like a good idea. All that is needed is to delete the doc arguments for those five properties. Because those docstrings were marked unimportant, I think using the get? docs was intended anyway. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Fri Nov 27 13:34:44 2009 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 27 Nov 2009 19:34:44 +0100 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> Message-ID: On Fri, Nov 27, 2009 at 7:15 PM, wrote: > > Well put, Chris. It took me a long time get my head around these > > distinctions, and then only when others pointed out my errors in > > understanding. This kind of info might be useful to other newcomers > > somewhere... ? Thoughts on > > posting this on the wiki here? > +1 > > I also agree. It will improve with the newly redesigned website for > scipy.org > However, I cannot find the link right now for the development version of > the new website. > http://new.scipy.org/ Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Nov 27 13:36:36 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 Nov 2009 10:36:36 -0800 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <1259315273.4110.804.camel@talisman> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> Message-ID: <4B101C34.8030601@noaa.gov> > The point is that I don't think we can just decide to use Unicode or > Bytes in all places where PyString was used earlier. Agreed. I think it's helpful to remember the origins of all this: IMHO, there are two distinct types of data that Python2 strings support: 1) text: this is the traditional "string". 2) bytes: raw bytes -- they could represent anything. This, of course, is what the py3k string and bytes types are all about. However, when python started, it just so happened that text was represented by an array of unsigned single byte integers, so there really was no point in having a "bytes" type, as a string would work just as well. Enter unicode: Now we have multiple ways of representing text internally, but want a single interface to that -- one that looks and acts like a sequence of characters to user's code. The result is that the unicode type was introduced. In a way, unicode strings are a bit like arrays: they have an encoding associated with them (like a dtype in numpy). You can represent a given bit of text in multiple different arangements of bytes, but they are all supposed to mean the same thing and, if you know the encoding, you can convert between them. This is kind of like how one can represent 5 in any of many dtypes: uint8, int16, int32, float32, float64, etc. Not any value represented by one dtype can be converted to all other dtypes, but many can. Just like encodings. Anyway, all this brings me to think about the use of strings in numpy in this way: if it is meant to be a human-readable piece of text, it should be a unicode object. If not, then it is bytes. So: "fromstring" and the like should, of course, work with bytes (though maybe buffers really...) > Which one it will > be should depend on the use. Users will expect that eg. array([1,2,3], > dtype='f4') still works, and they don't have to do e.g. array([1,2,3], > dtype=b'f4'). Personally, I try to use np.float32 instead, anyway, but I digress. In this case, the "type code" is supposed to be a human-readable bit of text -- it should be a unicode object (convertible to ascii for interfacing with C...) If we used b'f4', it would confuse things, as it couldn't be printed right. Also: would the actual bytes involved potentially change depending on what encoding was used for the literal? i.e. if the code was written in utf16, would that byte string be 4 bytes long? > To summarize the use cases I've ran across so far: > > 1) For 'S' dtype, I believe we use Bytes for the raw data and the > interface. I don't think so here. 'S' is usually used to store human-readable strings, I'd certainly expect to be able to do: s_array = np.array(['this', 'that'], dtype='S10') And I'd expect it to work with non-literals that were unicode strings, i.e. human readable text. In fact, it's pretty rare that I'd ever want bytes here. So I'd see 'S' mapped to 'U' here. Francesc Alted wrote: > the next should still work: > > In [2]: s = np.array(['asa'], dtype="S10") > > In [3]: s[0] > Out[3]: 'asa' # will become b'asa' in Python 3 I don't like that -- I put in a string, and get a bytes object back? > In [4]: s.dtype.itemsize > Out[4]: 10 # still 1-byte per element But what it the the strings passed in aren't representable in one byte per character? Do we define "S" as only supporting ANSI-only string? what encoding? Pauli Virtanen wrote: > 'U' > is same as Python 3 unicode and probably in same internal representation > (need to check). Neither is associated with encoding info. Isn't it? I thought the encoding was always the same internally? so it is known? Francesc Alted wrote: > That could be a good idea because that would ensure compatibility with > existing NumPy scripts (i.e. old 'string' dtypes are mapped to 'bytes', as it > should). What do you mean by compatible? It wold mean a lot of user code would have to change with the 2->3 transition. > The only thing that I don't like is that that 'S' seems to be the > initial letter for 'string', which is actually 'unicode' in Python 3 :-/ > But, for the sake of compatibility, we can probably live with that. I suppose we could at least depricate it. >> Also, what will a bytes dtype mean within a py2 program context? Does >> it matter if the bytes dtype just fails somehow if used in a py2 >> program? well, it should work in 2.6 anyway. > Maybe we want to introduce a separate "bytes" dtype that's an alias > for 'S'? What do we need "bytes" for? does it support anything that np.uint8 doesn't? > 2) The field names: > > a = array([], dtype=[('a', int)]) > a = array([], dtype=[(b'a', int)]) > > This is somewhat of an internal issue. We need to decide whether we > internally coerce input to Unicode or Bytes. Unicode is clear to me here -- it really should match what Python does for variable names -- that is unicode in py3k, no? > 3) Format strings > > a = array([], dtype=b'i4') > > I don't think it makes sense to handle format strings in Unicode > internally -- they should always be coerced to bytes. This should be fine -- we control what is a valid format string, and thus they can always be ASCII-safe. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Fri Nov 27 13:38:07 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 Nov 2009 10:38:07 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> Message-ID: <4B101C8F.2020808@noaa.gov> josef.pktd at gmail.com wrote: > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold wrote: >> This kind of info might be useful to other newcomers >> somewhere... ? Thoughts on >> posting this on the wiki here? > > I also agree. It will improve with the newly redesigned website for scipy.org > However, I cannot find the link right now for the development version of > the new website. Feel free to crib whatever you want from my post for that -- or suggest a place for me to put it, and I'll do it. I'm just not sure where it should go at this point. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From sierra_mtnview at sbcglobal.net Fri Nov 27 14:21:18 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 11:21:18 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B101C8F.2020808@noaa.gov> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> Message-ID: <4B1026AE.60002@sbcglobal.net> Lots of good suggestions. I'll pull them into a document for further reference. -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From sebas0 at gmail.com Fri Nov 27 14:29:51 2009 From: sebas0 at gmail.com (Sebastian) Date: Fri, 27 Nov 2009 17:29:51 -0200 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B101C8F.2020808@noaa.gov> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> Message-ID: Did you try using the parameter range? I do something like this. regards ax = fig.add_subplot(1,1,1) > pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > normed=1, facecolor='y', alpha=0.5) > ax.set_xlabel(r'\Large$ \rm{values}$') > ax.set_ylabel(r'\Large Delatavalue/Value') > > > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > pylab.plot(gausx,gaus, color='red', lw=2) > ax.set_xlim(-1.5, 1.5) > ax.grid(True) > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker wrote: > josef.pktd at gmail.com wrote: > > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold > wrote: > > >> This kind of info might be useful to other newcomers > >> somewhere... ? Thoughts on > >> posting this on the wiki here? > > > > I also agree. It will improve with the newly redesigned website for > scipy.org > > However, I cannot find the link right now for the development version of > > the new website. > > Feel free to crib whatever you want from my post for that -- or suggest > a place for me to put it, and I'll do it. I'm just not sure where it > should go at this point. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Nov 27 15:08:41 2009 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Nov 2009 22:08:41 +0200 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <4B101C34.8030601@noaa.gov> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> <4B101C34.8030601@noaa.gov> Message-ID: <1259352521.4268.39.camel@idol> pe, 2009-11-27 kello 10:36 -0800, Christopher Barker kirjoitti: [clip] > > Which one it will > > be should depend on the use. Users will expect that eg. array([1,2,3], > > dtype='f4') still works, and they don't have to do e.g. array([1,2,3], > > dtype=b'f4'). > > Personally, I try to use np.float32 instead, anyway, but I digress. In > this case, the "type code" is supposed to be a human-readable bit of > text -- it should be a unicode object (convertible to ascii for > interfacing with C...) Yes, this would solve the repr() issue easily. Now that I look more closely, the format strings are not actually used anywhere else than in the descriptor user interface, so from an implementation POV Unicode is not any harder. [clip] > Pauli Virtanen wrote: > > 'U' > > is same as Python 3 unicode and probably in same internal representation > > (need to check). Neither is associated with encoding info. > > Isn't it? I thought the encoding was always the same internally? so it > is known? Yes, so it needs not be associated with a separate piece of encoding info. [clip] > > Maybe we want to introduce a separate "bytes" dtype that's an alias > > for 'S'? > > What do we need "bytes" for? does it support anything that np.uint8 > doesn't? It has a string representation, but that's probably it. Actually, in Python 3, when you index a bytes object, you get integers back, so we just aliasing bytes_ = uint8 and making sure array() handles byte objects appropriately would be more or less consistent. > > 2) The field names: > > > > a = array([], dtype=[('a', int)]) > > a = array([], dtype=[(b'a', int)]) > > > > This is somewhat of an internal issue. We need to decide whether we > > internally coerce input to Unicode or Bytes. > > Unicode is clear to me here -- it really should match what Python does > for variable names -- that is unicode in py3k, no? Yep, let's follow Python. So Unicode and only Unicode it is. *** Ok, thanks for the feedback. The right answers seem to be: 1) Unicode works as it is now, and Python3 strings are Unicode. Bytes objects are coerced to uint8 by array(). We don't do implicit conversions between Bytes and Unicode. The 'S' dtype character will be deprecated, never appear in repr(), and its usage will result to a warning. 2) Field names are unicode always. Some backward compatibility needs to be added in pickling, and maybe the npy file format needs a fixed encoding. 3) Dtype strings are an user interface detail, and will be Unicode. -- Pauli Virtanen From peridot.faceted at gmail.com Fri Nov 27 15:09:10 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 27 Nov 2009 15:09:10 -0500 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <4B101C34.8030601@noaa.gov> References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> <4B101C34.8030601@noaa.gov> Message-ID: 2009/11/27 Christopher Barker : > >> The point is that I don't think we can just decide to use Unicode or >> Bytes in all places where PyString was used earlier. > > Agreed. I only half agree. It seems to me that for almost all situations where PyString was used, the right data type is a python3 string (which is unicode). I realize there may be some few cases where it is appropriate to use bytes, but I think there needs to be a compelling reason for each one. > In a way, unicode strings are a bit like arrays: they have an encoding > associated with them (like a dtype in numpy). You can represent a given > bit of text in multiple different arangements of bytes, but they are all > supposed to mean the same thing and, if you know the encoding, you can > convert between them. This is kind of like how one can represent 5 in > any of many dtypes: uint8, int16, int32, float32, float64, etc. Not any > value represented by one dtype can be converted to all other dtypes, but > many can. Just like encodings. This is incorrect. Unicode objects do not have default encodings or multiple internal representations (within a single python interpreter, at least). Unicode objects use 2- or 4-byte internal representations internally, but this is almost invisible to the user. Encodings only become relevant when you want to convert a unicode object to a byte stream. It is usually an error to store text in a byte stream (for it to make sense you must provide some mechanism to specify the encoding). > Anyway, all this brings me to think about the use of strings in numpy in > this way: if it is meant to be a human-readable piece of text, it should > be a unicode object. If not, then it is bytes. > > So: "fromstring" and the like should, of course, work with bytes (though > maybe buffers really...) I think if you're going to call it fromstring, it should onvert from strings (i.e. unicode strings). But really, I think it makes more sense to rename it frombytes() and have it convert bytes objects. One could then have def fromstring(s, encoding="utf-8"): return frombytes(s.encode(encoding)) as a shortcut. Maybe ASCII makes more sense as a default encoding. But really, think about where the user's going to get the srting: most of the time it's coming from a disk file or a network stream, so it will be a byte string already, so they should use frombytes. >> To summarize the use cases I've ran across so far: >> >> 1) For 'S' dtype, I believe we use Bytes for the raw data and the >> ? ?interface. > > I don't think so here. 'S' is usually used to store human-readable > strings, I'd certainly expect to be able to do: > > s_array = np.array(['this', 'that'], dtype='S10') > > And I'd expect it to work with non-literals that were unicode strings, > i.e. human readable text. In fact, it's pretty rare that I'd ever want > bytes here. So I'd see 'S' mapped to 'U' here. +1 > Francesc Alted wrote: >> the next ?should still work: >> >> In [2]: s = np.array(['asa'], dtype="S10") >> >> In [3]: s[0] >> Out[3]: 'asa' ?# will become b'asa' in Python 3 > > I don't like that -- I put in a string, and get a bytes object back? I agree. >> In [4]: s.dtype.itemsize >> Out[4]: 10 ? ? # still 1-byte per element > > But what it the the strings passed in aren't representable in one byte > per character? Do we define "S" as only supporting ANSI-only string? > what encoding? Itemsize will change. That's fine. >> 3) Format strings >> >> ? ? ? a = array([], dtype=b'i4') >> >> I don't think it makes sense to handle format strings in Unicode >> internally -- they should always be coerced to bytes. > > This should be fine -- we control what is a valid format string, and > thus they can always be ASCII-safe. I have to disagree. Why should we force the user to use bytes? The format strings are just that, strings, and we should be able to supply python strings to them. Keep in mind that "coercing" strings to bytes requires extra information, namely the encoding. If you want to emulate python2's value-dependent coercion - raise an exception only if non-ASCII is present - keep in mind that python3 is specifically removing that behaviour because of the problems it caused. Anne From Chris.Barker at noaa.gov Fri Nov 27 15:52:43 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 Nov 2009 12:52:43 -0800 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: References: <1259276898.8494.18.camel@idol> <1259315273.4110.804.camel@talisman> <4B101C34.8030601@noaa.gov> Message-ID: <4B103C1B.3040408@noaa.gov> Anne Archibald wrote: >>> I don't think it makes sense to handle format strings in Unicode >>> internally -- they should always be coerced to bytes. >> This should be fine -- we control what is a valid format string, and >> thus they can always be ASCII-safe. > > I have to disagree. Why should we force the user to use bytes? One of us mis-understood that -- I THINK the idea was that internally numpy would use bytes (for easy conversion to/from char*), but they would get converted, so the use could pass in unicode strings (or bytes). I guess the questions remains as to what you'd get when you printed a format string. > Keep in mind that "coercing" strings to bytes > requires extra information, namely the encoding. but that is built-in to the unicode object. I think the idea is that a format string is ALWAYS ASCII -f there are any other characters in there, it's an invalid format anyway. Unless I mis-understand what a format string is. I think it's a string you use to represent a custom dtype -- it that right? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dwf at cs.toronto.edu Fri Nov 27 16:04:55 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 27 Nov 2009 16:04:55 -0500 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: Message-ID: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> On 26-Nov-09, at 8:56 PM, Charles R Harris wrote: > I never had luck with the netlib-lapack-tarfile= > option, it didn't build lapack correctly. Try doing them separately. I just tried that. -Ss flapack path/to/lapack_LINUX.a didn't seem to build a proper LAPACK (it was missing all sorts of stuff and was only about 800kb) and so In unarchived the liblapack.a that ATLAS spit out and added those files to the one build by Netlib's makefile. The result is still the same. I'm wondering if this is maybe a gfortran version issue, somehow. David From charlesr.harris at gmail.com Fri Nov 27 17:02:32 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 27 Nov 2009 15:02:32 -0700 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> Message-ID: On Fri, Nov 27, 2009 at 2:04 PM, David Warde-Farley wrote: > On 26-Nov-09, at 8:56 PM, Charles R Harris wrote: > > > I never had luck with the netlib-lapack-tarfile= > > option, it didn't build lapack correctly. Try doing them separately. > > I just tried that. > > -Ss flapack path/to/lapack_LINUX.a didn't seem to build a proper > > --with-netlib-lapack= What version of ATLAS? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Fri Nov 27 17:19:58 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Nov 2009 23:19:58 +0100 Subject: [Numpy-discussion] Bytes vs. Unicode in Python3 In-Reply-To: <200911271704.57927.faltet@pytables.org> References: <1259276898.8494.18.camel@idol> <200911271633.47281.faltet@pytables.org> <1259336464.4110.825.camel@talisman> <200911271704.57927.faltet@pytables.org> Message-ID: <4B10508E.4040200@student.matnat.uio.no> Francesc Alted wrote: > A Friday 27 November 2009 16:41:04 Pauli Virtanen escrigu?: >>>> I think so. However, I think S is probably closest to bytes... and >>>> maybe S can be reused for bytes... I'm not sure though. >>> That could be a good idea because that would ensure compatibility with >>> existing NumPy scripts (i.e. old 'string' dtypes are mapped to 'bytes', >>> as it should). The only thing that I don't like is that that 'S' seems >>> to be the initial letter for 'string', which is actually 'unicode' in >>> Python 3 :-/ But, for the sake of compatibility, we can probably live >>> with that. >> Well, we can "deprecate" 'S' (ie. never show it in repr, always only 'B' >> or 'U'). > > Well, deprecating 'S' seems a sensible option too. But why only avoiding > showing it in repr? Why not issue a DeprecationWarning too? One thing to keep in mind here is that PEP 3118 actually defines a standard dtype format string, which is (mostly) incompatible with NumPy's. It should probably be supported as well when PEP 3118 is implemented. Just something to keep in the back of ones mind when discussing this. For instance one could, instead of inventing something new, adopt the characters PEP 3118 uses (if there isn't a conflict): - b: Raw byte - c: ucs-1 encoding (latin 1, one byte) - u: ucs-2 encoding, two bytes - w: ucs-4 encoding, four bytes Long-term I hope the NumPy-specific format string will be deprecated, so that repr print out the PEP 3118 format string etc. But, I'm aware that API breakage shouldn't happen when porting to Python 3. -- Dag Sverre From sierra_mtnview at sbcglobal.net Fri Nov 27 17:47:46 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 14:47:46 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> Message-ID: <4B105712.7070701@sbcglobal.net> I tried this and it put ranges on y from 0 to 0.45 and x from 5 to 50. import numpy as np import pylab v = np.array([20, 15,10,30, 50, 30, 20, 25, 10]) #Plot a normalized histogram print np.linspace(0,50,10) pylab.hist(v, normed=1, bins=np.linspace(0,9,10), range=(0,100)) pylab.show() I added the two imports. I got a fig error on the first line. import pylab import numpy Shouldn't there by a pylab.Show in there? ax = fig.add_subplot(1,1,1) pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), normed=1, facecolor='y', alpha=0.5) ax.set_xlabel(r'\Large$ \rm{values}$') ax.set_ylabel(r'\Large Delatavalue/Value') gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) pylab.plot(gausx,gaus, color='red', lw=2) ax.set_xlim(-1.5, 1.5) ax.grid(True) Sebastian wrote: > Did you try using the parameter range? > I do something like this. > regards > > ax = fig.add_subplot(1,1,1) > pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > normed=1, facecolor='y', alpha=0.5) > ax.set_xlabel(r'\Large$ \rm{values}$') > ax.set_ylabel(r'\Large Delatavalue/Value') > > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > pylab.plot(gausx,gaus, color='red', lw=2) > ax.set_xlim(-1.5, 1.5) > ax.grid(True) > > > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker > > wrote: > > josef.pktd at gmail.com wrote: > > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold > > wrote: > > >> This kind of info might be useful to other newcomers > >> somewhere... ? Thoughts on > >> posting this on the wiki here? > > > > I also agree. It will improve with the newly redesigned website > for scipy.org > > However, I cannot find the link right now for the development > version of > > the new website. > > Feel free to crib whatever you want from my post for that -- or > suggest > a place for me to put it, and I'll do it. I'm just not sure where it > should go at this point. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From dwf at cs.toronto.edu Fri Nov 27 18:18:43 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Fri, 27 Nov 2009 18:18:43 -0500 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> Message-ID: On 27-Nov-09, at 5:02 PM, Charles R Harris wrote: > What version of ATLAS? 3.9.17, the latest development branch version. At some point in the changelog the author mentions he removed other methods building with LAPACK, leaving only the tarfile one. I'm giving the entire build another try with gfortran-4.2 rather than gfortran-4.4, since I already had a problem on another Ubuntu 9.10 machine with gfortran 4.4 and that exact same function, except it wasn't an unresolved symbol issue, rather it was an infinite loop: http://mail.scipy.org/pipermail/numpy-discussion/2009-November/046373.html David From charlesr.harris at gmail.com Fri Nov 27 18:29:47 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 27 Nov 2009 16:29:47 -0700 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> Message-ID: On Fri, Nov 27, 2009 at 4:18 PM, David Warde-Farley wrote: > On 27-Nov-09, at 5:02 PM, Charles R Harris wrote: > > > What version of ATLAS? > > 3.9.17, the latest development branch version. At some point in the 3.9.12 segfaulted on me while running, so I haven't bothered with versions after that. Why not try the stable version 3.8.3? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Nov 27 19:49:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 28 Nov 2009 09:49:09 +0900 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> Message-ID: <5b8d13220911271649t7ecff43fn734d41b791f2a5c8@mail.gmail.com> On Sat, Nov 28, 2009 at 8:29 AM, Charles R Harris wrote: > > > On Fri, Nov 27, 2009 at 4:18 PM, David Warde-Farley > wrote: >> >> On 27-Nov-09, at 5:02 PM, Charles R Harris wrote: >> >> > What version of ATLAS? >> >> 3.9.17, the latest development branch version. At some point in the > > 3.9.12 segfaulted on me while running, so I haven't bothered with versions > after that. Why not try the stable version 3.8.3? I guess because of the updated threading support. I think the solution is to simply bypass the atlas mechanism to build lapack, and do it manually. David From sebas0 at gmail.com Fri Nov 27 21:05:20 2009 From: sebas0 at gmail.com (Sebastian) Date: Sat, 28 Nov 2009 00:05:20 -0200 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B105712.7070701@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> Message-ID: Hi Chris, yeah there should, try the following: import numpy import matplotlib.pyplot as pylab regards On Fri, Nov 27, 2009 at 8:47 PM, Wayne Watson wrote: > I tried this and it put ranges on y from 0 to 0.45 and x from 5 to 50. > > import numpy as np > import pylab > > v = np.array([20, 15,10,30, 50, 30, 20, 25, 10]) > #Plot a normalized histogram > print np.linspace(0,50,10) > pylab.hist(v, normed=1, bins=np.linspace(0,9,10), range=(0,100)) > pylab.show() > > I added the two imports. I got a fig error on the first line. > import pylab > import numpy > > Shouldn't there by a pylab.Show in there? > > ax = fig.add_subplot(1,1,1) > pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), > > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > normed=1, facecolor='y', alpha=0.5) > ax.set_xlabel(r'\Large$ \rm{values}$') > ax.set_ylabel(r'\Large Delatavalue/Value') > > > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > pylab.plot(gausx,gaus, color='red', lw=2) > ax.set_xlim(-1.5, 1.5) > ax.grid(True) > > Sebastian wrote: > > Did you try using the parameter range? > > I do something like this. > > regards > > > > ax = fig.add_subplot(1,1,1) > > pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), > > > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > > normed=1, facecolor='y', alpha=0.5) > > ax.set_xlabel(r'\Large$ \rm{values}$') > > ax.set_ylabel(r'\Large Delatavalue/Value') > > > > > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > > > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > > pylab.plot(gausx,gaus, color='red', lw=2) > > ax.set_xlim(-1.5, 1.5) > > ax.grid(True) > > > > > > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker > > > wrote: > > > > josef.pktd at gmail.com wrote: > > > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold > > > wrote: > > > > >> This kind of info might be useful to other newcomers > > >> somewhere... ? Thoughts > on > > >> posting this on the wiki here? > > > > > > I also agree. It will improve with the newly redesigned website > > for scipy.org > > > However, I cannot find the link right now for the development > > version of > > > the new website. > > > > Feel free to crib whatever you want from my post for that -- or > > suggest > > a place for me to put it, and I'll do it. I'm just not sure where it > > should go at this point. > > > > -Chris > > > > > > -- > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet > > 350 350 350 350 350 350 350 350 350 350 > Make the number famous. See 350.org > The major event has passed, but keep the number alive. > > Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Nov 27 21:18:12 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 21:18:12 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> Message-ID: <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> On Fri, Nov 27, 2009 at 9:05 PM, Sebastian wrote: > Hi Chris, yeah there should, try the following: > import numpy > import matplotlib.pyplot as pylab > regards > > On Fri, Nov 27, 2009 at 8:47 PM, Wayne Watson > wrote: >> >> I tried this and it put ranges on y from 0 to 0.45 and x from 5 to 50. >> >> import numpy as np >> import pylab >> >> v = np.array([20, 15,10,30, 50, 30, 20, 25, 10]) >> #Plot a normalized histogram >> print np.linspace(0,50,10) >> pylab.hist(v, normed=1, bins=np.linspace(0,9,10), range=(0,100)) >> pylab.show() >> >> I ?added the two imports. I got a fig error on the first line. >> import pylab >> import numpy >> >> Shouldn't there by a pylab.Show in there? >> you need to create a figure, before you can use it fig = pylab.figure() Josef >> ax = fig.add_subplot(1,1,1) >> pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >> >> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> normed=1, facecolor='y', alpha=0.5) >> ax.set_xlabel(r'\Large$ \rm{values}$') >> ax.set_ylabel(r'\Large Delatavalue/Value') >> >> >> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> >> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> pylab.plot(gausx,gaus, color='red', lw=2) >> ax.set_xlim(-1.5, 1.5) >> ax.grid(True) >> >> Sebastian wrote: >> > Did you try using the parameter range? >> > I do something like this. >> > regards >> > >> > ? ? ax = fig.add_subplot(1,1,1) >> > ? ? pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >> > ? ? n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >> > >> > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> > ? ? normed=1, facecolor='y', alpha=0.5) >> > ? ? ax.set_xlabel(r'\Large$ \rm{values}$') >> > ? ? ax.set_ylabel(r'\Large Delatavalue/Value') >> > >> > >> > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> > >> > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> > ? ? pylab.plot(gausx,gaus, color='red', lw=2) >> > ? ? ax.set_xlim(-1.5, 1.5) >> > ? ? ax.grid(True) >> > >> > >> > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker >> > > wrote: >> > >> > ? ? josef.pktd at gmail.com wrote: >> > ? ? > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold >> > ? ? > wrote: >> > >> > ? ? >> ?This kind of info might be useful to other newcomers >> > ? ? >> somewhere... ?? ?Thoughts >> > on >> > ? ? >> posting this on the wiki here? >> > ? ? > >> > ? ? > I also agree. It will improve with the newly redesigned website >> > ? ? for scipy.org >> > ? ? > However, I cannot find the link right now for the development >> > ? ? version of >> > ? ? > the new website. >> > >> > ? ? Feel free to crib whatever you want from my post for that -- or >> > ? ? suggest >> > ? ? a place for me to put it, and I'll do it. I'm just not sure where it >> > ? ? should go at this point. >> > >> > ? ? -Chris >> > >> > >> > ? ? -- >> > ? ? Christopher Barker, Ph.D. >> > ? ? Oceanographer >> > >> > ? ? Emergency Response Division >> > ? ? NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice >> > ? ? 7600 Sand Point Way NE ? (206) 526-6329 ? fax >> > ? ? Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception >> > >> > ? ? Chris.Barker at noaa.gov >> > ? ? _______________________________________________ >> > ? ? NumPy-Discussion mailing list >> > ? ? NumPy-Discussion at scipy.org >> > ? ? http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> -- >> ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 >> ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org >> ? ? ? ? ? ?The major event has passed, but keep the number alive. >> >> ? ? ? ? ? ? ? ? ? ?Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From gokhansever at gmail.com Fri Nov 27 21:28:55 2009 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 27 Nov 2009 20:28:55 -0600 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> Message-ID: <49d6b3500911271828r6fc15da7v43b546f88286221b@mail.gmail.com> On Fri, Nov 27, 2009 at 8:05 PM, Sebastian wrote: > Hi Chris, yeah there should, try the following: > import numpy > import matplotlib.pyplot as pylab > regards > > Shouldn't the 2nd import be: import matplotlib.pyplot as plt ? > > On Fri, Nov 27, 2009 at 8:47 PM, Wayne Watson < > sierra_mtnview at sbcglobal.net> wrote: > >> I tried this and it put ranges on y from 0 to 0.45 and x from 5 to 50. >> >> import numpy as np >> import pylab >> >> v = np.array([20, 15,10,30, 50, 30, 20, 25, 10]) >> #Plot a normalized histogram >> print np.linspace(0,50,10) >> pylab.hist(v, normed=1, bins=np.linspace(0,9,10), range=(0,100)) >> pylab.show() >> >> I added the two imports. I got a fig error on the first line. >> import pylab >> import numpy >> >> Shouldn't there by a pylab.Show in there? >> >> ax = fig.add_subplot(1,1,1) >> pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') >> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >> >> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> normed=1, facecolor='y', alpha=0.5) >> ax.set_xlabel(r'\Large$ \rm{values}$') >> ax.set_ylabel(r'\Large Delatavalue/Value') >> >> >> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> >> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> pylab.plot(gausx,gaus, color='red', lw=2) >> ax.set_xlim(-1.5, 1.5) >> ax.grid(True) >> >> Sebastian wrote: >> > Did you try using the parameter range? >> > I do something like this. >> > regards >> > >> > ax = fig.add_subplot(1,1,1) >> > pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') >> > n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >> > >> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> > normed=1, facecolor='y', alpha=0.5) >> > ax.set_xlabel(r'\Large$ \rm{values}$') >> > ax.set_ylabel(r'\Large Delatavalue/Value') >> > >> > >> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> > >> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> > pylab.plot(gausx,gaus, color='red', lw=2) >> > ax.set_xlim(-1.5, 1.5) >> > ax.grid(True) >> > >> > >> > On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker >> > > wrote: >> > >> > josef.pktd at gmail.com wrote: >> > > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold >> > > wrote: >> > >> > >> This kind of info might be useful to other newcomers >> > >> somewhere... ? Thoughts >> on >> > >> posting this on the wiki here? >> > > >> > > I also agree. It will improve with the newly redesigned website >> > for scipy.org >> > > However, I cannot find the link right now for the development >> > version of >> > > the new website. >> > >> > Feel free to crib whatever you want from my post for that -- or >> > suggest >> > a place for me to put it, and I'll do it. I'm just not sure where it >> > should go at this point. >> > >> > -Chris >> > >> > >> > -- >> > Christopher Barker, Ph.D. >> > Oceanographer >> > >> > Emergency Response Division >> > NOAA/NOS/OR&R (206) 526-6959 voice >> > 7600 Sand Point Way NE (206) 526-6329 fax >> > Seattle, WA 98115 (206) 526-6317 main reception >> > >> > Chris.Barker at noaa.gov >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> 350 350 350 350 350 350 350 350 350 350 >> Make the number famous. See 350.org >> The major event has passed, but keep the number alive. >> >> Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From sierra_mtnview at sbcglobal.net Fri Nov 27 21:44:44 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 18:44:44 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> Message-ID: <4B108E9C.4090808@sbcglobal.net> Joseph, That got it by the fig problem but there is yet another one. value is not defined on the very long line: range = ... Wayne josef.pktd at gmail.com wrote: > On Fri, Nov 27, 2009 at 9:05 PM, Sebastian wrote: > > ... > you need to create a figure, before you can use it > > fig = pylab.figure() > > Josef > > >>> ax = fig.add_subplot(1,1,1) >>> pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') >>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >>> >>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >>> normed=1, facecolor='y', alpha=0.5) >>> ax.set_xlabel(r'\Large$ \rm{values}$') >>> ax.set_ylabel(r'\Large Delatavalue/Value') >>> >>> >>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >>> >>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >>> pylab.plot(gausx,gaus, color='red', lw=2) >>> ax.set_xlim(-1.5, 1.5) >>> ax.grid(True) >>> >>> Sebastian wrote: >>> >>>> Did you try using the parameter range? >>>> I do something like this. >>>> regards >>>> >>>> ax = fig.add_subplot(1,1,1) >>>> pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') >>>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >>>> >>>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >>>> normed=1, facecolor='y', alpha=0.5) >>>> ax.set_xlabel(r'\Large$ \rm{values}$') >>>> ax.set_ylabel(r'\Large Delatavalue/Value') >>>> >>>> >>>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >>>> >>>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >>>> pylab.plot(gausx,gaus, color='red', lw=2) >>>> ax.set_xlim(-1.5, 1.5) >>>> ax.grid(True) >>>> >>>> >>>> On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker >>>> > wrote: >>>> >>>> josef.pktd at gmail.com wrote: >>>> > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold >>>> > wrote: >>>> >>>> >> This kind of info might be useful to other newcomers >>>> >> somewhere... ? Thoughts >>>> on >>>> >> posting this on the wiki here? >>>> > >>>> > I also agree. It will improve with the newly redesigned website >>>> for scipy.org >>>> > However, I cannot find the link right now for the development >>>> version of >>>> > the new website. >>>> >>>> Feel free to crib whatever you want from my post for that -- or >>>> suggest >>>> a place for me to put it, and I'll do it. I'm just not sure where it >>>> should go at this point. >>>> >>>> -Chris >>>> >>>> >>>> -- >>>> Christopher Barker, Ph.D. >>>> Oceanographer >>>> >>>> Emergency Response Division >>>> NOAA/NOS/OR&R (206) 526-6959 voice >>>> 7600 Sand Point Way NE (206) 526-6329 fax >>>> Seattle, WA 98115 (206) 526-6317 main reception >>>> >>>> Chris.Barker at noaa.gov >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>> -- >>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>> >>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >>> >>> 350 350 350 350 350 350 350 350 350 350 >>> Make the number famous. See 350.org >>> The major event has passed, but keep the number alive. >>> >>> Web Page: >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From sierra_mtnview at sbcglobal.net Fri Nov 27 21:47:22 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 18:47:22 -0800 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known Message-ID: <4B108F3A.4080008@sbcglobal.net> How do I compute avg, std dev, min, max and other simple stats if I only know the frequency distribution? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From josef.pktd at gmail.com Fri Nov 27 22:01:22 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 22:01:22 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B108E9C.4090808@sbcglobal.net> References: <4B0F60A5.7040302@sbcglobal.net> <4B100F5E.7080409@noaa.gov> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> <4B108E9C.4090808@sbcglobal.net> Message-ID: <1cd32cbb0911271901y1f6aaffaq46f4a9d4fb3a669b@mail.gmail.com> On Fri, Nov 27, 2009 at 9:44 PM, Wayne Watson wrote: > Joseph, > That got it by the fig problem but there is yet another one. value is > not defined on the very long line: > range = ... > ? ?Wayne (values is the data array, ... no idea about scientificstat.standardDeviation) Sebastian's example is only part of a larger script that defines many of the variables and functions that are used. If you are not yet familiar with these examples, maybe you look at the self contained examples in the matplotlib docs. At least that's what I do when I only have a rough idea about what graph I want to do but don't know how to do it with matplotlib. I usually just copy a likely looking candidate and change it until it (almost) produces what I want. For example look at histogram examples in http://matplotlib.sourceforge.net/examples/index.html Josef > josef.pktd at gmail.com wrote: >> On Fri, Nov 27, 2009 at 9:05 PM, Sebastian wrote: >> >> ... >> you need to create a figure, before you can use it >> >> fig = pylab.figure() >> >> Josef >> >> >>>> ax = fig.add_subplot(1,1,1) >>>> pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >>>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >>>> >>>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >>>> normed=1, facecolor='y', alpha=0.5) >>>> ax.set_xlabel(r'\Large$ \rm{values}$') >>>> ax.set_ylabel(r'\Large Delatavalue/Value') >>>> >>>> >>>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >>>> >>>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >>>> pylab.plot(gausx,gaus, color='red', lw=2) >>>> ax.set_xlim(-1.5, 1.5) >>>> ax.grid(True) >>>> >>>> Sebastian wrote: >>>> >>>>> Did you try using the parameter range? >>>>> I do something like this. >>>>> regards >>>>> >>>>> ? ? ax = fig.add_subplot(1,1,1) >>>>> ? ? pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >>>>> ? ? n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >>>>> >>>>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >>>>> ? ? normed=1, facecolor='y', alpha=0.5) >>>>> ? ? ax.set_xlabel(r'\Large$ \rm{values}$') >>>>> ? ? ax.set_ylabel(r'\Large Delatavalue/Value') >>>>> >>>>> >>>>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >>>>> >>>>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >>>>> ? ? pylab.plot(gausx,gaus, color='red', lw=2) >>>>> ? ? ax.set_xlim(-1.5, 1.5) >>>>> ? ? ax.grid(True) >>>>> >>>>> >>>>> On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker >>>>> > wrote: >>>>> >>>>> ? ? josef.pktd at gmail.com wrote: >>>>> ? ? > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold >>>>> ? ? > wrote: >>>>> >>>>> ? ? >> ?This kind of info might be useful to other newcomers >>>>> ? ? >> somewhere... ?? ?Thoughts >>>>> on >>>>> ? ? >> posting this on the wiki here? >>>>> ? ? > >>>>> ? ? > I also agree. It will improve with the newly redesigned website >>>>> ? ? for scipy.org >>>>> ? ? > However, I cannot find the link right now for the development >>>>> ? ? version of >>>>> ? ? > the new website. >>>>> >>>>> ? ? Feel free to crib whatever you want from my post for that -- or >>>>> ? ? suggest >>>>> ? ? a place for me to put it, and I'll do it. I'm just not sure where it >>>>> ? ? should go at this point. >>>>> >>>>> ? ? -Chris >>>>> >>>>> >>>>> ? ? -- >>>>> ? ? Christopher Barker, Ph.D. >>>>> ? ? Oceanographer >>>>> >>>>> ? ? Emergency Response Division >>>>> ? ? NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice >>>>> ? ? 7600 Sand Point Way NE ? (206) 526-6329 ? fax >>>>> ? ? Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception >>>>> >>>>> ? ? Chris.Barker at noaa.gov >>>>> ? ? _______________________________________________ >>>>> ? ? NumPy-Discussion mailing list >>>>> ? ? NumPy-Discussion at scipy.org >>>>> ? ? http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>>> >>>> -- >>>> ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>>> >>>> ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>>> ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >>>> >>>> ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 >>>> ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org >>>> ? ? ? ? ? ?The major event has passed, but keep the number alive. >>>> >>>> ? ? ? ? ? ? ? ? ? ?Web Page: >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Fri Nov 27 22:14:25 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 27 Nov 2009 22:14:25 -0500 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <4B108F3A.4080008@sbcglobal.net> References: <4B108F3A.4080008@sbcglobal.net> Message-ID: <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> On Fri, Nov 27, 2009 at 9:47 PM, Wayne Watson wrote: > How do I compute avg, std dev, min, max and other simple stats if I only > know the frequency distribution? If you are willing to assign to all observations in a bin the value at the bin midpoint, then you could do it with weights in the statistics calculations. However, numpy.average is, I think, the only statistic that takes weights. min max are independent of weight, but std and var need to be calculated indirectly. If you need more stats with weights, then the attachment in http://projects.scipy.org/scipy/ticket/604 is a good start. Josef > > -- > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org > ? ? ? ? ? ?The major event has passed, but keep the number alive. > > ? ? ? ? ? ? ? ? ? ?Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sierra_mtnview at sbcglobal.net Sat Nov 28 00:25:30 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Fri, 27 Nov 2009 21:25:30 -0800 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> Message-ID: <4B10B44A.8030700@sbcglobal.net> I actually wrote my own several days ago. When I began getting myself more familiar with numpy, I was hoping there would be an easy to use version in it for this frequency approach. If not, then I'll just stick with what I have. It seems something like this should be common. A simple way to do it with the present capabilities would be to "unwind" the frequencies, For example, given [2,1,3] for some corresponding set of x, say, [1,2,3], produce[1, 1, 2, 3, 3, 3]. I have no idea if numpy does anything like that, but, if so, the typical mean, std, ... could be used. In my case, it's sort of pointless. It would produce an array of 307,200 items for 256 x (0,1,2,...,255), and just slow down the computations "unwinding" it in software. The sub-processor hardware already produced the 256 frequencies. Basically, this amounts to having a pdf, and values of x. Mathematically, the statistics are produced directly from it. josef.pktd at gmail.com wrote: > On Fri, Nov 27, 2009 at 9:47 PM, Wayne Watson > wrote: > >> How do I compute avg, std dev, min, max and other simple stats if I only >> know the frequency distribution? >> > > If you are willing to assign to all observations in a bin the value at > the bin midpoint, then you could do it with weights in the statistics > calculations. However, numpy.average is, I think, the only statistic > that takes weights. min max are independent of weight, but std and var > need to be calculated indirectly. > > If you need more stats with weights, then the attachment in > http://projects.scipy.org/scipy/ticket/604 is a good start. > > Josef > > > >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> 350 350 350 350 350 350 350 350 350 350 >> Make the number famous. See 350.org >> The major event has passed, but keep the number alive. >> >> Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From dwf at cs.toronto.edu Sat Nov 28 01:12:20 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sat, 28 Nov 2009 01:12:20 -0500 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: <5b8d13220911271649t7ecff43fn734d41b791f2a5c8@mail.gmail.com> References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> <5b8d13220911271649t7ecff43fn734d41b791f2a5c8@mail.gmail.com> Message-ID: <2AF3C935-C08C-4856-AFDB-3E18B3996A02@cs.toronto.edu> On 27-Nov-09, at 7:49 PM, David Cournapeau wrote: > I guess because of the updated threading support. Right you are; the 3.9 series is rather faster I find, at least in parallel. > I think the solution is to simply bypass the atlas mechanism to build > lapack, and do it manually. Tried that... see below: > "and so I unarchived the liblapack.a that ATLAS spit out and added > those files to the one built by Netlib's makefile. The result is still > the same." I'm gonna try what Chuck suggested and go back a few versions at least. David From renesd at gmail.com Sat Nov 28 04:34:48 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sat, 28 Nov 2009 10:34:48 +0100 Subject: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux Message-ID: <64ddb72c0911280134m48d5fec5u40540f35c923318a@mail.gmail.com> Hi, the other day I tried installing numpy with easy_install, and did not have much luck with python2.6. To reproduce, try installing with easy_install-2.6, or with buildout. The work around is to just install it in another way (eg, dmg or with setup.py install). cheers, -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Nov 28 05:16:06 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 28 Nov 2009 19:16:06 +0900 Subject: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux In-Reply-To: <64ddb72c0911280134m48d5fec5u40540f35c923318a@mail.gmail.com> References: <64ddb72c0911280134m48d5fec5u40540f35c923318a@mail.gmail.com> Message-ID: <5b8d13220911280216v19806cd1wbb8cea4b09c3298f@mail.gmail.com> On Sat, Nov 28, 2009 at 6:34 PM, Ren? Dudfield wrote: > Hi, > > the other day I tried installing numpy with easy_install, and did not have > much luck with python2.6. What problem exactly do you have ? Since 1.3.0, the release process for numpy uses virtualenv internally to build the installers, so at least python setupegg.py install works as far as setuptools is concerned. > The work around is to just install it in another way (eg, dmg or with > setup.py install). That's the recommended way anyway. I am trying to make sure we work under as many cases as possible for packaging, but the underlying python architecture for packaging is so fragile and flawed that things will break from time to time. David From sebas0 at gmail.com Sat Nov 28 06:18:34 2009 From: sebas0 at gmail.com (Sebastian) Date: Sat, 28 Nov 2009 09:18:34 -0200 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <1cd32cbb0911271901y1f6aaffaq46f4a9d4fb3a669b@mail.gmail.com> References: <4B0F60A5.7040302@sbcglobal.net> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> <4B108E9C.4090808@sbcglobal.net> <1cd32cbb0911271901y1f6aaffaq46f4a9d4fb3a669b@mail.gmail.com> Message-ID: On Sat, Nov 28, 2009 at 1:01 AM, wrote: > On Fri, Nov 27, 2009 at 9:44 PM, Wayne Watson > wrote: > > Joseph, > > That got it by the fig problem but there is yet another one. value is > > not defined on the very long line: > > range = ... > > Wayne > > (values is the data array, ... no idea about > scientificstat.standardDeviation) > > Sebastian's example is only part of a larger script that defines many > of the variables and functions that are used. > > If you are not yet familiar with these examples, maybe you look at the > self contained examples in the matplotlib docs. At least that's what I > do when I only have a rough idea about what graph I want to do but > don't know how to do it with matplotlib. I usually just copy a likely > looking candidate and change it until it (almost) produces what I > want. > For example look at histogram examples in > > http://matplotlib.sourceforge.net/examples/index.html > > Josef > > > > josef.pktd at gmail.com wrote: > >> On Fri, Nov 27, 2009 at 9:05 PM, Sebastian wrote: > >> > >> ... > >> you need to create a figure, before you can use it > >> > >> fig = pylab.figure() > >> > >> Josef > >> > >> > >>>> ax = fig.add_subplot(1,1,1) > >>>> pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > >>>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), > >>>> > >>>> > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > >>>> normed=1, facecolor='y', alpha=0.5) > >>>> ax.set_xlabel(r'\Large$ \rm{values}$') > >>>> ax.set_ylabel(r'\Large Delatavalue/Value') > >>>> > >>>> > >>>> > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > >>>> > >>>> > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > >>>> pylab.plot(gausx,gaus, color='red', lw=2) > >>>> ax.set_xlim(-1.5, 1.5) > >>>> ax.grid(True) > >>>> > >>>> Sebastian wrote: > >>>> > >>>>> Did you try using the parameter range? > >>>>> I do something like this. > >>>>> regards > >>>>> > >>>>> ax = fig.add_subplot(1,1,1) > >>>>> pylab.title(r'\Large BCG NO radio distribution $ \rm{TITLE}$') > >>>>> n, bins, patches = pylab.hist(values, > bins=math.sqrt(len(values)), > >>>>> > >>>>> > range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), > >>>>> normed=1, facecolor='y', alpha=0.5) > >>>>> ax.set_xlabel(r'\Large$ \rm{values}$') > >>>>> ax.set_ylabel(r'\Large Delatavalue/Value') > >>>>> > >>>>> > >>>>> > gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) > >>>>> > >>>>> > gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) > >>>>> pylab.plot(gausx,gaus, color='red', lw=2) > >>>>> ax.set_xlim(-1.5, 1.5) > >>>>> ax.grid(True) > >>>>> > >>>>> > >>>>> On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker > >>>>> > wrote: > >>>>> > >>>>> josef.pktd at gmail.com wrote: > >>>>> > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold > >>>>> > wrote: > >>>>> > >>>>> >> This kind of info might be useful to other newcomers > >>>>> >> somewhere... ? > Thoughts > >>>>> on > >>>>> >> posting this on the wiki here? > >>>>> > > >>>>> > I also agree. It will improve with the newly redesigned website > >>>>> for scipy.org > >>>>> > However, I cannot find the link right now for the development > >>>>> version of > >>>>> > the new website. > >>>>> > >>>>> Feel free to crib whatever you want from my post for that -- or > >>>>> suggest > >>>>> a place for me to put it, and I'll do it. I'm just not sure where > it > >>>>> should go at this point. > >>>>> > >>>>> -Chris > >>>>> > >>>>> > >>>>> -- > >>>>> Christopher Barker, Ph.D. > >>>>> Oceanographer > >>>>> > >>>>> Emergency Response Division > >>>>> NOAA/NOS/OR&R (206) 526-6959 voice > >>>>> 7600 Sand Point Way NE (206) 526-6329 fax > >>>>> Seattle, WA 98115 (206) 526-6317 main reception > >>>>> > >>>>> Chris.Barker at noaa.gov > >>>>> _______________________________________________ > >>>>> NumPy-Discussion mailing list > >>>>> NumPy-Discussion at scipy.org > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>>> > >>>>> > >>>>> > ------------------------------------------------------------------------ > >>>>> > >>>>> _______________________________________________ > >>>>> NumPy-Discussion mailing list > >>>>> NumPy-Discussion at scipy.org > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>>> > >>>>> > >>>> -- > >>>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > >>>> > >>>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > >>>> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet > >>>> > >>>> 350 350 350 350 350 350 350 350 350 350 > >>>> Make the number famous. See 350.org > >>>> The major event has passed, but keep the number alive. > >>>> > >>>> Web Page: > >>>> > >>>> _______________________________________________ > >>>> NumPy-Discussion mailing list > >>>> NumPy-Discussion at scipy.org > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >>> > >>> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > > > -- > > Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > > > (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > > Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet > > > > 350 350 350 350 350 350 350 350 350 350 > > Make the number famous. See 350.org > > The major event has passed, but keep the number alive. > > > > Web Page: > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Chris I',m using the package Scientific.Stats to calculate the standard deviation. I believe it is from here: http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ I use the Scientific.Stats package for the standard deviation calculation because back when I wrote the code I realized that numpy's standard deviation seems to assume that you have all the distribution (the parent population),while I think the Scientific.Stats is more accurate for smaller samples. But maybe there is an equivalent numpy standard deviation. If I recall OK the difference is an (n-1) instead of an (n) in the formula. For larger samples both the numpy and the Scientific.Stats standard deviation shouldn't be too different . So I use the range to specify the values to bin over. It seems you might want your range parameter to be different. I'm choosing the range to be +/- 3-sigma, and that way ignore values that are too extreme so my bins a more concentrated about the distribution. OK so I also have to add the following import line in the code, too: import Scientific.Statistics as scientificstat Sebas -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Nov 28 08:27:30 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 28 Nov 2009 08:27:30 -0500 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: References: <4B0F60A5.7040302@sbcglobal.net> <1cd32cbb0911271015x3b8fcbc7n95d6db53ebe34cc5@mail.gmail.com> <4B101C8F.2020808@noaa.gov> <4B105712.7070701@sbcglobal.net> <1cd32cbb0911271818y5c289dbfn1fd3263cceac065a@mail.gmail.com> <4B108E9C.4090808@sbcglobal.net> <1cd32cbb0911271901y1f6aaffaq46f4a9d4fb3a669b@mail.gmail.com> Message-ID: <1cd32cbb0911280527j355bf50p3647b19f7b81a956@mail.gmail.com> On Sat, Nov 28, 2009 at 6:18 AM, Sebastian wrote: > > > On Sat, Nov 28, 2009 at 1:01 AM, wrote: >> >> On Fri, Nov 27, 2009 at 9:44 PM, Wayne Watson >> wrote: >> > Joseph, >> > That got it by the fig problem but there is yet another one. value is >> > not defined on the very long line: >> > range = ... >> > ? ?Wayne >> >> (values is the data array, ... no idea about >> scientificstat.standardDeviation) >> >> Sebastian's example is only part of a larger script that defines many >> of the variables and functions that are used. >> >> If you are not yet familiar with these examples, maybe you look at the >> self contained examples in the matplotlib docs. At least that's what I >> do when I only have a rough idea about what graph I want to do but >> don't know how to do it with matplotlib. I usually just copy a likely >> looking candidate and change it until it (almost) ?produces what I >> want. >> For example look at histogram examples in >> >> http://matplotlib.sourceforge.net/examples/index.html >> >> Josef >> >> >> > josef.pktd at gmail.com wrote: >> >> On Fri, Nov 27, 2009 at 9:05 PM, Sebastian wrote: >> >> >> >> ... >> >> you need to create a figure, before you can use it >> >> >> >> fig = pylab.figure() >> >> >> >> Josef >> >> >> >> >> >>>> ax = fig.add_subplot(1,1,1) >> >>>> pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >> >>>> n, bins, patches = pylab.hist(values, bins=math.sqrt(len(values)), >> >>>> >> >>>> >> >>>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> >>>> normed=1, facecolor='y', alpha=0.5) >> >>>> ax.set_xlabel(r'\Large$ \rm{values}$') >> >>>> ax.set_ylabel(r'\Large Delatavalue/Value') >> >>>> >> >>>> >> >>>> >> >>>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> >>>> >> >>>> >> >>>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> >>>> pylab.plot(gausx,gaus, color='red', lw=2) >> >>>> ax.set_xlim(-1.5, 1.5) >> >>>> ax.grid(True) >> >>>> >> >>>> Sebastian wrote: >> >>>> >> >>>>> Did you try using the parameter range? >> >>>>> I do something like this. >> >>>>> regards >> >>>>> >> >>>>> ? ? ax = fig.add_subplot(1,1,1) >> >>>>> ? ? pylab.title(r'\Large ?BCG NO radio distribution $ \rm{TITLE}$') >> >>>>> ? ? n, bins, patches = pylab.hist(values, >> >>>>> bins=math.sqrt(len(values)), >> >>>>> >> >>>>> >> >>>>> range=(numpy.mean(values)-3*scientificstat.standardDeviation(values),numpy.mean(values)+3*scientificstat.standardDeviation(values)), >> >>>>> ? ? normed=1, facecolor='y', alpha=0.5) >> >>>>> ? ? ax.set_xlabel(r'\Large$ \rm{values}$') >> >>>>> ? ? ax.set_ylabel(r'\Large Delatavalue/Value') >> >>>>> >> >>>>> >> >>>>> >> >>>>> gausx=numpy.arange(numpy.mean(Value)-3*scientificstat.standardDeviation(Value),numpy.mean(Value)+3*scientificstat.standardDeviation(bpty_plt),0.1) >> >>>>> >> >>>>> >> >>>>> gaus=normpdf(gausx,numpy.mean(Value),scientificstat.standardDeviation(Value)) >> >>>>> ? ? pylab.plot(gausx,gaus, color='red', lw=2) >> >>>>> ? ? ax.set_xlim(-1.5, 1.5) >> >>>>> ? ? ax.grid(True) >> >>>>> >> >>>>> >> >>>>> On Fri, Nov 27, 2009 at 4:38 PM, Christopher Barker >> >>>>> > wrote: >> >>>>> >> >>>>> ? ? josef.pktd at gmail.com wrote: >> >>>>> ? ? > On Fri, Nov 27, 2009 at 12:57 PM, Skipper Seabold >> >>>>> ? ? > wrote: >> >>>>> >> >>>>> ? ? >> ?This kind of info might be useful to other newcomers >> >>>>> ? ? >> somewhere... ?? >> >>>>> ?Thoughts >> >>>>> on >> >>>>> ? ? >> posting this on the wiki here? >> >>>>> ? ? > >> >>>>> ? ? > I also agree. It will improve with the newly redesigned >> >>>>> website >> >>>>> ? ? for scipy.org >> >>>>> ? ? > However, I cannot find the link right now for the development >> >>>>> ? ? version of >> >>>>> ? ? > the new website. >> >>>>> >> >>>>> ? ? Feel free to crib whatever you want from my post for that -- or >> >>>>> ? ? suggest >> >>>>> ? ? a place for me to put it, and I'll do it. I'm just not sure >> >>>>> where it >> >>>>> ? ? should go at this point. >> >>>>> >> >>>>> ? ? -Chris >> >>>>> >> >>>>> >> >>>>> ? ? -- >> >>>>> ? ? Christopher Barker, Ph.D. >> >>>>> ? ? Oceanographer >> >>>>> >> >>>>> ? ? Emergency Response Division >> >>>>> ? ? NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice >> >>>>> ? ? 7600 Sand Point Way NE ? (206) 526-6329 ? fax >> >>>>> ? ? Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception >> >>>>> >> >>>>> ? ? Chris.Barker at noaa.gov >> >>>>> ? ? _______________________________________________ >> >>>>> ? ? NumPy-Discussion mailing list >> >>>>> ? ? NumPy-Discussion at scipy.org >> >>>>> ? ? http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >>>>> >> >>>>> >> >>>>> >> >>>>> ------------------------------------------------------------------------ >> >>>>> >> >>>>> _______________________________________________ >> >>>>> NumPy-Discussion mailing list >> >>>>> NumPy-Discussion at scipy.org >> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >>>>> >> >>>>> >> >>>> -- >> >>>> ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >>>> >> >>>> ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> >>>> ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >>>> >> >>>> ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 >> >>>> ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org >> >>>> ? ? ? ? ? ?The major event has passed, but keep the number alive. >> >>>> >> >>>> ? ? ? ? ? ? ? ? ? ?Web Page: >> >>>> >> >>>> _______________________________________________ >> >>>> NumPy-Discussion mailing list >> >>>> NumPy-Discussion at scipy.org >> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >>>> >> >>> _______________________________________________ >> >>> NumPy-Discussion mailing list >> >>> NumPy-Discussion at scipy.org >> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >>> >> >>> >> >>> >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> >> > >> > -- >> > ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> > >> > ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> > ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >> > >> > ? ? ? ? ? ? ? ? ? 350 350 350 350 350 350 350 350 350 350 >> > ? ? ? ? ? ? ? ? ? ? Make the number famous. See 350.org >> > ? ? ? ? ? ?The major event has passed, but keep the number alive. >> > >> > ? ? ? ? ? ? ? ? ? ?Web Page: >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Chris I',m using the package Scientific.Stats to calculate the standard > deviation. > I believe it is from here: > http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ > I use the Scientific.Stats package for the standard deviation calculation > because back when I wrote the code I realized that numpy's standard > deviation seems to assume that you have all the distribution (the parent > population),while I think the Scientific.Stats is more accurate for smaller > samples. But maybe there is an equivalent numpy standard deviation. > If I recall OK the difference is an (n-1) instead of an (n) in the formula. > For larger samples both the numpy and the? Scientific.Stats standard > deviation shouldn't be too different numpy std and var have a ddof option to adjust the degrees of freedom. ddof=0 by default, but you can set ddof=1 to get denominator (n-1) there was a long discussion on bias and bias correction on the mailing list a few months ago. Josef > . > So I use the range to specify? the values to bin over. It seems you might > want your range parameter to be different. > I'm choosing the range to be +/- 3-sigma, and that way ignore values that > are too extreme so my bins a more concentrated about the distribution.? OK > so I also have to add the following import line in the code, too: > > import Scientific.Statistics as scientificstat > > Sebas > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From renesd at gmail.com Sat Nov 28 08:42:30 2009 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sat, 28 Nov 2009 14:42:30 +0100 Subject: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux In-Reply-To: <5b8d13220911280216v19806cd1wbb8cea4b09c3298f@mail.gmail.com> References: <64ddb72c0911280134m48d5fec5u40540f35c923318a@mail.gmail.com> <5b8d13220911280216v19806cd1wbb8cea4b09c3298f@mail.gmail.com> Message-ID: <64ddb72c0911280542n4521b822madd4b336e3c9b5f5@mail.gmail.com> On Sat, Nov 28, 2009 at 11:16 AM, David Cournapeau wrote: > On Sat, Nov 28, 2009 at 6:34 PM, Ren? Dudfield wrote: > > Hi, > > > > the other day I tried installing numpy with easy_install, and did not > have > > much luck with python2.6. > > What problem exactly do you have ? > > Since 1.3.0, the release process for numpy uses virtualenv internally > to build the installers, so at least python setupegg.py install works > as far as setuptools is concerned. > > > The work around is to just install it in another way (eg, dmg or with > > setup.py install). > > That's the recommended way anyway. I am trying to make sure we work > under as many cases as possible for packaging, but the underlying > python architecture for packaging is so fragile and flawed that things > will break from time to time. > > David > > > Hello David, yeah, I completely understand the unfortunate packaging situation (eg, some of my packages do not work with this install method). Here is a simple package requiring numpy. It uses buildout ( http://www.buildout.org/). To help easily reproduce the problem, here are the commands to reproduce. curl -O http://rene.f0o.com/~rene/numpy_buildout.tar.gz #wget http://rene.f0o.com/~rene/numpy_buildout.tar.gz tar xf numpy_buildout.tar.gz cd numpy_buildout python2.6 bootstrap.py ./bin/buildout It prints out heaps of stuff... then finally fails with: /var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py:219: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py", line 219, in clean_up_temporary_directory ImportError: No module named numpy.distutils Error in sys.exitfunc: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/var/folders/d1/d1p0zeCxF7Kz3G5FZJsh6E+++TM/-Tmp-/easy_install-2WsZVp/numpy-1.3.0/numpy/distutils/misc_util.py", line 219, in clean_up_temporary_directory I hope that helps if someone wants to reproduce the problem very easily some time. notes: * you should not have numpy installed already to reproduce the problem, otherwise the buildout uses the installed version and works. * use `python bootstrap.py -d` if you want to use distribute instead of setuptools (which has python3.1 support). * numpy with buildout works ok with python2.5 cheers, -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Nov 28 08:56:30 2009 From: cournape at gmail.com (David Cournapeau) Date: Sat, 28 Nov 2009 22:56:30 +0900 Subject: [Numpy-discussion] numpy 1.3.0 eggs with python2.6 seem broken on osx, and linux In-Reply-To: <64ddb72c0911280542n4521b822madd4b336e3c9b5f5@mail.gmail.com> References: <64ddb72c0911280134m48d5fec5u40540f35c923318a@mail.gmail.com> <5b8d13220911280216v19806cd1wbb8cea4b09c3298f@mail.gmail.com> <64ddb72c0911280542n4521b822madd4b336e3c9b5f5@mail.gmail.com> Message-ID: <5b8d13220911280556s475ed2e5hec24768e20602210@mail.gmail.com> On Sat, Nov 28, 2009 at 10:42 PM, Ren? Dudfield wrote: > > yeah, I completely understand the unfortunate packaging situation (eg, some > of my packages do not work with this install method). > > Here is a simple package requiring numpy.? It uses buildout > (http://www.buildout.org/).? To help easily reproduce the problem, here are > the commands to reproduce. I have not tried it, but from your explanation, I would suspect that buildout does some monkeypatching which is not exactly the same as setuptools itself, and that would break numpy.distutils. The distutils architecture unfortunately requires to take into account and special case any distutils extensions, because of the "extension as command inheritance" approach. I know I prefer spending my time on new packaging solution, but I would take patches. David From cgohlke at uci.edu Sat Nov 28 13:27:49 2009 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 28 Nov 2009 10:27:49 -0800 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 Message-ID: <4B116BA5.9020208@uci.edu> The current numpy sources from svn trunk (r7772 or newer) and the 1.4.x branch fail to build with Visual Studio 2008 with the following compiler error: numpy\core\src\multiarray\descriptor.h(16) : error C2133: '_datetime_strings' : unknown size A quick fix is to specify the size of the _datetime_strings array, e.g. _datetime_strings[14]. Christoph From charlesr.harris at gmail.com Sat Nov 28 13:47:08 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Nov 2009 11:47:08 -0700 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 In-Reply-To: <4B116BA5.9020208@uci.edu> References: <4B116BA5.9020208@uci.edu> Message-ID: On Sat, Nov 28, 2009 at 11:27 AM, Christoph Gohlke wrote: > The current numpy sources from svn trunk (r7772 or newer) and the 1.4.x > branch fail to build with Visual Studio 2008 with the following compiler > error: > > numpy\core\src\multiarray\descriptor.h(16) : error C2133: > '_datetime_strings' : unknown size > > A quick fix is to specify the size of the _datetime_strings array, e.g. > _datetime_strings[14]. > > What happens if you just replace: *_datetime_strings[]; by **_datetime_strings; Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sat Nov 28 14:07:29 2009 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 28 Nov 2009 11:07:29 -0800 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 References: 4B116BA5.9020208@uci.edu Message-ID: <4B1174F1.2090909@uci.edu> Changing *_datetime_strings[] to **_datetime_strings in descriptor.h results in the following compiler error: numpy\core\src\multiarray\descriptor.c(472) : error C2372: '_datetime_strings' : redefinition; different types of indirection numpy\core\src\multiarray\descriptor.h(16) : see declaration of '_datetime_strings' Christoph From charlesr.harris at gmail.com Sat Nov 28 14:21:54 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Nov 2009 12:21:54 -0700 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 In-Reply-To: <4B1174F1.2090909@uci.edu> References: <4B1174F1.2090909@uci.edu> Message-ID: On Sat, Nov 28, 2009 at 12:07 PM, Christoph Gohlke wrote: > Changing *_datetime_strings[] to **_datetime_strings in descriptor.h > results in the following compiler error: > > numpy\core\src\multiarray\descriptor.c(472) : error C2372: > '_datetime_strings' : redefinition; different types of indirection > > numpy\core\src\multiarray\descriptor.h(16) : see declaration of > '_datetime_strings' > > The problem is that the bit in the header looks like a declaration. Try commenting out the line like so: /*NPY_NO_EXPORT char *_datetime_strings[];*/ Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Nov 28 14:29:24 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Nov 2009 12:29:24 -0700 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 In-Reply-To: References: <4B1174F1.2090909@uci.edu> Message-ID: On Sat, Nov 28, 2009 at 12:21 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Nov 28, 2009 at 12:07 PM, Christoph Gohlke wrote: > >> Changing *_datetime_strings[] to **_datetime_strings in descriptor.h >> results in the following compiler error: >> >> numpy\core\src\multiarray\descriptor.c(472) : error C2372: >> '_datetime_strings' : redefinition; different types of indirection >> >> numpy\core\src\multiarray\descriptor.h(16) : see declaration of >> '_datetime_strings' >> >> > The problem is that the bit in the header looks like a declaration. Try > commenting out the line like so: > > /*NPY_NO_EXPORT char *_datetime_strings[];*/ > > In case that wasn't precise enough, it should look like this: #ifdef NPY_ENABLE_SEPARATE_COMPILATION extern NPY_NO_EXPORT char *_datetime_strings[]; #else /*NPY_NO_EXPORT char *_datetime_strings[];*/ #endif Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sat Nov 28 14:32:57 2009 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 28 Nov 2009 11:32:57 -0800 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 References: 4B1174F1.2090909@uci.edu Message-ID: <4B117AE9.70008@uci.edu> Commenting out the declaration in line 16 of decriptor.h works. Christoph From charlesr.harris at gmail.com Sat Nov 28 15:51:23 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Nov 2009 13:51:23 -0700 Subject: [Numpy-discussion] Numpy trunk and 1.4.x branch fail to build with VS2008 In-Reply-To: <4B117AE9.70008@uci.edu> References: <4B117AE9.70008@uci.edu> Message-ID: On Sat, Nov 28, 2009 at 12:32 PM, Christoph Gohlke wrote: > Commenting out the declaration in line 16 of decriptor.h works. > > OK, I went ahead and committed this change. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cheronetolivia at yahoo.com Sat Nov 28 16:50:53 2009 From: cheronetolivia at yahoo.com (Olivia Cheronet) Date: Sat, 28 Nov 2009 13:50:53 -0800 (PST) Subject: [Numpy-discussion] Installing numpy under cygwin In-Reply-To: <5b8d13220911270332n5f208024p272eeee00f77b4fd@mail.gmail.com> References: <432197.42695.qm@web51001.mail.re2.yahoo.com> <5b8d13220911241857q52988ee5sd53faac42da33a86@mail.gmail.com> <544434.51110.qm@web51003.mail.re2.yahoo.com> <4B0CE64E.7020106@ar.media.kyoto-u.ac.jp> <196130.37019.qm@web51009.mail.re2.yahoo.com> <5b8d13220911250159i9f0c2b7u3df5089f984c34ef@mail.gmail.com> <305713.71840.qm@web51008.mail.re2.yahoo.com> <5b8d13220911250256k62a3c13ej98ac04e89ccacfeb@mail.gmail.com> <887521.34374.qm@web51012.mail.re2.yahoo.com> <5b8d13220911270332n5f208024p272eeee00f77b4fd@mail.gmail.com> Message-ID: <823907.28119.qm@web51006.mail.re2.yahoo.com> Thank you. This time the building of Numpy worked with no error message. However, when I now try to import Numpy in Python, there is a problem with lapack_lite (see below). Thanks again! Olivia $ python Python 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/site-packages/numpy/__init__.py", line 132, in import add_newdocs File "/usr/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "/usr/lib/python2.5/site-packages/numpy/lib/__init__.py", line 13, in from polynomial import * File "/usr/lib/python2.5/site-packages/numpy/lib/polynomial.py", line 17, in < module> from numpy.linalg import eigvals, lstsq File "/usr/lib/python2.5/site-packages/numpy/linalg/__init__.py", line 47, in from linalg import * File "/usr/lib/python2.5/site-packages/numpy/linalg/linalg.py", line 22, in from numpy.linalg import lapack_lite ImportError: No such file or directory ----- Original Message ---- > From: David Cournapeau > > Please update to the trunk - I can see the error as well for 1.3.0, > and the trunk does build correctly on cygwin. I don't understand where > the error is coming from in 1.3.0, it almost look like a cpp bug. > > cheers, > > David From fonnesbeck at gmail.com Sat Nov 28 20:02:06 2009 From: fonnesbeck at gmail.com (Chris) Date: Sun, 29 Nov 2009 01:02:06 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> Message-ID: Chris gmail.com> writes: > Nothing looks out of order, but I still get the > Symbol not found: _npy_cexp > errors. > This problem still persists through rev 7803. Is there no solution for this? I have no clue where its coming from. Thanks in advance. From cournape at gmail.com Sat Nov 28 20:44:29 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 29 Nov 2009 10:44:29 +0900 Subject: [Numpy-discussion] Import error in builds of 7726 In-Reply-To: References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> Message-ID: <5b8d13220911281744ib60ab05j9fefdd5c7749439@mail.gmail.com> On Sun, Nov 29, 2009 at 10:02 AM, Chris wrote: > Chris gmail.com> writes: >> Nothing looks out of order, but I still get the >> Symbol not found: _npy_cexp >> errors. >> > > This problem still persists through rev 7803. Is there no solution > for this? The problem is that I don't see this issue at all, so I have no idea what may cause it. I again checked with your build script on my macbook, and everything works as expected. David From sierra_mtnview at sbcglobal.net Sat Nov 28 20:56:26 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sat, 28 Nov 2009 17:56:26 -0800 Subject: [Numpy-discussion] A Simple Example of histogram Using Range? Message-ID: <4B11D4CA.40001@sbcglobal.net> See Subject. I don't seem able to produce a simple example of using (Matplotlib) histogram that uses range. I tried a variety of ranges, range=(5,22), range=(0, 50.2), ... and I see no difference between any of the x values scale on the plot. Can someone provide an example that shows how it works? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From charlesr.harris at gmail.com Sat Nov 28 21:46:43 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Nov 2009 19:46:43 -0700 Subject: [Numpy-discussion] Import error in builds of 7726 In-Reply-To: <5b8d13220911281744ib60ab05j9fefdd5c7749439@mail.gmail.com> References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> <5b8d13220911281744ib60ab05j9fefdd5c7749439@mail.gmail.com> Message-ID: On Sat, Nov 28, 2009 at 6:44 PM, David Cournapeau wrote: > On Sun, Nov 29, 2009 at 10:02 AM, Chris wrote: > > Chris gmail.com> writes: > >> Nothing looks out of order, but I still get the > >> Symbol not found: _npy_cexp > >> errors. > >> > > > > This problem still persists through rev 7803. Is there no solution > > for this? > > The problem is that I don't see this issue at all, so I have no idea > what may cause it. I again checked with your build script on my > macbook, and everything works as expected. > > Maybe there is a stray old file floating about somewhere. Chris, is there a locate command for the mac? Could you track down numpy related files and make sure none are sitting in some dusty old corner? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Sat Nov 28 22:20:15 2009 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 28 Nov 2009 19:20:15 -0800 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <4B10B44A.8030700@sbcglobal.net> References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> Message-ID: <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> On Fri, Nov 27, 2009 at 9:25 PM, Wayne Watson wrote: > I actually wrote my own several days ago. When I began getting myself > more familiar with numpy, I was hoping there would be an easy to use > version in it for this frequency approach. If not, then I'll just stick > with what I have. It seems something like this should be common. > > A simple way to do it with the present capabilities would be to "unwind" > the frequencies, For example, given [2,1,3] for some corresponding set > of x, say, [1,2,3], produce[1, 1, 2, 3, 3, 3]. I have no idea if numpy > does anything like that, but, if so, the typical mean, std, ... could be > used. In my case, it's sort of pointless. It would produce an array of > 307,200 items for 256 x (0,1,2,...,255), and just slow down the > computations "unwinding" it in software. The sub-processor hardware > already produced the 256 frequencies. > > Basically, this amounts to having a pdf, and values of x. > Mathematically, the statistics are produced directly from it. > > josef.pktd at gmail.com wrote: > > On Fri, Nov 27, 2009 at 9:47 PM, Wayne Watson > > wrote: > > > >> How do I compute avg, std dev, min, max and other simple stats if I only > >> know the frequency distribution? > >> > > > > If you are willing to assign to all observations in a bin the value at > > the bin midpoint, then you could do it with weights in the statistics > > calculations. However, numpy.average is, I think, the only statistic > > that takes weights. min max are independent of weight, but std and var > > need to be calculated indirectly. > > > > If you need more stats with weights, then the attachment in > > http://projects.scipy.org/scipy/ticket/604 is a good start. > > > > Josef > Wayne: There is no need to "unwind": If Y(X) is the (unnormalized) freq. distr. of random variable/data X, start by computing y = Y/(Y.sum()) (if Y is already normalized, skip this step). Then: av(X) = np.dot(X, y), sd(X) = np.sqrt(np.dot((X*X), y) - (av(X))^2), and higher moment statistics can be calculated utilizing similar formulae. DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From sierra_mtnview at sbcglobal.net Sat Nov 28 22:45:28 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sat, 28 Nov 2009 19:45:28 -0800 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> Message-ID: <4B11EE58.3030808@sbcglobal.net> David Goldsmith wrote: > On Fri, Nov 27, 2009 at 9:25 PM, Wayne Watson > > > wrote: > > I actually wrote my own several days ago. When I began getting myself > more familiar with numpy, I was hoping there would be an easy to use > version in it for this frequency approach. If not, then I'll just > stick > with what I have. It seems something like this should be common. > > ... > > > If you need more stats with weights, then the attachment in > > http://projects.scipy.org/scipy/ticket/604 is a good start. > > > > Josef > > > Wayne: > > There is no need to "unwind": If Y(X) is the (unnormalized) freq. > distr. of random variable/data X, start by computing y = Y/(Y.sum()) > (if Y is already normalized, skip this step). Then: > > av(X) = np.dot(X, y), sd(X) = np.sqrt(np.dot((X*X), y) - (av(X))^2), > and higher moment statistics can be calculated utilizing similar > formulae. > > DG I was only illustrating a way that I would not consider, since the hardware has already created the pdf. I've already coded it pretty much as you have suggested. As I think I mention ed above, I'm a bit surprised numpy doesn't provide the code you suggest as part of some function. CalcSimplefromPDF(xvalues=mydatarray, avg=ture, minmax=true, ...). -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From peridot.faceted at gmail.com Sat Nov 28 22:50:57 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 28 Nov 2009 22:50:57 -0500 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <4B11EE58.3030808@sbcglobal.net> References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> <4B11EE58.3030808@sbcglobal.net> Message-ID: 2009/11/28 Wayne Watson : > > I was only illustrating a way that I would not consider, since the > hardware has already created the pdf. I've already coded it pretty much > as you have suggested. As I think I mention ed above, I'm a bit > surprised numpy doesn't provide the code you suggest as part of some > function. CalcSimplefromPDF(xvalues=mydatarray, avg=ture, minmax=true, > ...). Feel free to submit an implementation to numpy's issue tracker. I suggest modifying mean, std, and var (at least) so that, like average, they take an array of weights. Anne From sierra_mtnview at sbcglobal.net Sat Nov 28 23:58:06 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sat, 28 Nov 2009 20:58:06 -0800 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> <4B11EE58.3030808@sbcglobal.net> Message-ID: <4B11FF5E.3020100@sbcglobal.net> How would I do that? Anne Archibald wrote: > 2009/11/28 Wayne Watson : > >> I was only illustrating a way that I would not consider, since the >> hardware has already created the pdf. I've already coded it pretty much >> as you have suggested. As I think I mention ed above, I'm a bit >> surprised numpy doesn't provide the code you suggest as part of some >> function. CalcSimplefromPDF(xvalues=mydatarray, avg=ture, minmax=true, >> ...). >> > > Feel free to submit an implementation to numpy's issue tracker. I > suggest modifying mean, std, and var (at least) so that, like average, > they take an array of weights. > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From sierra_mtnview at sbcglobal.net Sun Nov 29 00:30:34 2009 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Sat, 28 Nov 2009 21:30:34 -0800 Subject: [Numpy-discussion] Producing a Histogram When Bins Are Known In-Reply-To: <4B100F5E.7080409@noaa.gov> References: <4B0F60A5.7040302@sbcglobal.net> <4B0FD797.70506@sbcglobal.net> <4B100F5E.7080409@noaa.gov> Message-ID: <4B1206FA.7030401@sbcglobal.net> Yes, the book description is here Christopher Barker wrote: > ... > FWIW, a book about MPL has just been published -- I don't know any more > about it, but I'm sure google will tell you. > > >> Is there a matplotlib or Pylab mailing list? >> > > There certainly is: > > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > And yes, that is the place for such questions. > > > HTH, > > -Chris > > > > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet 350 350 350 350 350 350 350 350 350 350 Make the number famous. See 350.org The major event has passed, but keep the number alive. Web Page: From eg at fft.be Sun Nov 29 07:16:03 2009 From: eg at fft.be (Eloi Gaudry) Date: Sun, 29 Nov 2009 13:16:03 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> Message-ID: <4B126603.7040203@fft.be> On 27/11/2009 18:21, David Cournapeau wrote: > On Fri, Nov 27, 2009 at 11:01 PM, Eloi Gaudry wrote: > >> David, >> >> I know this discussion first took place months ago, but I'd like to know >> whether or not you find a solution to the SxS assemblies issues using >> MSVC9.0. >> > Which issue exactly are you talking about ? When python is installed > for one user (when installed for all users, the C runtime is installed > in the SxS) ? > Sorry, I was thinking of the "single-user" installation one, where no C runtime can be installed to the SxS folder but rather made available next to python.exe executable. >> In case you haven't (the binaries package for windows is built using >> mingw), I'd like to know if this would be possible to relocate all numpy >> *.pyd from their original location (i.e. site-packages/numpy/core/, >> etc.) to the main executable one (i.e. python.exe). >> > This sounds like a very bad idea: if packages start doing that, there > will quickly be clashes between extensions. > well, the DLLs directory of the nt python install is somehow using the very same concept. Moreover this could offer a proper solution to the single-user installation C runtime "issue". By making sure every binaries are located in the same directory where the C runtime are, one doesn't need to install the C runtime libraries in the SxS folder, thus no administrative rights are needed at set-up. > You would need to describe the exact scenario which is failing: how > python is installed, how you build numpy, what is not working with > which message, etc... > I'm distributing a software using python as its main interpreter and taking advantage of the several python extensions available (such as numpy, matplotlib, reportlab, vtk bindings, etc.). Python and its extension binaries (*.pyd, *.dll and *.exe) are dynamically built using msvc 2008 and thus rely on the SxS assemblies at runtime. As this software need to be installed without administrative privileges, I cannot install the mandatory runtimes libraries in the shared SxS assembly directory (if missing). I then need to directly provide the runtime libraries in a private directory of the software I'm distributing. Actually, this is why I provide the C runtime libraries in the 'bin' directory, where all binaries are stored (python.exe, *.dll, vtk*.pyd, etc.). Eloi > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From zachary.pincus at yale.edu Sun Nov 29 10:36:24 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 29 Nov 2009 10:36:24 -0500 Subject: [Numpy-discussion] convert strides/shape/offset into nd index? Message-ID: <6DE05582-D10B-4632-9C9C-6B07DAF321BE@yale.edu> Hi all, I'm curious as to what the most straightforward way is to convert an offset into a memory buffer representing an arbitrarily strided array into the nd index into that array. (Let's assume for simplicity that each element is one byte...) Does sorting the strides from largest to smallest and then using integer division and mod (in the obvious way) always work? (I can't seem to find a counterexample, but I am not used to thinking too deeply about bizarrely-strided configurations.) Is there a method that doesn't involve sorting? Thanks, Zach From cgohlke at uci.edu Sun Nov 29 14:25:23 2009 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 29 Nov 2009 11:25:23 -0800 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B126603.7040203@fft.be> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> Message-ID: <4B12CAA3.50804@uci.edu> Eloi: take a look at , which contains a discussion on, and a patch for, not embedding manifest files in *.pyd when compiling with MSVC. You could recompile the PYD/DLL files that depend on the VC90.CRT in winSXS with the patch applied, or manually remove the manifests from those PYD/DLL files using a HEX editor or script. Also make sure that all DLLs that your PYD files depend on are in the Windows search PATH. Christoph From dwf at cs.toronto.edu Sun Nov 29 16:46:46 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sun, 29 Nov 2009 16:46:46 -0500 Subject: [Numpy-discussion] another numpy/ATLAS problem In-Reply-To: References: <662818A6-E8EE-4392-9BED-D281D07D1DFC@cs.toronto.edu> Message-ID: <1CF7EB5A-F17B-4E2C-BD19-35AED576DEC8@cs.toronto.edu> On 27-Nov-09, at 6:29 PM, Charles R Harris wrote: > 3.9.12 segfaulted on me while running, so I haven't bothered with > versions after that. Why not try the stable version 3.8.3? Just to follow up, I went back to 3.9.11 and numpy works without incident, using the exact same ATLAS build procedure; I guess it wasn't something I was doing wrong after all. I've filed it on the ATLAS tracker, so hopefully it'll be addressed sooner or later. David From pfeldman at verizon.net Sun Nov 29 17:10:14 2009 From: pfeldman at verizon.net (Dr. Phillip M. Feldman) Date: Sun, 29 Nov 2009 14:10:14 -0800 (PST) Subject: [Numpy-discussion] non-standard standard deviation Message-ID: <26566808.post@talk.nabble.com> All of the statistical packages that I am currently using and have used in the past (Matlab, Minitab, R, S-plus) calculate standard deviation using the sqrt(1/(n-1)) normalization, which gives a result that is unbiased when sampling from a normally-distributed population. NumPy uses the sqrt(1/n) normalization. I'm currently using the following code to calculate standard deviations, but would much prefer if this could be fixed in NumPy itself: def mystd(x=numpy.array([]), axis=None): """This function calculates the standard deviation of the input using the definition of standard deviation that gives an unbiased result for samples from a normally-distributed population.""" -- View this message in context: http://old.nabble.com/non-standard-standard-deviation-tp26566808p26566808.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From peridot.faceted at gmail.com Sun Nov 29 17:13:05 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 29 Nov 2009 17:13:05 -0500 Subject: [Numpy-discussion] non-standard standard deviation In-Reply-To: <26566808.post@talk.nabble.com> References: <26566808.post@talk.nabble.com> Message-ID: 2009/11/29 Dr. Phillip M. Feldman : > All of the statistical packages that I am currently using and have used in > the past (Matlab, Minitab, R, S-plus) calculate standard deviation using the > sqrt(1/(n-1)) normalization, which gives a result that is unbiased when > sampling from a normally-distributed population. ?NumPy uses the sqrt(1/n) > normalization. ?I'm currently using the following code to calculate standard > deviations, but would much prefer if this could be fixed in NumPy itself: This issue was the subject of lengthy discussions on the mailing list, the upshot of which is that in current versions of scipy, std and var take an optional argument "ddof", into which you can supply 1 to get the normalization you want. Anne > def mystd(x=numpy.array([]), axis=None): > ? """This function calculates the standard deviation of the input using the > ? definition of standard deviation that gives an unbiased result for > samples > ? from a normally-distributed population.""" > -- > View this message in context: http://old.nabble.com/non-standard-standard-deviation-tp26566808p26566808.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pfeldman at verizon.net Sun Nov 29 17:13:11 2009 From: pfeldman at verizon.net (Dr. Phillip M. Feldman) Date: Sun, 29 Nov 2009 14:13:11 -0800 (PST) Subject: [Numpy-discussion] non-standard standard deviation Message-ID: <26566843.post@talk.nabble.com> All of the statistical packages that I am currently using and have used in the past (Matlab, Minitab, R, S-plus) calculate standard deviation using the sqrt(1/(n-1)) normalization, which gives a result that is unbiased when sampling from a normally-distributed population. NumPy uses the sqrt(1/n) normalization. I'm currently using the following code to calculate standard deviations, but would much prefer if this could be fixed in NumPy itself: def mystd(x=numpy.array([]), axis=None): """This function calculates the standard deviation of the input using the definition of standard deviation that gives an unbiased result for samples from a normally-distributed population.""" xd= x - x.mean(axis=axis) return sqrt( (xd*xd).sum(axis=axis) / (numpy.size(x,axis=axis)-1.0) ) -- View this message in context: http://old.nabble.com/non-standard-standard-deviation-tp26566843p26566843.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From cournape at gmail.com Sun Nov 29 19:12:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Nov 2009 09:12:27 +0900 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B12CAA3.50804@uci.edu> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> Message-ID: <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> On Mon, Nov 30, 2009 at 4:25 AM, Christoph Gohlke wrote: > Eloi: take a look at , which contains > a discussion on, and a patch for, not embedding manifest files in *.pyd > when compiling with MSVC. Note that the official binaries are built with mingw, and thus do not embed any manifest in the .pyd. David From cjw at ncf.ca Sun Nov 29 19:30:49 2009 From: cjw at ncf.ca (Colin J. Williams) Date: Sun, 29 Nov 2009 19:30:49 -0500 Subject: [Numpy-discussion] non-standard standard deviation In-Reply-To: <26566843.post@talk.nabble.com> References: <26566843.post@talk.nabble.com> Message-ID: <4B131239.7080801@ncf.ca> On 29-Nov-09 17:13 PM, Dr. Phillip M. Feldman wrote: > All of the statistical packages that I am currently using and have used in > the past (Matlab, Minitab, R, S-plus) calculate standard deviation using the > sqrt(1/(n-1)) normalization, which gives a result that is unbiased when > sampling from a normally-distributed population. NumPy uses the sqrt(1/n) > normalization. I'm currently using the following code to calculate standard > deviations, but would much prefer if this could be fixed in NumPy itself: > > def mystd(x=numpy.array([]), axis=None): > """This function calculates the standard deviation of the input using the > definition of standard deviation that gives an unbiased result for > samples > from a normally-distributed population.""" > > xd= x - x.mean(axis=axis) > return sqrt( (xd*xd).sum(axis=axis) / (numpy.size(x,axis=axis)-1.0) ) > Anne Archibald has suggested a work-around. Perhaps ddof could be set, by default to 1 as other values are rarely required. Where the distribution of a variate is not known a priori, then I believe that it can be shown that the n-1 divisor provides the best estimate of the variance. Colin W. From fonnesbeck at gmail.com Sun Nov 29 19:59:51 2009 From: fonnesbeck at gmail.com (Chris) Date: Mon, 30 Nov 2009 00:59:51 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> <5b8d13220911281744ib60ab05j9fefdd5c7749439@mail.gmail.com> Message-ID: Charles R Harris gmail.com> writes: > > Maybe there is a stray old file floating about somewhere. Chris, is there a locate command for the mac? Could you track down numpy related files and make sure none are sitting in some dusty old corner?Chuck > > Sorry to be a pain on this, but it has me baffled. I just deleted my entire numpy source tree and pulled it down again. I also tracked down and deleted anything related to numpy anywhere in my python path. Now, I used the following script to build numpy: #!/bin/sh export MACOSX_DEPLOYMENT_TARGET=10.6 export CFLAGS="-arch x86_64" #export FFLAGS="-arch i386 -arch x86_64" export FFLAGS="-arch x86_64" export LDFLAGS="-Wall -lgfortran -undefined dynamic_lookup -bundle -arch x86_64" rm -rf build python setup.py config -L/Users/chris/Code/freetype -L/Users/chris/Code/libpng build python setupegg.py egg_info --tag-date bdist_egg But, installing the resulting egg, then importing numpy gives: In [1]: import numpy ------------------------------------------------------------ Traceback (most recent call last): File "", line 1, in File "/Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/__init__.py", line 132, in import add_newdocs File "/Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "/Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/lib/__init__.py", line 4, in from type_check import * File "/Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "/Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/core/__init__.py", line 6, in import umath ImportError: dlopen(/Library/Python/2.6/site-packages/ numpy-1.4.0.dev_20091130-py2.6-macosx-10.6-universal.egg/ numpy/core/umath.so, 2): Symbol not found: _npy_cexp Referenced from: /Library/Python/2.6/site-packages/ numpy-1.4.0.dev_20091130-py2.6-macosx-10.6-universal.egg/ numpy/core/umath.so Expected in: flat namespace in /Library/Python/2.6/site-packages/numpy-1.4.0.dev_ 20091130-py2.6-macosx-10.6-universal.egg/numpy/core/umath.so The problem is clearly in the .egg that I just built, as the filenames match. For some reason umath is not being built properly. Thanks in advance to everyone. From robince at gmail.com Sun Nov 29 20:15:35 2009 From: robince at gmail.com (Robin) Date: Mon, 30 Nov 2009 01:15:35 +0000 Subject: [Numpy-discussion] non-standard standard deviation In-Reply-To: <4B131239.7080801@ncf.ca> References: <26566843.post@talk.nabble.com> <4B131239.7080801@ncf.ca> Message-ID: <2d5132a50911291715k703a0b32t1deffe67cf8947e2@mail.gmail.com> On Mon, Nov 30, 2009 at 12:30 AM, Colin J. Williams wrote: > On 29-Nov-09 17:13 PM, Dr. Phillip M. Feldman wrote: >> All of the statistical packages that I am currently using and have used in >> the past (Matlab, Minitab, R, S-plus) calculate standard deviation using the >> sqrt(1/(n-1)) normalization, which gives a result that is unbiased when >> sampling from a normally-distributed population. ?NumPy uses the sqrt(1/n) >> normalization. ?I'm currently using the following code to calculate standard >> deviations, but would much prefer if this could be fixed in NumPy itself: >> >> def mystd(x=numpy.array([]), axis=None): >> ? ? """This function calculates the standard deviation of the input using the >> ? ? definition of standard deviation that gives an unbiased result for >> samples >> ? ? from a normally-distributed population.""" >> >> ? ? xd= x - x.mean(axis=axis) >> ? ? return sqrt( (xd*xd).sum(axis=axis) / (numpy.size(x,axis=axis)-1.0) ) >> > Anne Archibald has suggested a work-around. ?Perhaps ddof could be set, > by default to > 1 as other values are rarely required. > > Where the distribution of a variate is not known a priori, then I > believe that it can be shown > that the n-1 divisor provides the best estimate of the variance. There have been previous discussions on this (but I can't find them now) and I believe the current default was chosen deliberately. I think it is the view of the numpy developers that the n divisor has more desireable properties in most cases than the traditional n-1 - see this paper by Travis Oliphant for details: http://hdl.handle.net/1877/438 Cheers Robin From peridot.faceted at gmail.com Sun Nov 29 20:15:46 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 29 Nov 2009 20:15:46 -0500 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: <4B11FF5E.3020100@sbcglobal.net> References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> <4B11EE58.3030808@sbcglobal.net> <4B11FF5E.3020100@sbcglobal.net> Message-ID: 2009/11/28 Wayne Watson : > Anne Archibald wrote: >> 2009/11/28 Wayne Watson : >> >>> I was only illustrating a way that I would not consider, since the >>> hardware has already created the pdf. I've already coded it pretty much >>> as you have suggested. As I think I mention ed above, I'm a bit >>> surprised numpy doesn't provide the code you suggest as part of some >>> function. CalcSimplefromPDF(xvalues=mydatarray, avg=ture, minmax=true, >>> ...). >>> >> >> Feel free to submit an implementation to numpy's issue tracker. I >> suggest modifying mean, std, and var (at least) so that, like average, >> they take an array of weights. > > How would I do that? > Obtain a copy of recent numpy source code - a .tar file from the website, or using SVN, or git. Then add the feature plus some tests, confirm that the tests pass, and post a request and your patch to the bug tracker. Anne From fonnesbeck at gmail.com Sun Nov 29 20:16:52 2009 From: fonnesbeck at gmail.com (Chris) Date: Mon, 30 Nov 2009 01:16:52 +0000 (UTC) Subject: [Numpy-discussion] Import error in builds of 7726 References: <4AFA285A.6000105@ar.media.kyoto-u.ac.jp> <5b8d13220911120516j521e70f8jfac325ef5e157e33@mail.gmail.com> <5b8d13220911121950p4c52e60ej93040405e7296b40@mail.gmail.com> <5b8d13220911281744ib60ab05j9fefdd5c7749439@mail.gmail.com> Message-ID: Chris gmail.com> writes: > > Charles R Harris gmail.com> writes: > > > > Maybe there is a stray old file floating about somewhere. > Chris, is there a locate command for the mac? Could you track > down numpy related files and make sure none are sitting in some > dusty old corner?Chuck Additionally, if its helpful, here is a log of the build itself: http://files.me.com/fonnesbeck/y47rzz From charlesr.harris at gmail.com Sun Nov 29 20:33:49 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 29 Nov 2009 18:33:49 -0700 Subject: [Numpy-discussion] Computing Simple Statistics When Only they Frequency Distribution is Known In-Reply-To: References: <4B108F3A.4080008@sbcglobal.net> <1cd32cbb0911271914y46f9a1f0mb4ab34a04da19436@mail.gmail.com> <4B10B44A.8030700@sbcglobal.net> <45d1ab480911281920td6e58eftf4e94f0aa6172637@mail.gmail.com> <4B11EE58.3030808@sbcglobal.net> <4B11FF5E.3020100@sbcglobal.net> Message-ID: On Sun, Nov 29, 2009 at 6:15 PM, Anne Archibald wrote: > 2009/11/28 Wayne Watson : > > Anne Archibald wrote: > >> 2009/11/28 Wayne Watson : > >> > >>> I was only illustrating a way that I would not consider, since the > >>> hardware has already created the pdf. I've already coded it pretty much > >>> as you have suggested. As I think I mention ed above, I'm a bit > >>> surprised numpy doesn't provide the code you suggest as part of some > >>> function. CalcSimplefromPDF(xvalues=mydatarray, avg=ture, minmax=true, > >>> ...). > >>> > >> > >> Feel free to submit an implementation to numpy's issue tracker. I > >> suggest modifying mean, std, and var (at least) so that, like average, > >> they take an array of weights. > > > > How would I do that? > > > > Obtain a copy of recent numpy source code - a .tar file from the > website, or using SVN, or git. Then add the feature plus some tests, > confirm that the tests pass, and post a request and your patch to the > bug tracker. > > You might also want to use average as a starting point. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Mon Nov 30 01:57:33 2009 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Mon, 30 Nov 2009 01:57:33 -0500 Subject: [Numpy-discussion] non-standard standard deviation In-Reply-To: <2d5132a50911291715k703a0b32t1deffe67cf8947e2@mail.gmail.com> References: <26566843.post@talk.nabble.com> <4B131239.7080801@ncf.ca> <2d5132a50911291715k703a0b32t1deffe67cf8947e2@mail.gmail.com> Message-ID: <38B0CE97-4725-4FC4-8087-E0164564F3A2@cs.toronto.edu> On 29-Nov-09, at 8:15 PM, Robin wrote: > There have been previous discussions on this (but I can't find them > now) and I believe the current default was chosen deliberately. I > think it is the view of the numpy developers that the n divisor has > more desireable properties in most cases than the traditional n-1 - > see this paper by Travis Oliphant for details: > http://hdl.handle.net/1877/438 One such relevant discussion preceded this informative rant by Robert Kern: http://mail.scipy.org/pipermail/scipy-user/2008-December/019118.html David From eg at fft.be Mon Nov 30 04:11:25 2009 From: eg at fft.be (Eloi Gaudry) Date: Mon, 30 Nov 2009 10:11:25 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B12CAA3.50804@uci.edu> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> Message-ID: <4B138C3D.5030500@fft.be> Christoph, thanks for pointing this discussion. That's a perfect match. If the workaround provided offers a solution to the current redistribution issue, I'm wondering if it will still be the case when an update to the assembly check function will be activated/implemented (within Windows). The manifest edition (removing the "assemblyIdentity" tag) doesn't seem to be a popular/official/supported way of dealing with the whole runtime libraries issue. Don't you think ? Christoph Gohlke wrote: > Eloi: take a look at , which contains > a discussion on, and a patch for, not embedding manifest files in *.pyd > when compiling with MSVC. > > You could recompile the PYD/DLL files that depend on the VC90.CRT in > winSXS with the patch applied, or manually remove the manifests from > those PYD/DLL files using a HEX editor or script. Also make sure that > all DLLs that your PYD files depend on are in the Windows search PATH. > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Eloi Gaudry Free Field Technologies Axis Park Louvain-la-Neuve Rue Emile Francqui, 1 B-1435 Mont-Saint Guibert BELGIUM Company Phone: +32 10 487 959 Company Fax: +32 10 454 626 From eg at fft.be Mon Nov 30 04:16:31 2009 From: eg at fft.be (Eloi Gaudry) Date: Mon, 30 Nov 2009 10:16:31 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> Message-ID: <4B138D6F.60907@fft.be> David, I think that what Christoph pointed here applies to the msvc built libraries only (I need to build the whole python/extensions using msvc). May I ask you what re the "clashes between extensions" you were referring to in your previous answer ? Have you already encountered such issues ? I'm wondering how it could break things asp roviding libraries in an unique location is a known and common practice. Eloi David Cournapeau wrote: > On Mon, Nov 30, 2009 at 4:25 AM, Christoph Gohlke wrote: > >> Eloi: take a look at , which contains >> a discussion on, and a patch for, not embedding manifest files in *.pyd >> when compiling with MSVC. >> > > Note that the official binaries are built with mingw, and thus do not > embed any manifest in the .pyd. > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Eloi Gaudry Free Field Technologies Axis Park Louvain-la-Neuve Rue Emile Francqui, 1 B-1435 Mont-Saint Guibert BELGIUM Company Phone: +32 10 487 959 Company Fax: +32 10 454 626 From david at ar.media.kyoto-u.ac.jp Mon Nov 30 04:12:05 2009 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 30 Nov 2009 18:12:05 +0900 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B138D6F.60907@fft.be> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> Message-ID: <4B138C65.5000401@ar.media.kyoto-u.ac.jp> Eloi Gaudry wrote: > David, I think that what Christoph pointed here applies to the msvc > built libraries only (I need to build the whole python/extensions using > msvc). > Ok, I did not understand that point. Do you know if the patch discussed in 4120 has been applied to 2.6.4 ? > May I ask you what re the "clashes between extensions" you were > referring to in your previous answer ? Have you already encountered such > issues ? > If every python package starts to put its extensions (*.pyd) into a directory, what happens when two different packages have an extension with the same name (e.g. package foo has a package multiarray.pyd) ? I would also be really annoyed if a 3rd party extension starts polluting my C:\Python26. > I'm wondering how it could break things asp roviding libraries in an > unique location is a known and common practice. The problem is that numpy does not "own" C:\Python*, and it is not expected that projects start dumping their stuff there. It usually "works" on windows because each applications is independent of each other. But here, numpy extends python, so we cannot just do as we please - especially to deal with this whole manifest stupidity. Given that even MS has implicitly recognized how broken manifests were (they are giving it up for VS 2010), I don't think it worths spending time on this issue which may well be unsolvable in a reliable way anyway. AFAIK, the related issue of broken "install for me" python installers on vista and windows 7 has not been solved yet. cheers, David From eg at fft.be Mon Nov 30 06:52:29 2009 From: eg at fft.be (Eloi Gaudry) Date: Mon, 30 Nov 2009 12:52:29 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B138C65.5000401@ar.media.kyoto-u.ac.jp> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> Message-ID: <4B13B1FD.6080404@fft.be> David Cournapeau wrote: > Eloi Gaudry wrote: > >> David, I think that what Christoph pointed here applies to the msvc >> built libraries only (I need to build the whole python/extensions using >> msvc). >> >> > > Ok, I did not understand that point. Do you know if the patch discussed > in 4120 has been applied to 2.6.4 ? > I don't think so... I'm using 2.6.4 and still wondering how to circumvent the runtime redistribution issue. > >> May I ask you what re the "clashes between extensions" you were >> referring to in your previous answer ? Have you already encountered such >> issues ? >> >> > > If every python package starts to put its extensions (*.pyd) into a > directory, what happens when two different packages have an extension > with the same name (e.g. package foo has a package multiarray.pyd) ? I > would also be really annoyed if a 3rd party extension starts polluting > my C:\Python26. > You made your point. > >> I'm wondering how it could break things asp roviding libraries in an >> unique location is a known and common practice. >> > > The problem is that numpy does not "own" C:\Python*, and it is not > expected that projects start dumping their stuff there. It usually > "works" on windows because each applications is independent of each > other. But here, numpy extends python, so we cannot just do as we please > - especially to deal with this whole manifest stupidity. Given that even > MS has implicitly recognized how broken manifests were (they are giving > it up for VS 2010), I don't think it worths spending time on this issue > which may well be unsolvable in a reliable way anyway. AFAIK, the > related issue of broken "install for me" python installers on vista and > windows 7 has not been solved yet. > Well, I wasn't aware of Microsoft willing to giving up the whole SxS/manifest thing. Is there any MSDN information available? Thanks for your input David, Eloi > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Eloi Gaudry Free Field Technologies Axis Park Louvain-la-Neuve Rue Emile Francqui, 1 B-1435 Mont-Saint Guibert BELGIUM Company Phone: +32 10 487 959 Company Fax: +32 10 454 626 From cournape at gmail.com Mon Nov 30 07:08:52 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Nov 2009 21:08:52 +0900 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B13B1FD.6080404@fft.be> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> <4B13B1FD.6080404@fft.be> Message-ID: <5b8d13220911300408t4f5c9c9dm59eb4391c26c0149@mail.gmail.com> On Mon, Nov 30, 2009 at 8:52 PM, Eloi Gaudry wrote: > > Well, I wasn't aware of Microsoft willing to giving up the whole > SxS/manifest thing. Is there any MSDN information available? I have seen this mentioned for the first time on the python-dev ML: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3764855 The mention of including the version in the dll file, if true, is tragically comic. Maybe in 20 years windows will be able to have a system which exists for more than a decade on conventional unix... The link given by M.A Lemburg has changed since, though, as the description is nowhere to be found in the link. I think I have read that VS 2010 will never install the runtime in the SxS configuration, but I of course cannot find this information anymore. Maybe it is not true anymore, VS 2010 has not yet been released. You can also find useful manifest troubleshooting information there: http://blogs.msdn.com/junfeng/archive/2006/04/14/576314.aspx cheers, David From sturla at molden.no Mon Nov 30 08:19:59 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Nov 2009 14:19:59 +0100 Subject: [Numpy-discussion] non-standard standard deviation In-Reply-To: <4B131239.7080801@ncf.ca> References: <26566843.post@talk.nabble.com> <4B131239.7080801@ncf.ca> Message-ID: <4B13C67F.4080308@molden.no> Colin J. Williams skrev: > Where the distribution of a variate is not known a priori, then I > believe that it can be shown > that the n-1 divisor provides the best estimate of the variance. > Have you ever been shooting with a rifle? What would you rather do: - Hit 9 or 10, with a bias to the right. - Hit 7 or better, with no bias. Do you think it can be shown that the latter option is the better? No? Sturla Molden From sturla at molden.no Mon Nov 30 08:48:02 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Nov 2009 14:48:02 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B138C65.5000401@ar.media.kyoto-u.ac.jp> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> Message-ID: <4B13CD12.4080401@molden.no> David Cournapeau skrev: > If every python package starts to put its extensions (*.pyd) into a > directory, what happens when two different packages have an extension > with the same name (e.g. package foo has a package multiarray.pyd) ? I > would also be really annoyed if a 3rd party extension starts polluting > my C:\Python26. > > In disutils, data_files can install DLLs where Python wants them. Just pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes your c:\Python26 it is distutils, not a third party extension. This is not any different from installing other data files. Sturla From cournape at gmail.com Mon Nov 30 08:54:05 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Nov 2009 22:54:05 +0900 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B13CD12.4080401@molden.no> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> <4B13CD12.4080401@molden.no> Message-ID: <5b8d13220911300554x2d8f00f6w3395f34cb2c9d10b@mail.gmail.com> On Mon, Nov 30, 2009 at 10:48 PM, Sturla Molden wrote: > David Cournapeau skrev: >> If every python package starts to put its extensions (*.pyd) into a >> directory, what happens when two different packages have an extension >> with the same name (e.g. package foo has a package multiarray.pyd) ? I >> would also be really annoyed if a 3rd party extension starts polluting >> my C:\Python26. >> >> > In disutils, data_files can install DLLs where Python wants them. Just > pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes > your c:\Python26 it is distutils, not a third party extension. This is > not any different from installing other data files. We are talking about the numpy extensions here, which are not installed through the install_data command. The problem is about how windows looks for dll with the manifest mechanism, and how to build/install extensions when the C runtime (or any other "system" dll) is not installed in the SxS cache. David From sturla at molden.no Mon Nov 30 09:07:43 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Nov 2009 15:07:43 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <5b8d13220911300554x2d8f00f6w3395f34cb2c9d10b@mail.gmail.com> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> <4B13CD12.4080401@molden.no> <5b8d13220911300554x2d8f00f6w3395f34cb2c9d10b@mail.gmail.com> Message-ID: <4B13D1AF.9090302@molden.no> David Cournapeau skrev: > We are talking about the numpy extensions here, which are not > installed through the install_data command. The problem is about how > windows looks for dll with the manifest mechanism, and how to > build/install extensions when the C runtime (or any other "system" > dll) is not installed in the SxS cache. > Do we need to support Windows 2000? S.M. From sturla at molden.no Mon Nov 30 09:10:52 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Nov 2009 15:10:52 +0100 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <5b8d13220911300554x2d8f00f6w3395f34cb2c9d10b@mail.gmail.com> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <5b8d13220911291612o7b57dff3xca4b04cdd5277fa2@mail.gmail.com> <4B138D6F.60907@fft.be> <4B138C65.5000401@ar.media.kyoto-u.ac.jp> <4B13CD12.4080401@molden.no> <5b8d13220911300554x2d8f00f6w3395f34cb2c9d10b@mail.gmail.com> Message-ID: <4B13D26C.7070802@molden.no> David Cournapeau skrev: > We are talking about the numpy extensions here, which are not > installed through the install_data command. The problem is about how > windows looks for dll with the manifest mechanism, and how to > build/install extensions when the C runtime (or any other "system" > dll) is not installed in the SxS cache. > Is COM (aka ActiveX) out of the question? From cgohlke at uci.edu Mon Nov 30 13:44:25 2009 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 30 Nov 2009 10:44:25 -0800 Subject: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ? In-Reply-To: <4B138C3D.5030500@fft.be> References: <4B0FDBAC.3030108@fft.be> <5b8d13220911270921l78b74d45s93a25f692a7c1c22@mail.gmail.com> <4B126603.7040203@fft.be> <4B12CAA3.50804@uci.edu> <4B138C3D.5030500@fft.be> Message-ID: <4B141289.10400@uci.edu> The most popular/simple way to deal with the VC90.CRT dependency issue is to have the user install the runtime redistributable on their system. If you don't want to put that burden on the user, which I understand, you have to make adjustments to the assembly manifests. This is not unofficial or unsupported. It is a bug in Python that it embeds the assemblyIdentity for VC90.CRT in all extensions build with distutils/msvc9compiler.py. In fact, the *.pyd distributed with Python 2.6.3+ don't have that problem. Maybe you can raise your concerns about future compatibility at . Christoph On 11/30/2009 1:11 AM, Eloi Gaudry wrote: > Christoph, thanks for pointing this discussion. That's a perfect match. > > If the workaround provided offers a solution to the current > redistribution issue, I'm wondering if it will still be the case when an > update to the assembly check function will be activated/implemented > (within Windows). > The manifest edition (removing the "assemblyIdentity" tag) doesn't seem > to be a popular/official/supported way of dealing with the whole runtime > libraries issue. Don't you think ? From charlesr.harris at gmail.com Mon Nov 30 15:04:01 2009 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Nov 2009 13:04:01 -0700 Subject: [Numpy-discussion] Python 3K merge Message-ID: Hi Pauli, It looks like you doing great stuff with the py3k transition. Do you and David have any sort of merge schedule in mind? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From donovan.parks at gmail.com Mon Nov 30 16:06:10 2009 From: donovan.parks at gmail.com (Donovan Parks) Date: Mon, 30 Nov 2009 17:06:10 -0400 Subject: [Numpy-discussion] Packaging numpy with py2app v0.4.3 In-Reply-To: References: Message-ID: <8fd818a30911301306t7fb644dbwffe7d2135a73de20@mail.gmail.com> Hello, I am trying to package a simple python script which uses numpy using py2app. There was a lengthy discussion about this back in 2006 on this list, but it doesn't seem relevant anymore given the changes to numpy and py2app over the last few years. I am hoping someone on this list has experience with py2app and can give me a hand. My system is as follows: OS X Leopard MacPort Python 2.6.4 (installed via MacPort two days ago) py26-py2app 0.4.3 (installed via MacPort two days ago) py26-numpy 1.3.0 (installed via MacPort two days ago) The minimal numpy script I've been working with is: from numpy import array a = array( [ 10,20,30,40 ] ) print a I've been trying to package this with numpy using the following setup.py file: from setuptools import setup APP = ['numpyTest.py'] OPTIONS = {'argv_emulation': True} setup( app=APP, options={'py2app': OPTIONS}, setup_requires=['py2app'], ) When I run 'python setup.py py2app' I get the following error after py2app starts stripping out debug information: /usr/bin/strip: for architecture i386 object: /Users/dparks/test/dist/numpyTest.app/Contents/Frameworks/libgcc_s.1.dylib truncated or malformed object (LC_SEGMENT command 2 fileoff field plus filesize field extends past the end of the file) stripping saved 2592052 bytes (32431544 / 35023596) A numpyTest.app file is created in the dist directory. However, when I try to run this file I get the following error: ZipImportError: can't decompress data; zlib not available I tried --no-strip which does cause a clean 'build' from py2app, but I still get the same error message about zlib. Explicitly adding zlib to setup.py doesn't change anything. I've also tried going into numpyTest.app and unzipping the site-packages.zip file, but again this makes no difference. I'm at a bit of a lost at what to try next since my understanding is that py2app has a built-in recipe for numpy so should more-or-less work out of the box. Any insight into what the stripping error relates to? Perhaps I am missing something simple here. Does one need to explictly tell py2app what recipes to use (the documentation doesn't make it sound that way)? Do I need to explicitly indicate I have a 32-bit or 64-bit system (actually not sure what it is at the moment)? Thanks for any and all help. Regards, Donovan From Chris.Barker at noaa.gov Mon Nov 30 16:29:22 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 30 Nov 2009 13:29:22 -0800 Subject: [Numpy-discussion] Packaging numpy with py2app v0.4.3 In-Reply-To: <8fd818a30911301306t7fb644dbwffe7d2135a73de20@mail.gmail.com> References: <8fd818a30911301306t7fb644dbwffe7d2135a73de20@mail.gmail.com> Message-ID: <4B143932.5000801@noaa.gov> Donovan Parks wrote: I think this may have nothing at all to do with numpy. Have you tried a simple script that doesn't use numpy? > OS X Leopard > MacPort I use neither Leopard nor macports, so it may be an issue there, but with the python.org python25 and numpy1.3, I have no problems. > When I run 'python setup.py py2app' I get the following error after > py2app starts stripping out debug information: > > /usr/bin/strip: for architecture i386 object: > /Users/dparks/test/dist/numpyTest.app/Contents/Frameworks/libgcc_s.1.dylib > truncated or malformed object (LC_SEGMENT command 2 fileoff field plus note that this is a libgcc error -- you wouldn't need libgcc if you used the python.org build... > ZipImportError: can't decompress data; zlib not available > I tried --no-strip which does cause a clean 'build' from py2app, I don't think this has anything to do with the stripping (or numpy). py2app, by default, puts all the modules you need into a zipped archive, so any import you do is going to be a problem if zlib is not available. I suspect this may be a macports/py2app incompatibility -- with the python.org python, it probably uses a system zlib, so it hasn't bundled up zlib from macports for you. I'd try asking on the pythonmac list. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bergstrj at iro.umontreal.ca Mon Nov 30 22:04:44 2009 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Mon, 30 Nov 2009 22:04:44 -0500 Subject: [Numpy-discussion] convert strides/shape/offset into nd index? In-Reply-To: <6DE05582-D10B-4632-9C9C-6B07DAF321BE@yale.edu> References: <6DE05582-D10B-4632-9C9C-6B07DAF321BE@yale.edu> Message-ID: <7f1eaee30911301904r4be694e6x85eb47319bebea67@mail.gmail.com> Your question involves a few concepts: - an integer vector describing the position of an element - the logical shape (another int vector) - the physical strides (another int vector) Ignoring the case of negative offsets, a physical offset is the inner product of the physical strides with the position vector. In these terms, you are asking how to solve the inner-product equation for the position vector. There can be many possible solutions (like, if there is a stride of 1, then you can make that dimension account for the entire offset. This is often not the solution you want.). For valid ndarrays though, there is at most one solution though with the property that every position element is less than the shape. You will also need to take into account that for certain stride vectors, there is no way to get certain offsets. Imagine all the strides were even, and you needed to get at an odd offset... it would be impossible. It would even be impossible if there were a dimension with stride 1 but it had shape of 1 too. I can't think of an algorithm off the top of my head that would do this in a quick and elegant way. James On Sun, Nov 29, 2009 at 10:36 AM, Zachary Pincus wrote: > Hi all, > > I'm curious as to what the most straightforward way is to convert an > offset into a memory buffer representing an arbitrarily strided array > into the nd index into that array. (Let's assume for simplicity that > each element is one byte...) > > Does sorting the strides from largest to smallest and then using > integer division and mod (in the obvious way) always work? (I can't > seem to find a counterexample, but I am not used to thinking too > deeply about bizarrely-strided configurations.) Is there a method that > doesn't involve sorting? > > Thanks, > Zach > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- http://www-etud.iro.umontreal.ca/~bergstrj From cournape at gmail.com Mon Nov 30 22:47:19 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 1 Dec 2009 12:47:19 +0900 Subject: [Numpy-discussion] Numpy 1.4.0 rc1 released Message-ID: <5b8d13220911301947m7fc20c83v59d84a035cb6e6ac@mail.gmail.com> Hi, The first release candidate for 1.4.0 has been released. The sources, as well as mac and windows installers may be found here: https://sourceforge.net/projects/numpy/files/ The main improvements compared to 1.3.0 are: * Faster import time * Extended array wrapping mechanism for ufuncs * New Neighborhood iterator (C-level only) * C99-like complex functions in npymath As well as more than 50 bug fixes. The detailed list of changes may be found on trac: http://projects.scipy.org/numpy/roadmap cheers, David From peridot.faceted at gmail.com Mon Nov 30 23:14:09 2009 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 30 Nov 2009 23:14:09 -0500 Subject: [Numpy-discussion] convert strides/shape/offset into nd index? In-Reply-To: <7f1eaee30911301904r4be694e6x85eb47319bebea67@mail.gmail.com> References: <6DE05582-D10B-4632-9C9C-6B07DAF321BE@yale.edu> <7f1eaee30911301904r4be694e6x85eb47319bebea67@mail.gmail.com> Message-ID: 2009/11/30 James Bergstra : > Your question involves a few concepts: > > - an integer vector describing the position of an element > > - the logical shape (another int vector) > > - the physical strides (another int vector) > > Ignoring the case of negative offsets, a physical offset is the inner > product of the physical strides with the position vector. > > In these terms, you are asking how to solve the inner-product equation > for the position vector. ?There can be many possible solutions (like, > if there is a stride of 1, then you can make that dimension account > for the entire offset. ?This is often not the solution you want.). > For valid ndarrays though, there is at most one solution though with > the property that every position element is less than the shape. > > You will also need to take into account that for certain stride > vectors, there is no way to get certain offsets. ?Imagine all the > strides were even, and you needed to get at an odd offset... it would > be impossible. ?It would even be impossible if there were a dimension > with stride 1 but it had shape of 1 too. > > I can't think of an algorithm off the top of my head that would do > this in a quick and elegant way. Not to be a downer, but this problem is technically NP-complete. The so-called "knapsack problem" is to find a subset of a collection of numbers that adds up to the specified number, and it is NP-complete. Unfortunately, it is exactly what you need to do to find the indices to a particular memory location in an array of shape (2,2,...,2). What that means in practice is that either you have to allow potentially very slow algorithms (though you know that there will never be more than 32 different values in the knapsack, which might or might not be enough to keep things tractable) or use heuristic algorithms that don't always work. There are probably fairly good heuristics, particularly if the array elements are all at distinct memory locations (arrays with overlapping elements can arise from broadcasting and other slightly more arcane operations). Anne > > James > > On Sun, Nov 29, 2009 at 10:36 AM, Zachary Pincus > wrote: >> Hi all, >> >> I'm curious as to what the most straightforward way is to convert an >> offset into a memory buffer representing an arbitrarily strided array >> into the nd index into that array. (Let's assume for simplicity that >> each element is one byte...) >> >> Does sorting the strides from largest to smallest and then using >> integer division and mod (in the obvious way) always work? (I can't >> seem to find a counterexample, but I am not used to thinking too >> deeply about bizarrely-strided configurations.) Is there a method that >> doesn't involve sorting? >> >> Thanks, >> Zach >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > http://www-etud.iro.umontreal.ca/~bergstrj > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >